Revisiting neural information, computing and linking capacity

János Végh; Ádám József Berki; János Végh; Ádám József Berki

doi:10.3934/mbe.2023551

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 7: 12380-12403. doi: 10.3934/mbe.2023551

Previous Article Next Article

Review Special Issues

Revisiting neural information, computing and linking capacity

János Végh ¹,
Ádám József Berki ^{2,3
,
,}

1.
Kalimános BT, 4028 Debrecen, Hungary
2.
Department of Neurology, Semmelweis University, 1085 Budapest, Hungary
3.
János Szentágothai Doctoral School of Neurosciences, Semmelweis University, 1085 Budapest, Hungary

Received: 19 February 2023 Revised: 26 April 2023 Accepted: 03 May 2023 Published: 22 May 2023

Neural information theory represents a fundamental method to model dynamic relations in biological systems. However, the notion of information, its representation, its content and how it is processed are the subject of fierce debates. Since the limiting capacity of neuronal links strongly depends on how neurons are hypothesized to work, their operating modes are revisited by analyzing the differences between the results of the communication models published during the past seven decades and those of the recently developed generalization of the classical information theory. It is pointed out that the operating mode of neurons is in resemblance with an appropriate combination of the formerly hypothesized analog and digital working modes; furthermore that not only the notion of neural information and its processing must be reinterpreted. Given that the transmission channel is passive in Shannon's model, the active role of the transfer channels (the axons) may introduce further transmission limits in addition to the limits concluded from the information theory. The time-aware operating model enables us to explain why (depending on the researcher's point of view) the operation can be considered either purely analog or purely digital.
- information theory,
- neural information,
- neuron operating mode,
- neural communication,
- information content,
- limiting information capacity,
- neural bandwidth
Citation: János Végh, Ádám József Berki. Revisiting neural information, computing and linking capacity[J]. Mathematical Biosciences and Engineering, 2023, 20(7): 12380-12403. doi: 10.3934/mbe.2023551

Related Papers:

Abstract

Neural information theory represents a fundamental method to model dynamic relations in biological systems. However, the notion of information, its representation, its content and how it is processed are the subject of fierce debates. Since the limiting capacity of neuronal links strongly depends on how neurons are hypothesized to work, their operating modes are revisited by analyzing the differences between the results of the communication models published during the past seven decades and those of the recently developed generalization of the classical information theory. It is pointed out that the operating mode of neurons is in resemblance with an appropriate combination of the formerly hypothesized analog and digital working modes; furthermore that not only the notion of neural information and its processing must be reinterpreted. Given that the transmission channel is passive in Shannon's model, the active role of the transfer channels (the axons) may introduce further transmission limits in addition to the limits concluded from the information theory. The time-aware operating model enables us to explain why (depending on the researcher's point of view) the operation can be considered either purely analog or purely digital.

References

[1]	W. S. McCulloch, W. Pitts, A logical calculus of the ideas immanent in nervous activity, J. Bull. Math. Biophys, 5 (1943), 115–133. https://doi.org/10.1007/BF02478259 doi: 10.1007/BF02478259
[2]	W. Pitts, W. S. McCulloch, How we know universals the perception of auditory and visual forms, J. Bull. Math. Biophys, 9 (1947), 127–147. https://doi.org/10.1007/BF02478291 doi: 10.1007/BF02478291
[3]	C. E. Shannon, A mathematical theory of communication, Bell System Techn. J., 27 (1948), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x doi: 10.1002/j.1538-7305.1948.tb01338.x
[4]	J. Végh, Á. J. Berki, Towards generalizing the information theory for neural communication, Entropy, 24 (2022), 1086. https://doi.org/10.3390/e24081086 doi: 10.3390/e24081086
[5]	L. Nizami, Information theory is abused in neuroscience, Cybern. Human Knowing, 26 (2019), 47–97.
[6]	C. E. Shannon, The Bandwagon, IRE Trans. Inf. Theory, 2 (1956), 3.
[7]	D. H. Johnson, Information theory and neuroscience: Why is the intersection so small?, in 2008 IEEE Information Theory Workshop, (2008), 104–108. https://doi.org/10.1109/ITW.2008.4578631
[8]	M. D. McDonnell, S. Ikeda, J. H. Manton, An introductory review of information theory in the context of computational neuroscience, Biol. Cybern., 105 (2011). https://doi.org/10.1007/s00422-011-0451-9 doi: 10.1007/s00422-011-0451-9
[9]	J. N. Carbone, J. A. Crowder, The great migration: Information content to knowledge using cognition based frameworks, in Biomedical Engineering, Springer, New York, (2011), 17–46. https://doi.org/10.1007/978-1-4614-0116-2_2
[10]	J. Végh, Why do we need to Introduce Temporal Behavior in both Modern Science and Modern Computing, Global J. Comput. Sci. Technol., 20 (2020), 13–29. https://doi.org/10.34257/GJCSTAVOL20IS1PG13 doi: 10.34257/GJCSTAVOL20IS1PG13
[11]	J. Végh, Revising the classic computing paradigm and its technological implementations, Informatics, 8 (2021). https://doi.org/10.3390/informatics8040071 doi: 10.3390/informatics8040071
[12]	J. Végh, Á. J. Berki, On the role of speed in technological and biological information transfer for computations, Acta Biotheor., 70 (2022), 26. https://doi.org/10.1007/s10441-022-09450-6 doi: 10.1007/s10441-022-09450-6
[13]	G. Buzsáki, J. Végh, Space, Time and Memory, 1st edition, Oxford University Press, in print, 2023.
[14]	H. Minkowski, Die Grundgleichungen für die electromagnetischen Vorgänge in bewegten Körpern, Nachr. Königl., Ges. der Wissenschaften zu Göttingen (in German), (1908), 53–111.
[15]	L. Pyenson, Hermann Minkowski and Einstein's special theory of relativity, Arch. Hist. Exact Sci., 17 (1977), 71–95. https://doi.org/10.1007/BF00348403 doi: 10.1007/BF00348403
[16]	J. M. Gomes, C. Bédard, S. Valtcheva, M. Nelson, V. Khokhlova, P. Pouget, et al., Intracellular impedance measurements reveal non-ohmic properties of the extracellular medium around neurons, Biophys. J., 110 (2016), 234–246. https://doi.org/10.1016/j.bpj.2015.11.019 doi: 10.1016/j.bpj.2015.11.019
[17]	D. Johnston, S. M. S. Wu, Foundations of Cellular Neurophysiology, Massachusetts Institute of Technology, 1995.
[18]	C. Koch, Biophysics of Computation, Oxford University Press, 1999.
[19]	B. Podobnik, M. Jusup, Z. Tiganj, W. X. Wang, J. M. Buld, H. E. Stanley, Biological conservation law as an emerging functionality in dynamical neuronal networks, PNAS, 45 (2017), 11826–11831. https://doi.org/10.1073/pnas.1705704114 doi: 10.1073/pnas.1705704114
[20]	J. von Neumann, First draft of a report on the EDVAC, IEEE Ann. Hist. Comput., 15 (1993), 27–75. https://doi.org/10.1109/85.238389 doi: 10.1109/85.238389
[21]	C. Koch, T. A. Poggio, A theoretical analysis of electrical properties of spines, Proc. R. Soc. Ser. B Biol. Sci., 218 (1983), 455–477.
[22]	G. Somjen, Sensory Coding in the Mammalian Nervous System, New York: Meredith Corporation, 1972.
[23]	C. Fiorillo, J. Kim, S. Hong, The meaning of spikes from the neuron's point of view: predictive homeostasis generates the appearance of randomness, Front. Comput. Neurosci., 8 (2014). https://doi.org/10.3389/fncom.2014.00049 doi: 10.3389/fncom.2014.00049
[24]	T. J. Sejnowski, The computer and the brain revisited, IEEE Ann. History of Computing, 11 (1989), 197–201. https://doi.org/10.1109/MAHC.1989.10028 doi: 10.1109/MAHC.1989.10028
[25]	D. Tsafrir, The context-switch overhead inflicted by hardware interrupts (and the enigma of do-nothing loops), in Proceedings of the 2007 workshop on Experimental computer science, ACM, New York, USA, (2007), 4–es.
[26]	F. M. David, J. C. Carlyle, R. H. Campbell, Context switch overheads for Linux on ARM platforms, in Proceedings of the 2007 workshop on Experimental computer science, ACM, New York, USA, (2007), 3–es. http://doi.acm.org/10.1145/1281700.1281703
[27]	J. von Neumann, The Computer and the Brain (The Silliman Memorial Lectures Series), New Haven, Yale University Press, 2012.
[28]	P. Mitra, Fitting elephants in modern machine learning by statistically consistent interpolation, Nat. Mach. Intell., 3 (2021), 378–386. https://doi.org/10.1038/s42256-021-00345-8 doi: 10.1038/s42256-021-00345-8
[29]	R. P. Feynman, Feynman Lectures on Computation, CRC Press, 2018.
[30]	Y. A. Cengel, On entropy, information, and conservation of information, Entropy, 23 (2021), 779. https://doi.org/10.3390/e23060779 doi: 10.3390/e23060779
[31]	A. Borst, F. E. Theunissen, Information theory and neural coding, Nat. Neurosci., 2 (1999), 947–957. https://doi.org/10.1038/14731 doi: 10.1038/14731
[32]	R. Brette, Is coding a relevant metaphor for the brain, Behav. Brain Sci., 42 (2018), e215. https://doi.org/10.1017/S0140525X19000049 doi: 10.1017/S0140525X19000049
[33]	N. Brenner, S. P. Strong, R. Koberle, W. Bialek, R. R. de Ruyter van Steveninck, Synergy in a neural code, Neural Comput., 12 (2000), 1531–1552. https://doi.org/10.1162/089976600300015259 doi: 10.1162/089976600300015259
[34]	S. P. Strong, R. R. de Ruyter van Steveninck, W. Bialek, R. Koberle, On the application of information theory to neural spike trains, Neural Comput., 1998 (1998), 621–632.
[35]	M. Li, J. Z. Tsien, Neural code—neural self-information theory on how cell-assembly code rises from spike time and neuronal variability, Front. Cell. Neurosci., 11 (2017). https://doi.org/10.3389/fncel.2017.00236 doi: 10.3389/fncel.2017.00236
[36]	I. Csiszár, J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems, Cambridge Universiy Press, 2011.
[37]	C. Wilson, Up and down states, Scholarpedia J., 6 (2008), 1410. https://doi.org/10.4249/scholarpedia.1410 doi: 10.4249/scholarpedia.1410
[38]	D. Levenstein, G. Girardeau, J. Gornet, A. Grosmark, R. Huszár, A. Peyrache, et al., Distinct ground state and activated state modes of spiking in forebrain neurons, bioRxiv, 2021. https://doi.org/10.1101/2021.09.20.461152
[39]	S. Eddy, What is a hidden markov model, Nat. Biotechnol., 22 (2004), 1315–1316. https://doi.org/10.1038/nbt1004-1315 doi: 10.1038/nbt1004-1315
[40]	S. B. Laughlin, Energy as a constraint on the coding and processing of sensory information, Curr. Opin. Neurobiol., 11 (2001), 475–480. https://doi.org/10.1016/S0959-4388(00)00237-3 doi: 10.1016/S0959-4388(00)00237-3
[41]	H. Barlow, Redundancy reduction revisited, Network: Comput. Neural Syst., 12 (2001), 241. https://doi.org/10.1088/0954-898X/12/3/301 doi: 10.1088/0954-898X/12/3/301
[42]	T. Berger, W. B. Levy, A mathematical theory of energy efficient neural computation and communication, IEEE Trans. Inf. Theory, 56 (2010), 852–874. https://doi.org/10.1109/TIT.2009.2037089 doi: 10.1109/TIT.2009.2037089
[43]	D. M. MacKay, W. S. McCulloch, The limiting information capacity of a neuronal link, Bull. Math. Biophys., 14 (1952), 127–135. https://doi.org/10.1007/BF02477711 doi: 10.1007/BF02477711
[44]	F. Rieke, D. Warland, W. Bialek, Spikes: Exploring the Neural Code, 2nd edition, The MIT Press, 1997.
[45]	J. V. Stone, Principles of Neural Information Theory, Sebtel Press, Sheffield, UK, 2018.
[46]	P. Sterling, S. Laughlin, Principles of Neural Design, 1st edition, The MIT Press, 2017.
[47]	P. M. DiLorenzo, J. D. Victor, Spike Timing: Mechanisms and Function, 1st edition, CRC Press, 2013.
[48]	I. Nemenman, G. D. Lewen, W. Bialek, R. R. de Ruyter van Steveninck, Neural coding of natural stimuli: Information at sub-millisecond resolution, PLoS Comput. Biol., 4 (2008), 1–12. https://doi.org/10.1371/journal.pcbi.1000025 doi: 10.1371/journal.pcbi.1000025
[49]	A. Losonczy, J. Magee, Integrative properties of radial oblique dendrites in hippocampal CA1 pyramidal neurons, Neuron, 50 (2006), 291–307. https://doi.org/10.1016/j.neuron.2006.03.016 doi: 10.1016/j.neuron.2006.03.016
[50]	R. B. Stein, The information capacity of nerve cells using a frequency code, Biophys. J., 6 (1967), 797–826. https://doi.org/10.1016/S0006-3495(67)86623-2 doi: 10.1016/S0006-3495(67)86623-2
[51]	S. P. Strong, R. Koberle, R. R. de Ruyter van Steveninck, W. Bialek, Entropy and information in neural spike trains, Phys. Rev. Lett., 80 (1998), 197–200. https://doi.org/10.1103/PhysRevLett.80.197 doi: 10.1103/PhysRevLett.80.197
[52]	R. Sarpeshkar, Analog versus digital: Extrapolating from electronics to neurobiology, Neural Comput., 10 (1998), 1601–1638. https://doi.org/10.1162/089976698300017052 doi: 10.1162/089976698300017052
[53]	S. B. Laughlin, R. R. de Ruyter van Steveninck, J. C. Anderson, The metabolic cost of neural information, Nat. Neurosci., 1 (1998), 36–41. https://doi.org/10.1038/236 doi: 10.1038/236
[54]	P. Singh, P. Sahoo, K. Saxena, J. S. Manna, K. Ray, S. Kanad, et al., Cytoskeletal filaments deep inside a neuron are not silent: They regulate the precise timing of nerve spikes using a pair of vortices, Symmetry, 13 (2021). https://doi.org/10.3390/sym13050821 doi: 10.3390/sym13050821
[55]	M. Stemmler, C. Koch, How voltage-dependent conductances can adapt to maximize the information encoded by neuronal firing rate, Nat. Neurosci., 2 (1999), 521–527. https://doi.org/10.1038/9173 doi: 10.1038/9173
[56]	P. Khorsand, F. Chance, Transient responses to rapid changes in mean and variance in spiking models, PLoS ONE, 3 (208), e3786. doi.org/10.1371/journal.pone.0003786 doi: 10.1371/journal.pone.0003786
[57]	K. Kar, S. Kornblith, E. Fedorenko, Interpretability of artificial neural network models in artificial intelligence versus neuroscience, Nature Mach. Intell., 4 (2022), 1065–1067. https://doi.org/10.1038/s42256-022-00592-3 doi: 10.1038/s42256-022-00592-3
[58]	R. Vicente, M. Wibral, M. Lindner, G. Pipa, Transfer entropy—a model-free measure of effective connectivity for the neurosciences, J. Comput. Neurosci., 30 (2011), 45–67. https://doi.org/10.1007/s10827-010-0262-3 doi: 10.1007/s10827-010-0262-3
[59]	K. Hlaváčková-Schindler, M. Paluš, M. Vejmelka, J. Bhattacharya, Causality detection based on information-theoretic approaches in time series analysis, Phys. Rep., 441 (2007), 1–46. https://doi.org/10.1016/j.physrep.2006.12.004 doi: 10.1016/j.physrep.2006.12.004
[60]	A. Abbott, Documentary follows implosion of billion-euro brain project, Nature, 588 (2020), 215–216. https://doi.org/10.1038/d41586-020-03462-3 doi: 10.1038/d41586-020-03462-3
[61]	A. G. Dimitrov, J. P. Miller, Neural coding and decoding: communication channels and quantization, Network: Comput. Neural Syst., 12 (2001), 441. https://doi.org/10.1088/0954-898X/12/4/303 doi: 10.1088/0954-898X/12/4/303
[62]	G. M. Shepherd, The Synaptic Organization of the Brain, 5 edition, Oxford Academic, New York, 2006.
[63]	W. B. Levy, V. G. Calvert, Communication consumes 35 times more energy than computation in the human cortex, but both costs are needed to predict synapse number, Proc. Nat. Acad. Sci., 118 (2021), e2008173118. https://doi.org/10.1073/pnas.2008173118 doi: 10.1073/pnas.2008173118
[64]	H. Simon, Why we need Exascale and why we won't get there by 2020, in Conference: AASCTS2: Exascale Radioastronomy Meeting, 2014. Accesse date: Oct. 23, 2021. Available from: https://www.researchgate.net/publication/261879110_Why_we_need_Exascale_and_why_we_won't_get_there_by_2020.
[65]	J. Végh, Finally, how many efficiencies the supercomputers have, J. Supercomput., 76 (2020), 9430–9455. https://doi.org/10.1007/s11227-020-03210-4 doi: 10.1007/s11227-020-03210-4
[66]	S. Williams, A. Waterman, D. Patterson, Roofline: An insightful visual performance model for multicore architectures, Commun. ACM, 52 (2009), 65–76. https://doi.org/10.1145/1498765.1498785 doi: 10.1145/1498765.1498785
[67]	F. Zeldenrust, S. de Knecht, W. J. Wadman, S. Denève, B. Gutkin, Estimating the information extracted by a single spiking neuron from a continuous input time series, Front. Comput. Neurosci., 11 (2017), 49. https://doi.org/10.3389/fncom.2017.00049 doi: 10.3389/fncom.2017.00049
[68]	L. Eisenman, C. Emnett, J. Mohan, C. Zorumski, S. Mennerick, Quantification of bursting and synchrony in cultured hippocampal neurons, J. Neurophysiol., 114 (2015). https://doi.org/10.1152/jn.00079.2015 doi: 10.1152/jn.00079.2015
[69]	D. H. Johnson, Dialogue Concerning Neural Coding and Information Theory, 2003. Available from: http://www.ece.rice.edu/dhj/dialog.pdf.
[70]	R. R. de Ruyter van Steveninck, G. D. Lewen, S. P. Strong, R. Koberle, W. Bialek, Reproducibility and variability in neural spike trains, Science, 275 (1997), 1805–1808. https://doi.org/10.1126/science.275.5307.1805 doi: 10.1126/science.275.5307.1805
[71]	B. Sengupta, S. Laughlin, J. Niven, Consequences of converting graded to action potentials upon neural information coding and energy efficiency, PLoS Comput. Biol., 1 (2014). https://doi.org/10.1371/journal.pcbi.1003439 doi: 10.1371/journal.pcbi.1003439
[72]	S. J. van Albada, A. G. Rowley, J. Senk, M. Hopkins, M. Schmidt, A. B. Stokes, et al., Performance comparison of the digital neuromorphic hardware spiNNaker and the neural network simulation software NEST for a full-scale cortical microcircuit model, Front. Neurosci., 12 (2018), 291. https://doi.org/10.3389/fnins.2018.00291 doi: 10.3389/fnins.2018.00291
[73]	J. Végh, How Amdahl's Law limits performance of large artificial neural networks, Brain Inf., 6 (2019), 1–11. https://doi.org/10.1186/s40708-019-0097-2 doi: 10.1186/s40708-019-0097-2
[74]	J. Végh, Which scaling rule applies to Artificial Neural Networks, Neural Comput. Appl., 33 (2021), 16847–16864. https://doi.org/10.1007/s00521-021-06456-y doi: 10.1007/s00521-021-06456-y
[75]	Human Brain Project, E. Human Brain Project, 2018. Available from: https://www.humanbrainproject.eu/en/.
[76]	A. Mehonic, A. Kenyon, Brain-inspired computing needs a master plan, Nature, 604 (2022), 255–260. https://doi.org/10.1038/s41586-021-04362-w doi: 10.1038/s41586-021-04362-w
[77]	D. Markovic, A. Mizrahi, D. Querlioz, J. Grollier, Physics for neuromorphic computing, Nat. Rev. Phys., 2 (2020), 499–510. https://doi.org/10.1038/s42254-020-0208-2 doi: 10.1038/s42254-020-0208-2

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)