Literary Insights

BOOK LINK:

Here is a summary of the key points from the book “The Computer and the Brain” by John von Neumann:

Von Neumann laid out the basic architecture of modern digital computers, known as the von Neumann architecture. This separates the central processing unit from the memory, with instructions and data stored in the same memory. This became the predominant design for digital computers.
He explores the mathematical foundations of digital computing, including binary representation of numbers, logic gates, abstraction of computer components, control systems, algorithms, memory hierarchies, parallel vs serial computing, and more.
Von Neumann draws direct analogies between biological neurons and abstract computing components like logic gates, seeing both as binary digital systems based on signaling pulses that stimulate binary responses.
He proposes that memory in the brain could physically take many forms, but at a logical level is analogous to computer memory storage and retrieval systems through addressing schemes.
von Neumann argues that while the brain and computers have differing physical properties, at the level of information processing and computation they are functionally equivalent based on Turing’s universality of computation.
He anticipates that understanding the brain’s computational principles through reverse engineering could allow replicating and enhancing its capabilities with artificial systems, foreshadowing modern artificial intelligence research.
Von Neumann was one of the first to recognize the exponential pace of technological progress and its implications for transforming human civilization through intelligence amplification.

So in summary, the book lays the groundwork for understanding the brain as an information processing system and exploring its parallels and relationships to modern computing based on von Neumann’s foundational work in computer architecture and digital logic.

The passage discusses how even with highly accurate communication channels, errors are likely in sending large files or messages. A single bit error can invalidate digital data like a computer program.
It describes how Claude Shannon’s pioneering work in information theory established that reliable communication is possible even over noisy channels, through the use of error correction codes and redundancy. This fundamental discovery enabled digital communication and computation.
Redundancy reduces errors by repeating bits or using error-checking codes. Shannon proved this approach can make the error rate as low as desired, achieving arbitrarily high accuracy from low-accuracy channels.
Applications of Shannon’s work include modems, CDs/DVDs, and ensuring reliable memory and computation despite potential errors. His ideas were crucial to the development of the digital information age.
Alan Turing’s 1936 paper established the theoretical foundations of computation by describing a simple conceptual “Turing machine” that could solve any solvable problem, thus proving the universality of computation.
John von Neumann’s 1945 paper presented the architecture of the stored-program computer, known as the von Neumann model, which has remained the dominant design underpinning all modern computers.

Here is a summary of the key points about time from the passage:

Turing’s original concept of a Turing machine was not very practical because it took an inefficient amount of time to move the sequential memory tape back and forth to store and retrieve data.
Von Neumann proposed improvements like a random access memory that allowed for faster retrieval of any data item without needing to move a tape. This made computational machines more practical.
Other innovations like the stored program concept (introduced a decade earlier by Von Neumann) allowed computers to be reprogrammed more easily and quickly for different tasks, fulfilling Turing’s vision of universality more effectively.
Babbage conceptualized the idea of a stored program over a century before Von Neumann in his Analytical Engine design from 1837, but the technology of his time prevented it from being built.
The passage emphasizes how innovations over time like Von Neumann’s architecture, Shannon’s information theory, and work on artificial intelligence have improved our ability to build intelligent computational machines and endow them with human-level intelligence, which was one of Turing’s original goals.

So in summary, the key ideas around time discussed are how different concepts and innovations over the decades have made computational machines and the pursuit of artificial intelligence more practical, efficient and capable of performing complex human-level tasks in reasonable amounts of time.

Von Neumann proposed that memory and thought are stored as patterns of activity across hierarchies of pattern recognition systems in the brain. Low-level systems recognize basic visual features, while higher-level systems recognize letters, words, objects, etc.
Memories are encoded as sequences of these higher-level patterns, not like a video playback. Recall involves reconstructing experiences from sparse cues, not explicitly remembering every detail.
The brain processes information massively in parallel across billions of neurons, allowing for rapid decisions. This was seen as key to its abilities, even in 1956 when computation was slow.
While the brain is analog, digital computers can simulate analog values precisely enough. Massive parallelism can also be simulated through parallel von Neumann machines like supercomputers.
Von Neumann predicted computers would one day match the brain’s computational speed and capabilities, laying the foundations for worrying about a technological singularity.
The core ideas about computationalism, neural hierarchies, parallelism, and eventual machine emulation of the brain remain remarkably valid considering when the work was written. It provides a foundational framework still relevant today.
From the 1950s until recently, computer science and neuroscience developed largely independently, with little cross-pollination. Computer scientists focused on programming and hardware, not biology, while neuroscientists studied brain tissue and neuron activity experimentally but not computationally.
This separate development occurred despite an apparent overlap in interests, suggesting the biological brain operates computationally very differently than traditional digital computers.
John von Neumann recognized key differences between neurons and digital logic gates that questioned applying a digital/von Neumann model to the brain. Neurons have many more inputs than outputs, operate much slower, and represent values less accurately than digital computers.
This led von Neumann to conclude the brain cannot perform complex computations sequentially as digital computers do due to speed and accuracy limitations of neurons. However, the brain clearly performs sophisticated computations quickly.
Von Neumann suggested the brain exploits “logical breadth” through highly parallel processing across many synaptic connections, allowing massive parallel operations to compensate for the lack of “logical depth” of sequential steps.
Modern neuroscience has since confirmed the brain performs roughly 1016 operations per second through highly parallel synaptic processing, making it a massively parallel analog computing machine rather than a slow, inaccurate serial digital one. Von Neumann presciently speculated on this alternative computational strategy.
Von Neumann is credited with developing the modern computer architecture that underlies most 20th century computers. This involves the separation of processing, memory, input/output, and control units.
He began to realize the limitations of this digital, serial model and saw potential in a “shallow-depth but extraordinary-breadth” computational approach similar to the brain’s massive parallel neural networks.
The brain uses a “synaptic” approach with millions of neural connections that can perform computational transformations quickly and without recursively amplified errors. Electronic versions of synapses and neural networks could realize artificial intelligence.
While small artificial neural networks have shown promising skills, current models still lack the complexity and variety of real biological neurons and synapses. Neuroscience continues to reveal new aspects of brain computation beyond our initial models.
Von Neumann acknowledged limitations in early models of the brain and remained open-minded about new discoveries, as neuroscience continues to provide surprises.
While his original computer architecture revolutionized computing, von Neumann made important contributions to understanding the brain and should be considered “the Newton of the mind” for his insights into alternative models of intelligence beyond traditional computing.
John von Neumann was a mathematician who became interested in studying the nervous system and brain from a mathematical perspective, particularly information theory and modeling the brain as a type of computational system.
During the postwar years, he did work in fields like meteorology and nuclear physics where numerical calculations could provide insights. He joined the Atomic Energy Commission in 1955.
In 1955, von Neumann was diagnosed with bone cancer after developing pain in his shoulder. The cancer later spread to his spine. Though he continued working while receiving treatment, his health steadily declined over the next two years.
Von Neumann had been tasked with delivering the Silliman Lectures at Yale in 1956. However, as his illness progressed he was unable to complete and deliver the full lectures. Yale allowed him to submit an unfinished manuscript, which forms the basis of this work.
The introduction outlines von Neumann’s “mathematician’s point of view” for studying the nervous system, with a focus on information theory, logical systems, statistics and modeling the brain as a type of complex computational automata or machine.
The first part discusses different types of computing machines, particularly analog computers that represent numbers via continuous physical variables versus digital computers that use discrete representations. It provides some context for how von Neumann viewed modeling the brain computationally.
Digital machines represent numbers as sequences of decimal digits, which are in turn represented by groups of binary markers (typically electrical pulses).
Basic arithmetic operations like addition, subtraction, multiplication and division are done through logical rules and repetitive sequences rather than physical processes. Especially multiplication and division have complex logical structures.
Early analog machines performed operations sequentially by having enough physical components pre-wired to represent the necessary calculations.
Later analog machines used electrical connections that could be changed by plugging, controlled by electromechanical relays responding to paper tapes.
The tapes could start, stop and restart the calculation based on predetermined numerical conditions in the machine, like a number becoming negative or exceeding a threshold.
This introduced a level of programmable, logical control over the sequencing of operations to solve problems, moving beyond just fixed wiring of components. It paved the way for fully programmable digital machines controlled by logical instructions on tapes or other media.
Digital machines typically have only one component/organ that performs each basic operation, unlike analog machines which may need multiple components per operation depending on the problem.
This necessitates a memory organ/registers in digital machines to store intermediate and final results passively. Memory registers can store, retrieve, and output numbers.
There are two basic types of control systems for digital machines:
1. Control sequence point system - Points are connected to operation organs and memory registers. They activate successors in sequence to control the calculation flow. Branching points allow conditional flows.
2. Memory-stored control system - “Orders” stored in memory registers indicate operations, input/output registers, and successor orders. Orders are essentially numbers specifying the control flow. Successors can be explicit or implicit based on order addresses. Conditional transfers allow branching.

The key aspects are that digital machines need memory to store results with their single components per operation, and the two basic control approaches of hard-wired sequence points vs programmatic orders stored in memory.

Memory-stored control is a more flexible mode of control for computing machines compared to plugged control. In this mode, the control sequence (i.e. the problem/instructions) are stored in memory rather than represented by physical connections.
This allows the machine to extract, process, and return numbers/instructions to memory, changing the contents of memory and thus modifying the control sequence and computational processes over time. More complex iterative and non-iterative processes become possible.
Control can combine plugged and memory-stored modes. The plugged configuration can be represented digitally in memory and loaded/modified under memory control. Orders allow resetting plugging from memory or changing individual connections.
Mixed analog/digital machines are also possible, where parts are analog and parts digital but can communicate numerically and be under common or separate control. Conversion is needed between digital and analog representations.
Numbers can also be represented partially analog and partially digital (e.g. using pulse density), and arithmetic operations can be designed accordingly, though precision may be limited compared to fully digital representations.
Digital machines can achieve much higher precision than analog machines, and precision can be more easily increased in digital systems through relatively simple design changes. High precision (e.g. 12 decimal places) becomes typical and important for digital machines.
Higher precision levels like 1:10^5 are often inadequate for complex problems in fields like mathematics, science and economics. Precision levels of 1:10^10 to 1:10^12 are routinely needed for important problems.
This is due to the inherent structure of current mathematical and numerical procedures, which break problems down into long sequences of basic arithmetic operations. Errors accumulate non-linearly as operations are chained together.
Modern analog machines typically have 100-200 operational components. Digital machines have thousands of active components that perform logical and arithmetic functions. Memory components are classified by access time.
Registers built from active components like vacuum tubes have the fastest access time but are expensive. They are needed for short-term storage but not large memory capacities. Other memory types have slower access but larger storage at lower cost.
Machine speeds have improved from relay machines at 10-2 seconds per operation to vacuum tubes at 10-5-10-6 seconds to solid state devices now at 10-6 seconds and expected to reach 10-8-10-9 seconds. Larger machines have thousands to tens of thousands of active components.
The human nervous system is composed of nerve cells called neurons that communicate through nerve impulses.
When stimulated, a neuron generates a nerve impulse that propagates along its branch called an axon.
The nerve impulse has electrical, chemical, and mechanical aspects. It involves an electrical potential change along the axon as well as changes in ion concentrations, chemical properties of the axon membrane, and likely mechanical changes to membrane molecules.
The nerve impulse is a reproducible, unitary response that propagates at a fixed or variable speed depending on the neuron. It lasts around 1 millisecond.
After the impulse passes, all conditions and components of the neuron revert back to their original states.
The ability of neurons to generate digital, all-or-nothing responses suggests the nervous system functions in a manner similar to digital computers, which will be explored further in the comparison to modern computing machines.
Nerve impulses occur on a molecular scale, at the thickness of cell membranes (a few thousandths of a micron). On this scale, electrical, chemical and mechanical processes are not clearly distinct - changes in one always affect the others.
Stimulation of a nerve impulse normally occurs near the cell body. If successful, a standard nerve impulse is generated and propagates along the axon, relatively independently of the stimulation details.
Nerve impulses can be viewed digitally as binary markers - the presence or absence of a pulse representing 1 or 0. Nerve cells respond conditionally to combinations/timing of input pulses by emitting a pulse or not.
Stimulation reaction time is 10-4 to 10-2 seconds, depending on definition. Significant recovery time is needed before a second standard response. Vacuum tubes and transistors can respond orders of magnitude faster, 10-6 to 10-7 seconds.
Individual neuron sizes vary greatly but their active membrane is on the order of 10-5 cm, while artificial components like vacuum tubes and transistors have active distances of 10-1 to 10-2 cm, so natural components are smaller by about a factor of 1000.
The human central nervous system (brain) occupies about 1 liter of space and contains around 100 billion (1010) neurons. This allows around 10-7 cm3 of space per neuron.
Vacuum tubes and transistors can be packed more densely than neurons. Present technologies allow packing around 1000-10,000 vacuum tubes in a few cubic feet, or around 1000-10,000 transistors in around 1 cubic foot. This corresponds to around 10-2-10 cm3 per component. So natural components (neurons) are around 108-109 times more space-efficient than artificial components.
The human brain dissipates around 10 watts of power. Given the 100 billion neurons, this corresponds to around 10-9 watts per neuron. Vacuum tubes dissipate 5-10 watts each and transistors around 10-1 watts each. So natural components are 108-109 times more energy-efficient.
In terms of number of logical operations per unit size/power, natural components are around 108-109/104-105 = 104 times more efficient than artificial ones.
Large natural systems favor parallel processing while large artificial systems favor serial processing due to these efficiency factors. Parallelism is not always substitutable for serialism though.
Actual neuron behavior is more complex than simple “AND” and “OR” logic. Neurons have many synapses and stimulus patterns can depend on synapse locations and numbers, not just counts.
The stimulation of neurons can be complex and not simply defined by attaining a threshold number of pulses. Factors like the number, coverage, and spatial relations of stimulating regions on the neuron can impact stimulation.
The threshold refers to the minimum number of simultaneous stimulating pulses required to activate a neuron. However, stimulation criteria may be more complicated than just reaching a threshold.
The summation time refers to the period within which two pulses can be considered simultaneous even if not exactly overlapping. Summation times may vary and be non-sharp concepts.
Neurons can have properties beyond simple stimulus-response, like fatigue and recovery states that change response characteristics.
Receptors respond to stimuli other than nerve pulses, like light, sound, pressure. Their stimulation criteria can be a threshold intensity or a response to changes in the stimuli, not just its level.
The complexities may allow analog or “mixed” behaviors in addition to the digital organization of the nervous system. They also increase the count of basic active organs beyond just counting neurons.
The presence of memory in the nervous system is hypothesized based on artificial systems, but its nature and location are unknown. It is likely a large-capacity memory necessary for complicated functions.
Memory capacity in artificial computing machines is usually estimated in bits, with typical modern machines having capacities between 105 to 106 bits.
Estimating the nervous system’s memory capacity using the same conventions, it would likely be much larger given the nervous system is a more complex “automaton” than computers.
A rough estimate based on assumptions about number of neurons, inputs per second, and lifespan suggests a capacity of around 2.8×1020 bits for a human nervous system.
The physical embodiment of memory in the nervous system is unknown. Some suggestions include variable neuron thresholds/connections over time, genetic/chemical factors, or cyclical neuron interactions similar to computer memory.
Analogies to computer memory technologies show the memory components need not be the same as the basic active neurons. Computer memories later moved to different technologies than just logic gates/tubes for efficiency.
In summary, while memory capacity can be estimated, the specific physical implementation of memory storage in the nervous system remains an open fundamental question. Both digital and analog components may play a role.
The nervous system involves both digital (nerve pulses) and analog (chemical/mechanical) processes that alternate and interact, giving it a mixed character.
Genetic phenomena provide an example, as genes are digital but their effects involve analog chemical/enzyme processes.
Codes are logical instruction systems that cause an automaton to perform organized tasks. Complete codes specify all details, while short codes make one machine imitate another’s behaviors.
Short codes examine each order to see if it matches the imitated machine, then provide instructions to replicate that machine’s actions. This allows more complexity than the machine’s native instructions.
The nervous system has both logical and numerical/arithmetic components, as complicated functions often involve quantities and formulas. Numerical precision is important for tasks like maintaining physiological constants.
Logic alone cannot fully describe complex situations, so the nervous system’s arithmetic abilities are as important as its logical ones when viewed as an automaton controlling biological processes.
The nervous system processes numerical data using frequency-modulated pulse trains rather than a digital system. Intensities are encoded as pulse frequencies between 50-200 pulses per second.
This statistical, analog encoding allows only low precision of around 2-3 decimals, unlike digital systems which require much higher precision like 10-12 decimals for complex tasks.
However, it provides higher reliability since losing a single pulse does not completely distort the meaning, only slightly changes the frequency/intensity encoded.
The low precision is due to the statistical nature of the encoding and the “great logical depth” of the computations, i.e. many operations performed in series which allows errors to accumulate.
The nervous system trades off arithmetical precision for logical reliability by using this statistical encoding where message meaning depends on frequency distributions rather than precise pulse positions.
Other statistical properties of pulse trains beyond just frequency could also encode information, like correlation coefficients between multiple fibers.
The “language” used in the brain is not mathematical but differs structurally from formal logics/math through its lower depth and statistical nature. It may correspond to a “short code” rather than a complete symbolic system.

When discussing mathematics, we are often using a secondary language built on top of the primary language actually used by the central nervous system. The outward forms of mathematics we use are not entirely relevant for understanding the mathematical/logical language of the CNS. However, the points made about reliability, logical depth and arithmetic show that whatever system the CNS uses internally, it must differ meaningfully from what we consciously consider mathematics. While we express mathematics through our external languages, the internal language of the mind processing mathematics potentially has its own nature we do not fully grasp.

#book-summary

The Computer and the Brain - John von Neumann