Jeff Hawkins had an interesting conversation with Lex Fridman, referring to Hawkins’s two books — On Intelligence, and A Thousand Brains.
Hawkins proposes that the same mechanism that we use to model a water bottle can be used to model high-level thoughts. This means that knowledge itself is fractal, reflecting the structure of the physical world. “The demise of humanity” or any other high-level abstraction, he proposes, is modeled the same way.
This makes sense. Whatever the higher levels of abstraction are that are at play, it makes sense that they would follow the same mechanism and process as less high-level understanding. A “higher order concept” is hierarchical, but so is a physical object. Since we function in the physical world, it stands to reason that the structures that underlie our understanding be derived from the relationships among objects in that world.
It is also likely that language reflects the relationships among objects in the world. That is probably why large language models work as well as they do: language emerged as a way to store and pass information from one human being to another, reflecting relationships in the physical world. English is replete with examples. “Replete with” means “full of,” just as a bucket is full of apples. “Examples” is a general term which can be applied to any number of things. “What’s an example of a word that’s difficult to spell?” “A professional athlete has an obligation to set an example as a role model”. One way to understand what a word means is to use it in several examples.
This is how we learn, how we understand, and how we communicate.
The question then becomes: what is a mechanism of generalization/abstraction that is flexible enough to be applied in multiple domains, at multiple levels within and across domains? Human beings are able to generalize so well for this reason — that when you come upon a new or unexpected event, you are able to adapt to it. It is a generalization of something you know, something you’re already familiar with. It’s a variation on a theme — just as words are variations on themes.
This also suggests (though does not prove) that it is continuous. The idea of a continuous function is one of the critical backbones of deep learning. Because the function is continuous, a small change in x can only result in a small change in y —that’s a feature of continuity. And because the function is continuous, it is differentiable. That is what allows us to use back propagation to adjust the loss and get closer to the truth (greater accuracy) in solving deep learning problems.
The world itself is hierarchical. You just go into a room — it’s composed of other components. The kitchen has a refrigerator, the refrigerator has a door, the door has a hinge, the hinge has screws and pin. The modeling system that exists in every cortical column learns the hierarchical structure of objects. Hawkins suggests that Vernon Mountcastle made a discovery many years ago, which is that there’s a single cortical algorithm underlying everything we’re doing. So common sense concepts and higher level concepts are all represented in the same way; they’re set in the same mechanisms.
https://www.youtube.com/watch?v=Z1KwkpTUbkg&ab_channel=LexFridman