What is learning?
Really, what is it?
I think it might be helpful to ask a broader probing question first: “what can we definitively say about anything which might be categorized as learning?” This should help provide at least a rough outline of what it is which can inform further exploration.
So, what can we definitively say about learning?
Learning involves a change in the internal states of a system.
Why can we say this? Well, if we’re saying a system has learned, that implies there was a time where it hadn’t learned yet. If we’re now saying that learning has occurred, something must have changed (otherwise it would still be in its previous unlearned state). But what changed? It’s definitely not a state external to the system. If it were, we couldn’t say the system we’re talking about learned since a system only consists of its internal states. Since we know something must have changed and we know it can’t be any external states, then the only possible states we’re left with are the internal states of the system. Thus, saying “this system has learned” implies “some of the internal states of the system have changed.”
Some examples:
- In the case of human beings, an individual human is the system and the internal states being modified are brain states.
- In the case of a computer, the system is whatever is required to run some ML model and the internal states being modified may be the weights of a neural network.
What is learned by a system is dependent, at least in part, on information external to that system.
Why can we say this? Consider the situation where we say that learning depends exclusively on information inside the system. If this were the case then no learning would need to take place. As we already showed, learning is a change to the internal states of a system. If the system already has what it needs then it doesn’t need to change and putting the system through a process of change just to end up back where it already was would just be a waste of time and energy. Thus, the changes in a system which we call learning must be at least somewhat dependent on external states.
Also, this doesn’t mean that you can’t learn from information already integrated somewhere in your mind. The context defines what the “system” is in that case. For example, you may be able to picture in your mind what was on the board in your calculus lecture. That doesn’t mean you’ve learned the information though. The system here is (loosely) the parts of your mind involved in understanding and executing on the concepts and the external states are (again, loosely) whichever parts of your mind are involved in recreating that image of the board.
The external states which the changes to internal states (aka learning) are dependent on must convey information.
Why can we say this? First we need to consider what it would mean for those external states to not convey information to the system.
Consider this informal description from Gregory Bateson of the formal definition of what information is: “Information is a difference that makes a difference.” The first difference is the difference between what the internal states say about what the external states would be and the actual values of the external states. The second difference is the difference between the previous internal states and the new ones which are the result of integrating the previous state with the conveyed information.
Now say the first difference was 0. In other words, the internal states and the external states are already completely aligned. If this was the case then the change to cause the second difference would never get generated because it would be unneeded. Since learning requires change to occur in the internal states of the system, we can’t say learning has occurred if an external state conveys 0 bits of information.