For a decision-maker, the nature of human cognition creates the challenges of transformational experience and the necessity for self-altering decisions. But the nature of connection systems also creates a series of important hurdles to both optimal decision-making in preference optimization and thoughtful decision-making around self-change.
Since connection systems can do amazing things (like become unbeatable Go players), it’s reasonable to wonder why, if our brains are a massive network of connection systems, they don’t consistently mimic perfect rationality (because obviously they don’t). Well…in theory they can, but there are reasons why they don’t. Most of the rationality problems that plague us are due to resource constraints on decision-making or the nature of connection systems when combined with limited experience. Connection systems are about learning, and we do not always or even often learn to be perfectly rational.
One of the central problems of decision-making is that a huge amount of important decision-making happens in real-time. This was true for our distant ancestors who lived in a dangerous and unpredictable world where we doubled as both predators and prey. It’s true for us as well as we drive down busy San Francisco streets or navigate TikTok. The only way to make fast decisions is by having a dedicated, well-trained connection system (real estate set aside in the brain) to solve the problem at a subconscious level. On the other hand, there’s only so much real-estate in the brain. For generalized problem solvers, it isn’t practical to have a dedicated connection system for every kind of problem. Especially for non-replicating or infrequent problems, we need a generalized problem-solving system. Fortunately, humans have both these things. Psychologists call these System 1 and System 2 (keep in mind that when psychologists use these terms it’s as a model of decision-making not cognitive structure).
System 1 is our decision-making apparatus for fast, low-cost decision-making. It’s sub-conscious, parallel, and requires little or no cognitive load. System 2 is the generalized problem-solving system. It can tackle anything, but it will always be slower to produce answers than System 1. Not only is System 2 slower, it’s also a lot more work intensive – what’s called cognitive load. The lack of speed and the cognitive load of System 2 make it impractical to use for many tasks. If you use System 2 to walk, you will walk very poorly and slowly. Ditto if you use it to type. Or ski. Or read. Or catch a baseball.
Of course, when you have two systems for decision-making, there must be ways for the two systems to interact. And interact they do. In general, System 2 (which typically requires conscious effort), can seize control of our decision-making when it wants to. However, it doesn’t always want to. Sometimes it will just approve System 1 decisions. Or reject them. It may incorporate System 1 decisions into its own deliberation. Or it may practice a kind of benign neglect – letting System 1 drive the ship. This can be beneficial (if you’re thinking about how to shoot a free-throw, you’re highly likely to miss) or harmful (if you’re distracted you may let unconscious decisions slip-by that you would have overridden if you were paying attention).
Because many of our basic preference judgements are embedded in System 1 processes and System 2 is often used to override those preferences with broader situational awareness, it’s easy to think of System 1 as the “desiring” part of our cognitive machinery and System 2 as the “rational” part of our decision-making apparatus. That’s not how it is. System 2 is the generalized part of our decision-making apparatus. It’s no more “rational” than System 1 unless we train it properly. What’s more, System 1, when properly applied, can be nearly perfect in execution and rationality. When a concert pianist is playing a piece, very little of the playing is driven by System 2 but none of it is the slightest bit irrational or desiring.
It is also misleading to think about these as two fundamentally distinct types of system. First, it’s unclear if much of System 2 capabilities are inborn, though we almost certainly have real-estate naturally reserved for these capabilities (as we do for language). The fundamental architecture of learning (tuning a connection system based on experience) is obviously inborn, but most of how we think in System 2 must be learned. Nor is System 1 a fixed thing. We can turn things we can do in System 2 into a System 1 process by repetition and practice. When we first learn to walk, we use our early System 2 capabilities to drive the process. A toddler has to think about walking to do it. But as every parent knows, a toddler will quickly learn to get better and better at walking. The brain is creating a dedicated system for managing walking, and the better that system becomes, the less System 2 (general cognitive effort) is involved. Eventually, there is no conscious thought involved – and that’s a good thing. It’s the same for almost every activity we do repeatedly. A huge part of what System 2 does is create trained System 1 decision-makers (system 2 provides the evaluative reinforcement) that can handle real-time processes.
After all, System 2 is itself a complex set of connection systems that require training and development. We are not born with the ability to do math, reason deductively, or evaluate statistical evidence – we must learn how and when to do these things. And while it’s possible that the extent to which System 2 overrides System 1 decisions is partially innate, that too is probably a function of learned experience.
The existence of these two systems explains a lot of sub-optimal decision-making. System 1 does what it’s been trained to do. That means it gives us simple, obvious decisions. Those decisions may reflect known preferences (I see an In-N-Out, I want a burger), past experiences (this road is safe to drive), or snap judgements based on limited data (she looks mean). System 1 connection systems are great at managing the situations they’ve been trained for, but they do NOT generalize effectively. When we don’t have the time or energy to reflect on those judgements, all we’re getting is a really simple connection system match of whatever seems closest. If you train a connection system to recognize dogs and then you give it a picture of an elephant standing in a field and tell it to find the dog…well, it’s got four legs, a head, and a tail! A lot of the time, when it seems like people aren’t thinking about their decisions it’s because, consciously, they really aren’t thinking about their decisions. They are minimizing cognitive load. That’s its own kind of decision and it can be a mistake, but it can also make perfect sense and sometimes it’s necessary.
The existence of generalized (S2) and dedicated (S1) systems in our cognitive machinery has implications for our understanding of cognitive dispositions and character. All decision-making is driven by interactions in our cognitive machinery between whatever S1 machinery (heavily trained connection systems) is active and however our S2 (generalized reasoning capabilities) has been trained. A decision-maker looking to acquire skills, improve dispositions, or develop virtues must make sure that they do so at both system levels if they expect to operate consistently in a certain way. This is part of the reason why virtue theorists and behavioral psychologists emphasize the role of habit and repetition. Only by doing things over and over do we burn them into System 1 responses that will happen reliably in real-time without undue cognitive load.
There is also a different, higher-level form of training here. A great deal of practical virtue training is about building up proper habits of System 2 intervention. This includes building cognitive endurance so that a decision-maker doesn’t over-rely on System1 decisions in stressful and complicated situations, and also training oneself to deploy System 2 when it will do the most good. Both these cases are relevant to situationist critiques of theories of virtue and personality. We should not expect the degree to which a decision-maker’s habits and dispositions are consistent to be, itself, consistent across people. The extent to which a person might be said to “have” a personality (at least in terms of a consistent set of dispositions and decision tendencies) may be quite variable. Inconsistency might, possibly, be described as its own kind of personality type, but it’s a very different sort of beast.
If the problems inherent in juggling two separate decision-making systems aren’t enough, there are fundamental aspects of connection-system based learning that make it hard for us to be consistently rational even when we are taking the time and effort to use a well-trained System 2. Take, for example, the way connection systems tune their internal connections. Almost all connection systems use a similar strategy when it comes to learning. They heavily weight initial observations on the assumption that their initial state is random. Making big adjustments to the first few observations is usually the best general-purpose learning strategy, especially in situations where you can’t count on having a huge amount of training data. In the realm of AI and data science where connection systems are routinely trained on millions of observations, it doesn’t matter if that initial adjustment is mistaken because the system has millions more observations to correct a mistake before it’s ever used. Humans (all animals for that matter) aren’t so lucky. We are regularly forced to learn from, and make decisions based on, a very small number of experiences.
Because of this, initial observations always impact the system more than later observations. That means the experience you have when you first try something is more potent and impactful than an equivalent experience when it’s your hundredth or thousandth time at something. You might be tempted to say “Of course,” but to a different (more rational) kind of reasoner, the hundredth observation made is weighted equally to the first. When any decision system has a single observation, that observation will shape the system far more than when it has a hundred observations. But when a perfectly rational system has a hundred observations, it won’t weight the first observation more than the hundredth. Each observation will have the same weight. That’s not the way connection systems work. The first experiences we have change our cognitive structures more than later – equivalent – experiences.
If you never get a second chance to make a first impression, it’s because someone else’s brain is overweighting the first fact!
The rapid adjustment of connection systems to early experience has impacts outside of rationality. Childhood learning is remarkably fast across a range of experiences because a child’s brain isn’t yet structured by the weight of past experience – so the connection systems in the brain are making large, rapid adjustments to data. This isn’t a sign of a faster brain, just a less trained connection system. Children have a lot of untrained connection networks, and they get a lot of “new” experiences. The fast-training rates that children enjoy (or suffer from) provide good grounds for extra care in controlling the training experiences they have and form the basis of a great deal of practical educational theory.
Which brings up another aspect of connection systems that impacts both rationality and education: it’s harder to undo than create strong connections in the brain. This general rule applies at any age and it’s a natural implication of slowing rates of change as a connection system garners experience. Habits are easy to create and hard to break, and constant repetition especially when combined with positive reinforcement can create connections in the brain that are extraordinarily difficult to change. We are creatures of habit not from weakness of will or sloth but as a natural result of our cognitive function. What’s more, human brains are significantly less plastic than their digital connection system counterparts. To speed decision-processing, the brain creates physical structures that are less changeable once codified than the digital weights in a computer system. When we learn something deeply, it’s hard to change.
The power of repetition and reinforcement to create strong connections in cognitive structure has both positive and negative implications. It’s the secret behind the seemingly effortless performance of complex skills, our ability to perform under pressure or duress (whether that performance is playing piano piece in a competition, shooting a free-throw at the end of game, or firing a rifle in combat), and the strength of good character dispositions even in the teeth of terrible misfortunes or great temptations. Yet this same tendency can make addictive habits almost impossible to break, plague our adult thinking with childish fears and learnings that we’ve long outgrown, and can make us inflexible, narrow, and incapable of good learning as we age. Part of thinking clearly about how to live must be a recognition of the benefits and drawbacks of habit and repetition in cognitive change.
Continuing the irrationality parade, there’s an aspect of connection systems that’s easy to miss and yet has big implications for an adaptive learner. People often lament the persistence of memory. Most of us have a steamer trunk of experiences we’d like to forget. Nor is this just an emotional preference. We sometimes learn important lessons from bad experiences. But not every bad experience has a positive outcome. There are a ton of experiences it would be preferable not to have had and that one knows have had a deleterious impact on your dispositions or rationality. There are experiences that make you afraid, angry, or prejudiced. But in this regard, the persistence of memory is only one of our problems. Memories do persist quite unwillingly, but so do the negative learnings that came along with them. In fact, our tendency to re-litigate memories can make this even worse. We keep re-living the experience and learning the wrong lessons over and over – making the cognitive impact of those lessons stronger and stronger. A single, powerful negative experience can sometimes destroy a person or make them – by their own standards – much worse.
We cannot expunge the countless impacts of a disastrous relationship on our mind. There is no eternal sunshine and no spotless adult mind. The removal of memories wouldn’t restore our nature to a state of hopefulness or innocence (though it might make it easier to regain some aspects of those states). Is there any adult who hasn’t struggled to resist thoughts they don’t want to have? Or found themselves unable to escape ways of thinking that are painful or even abhorrent? Surely not.
The fundamental nature of connection systems is that once you’ve had an experience you can’t un-have it. A memory can be suppressed, but the impact on other cognitive structures will remain. Only time and other experience can diminish learning and every impactful event will remain in the causal chain that created the “Who you are now” of your cognitive structures.[1]
Given our propensity to reinforcement learning, this is particularly true of experiences that have strong positive or negative outcomes. In reinforcement learning, the impact of experience on cognitive structure is larger the more intense the experience – sometimes so large that it can never be effectively removed by ordinary experience.
So why does this matter to an adaptive learner? It raises the stakes on experience selection – particularly in long-term life projects. Every experience selection choice is a trade-off, and when we choose some path that involves many years of cumulative experience (like enlistment, college, or marriage), the opportunity cost is substantial – we’re giving up years of alternative experiences. That’s bad enough, but the inability to undo our learnings puts even more pressure on the selection of experience. A soldier unhappy with the version of their self fashioned by the crucible of combat hasn’t just wasted time exploring a path, they may have permanently damaged their ability to be somebody else. The downside to bad experiences is much, much worse than the opportunity cost.
You can’t unlive the experience of being raped or bullied. You can’t erase the stress and fear and violence of combat. With work, you may be able manage the impact of those experiences on your brain, but you can never unlearn them.
With a connectionist mind, you really can’t go home again.
The “stateful” nature of a connection system – the fact that it’s transformed by and not independent of the data it has processed (experienced) explains much that is otherwise puzzling about human behavior and rationality. We often imagine human minds working like Excel spreadsheets. Change the data in one cell, and every other connected cell instantly recalculates to an entirely new and correct value. This isn’t, of course, the way people behave at all. People can, and frequently do, hold two obviously contradictory beliefs even though believing P and Not P is almost a definition of irrationality. Nor does changing someone’s mind about a single fact seem to have the waterfall like “Excel” effect. Changing a single fact in someone’s belief set doesn’t necessarily change their thinking much or at all – even if it logically should. Individual facts have a tenuous connection to beliefs that would be impossible in a deductive or algorithmic thinker whose thinking processes were independent of prior experience and data.
The tenacity of beliefs, our susceptibility to contradiction, and the lack of any direction connection between acknowledged facts and beliefs are hard to understand on any other model of cognition and have driven philosophers to despair. How can a rational animal be so irrational? The culprit isn’t bad culture or lack of education or breathtaking stupidity. It’s just the natural way connection systems tend to work. Changing a single fact won’t undo the training caused by the initial acquisition of the fact and won’t necessarily move the needle very much in terms of actual thinking processes. In a connection system, the relationship between individual facts or beliefs and habits of thought is obscure and amorphous. Our cognitive structures can’t be easily related to a single fact or set of facts – meaning that changing our mind about those facts doesn’t necessarily change any of the underlying structures they shaped. And self-contradiction isn’t surprising in a complex network of connection systems. If your brain contained a single connection system, it wouldn’t be so prone to self-contradiction. But with many separate connection systems each structured in a different way and being trained on different data flows, it’s hardly surprising that different systems would conflict and, of course, we have no easy conscious way to change the workings of either conflicting network.
None of these tendencies condemn us to stupidity. We can learn to do a better job of abandoning unsupported beliefs, of consciously suppressing contradictions and of adjusting related facts to changing data. But these don’t come effortlessly to our ways of thinking, so it’s hardly surprising that people fall far short of perfection. More significant are the implications for transformational choice. Between the influence of initial experience and our inability to easily undo cognitive changes, the stakes on transformative experiences will always be high. Mistakes in transformative choice bear a significant cost in our most precious resource – time. Worse, they can leave us changed in ways that may be nearly impossible to correct or that dramatically restrict our opportunities for growth or change.
Understanding how our amazing cognitive capabilities let us learn almost anything and make transformational experience our reality is nice. Understanding how those same capabilities may limit us, box us in, or damage us is important.
Using this knowledge to better think about and select transformational experiences?
That is the work to be rational.
[1] This should also make plain that memory is not the whole of identity. Identity is much more. An amnesiac will still have an enormous number of cognitive structures that are