This post is a continuation of a chain of posts detailing concepts which can act as a foundation for new kinds of research in linguistics, bridging the atomic, structural approaches of generative linguistics with the somewhat loose approaches of more modern theories such as Optimality Theory (OT) and Dependency Grammar. In this post, I will be discussing the contents of a blog post made by Stephen Wolfram, who is well known for several contributions to the fields of physics, mathematics, and computing. In my own life, I used his “Wolfram Alpha” to supplement my learning of Calculus in high school, long before I would have even recognized his name.
Stephen Wolfram’s post explores further a topic that I introduced in my previous post about Ken Stanley’s work on evolutionary systems. Evolutionary systems explore a space of possibilities through mutation and selection, optimizing for characteristics that increase survival fitness from generation to generation. But what does the landscape of possibilities even look like? Well, talking about his evolutionary experiment “Picbreeder” in an interview, Ken Stanley states that “99.9999999% of this space is just total garbage blobs.” It is a massive sea of essentially useless, uninteresting, non-solutions! He further lays out a question: “How do [evolutionary systems] work when there are these extremely complex and obscure discoveries lurking in vast spaces of total garbage?” Stephen Wolfram begins to develop a way of understanding this process using “cellular automata”, which would be most famously recognized in “Conway’s Game of Life“. It’s fun and simple, go check it out!
Cellular automata are a good way to demonstrate how super simple systems can lead to unpredictably complex outcomes. With a grid of colored squares and a small set of rules about how each square should change in the next turn, you can generate a series of events that goes on for thousands of turns. Plenty of setups end up completing in a few turns, and plenty of them go on forever, but a good handful land somewhere in between. They proceed turn after turn, never repeating, until — *poof* — they suddenly stop with no clear explanation except for “that’s just what the rules end up doing!” I know this is a hard thing to grasp intuitively, because we always want to come up with a narrative explanation that trails alongside the process, boiling it down to a series of steps we can keep in order. However, the trouble here is that there is no simple explanation. Each step of the process is essential, despite how apparently random it is, and the only thing that can determine the outcome is actually running the system and finding out!
We can predict some setups, such as ones with repeating patterns that grow to infinity, but past a certain point of complexity we cannot — even with really great mathematics at our disposal! And among the long, unpredictable runs, of course there are some patterns and repeating characteristics, but the bulk of the process cannot be explained beyond the tiny, incremental, apparently unrelated steps it takes to reach a conclusion. Stephen Wolfram uses these cases to introduce an incredibly useful concept: “Computational Irreducibility”. Computational irreducibility just means that a pattern cannot be simplified, or “reduced”, to anything less than the long, complex path of computation that it is.
For the simplest sets of rules, like ones with only two colors, one can just try out every possible combination, running the simulation and holding onto the interesting outcomes. But for even slightly more complex sets of rules, it becomes practically unreasonable to try out every single combination. With just four colors in the ruleset, the number of possible rule combinations inflates to 3.022 x 1023. That’s roughly as many stars as there are in the observable universe! Now, in terms of being long-lasting and unpredictable, you may have guessed that most of these combinations result in “garbage blobs” just like in Ken Stanley’s Picbreeder. So in order to explore this space, Stephen Wolfram took an evolutionary approach, making his own process a fantastic mirror for Ken Stanley’s work!
By turning the rulesets into little strips of DNA, Wolfram ran a program that would hold onto a small population of rules and introduce mutations in each generation. The program would keep any rulesets that lasted longer than their parent did and remove any which were shorter. If a ruleset ended up repeating infinitely, it would also be removed. By running this evolutionary process, he discovered rulesets that solve plenty of different kinds of problems. He evolved rulesets whose runs were long, ones with a specific runtime (like 100 turns), ones which made patterns with a certain ratio of colors, and ones whose patterns had a certain aspect ratio (like 3:2).
Wolfram uses this whole evolutionary experiment to highlight how engineering practices differ from adaptive evolution, while both can function as methods of discovering solutions to specific problems:
“Something that we’ve seen over and over again here is that, yes, adaptive evolution manages to “solve a problem”. But its solution looks very complex to us. There might be some “simple engineering solution”—involving, say, a very regular pattern of behavior. But that’s not what adaptive evolution finds; instead it finds something that to us is typically very surprising—very often an “unexpectedly clever” solution in which lots of pieces fit together just right, in a way that our usual “understand-what’s-going-on” engineering practices would never let us invent.”
Yet he continues, revealing another incredible insight:
“One has the general impression that computational irreducibility is a strong enough phenomenon that it’s the “dominant force” that determines behavior and produces complexity. But there’s still usually something a bit different about the patterns we see from rules we’ve found by adaptive evolution, compared to rules we pick at random. Often there seems to be a certain additional level of “apparent mechanism”. The details still look complicated and in some ways quite random, but there seems to be a kind of “overall orchestration” to what’s going on.
“And whenever we can identify such regularities it’s a sign of some kind of computational reducibility. There’s still plenty of computational irreducibility at work. But “high fitness” rules that we find through adaptive evolution typically seem to exhibit traces of their specialness—that manifests in at least a certain amount of computational reducibility.
“Whenever we manage to come up with a “narrative explanation” or a “natural law” for something, it’s a sign that we’ve found a pocket of computational reducibility. If we say that a cellular automaton manages to live long because it generates certain robust geometric patterns—or, for that matter, that an organism lives long because it proofreads its DNA—we’re giving a narrative that’s based on computational reducibility.”
To me, this illustrates something about the world that dramatically shifted my perspective. As organisms, we find ourselves embedded in irreducible complexity from day-to-day — such that every single moment is technically experientially unique. Nonetheless, we’ve found predictable methods for engaging with the world, society, our bodies, our minds, and the environment. Our own genome predictably produces arms, legs, mouths, eyes, and brains for us despite the ease with which a few compounding mutations could cause cancer or disease. In many cases, clinging to reducible phenomena is what has allowed us to survive. However, the real revelation here is that as long as we are floating in a sea of mostly computational irreducibility, further pockets of reducibility rest just out of our view. Not only that, but we may never predict them based on the ones we have already found. We must make small changes to our rules and let the program run its course, only then will we discover what reducible pockets arise.
This is not intended as a motivational speech, though it easily could become one — and in fact, I have used this way of thinking to reach several goals without anticipating how exactly it would happen. To the contrary, I want to use this concept of the sea of computational irreducibility for its explanatory power in the realm of linguistics and language modeling. It gives a tool whereby we can reconcile the atomizing approach of generative linguistics (which undoubtedly carries reasonable intuition) and the unpredictable flexibility of actual linguistic expression, attested by the thousands of extant languages. The key to this reconciliation would be the creation and marriage of three things: a set of re-writable rules, some starting conditions, and an evolutionary space to explore.
With the proper (albeit probably very complex) definitions for these things, could such a system “discover” English, or Hindi, or Bantu, or Äiwoo? What constellations of computational reducibility would have to be locked into place for any of these to develop, and what would those pockets tell us about what kinds of languages are possible for a human to learn or speak? It is no coincidence that languages “evolve” through time and it is no coincidence that our cognitive capacities are what give rise to them. What does our human mind do to make this possible for us? I don’t claim to be capable of answering all of these questions, but I do believe that by building on the foundation of this framework, they can be answered.
In my next post, I will continue developing this theme by introducing some of the work of Michael Levin, whose research and explanatory tools add a layer of necessary complexity to the framework I am applying to linguistic modeling. I intend to explain how evolutionary systems always seem to have a direction to travel in pursuit of their goals, despite the relatively simple rules that are unfolding.