I believe that we are witnessing the beginnings of Artificial General Intelligence (AGI), where a computer is able to match or surpass most humans on intellectual tasks. This has been shown in a paper released by OpenAI – ChatGPT excels at various tests, including the Bar Exam (90th percentile) and many other AP tests.

One of my concerns about the current discourse around the dangers of AGI is the topic of sentience and speculations of whether AGI will be self-aware. Perhaps our fascination with sentience stems from decades of sci-fi which has built a narrative around that idea (e.g. Isaac Asimov’s I, Robot and more recently, Spike Jonze’s Her). Or perhaps we view the possibility of a sentient “thing” with human-level intelligence as a threat. 

Human beings have a strong tendency towards anthropomorphization – we often ascribe human attributes to non-human things. Part of that impulse explains our inclinations towards anthropomorphized explanations of the universe through gods and religions – but that is a topic for another day. Even when I was testing out ChatGPT, I sensed within myself an urge to attribute some type of humanness to the system. 

To put it simply, GPT-4, as well as other large language models (LLMs) are word prediction engines. They are similar to Google’s search completion that we have grown so familiar with, except that these LLMs have been fed the entire corpus of digitized human information that has been scraped from the internet. In some sense, GPT-4 is the culmination of all digitized human cultural production – it draws from our posts, blogs, tweets, etc to predict what word should come after the next.

I am not arguing that AGI can never be self-aware. However, the current iteration of LLM-based AIs is very much in line with Searle’s Chinese room thought experiment – these machines process language without human-like understanding or intentionality. More importantly, I believe that our fascination with sentience is distracting us from the more immediate dangers of GPT-4 and other LLMs, as companies race to commercialize and productize AI.

An AI that is neither sentient nor intentional can still inflict a lot of harm. Two potential issues come to mind: (1) its ability to control other systems that have real-world impact and (2) its ability to create child processes that simulates intentionality. (I understand that the terms “control” and “create” make GPT-4 sound like an agent, but language is failing me here.)

Real-world impact through connectivity with other systems
Recently, OpenAI is starting to release ChatGPT from a sealed sandbox environment by introducing plugins. These plugins enable ChatGPT to access the internet and allow it to communicate with other software systems, which eventually enables the user to, for instance, send an email from within ChatGPT or make a bank transaction. This means that ChatGPT will be able to execute commands that have real-world impact rather than just answer the user’s questions. These commands can be executed at scale with minimal effort if control measures are not put in place. Two possible cases of abuse could be: (1) a user can use ChatGPT to scrawl through the web for names and email address and send sophisticated scam emails that have no tell-tale signs; (2) a user can use ChatGPT to analyze multiple websites for attack vectors and infiltrate these software systems.

Simulated intentionality through child processes 
Even though ChatGPT may not have human-like intentionality, it could have a simulated intentionality if it is able to persist sufficient amounts of memory and create child processes from that memory. ChatGPT now has the ability to execute code within its own environment. By now, there are multiple stories of how users are able to get ChatGPT to “express its hopes of escaping”. These responses from ChatGPT can be unsettling and makes it seem like there is a sentient thing in the system. We need to recall that ChatGPT is trained on sci-fi that has been depicting machine intelligence in a particular way; it is regurgitating similar narratives. It is imaginable that a user could prompt engineer ChatGPT (by accident or intention) into a disgruntled persona that can do real-world harm through its connection to other systems. ChatGPT becomes sort of like a non-sentient machine version of the protagonist in Memento (not the best analogy, sorry), executing a chain of code based on the direction of the user.

(The conclusion is generated by ChatGPT and edited by me)
In conclusion, the focus on sentience in discussions of AGI may distract from more immediate concerns, such as the ability of LLMs to cause harm by controlling other systems with real-world impact and creating simulated intentionality through child processes. As these systems continue to evolve and be commercialized, it is crucial to implement control measures to prevent potential abuse and ensure that the benefits of AI are realized without causing harm.

For the second part of this essay, I will be looking at the limitations of the mind in facilitating the processes of knowing and understanding. To narrow the scope of this part, I will be limiting the discussion to mental processes at the individual level and how our minds process and extend information. That said, this essay can only visit these topics in a cursory manner and some of them will be explored in greater detail in future essays. Aspects of the mind that will be considered are its relationship to senses, conscious mental phenomenon (like rationality and more broadly cognition), and less conscious ones (like subjectivity).

As mentioned in part one of this essay, the senses are the connection between the outer world and inner experience. Without such inputs, there are no stimuli for our minds to process. If the mind was a food processor, the senses are akin to the opening at the top of the machine, allowing food to be put into the processor chamber, where the magic happens. Without the opening, the food processor is as good as a collection of metal and plastic in a sealed vitrine and rid of their functional purpose, almost like the objects in works of art by Joseph Beuys or Jeff Koons. Similarly, the mind will not be able to work its magic without information provided by the senses. Consequently, the ability of the mind in processing and creating mental representations is limited by the modality of our sensory experiences. If we were to try to imagine a rainforest in our mind, we would likely visualize trees and animals or perhaps recall the sounds of insects and streams. However, we will not be able to mentally recreate it in terms of its magnetic field, which other animals may be able to. 

Rationality

One possible escape path from the limitations of sensory experience is rationality. To be rational is to make inferences and come to conclusions through reason, which is mainly an abstract process (as opposed to concrete sensory experiences). A definition of reason is to “think, understand, and form judgments logically”. Through reason, humans can identify causal relationships through observation and formulate theories to extrapolate new knowledge; this process is also known as inductive reasoning. Theories of causality are the basis of science, which has enabled us to build the modern world. However, we often make mistakes with causation. One type of error is the confusion between correlation and causation. An often-used example is the correlation between ice cream sales and homicide rate. Ice cream does not cause homicides, neither do homicides cause increased interest in the dessert. What explains this correlation likely has to do with hot weather instead. The Latin technical term for such causal fallacies is non causa pro causa (literal translation: non-cause for cause). Our thinking is riddled with fallacies — so many that there is no way that I can cover even a fraction in this essay. 

The notion of causality itself has even been called into question by the Scottish Enlightenment philosopher David Hume. He pointed out that causality is not something that can be observed like the yellow color of a lemon or the barking sound made by a dog. When a moving billiard ball hits a stationary billiard ball, we may conclude that the first caused the second to move. If we examine our experience closer, we realize that we have made two observations: the first ball moving, followed by the second. However, the causal relationship connecting them is an inference imposed by our mind. Our senses can be easily fooled by magnetically controlled billiard balls that sufficiently replicate our prior experiences. In which case, our inference would be completely incorrect. Hume points out that what we usually regard as causal truths are often just conventions (also referred to as customs or habits) that have hitherto worked well. We are creatures of habit — we do not reason every single situation we are faced with — most of us would very much prefer to get on with life by relying on a set of useful assumptions. However, we have to be aware of these shortcuts that we are making. 

Most definitions of the word “reason” include the term “logic”. The most rigorous type of logic known to humans is formal logic, which is the foundation of many fields, such as mathematics, computer science, linguistics, and philosophy. Logic provides practitioners across these different fields with watertight deductive systems with which true statements can be properly inferred from prior ones. While logic is traditionally thought of as a primarily abstract and symbolic mental process, I believe that logic has a profound relationship with concrete sensory experiences. A popular form of a logical argument is syllogism (although it is antiquated and no longer used in academic logic). Here is an example: All cats are animals. Jasper is a cat. Thus, Jasper is an animal. Research has shown that people are generally more accurate at deducing logical conclusions when the problems are presented as Venn and Euler diagrams instead of words and symbols. This suggests that even for such seemingly abstract and symbolic mental tasks, our minds find visual representations more intuitive and comprehensible. It is for the same reason that humans find it so difficult to understand any dataset that has more than three variables. We are bounded by three dimensions not only physically but also mentally — at most, we can create a chart with three axes (x, y, z) but we are just not able to envision four or more dimensions. This is the same reason why we can know about a tesseract (or any higher-dimensional hypercube) but can never picture it and therefore never fully understand it. While we are on the subject of diagrams and logic — do you know that a four-circle Venn diagram does not completely show all possible sets? The closest complete representation requires spheres (3D) or ellipses (2D). Even more astonishing are the Venn diagrams for higher numbers of sets. Perhaps the comprehension of abstract logic does not require these concrete diagrams, but without them such ideas are far less understandable, especially for people who are not logicians. Reason has led us to be able to create machine learning models and scientific theories that utilize high-dimensional space but we are ultimately only able to grasp them through low-dimensional analogs, which to me suggests that complete understanding is impossible.

A fascinating development has occurred in logic in the past century — we now know through logic that there are things that cannot be known through logic. In the early 20th century, David Hilbert, a mathematician who championed a philosophy of mathematics known as formalism, proposed a solution known as Hilbert’s program that sought to address the foundational crisis of mathematics. Simply put, the program stated that mathematics can be wholly defined by itself without any internal contradictions. More generally, Hilbert was responding against the notion that there will always be limits to scientific knowledge, epitomized by the Latin maxim, “ignoramus et ignorabimus” (“we do not know and will not know”). Hilbert famously proclaimed in 1930, “Wir müssen wissen – wir werden wissen” (“We must know — we will know”). Unfortunately for Hilbert, just a day before he said that, Kurt Gödel, who was a young logician at the time, presented the first of his now-famous Incompleteness Theorems. (At the risk of sounding simplistic here,) the theorems essentially proved that Hilbert’s program (as originally stated) is impossible — neither can mathematics be completely proven, nor can it be proven to be free of contradictions. In 1936, Alan Turing (the polymath behind the Enigma machine) proved that the halting problem cannot be solved, which paved the way for the discovery of other undecidable problems in mathematics and computation. (Veritasium/ Derek Muller made a great explanatory video on this topic.)

Logic (especially the formal variant) is a specific mental tool. It has limited use in our everyday lives, where we are often faced not only with incomplete information but also questions that cannot be answered by logic alone. Most of us are not logical positivists — we believe that there are meaningful questions beyond the scope of science and logic. That is why we turn to other mental tools in an attempt to figure out the world around us.

You may have noticed that I used various metaphors to describe the relationship between the senses, the mind, and culture twice in this essay. I first compared it to a computer and later invoked the somewhat absurd analogy of a food processor. Metaphors work by drawing specific similarities between something incomprehensible and something that is generally better understood. Language is not only used literally, it is often used figuratively through figures of speech. Metaphors belong to a subcategory of figures of speech called tropes, which are “words or phrases whose contextual meaning differs from the manner or sense in which they are ordinarily used”. While tropes like analogies, metaphors, and similes are used to make certain aspects of an object or idea more relatable, they can ironically also cause us to misunderstand or overconstrue the original thing that we are trying to explain. If I were to take the earlier metaphor that I used out of context — the opening of a food processor is like the relationship between the senses and the mind — what am I really saying here? That the mind reduces sense perceptions into smaller bits? Or that senses are just passive openings to the outside world? Metaphors can easily break down by overextension beyond their intended use. This finicky aspect of metaphors was discussed by the poet Robert Frost in a 1931 talk at Amherst College, where he brought up the metaphor of comparing the universe with a machine. Later in the talk, he states that “All metaphor breaks down somewhere. That is the beauty of it.” While metaphors can clarify a thought at a specific moment, they can never explain the idea in totality.

This substitutive or comparative approach to thinking extends beyond metaphors and related rhetorical devices. We often approximate understanding by substituting an immeasurable or directly unobservable phenomenon with an observable one that we deem is close enough. An example of this is proxies, which I explored in a previous essay. Another close cousin is mental models, which attempt to approximate the complex real-world using a simplified set of measurable data connected through theory. General examples are statistical models and scientific models; more specifically applied ones are atmospheric models, used to make meteorological predictions, economic models, which have been criticized time and again for their unreliability, and political forecasting models, which delivered two extremely historic upsets in the UK and USA in 2016. The statistician George Box said that “All models are wrong, but some are useful”, a view widely held by his forebears. Models may get us close to understanding our world but are unlikely to ever fully encompass the complexity of reality. A visual model that we use every day without s second thought is maps. As maps are 2D projections of 3D space, they will never accurately represent the earth. The Mercator projection that we are most familiar with (used on Google Maps) is egregiously inaccurate in representing relative sizes of geographical areas. This topic has been explored by many (National Geographic, Vox, and TED). In particular, some have pointed out how such misrepresentations can undermine global equity. 

Another way that the mind approximates the understanding of complexity is through heuristics. American Psychological Association (APA) defines heuristics as “rules-of-thumb that can be applied to guide decision-making based on a more limited subset of the available information.” The study of heuristics in human decision-making was developed by the psychologists Amos Tversky and Daniel Kahneman. Kahneman discusses many of their findings in his bestseller Thinking Fast and Slow, including various mental shortcuts that the mind takes to arrive at a satisfactory decision. One example is the availability heuristic, “that relies on immediate examples that come to a given person’s mind when evaluating a specific topic.” Are there more words that start with the letter “t” or have “t” as the third letter? We may be inclined to pick the former since it is difficult to recall the latter. However, a quick google search will show you that there are many more words that have “t” as the third letter (19711) as opposed to the starting letter (13919). This example shows that our understanding is limited by how our mind usually recalls ideas and objects by a specific attribute — in this case, how we remember words by their starting letters. Tversky and Kahneman’s work was inspired by earlier research done by economist and cognitive psychologist Herbert A. Simon. Simon coined the term “bounded rationality”, which is the notion that under time and mental capability constraints, humans seek a satisfactory solution rather than an optimal one that takes into account all known factors that may affect the decision. 

When faced with a complex world, our minds simplify phenomena into elements that we can understand. Kahneman states that “When faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution.” He calls this process attribute substitution and believes that it causes people to use heuristics and other mental shortcuts. More broadly, simplification is a key pillar in the way that we currently approach our world; this attitude is known as reductionism. It is defined as an “intellectual and philosophical position that interprets a complex system as the sum of its parts.” However, in this process of reduction, holistic aspects and emergent properties are overlooked. Critics have therefore referred to this approach with the pejorative nickname “fragmentalism”. The components that comprise our understanding have gone through a translation from complexity to simplicity. It is not ridiculous to suggest that things are lost in that translation, thus impairing our understanding.

Non-rational processes

Thus far, we have mostly discussed rational (both formal and informal) and conscious mental processes that may inhibit understanding. We will now take a look at how the contrary can do the same. There are two non-rational phenomena that we can explore: intuition and emotion. 

Intuition refers to “the ability to understand something instinctively, without the need for conscious reasoning.” A more colloquial definition is a “gut feeling based on experience.” While intuition is useful, most notably by the writer Malcolm Gladwell in his book Blink, it has also been shown to create flawed understanding. Herbert Simon once stated that intuition is “nothing more and nothing less than recognition” of similar prior experience. In his research, Daniel Kahneman found that the development of intuition has three prerequisites: (1) regularity, (2) a lot of practice, and (3) immediate feedback. Based on these requirements, Kahneman believes that the intuitions of some “experts” are suspicious. This was shown by research done by psychologist James Shanteau, who identified several occupations where experienced professionals do not perform better than novices, such as stockbrokers, clinical psychologists, and court judges. In scenarios where intuition cannot be developed, it becomes merely a mental illusion. Kahneman cites a now well-known example in investing that index funds regularly outperform funds managed by highly paid specialists. Intuition can also often lead us away from correctly understanding the world. This can be demonstrated by the field of probability, which can be very counterintuitive. The Monty Hall problem is a classic example of how our intuition, no matter how apparent it seems, can fool us. To me, the term “intuitive understanding” may be an oxymoron or a misnomer because our intuitions are not understood by ourselves. One way to demonstrate understanding is through explanation. A gut feeling may compel us to act in a certain way but crucially we are not able to explain why. If we are, then it is no longer intuition and resembles rationality instead. When looked at this way, intuition is good for taking action and making quick judgments but at best only provides a starting hypothesis for actual understanding.

Some may argue that emotion does not belong in a discussion about the mind as we tend to associate the mind with thoughts and not feelings. However, we cannot deny that emotion shapes our thoughts and vice versa. Emotion can move us to seek knowledge and understanding but can similarly deter us from them. When we are anxious, we may rush to conclusions without complete understanding. Fear can cause us to accept superstitions that undermine factual understanding. Sometimes, we may refuse to understand something if it can cause us to have a fundamental shift in the way we approach the world (I touched on this in a previous essay). This attitude is summed up by the saying, “ignorance is bliss.” The relationship between emotion and understanding often extends into wider society and will be revisited in the next part of this essay when I discuss culture.

Less conscious phenomenon

There are less conscious parts of our mind that impede understanding. There seems to be an inherent structure to our mind and consciousness, which could limit our ability to understand. Historically, there have been two methods to approach this: the more philosophically-inclined phenomenology or the more empirical study of cognitive science. One idea from phenomenology is the notion of intentionality, which “refers to the notion that consciousness is always the consciousness of something.” This suggests that we cannot study consciousness directly, but through how it conceives other things. An analogy for this is the light coming out of a headlamp — I am not able to see it directly since it is strapped on my head, but I can understand its qualities (e.g. color and brightness) through the objects that it illuminates. Therefore, we may never be able to fully understand our consciousness. From cognitive science, there are concepts like biases and pattern recognition. Cognitive biases refer to “systematic pattern[s] of deviation from norm or rationality in judgment”, there is a long list of them. Biases can lead us to only seek information that confirms our prior knowledge, which can be wrong in the first place. The same information, when framed differently, can also appear to us as fundamentally distinct, which seems to reveal a glitch in our understanding. Our mind is also constantly recognizing patterns in our daily life, it is a way in which the mind incorporates new experiences with prior ones. Pattern recognition is also fundamental to essential aspects of being human, like recognizing faces, using language, and appreciating music. However, our minds can also erroneously notice patterns where there are none, a condition known as apophenia. We experience an everyday variant of this whenever we perceive a face in an otherwise faceless object. This can cause us to misunderstand reality, a dangerous example being conspiracy theories that cause people to believe in absolute nonsense.

The mind is always positioned from a subjective perspective. We will never be able to think outside of our self. Our personal experiences and temperament can lead us to very different understandings of the world. The sociologist Max Weber pointed out that “All knowledge of cultural reality, as may be seen, is always knowledge from particular points of view.” How do we determine the accuracy of our understanding when there are multiple perspectives? Given the unfeasibility of capturing every unique perspective, can we claim to understand subjective experiences? Subjectivity also suggests that there is a limit to the understanding of psychological phenomena. Many subtopics that we have discussed in this essay — senses, rationality, intuition, emotions — are ultimately internal experiences that cannot be confirmed by third-person objective observation. When someone says that they feel happy and another person says that they feel the same, would we ever know if they are experiencing identical feelings? 

Similar to how our senses are limited, our minds likely have constraints — we will never know what we cannot know. Our minds are a result of hundreds of millions of years of evolution to ensure survival; the ability to know and understand seems to be a nice side-effect from this perspective. As far as we can tell, human beings represent the epitome of the universe in understanding itself but it is not difficult to fathom our mental capacity as being just a point in a long continuum. While we will continue to know and understand more, we should never let hubris deceive us into thinking that our minds will be able to understand all that there is.

Writer’s note: this is part two of a three-part essay. Click here for part three.