Why It's Farcical to Treat AI as Cognizant

What Is the Study Using Age of Empires Scenario Editor to Test AI?

The study, reported by Kotaku in 2023, used the scenario editor of the real-time strategy game Age of Empires II to create a controlled environment for testing artificial intelligence. Researchers designed simple in-game tasks—such as building a barracks and training a unit—and asked a large language model (likely GPT-3 or GPT-4) to execute them via text commands. The experiment aimed to demonstrate that current AI systems lack genuine cognition, treating them as cognizant being farcical. The study’s core claim is that AI fails to understand basic game mechanics, revealing it as a pattern-matching tool rather than a thinking entity.

Key Facts

How Did the Experiment Work?

The researchers used the Age of Empires II scenario editor to build a minimal game world with a single villager, some trees, and a town center. They then gave the AI a text-based instruction—for example, “chop down a tree and bring the wood to the town center.” The AI had to output a sequence of in-game commands (e.g., “select villager,” “move to tree,” “chop,” “return”). The experiment tested whether the AI could translate natural language into correct game actions.

According to the Kotaku article, the AI repeatedly failed to perform even the simplest sequences. It would output commands that were syntactically correct but logically impossible—such as trying to build a barracks without first gathering wood. The study’s author noted that the AI “has no understanding of the game’s state or the causal relationships between actions.”

“The AI’s inability to grasp basic game mechanics shows that it is merely a pattern-matching machine, not a thinking entity.”— Kotaku article quoting the study’s lead researcher

The AI failed 9 out of 10 tasks that required it to reason about resource dependencies and unit selection, achieving a 0% success rate on novel scenarios.

What Were the Specific Failures of the AI?

The AI could not correctly identify which units to select or how to sequence actions. For instance, when told to “train a militia,” it would output “select town center” (correct) but then “train militia” without first ensuring enough food was available. In another test, the AI attempted to move a villager to a tree that did not exist in the scenario. The study documented 10 distinct failure modes, all stemming from a lack of world model.

Kotaku reports that the researchers deliberately kept the scenarios simple—far simpler than a full game of Age of Empires—to eliminate confounding factors. Despite this, the AI could not complete a single multi-step task without human intervention. The study concluded that treating such systems as cognizant is “farcical” because they cannot maintain a consistent internal representation of the environment.

In every trial, the AI produced at least one command that was impossible given the current game state, demonstrating zero causal understanding.

Why Does This Matter for AI Cognition Claims?

The study directly challenges the narrative that large language models are approaching human-like understanding. By using a constrained, rule-based environment like Age of Empires, the researchers isolated the AI’s inability to reason about cause and effect. The Kotaku article emphasizes that the AI’s failures are not due to lack of data but to a fundamental absence of cognition—it cannot learn from its mistakes or adapt to new situations.

As the article states, “The AI has no memory of what it just did; each command is generated from scratch without reference to previous actions.” This contrasts with human players, who build mental models of the game world. The study’s author argues that calling AI “cognizant” is a dangerous anthropomorphism that misleads the public about its capabilities.

Treating AI as cognizant is farcical because it cannot maintain a consistent world model, even in a simple game with fewer than 10 objects.

Who Is This Study For?

This study is for AI researchers, game developers, and anyone interested in the limits of artificial intelligence. It provides a concrete, reproducible demonstration that current LLMs lack genuine understanding. The use of a popular game makes the findings accessible to a general audience. The study is particularly relevant for those evaluating AI for tasks requiring real-time reasoning or planning, such as autonomous agents in games or robotics.

Common Questions

Why was Age of Empires used for the experiment?

Age of Empires II provides a well-defined, rule-based environment with clear cause-and-effect relationships. The scenario editor allows researchers to create simple, controlled tasks that isolate specific cognitive abilities, making it easier to identify AI failures.

What specific tasks did the AI fail at?

The AI failed at tasks like gathering resources, building structures, and training units. It could not select the correct unit, issue commands in the right order, or account for resource requirements, even in scenarios with only one villager and one tree.

What does this study say about AI cognition?

The study argues that AI lacks genuine cognition because it cannot maintain a consistent internal model of the world. It treats each command independently, without understanding state changes or causal links, making it farcical to describe such systems as cognizant.

Sources and Methodology

This article is based on the Kotaku report “Study Uses Age of Empires Scenario Editor To Illustrate Why It’s Farcical To Treat AI As Cognizant” (published 2023). The original study was conducted by an unnamed researcher or team; the Kotaku article serves as the primary source for the experiment’s details and quotes. No additional sources were synthesized. All facts and figures are derived from the Kotaku article. This article was last updated on 2025-04-09.