How to teach AI to play mobile games?

Most modern mobile games are built around short gaming sessions and intense competition for player attention, where any graphical bug or unbalanced mechanic costs the studio revenue. To accelerate development, optimize testing across thousands of different smartphones, and make gameplay more flexible, gamedev studios are increasingly implementing ML models for specific, targeted tasks. However, the effectiveness of such solutions under mobile constraints depends entirely on how well, contextually, and with an understanding of mobile gameplay specifics, the training data is annotated.

Quick Take

Computer vision and frame-by-frame video annotation allow AI agents to play through games, catching bugs and reporting them.
The use of chains of reasoning teaches bots to make sound decisions in combat or strategy games while avoiding "cheating".
RLHF and NER annotation guarantee the preservation of game canon and the absence of hallucinations.
The effectiveness of mobile ML is measured by the depth, accuracy, and context of the training data.

Where AI Already Works in Mobile Gamedev

For an AI model to justify the investment in mobile development, it must solve one of two tasks: save hundreds of hours of team labor or directly impact audience retention metrics. In practice, this comes down to four main directions:

Technical control – automatic layout checking, detection of visual anomalies, passing scenarios, and reproducing non-standard sequences of actions.
NPC behavior – creation of bots and opponents that react to the situation on the map, assess risks, and change tactics.
Generative content – procedural creation of dialogues, quests, tips, and other elements that should change depending on the context.
Multimodal interaction – complex work with the game frame, UI, text commands, and other signals.

Let us examine in detail what kind of data work is involved in creating functional tools for each of these tasks.

Automated Game Testing

Mobile testing is a good example of why AI needs to be taught to "see" a game roughly the way a human sees it.

The exact same screen can look different on smartphones with varying resolutions, aspect ratios, camera cutouts, or system overlays. A button can shift, text can get cut off, and an important interface element can end up beneath the system UI.

For an automated agent, this means that a button's coordinates alone provide little value. It needs to understand what exactly is on the screen and how that element relates to the game.

Therefore, training data may utilize:

Bounding boxes and polygons – for marking buttons, characters, enemies, health bars, items, and other elements.
Semantic and instance segmentation – when different areas of a frame need to be separated to precisely determine where an object, UI element, or visual effect ends.
Video annotation – so the model can account for the cumulative sequence of events – when an error appeared, what preceded it, and how the game state changed afterward.

The last point is especially critical. A graphical artifact in a single frame might simply be an animation effect. However, if it recurs during a specific sequence of actions or is accompanied by a change in UI state, the situation looks entirely different.

Overall, automated testing is where artificial intelligence yields the fastest and most tangible ROI for mobile gamedev. Therefore, in such projects, it is essential to pay close attention to video context and action sequences, involving annotators with practical experience in Game QA whenever the task demands it.

GameAI & Reasoning

While testing frees up developers' hands, artificial intelligence algorithms inside the gameplay itself are responsible for the most valuable asset – player retention. In mobile games, people instantly sense artificiality: a bot operating on a basic script that senselessly exposes itself or starts "cheating" by reading data directly from game memory gets boring quite quickly. A genuinely interesting opponent must evaluate the situation, weigh risks, and build tactics several steps ahead – just like a human does.

For such tasks, the model needs data that reads the entire situation around the bot regarding the game state. One approach involves creating reasoning-oriented datasets, where both the correct action and the decision-making logic behind that action are recorded. Another involves working with trajectories: character routes, movement zones, cover locations, points of interest, and other spatial attributes that provide the bot with spatial understanding.

This allows the model to learn how to identify cause-and-effect patterns, but here it is important not to overstate the role of any single method. None of them will automatically make an NPC "smart". They are merely part of a broader data preparation process for reasoning, evaluation, and model training, where the environment and the rules of the game itself are equally vital.

Generative Content and Gaming LLMs

In narrative-heavy games, the problem is somewhat different. Standard dialogue trees with three pre-written response options have long been a bottleneck for developers: they are expensive to produce and quickly exhausted. Integrating gaming LLMs makes NPCs feel "alive", but it simultaneously opens new horizons for hallucinations and sudden bot shifts into random, meaningless chatter. Therefore, controlling the context of the generated response is vital for gaming LLMs, which is achieved through specialized annotation:

Human preferences and RLHF-style feedback – verifying how well a response matches the character's personality, scene context, and world rules.
Multi-turn dialogue – teaching the model to maintain conversational consistency rather than treating each line in isolation.
Intent and entity recognition – identifying which item, character, location, or game mechanic the player is referring to.

Here, too, the question of balance arises: the model must have room for improvisation without stepping outside the boundaries of game lore. This is precisely why specialized data generation approaches are used for training gaming LLMs. Involving narrative experts and linguists in the annotation process helps shape datasets that teach AI to speak the native language of a specific genre.

Multimodal AI in Gameplay

The evolution of VLMs allows an AI agent to interact with a game comprehensively: seeing the gameplay frame, reading the UI state, and simultaneously interpreting the player's text or voice commands in real time. For mobile gamedev, this opens the door to new mechanics, though it also introduces significant complexities, as a multimodal agent must first be taught how to establish all these connections. Therefore, training data for such models may combine:

Image-text pairs – gameplay frames accompanied by descriptions of what is happening in them.
UI-to-action mappings – the link between an interface element and the action it triggers.
Spatial grounding – the relationship between objects and their positioning – to the left, behind, near, above, below, etc.
Action sequences – the sequence of steps required to execute a command.

As a result, the model learns to establish connections between what is seen, what is spoken, and the possible action. However, creating such hybrid datasets requires specialized annotation pipelines where visual frames and text are processed as a single unit. For annotating multimodal datasets, we use our own platform, which supports multimodal tagging and allows us to combine interface recognition, frame geometry, and text context into a stable data matrix.

AI as a New Link in Mobile Gamedev

Ultimately, all games are a full dialogue between the developer and the player. Artificial intelligence in this dialogue has become a new conversational partner – one that can be attentive to the player and useful to the developer. It simply needs to be provided with sufficiently good data so it understands the specific task, sees the necessary context, and can act correctly within that context.

It is precisely this behind-the-scenes work of data preparation – invisible to the player yet decisive for a project's success – that we perform at Keymakr, annotating with a deep understanding of mechanics and gamedev specifics. In the end, artificial intelligence sees the game world only as clearly, contextually, and professionally as it has been annotated. We are convinced that when a model is trained on the right dataset, with annotation tailored to your project's exact requirements, technology becomes part of true gaming magic.

FAQ

How do AI models run on budget smartphones with limited resources?

Large models do not necessarily need to run entirely on the device itself. In practice, a combination of approaches is used: compact models can perform fast tasks directly on-device, while more complex operations are offloaded to edge or cloud infrastructure. The choice depends on how critical latency, privacy, connectivity, and computational constraints are for the specific game.

Why can synthetic data from Unity or Unreal Engine not completely replace real gameplay?

Synthetic data is exceptionally easy to control: you can create the necessary geometry, adjust lighting, modify camera angles or object positions, and generate large volumes of examples. However, a real smartphone introduces variables that are difficult to simulate fully: diverse device models, network issues, interrupted sessions, system notifications, non-standard user behavior, and other edge cases. Therefore, synthetic and real data work best when combined.

Does high-quality annotation help reduce AI model size?

Not directly. Model size is determined by its architecture and optimization techniques – such as distillation, quantization, or pruning. However, a well-prepared and narrowly specialized dataset can help adapt a more compact model to a specific task instead of relying on a large, generic model for everything at once.

To what extent can a single game dataset be applied to train models across different projects?

Partially. Basic visual tasks – such as recognizing standard UI elements – may share common patterns. However, mechanics, lore, NPC behavior, level structures, and specific bugs depend entirely on the individual game. Therefore, adaptation or fine-tuning on data from the new project is usually required.

How can game data be used to fight fraud and bots?

Beyond visual data, telemetry and time series can be used for this purpose: action frequency, movement paths, interaction sequences, purchase patterns, or other behavioral signals. An ML model can leverage such data to detect anomalies and suspicious patterns. However, even here, it is crucial not to rely on a single signal: unusual behavior does not automatically mean you are dealing with a bot or a cheater.