Audiovisual Feedback Based Simulation

and the death of the classical UI paradigm in gaming

The premise

This paper deals with UIs from the perspective of the gaming industry. We set out to describe the common classical UI paradigm as we see it. We also deal with the function of UIs in games - the problems they propose to resolve and the problems they represent to immersion in gaming.

What is an UI?

UI – or User Interface – is an abstract metaphor which tries to convey a two-way communication between the human experience and a machine through the use of symbolic representations and audiovisual cues.

Why do we need UIs?

To understand the solution we must first understand the problem. Games - and computers in general - have a very narrow band through which they can communicate with us. If we ignore the fringe cases we are left only with the audio and the visual. Of these two we will mostly concentrate on the visual since the audio is mostly on par with reality and has very little symbolic representations which concern us.
So why do we need UIs? Games cannot convey things like feelings, thoughts, thirst, hunger, desires or the sensation of touch directly. With only the visual experience to aid us it comes naturally to us to resort to symbolism.
The thirst and hunger become bars on the UI. Feelings and thoughts become words or facial expressions. Desires become text elements or icons on the UI. Touch – damage – becomes a red haze on the screen.
Sometimes this is absolutely necessary as there are no other ways to represent these things in a medium restricted only to audio and visual bands.

The classical UI paradigm in gaming

When I refer to the classical UI paradigm I refer to a very specific phenomena and approach to user interfaces which has become a common industry practice. More than anything this approach is the result of the natural evolution of hardware, software and gaming industries and continues to be supported as a standard practice in games - mostly by convenience.
We shall approach this paradigm through the use of five of the more obvious examples.
Example 1 – Making items
In many games making items has been abstracted away and reperesented as an UI element which portrays the process of manufacturing. Usually you open up an UI which is often a graphical box of some sort with a grid or list. You choose the item which you wish to make by a selection method the game offers, most often represented by a list of names sometimes accompanied by images. And you most often pay either with time or materials or both for the item. Sometimes making items requires special skills.
Example 2 – Numeric representations of condition
Item/player/structure condition is often represented by numbers. Either a fixed or sliding scale portrays how good or bad, how healthy or sick or how valuable something is. Sometimes these numbers are represented visually by a status bar or a color.
Example 3 – Inventories
Inventories of characters or storage containers are often represented by a list of names sometimes accompanied by images. The UI allows you to select and handle these items by selecting the item and applying the appropriate UI function for the task required.
Example 4 – Numeric representations of abundance
Abundance in games is sometimes represented by numbers next to the item. A character might have an apple in his inventory represented either by text or graphics – or both – and if the character has several of those apples this will often be displayed by a number in addition to the other symbols. Unit troop strength might as well be portrayed by a graphical representation of a unit with the addition of a number representing the number of individual troops in that unit.
Example 5 – Excessive information
Sometimes games offer excessive information about items, characters, skills or places which doesn’t have a direct equivalent in real world. This can include but is not limited to UI element with map information, skill points, attribute points, statistics, symbols showing active effects, etc.

The problems with the classical UI paradigm

I have shown you above in the examples how games often represent things symbolically. Depending on the nature of the game and the specific situation these UI metaphors can be useful and many of them are very easy and cheap to implement giving the developers the possibility to spend the time and money saved working on other game content.
However, selecting the classical UI paradigm for every use case can lead to excess gamification of things which could - in fact - have a direct visual equivalent in the game world. Unnecessary UI metaphors run the risk of alienating the players from an immersive gaming experience.
With careful case by case consideration it is possible to make the most use of the classical UI paradigm in situations where no other method is possible due to the limitations of the medium.
When an abstraction is not required to represent reality the classical UI paradigm stops making sense.

Audiovisual Feedback Based Simulation

AFBS stands for simplicity and immersion. Giving audiovisual feedback to the player to simulate things happening in the game as opposed to using excessive menu abstractions to achieve the same goal. AFBS design philosophy calls for:
Representing the reality as it is
One of the most important concepts of AFBS is representing the reality as it is - when ever possible. It’s all about game immersion and getting rid of the unnecessary UI metaphors. The game itself can be as unrealistic as it needs to be but when something can be represented without a metaphor it should be.
Example: If you can represent the act of carving a wooden spike with a knife as an animation – then don’t abstract it away as a menu and a button.
Avoiding unnecessary UI elements
If you don’t need an UI menu for something you should probably avoid having one and go for the more visually appealing immersive option.
For example: Context sensitive interactions can drastically reduce the need for graphical UI elements.
Avoiding the display of excess information
Excess displaying of numerical and other data easily distracts the players from the game immersion. Human beings are wired in such a way that we experience progress as rewarding. Unless your game is all about numbers and stats the rewards should be self evident in other more fulfilling ways. If you display numbers to the players the players will spend a considerable amount of time comparing the numbers to get the best possible advantage instead just enjoying the game.
Using the nearest possible UI metaphor
Things which don’t have a direct equivalent in audiovisuals should use the nearest possible UI metaphor which makes sense.
For example: We can’t sense when the player character is hungry so it needs to be represented somehow. The usual UI metaphor is to add a bar on the onscreen display which shows how far the character is from starvation. However the same goal can also be achieved by adding a simple audio cue of the character’s stomach growling plus some occasional monologue about being hungry and adding effects from the starvation - such as slowing down, etc.