Interactive storytelling: why character control is at odds with movie-like storytelling

This post considers why it is that, during gameplay segments of video games, when the player has control over their character (like of their movements, and what they interact with), it's not possible to have storytelling like found in movies and tv. Of course it's possible to have such storytelling in these games, such as by using cutscenes -- just not within the gameplay segments. Why?

The answer to this may seem fairly obvious. Nonetheless, I have found pinpointing it quite difficult.


The player can't make their character act.

A major component of visual storytelling like in tv and movies is the 'acting' component. This refers to both live-action performances as well as the performances of animated characters. Acting includes things like movement, posture, gesture, facial expression, vocal performance.

Here we're not concerned with other parts of visual storytelling, like cinematography (lenses, camera positioning and movement, shot composition, lighting, etc), editing (selecting, ordering, and timing shots, etc), sound effects, and music. Just the acting component.

Acting's arguably the core of visual storytelling. Those other elements (cinematography, editing, sound effects, and music) can be mostly or fully stripped away, leaving just the acting -- as is found in theater.

At each moment of a scene, each character has very specific, deliberate acting performance details (movement, or lack thereof, their posture, gesture, facial expression, vocal performance, etc). It's through these that their actions are conveyed. They give the character psychological and behavioral verisimilitude. They reflect the characters intentions and mental state.

But when the player has control over a character (like of their movements, and what they interact with), they don't have the fine-grained control that would allow control of all those acting performance details. It's just not practical with a controller, or keyboard and mouse.

Because the player can't control those acting performance details, such details are effectively absent during gameplay. In typical games, when the player is controlling a character, that character always stands and moves in the exact same way. They effectively have no body language or facial expressions. Their eyes are always looking straight ahead. It's like the performance details have been abstracted away. The character's actions reflect "neutral" intentions and a "neutral" emotional/mental state.

So as far as storytelling goes, having control over a character means there's a very limited "palette" to work with. When there's full control over a character, it's not possible to implement the typical scenes you'd find in a movie.


Strong storytelling requires that there's no control.

Here I'm specifically concerned with control like over a character's movements and what they interact with.

Imagine a game trying to have movie-like scenes but where the player controls their character.

Either all the details of the scene and story (including potential branching paths) are worked out in advanced, or the player is given more freedom and the game can simulate how the NPCs and game world to react to what the player does.

If the scene and story details, including the moment-to-moment acting performance details, are worked out in advance, the player has no freedom to control the character's movements and actions (other than some branching choices). There's a rigid 'performance' the player character has to follow. Which makes the player having control over the character's movements and actions incompatible with such strong storytelling.

What if the player had control over their character and the game simulated the NPCs and games world's responses to the players actions?

At present we don't have the tech to do this. Not where the NPCs respond with verisimilitude.

If the player had the usual sort of control over their character -- which strips away most of the performance details reflecting the emotional and mental state of the character -- that makes it difficult for the game to respond to. Imagine a movie scenes where one of the characters had all their acting performance details stripped away, and could only act in the way a video game character under player control can.

The player would need to be able to control the acting performance. Perhaps this could be done with some sort of sophisticated body tracking technology, that also tracks facial expressions. Future AI may be able to simulate suitable responses to the player's actions. That could be an interesting and enjoyable experience. However, from a storytelling perspective, it wouldn't lead to good stories. One reason is that strong narratives don't live near each other in narrative space. That is, in the space of all possible narratives, most of them are weak. The strong ones are spread out. Changing any particular narrative (to one near it in narrative space) usually weakens the narrative.

Also, if the player has such freedom, much of what they could do wouldn't have verisimilitude for their character and the situation that character finds themselves in. Consider the character being in a situation where they would be scared of something, and would act accordingly (e.g. scared of standing up to a bully, or asking a girl out). Because the player is experiencing a fictional world, there's no reason for them to be scared (or scared much), so they could play the character in an unrealistically confident way.

A player could try to use their control to play their character with verisimilitude, and the game world could realistically respond to that. This could be quite interesting and enjoyable. But for the reasons I mentioned above, I don't think it will lead to strong storytelling.


How do games deal with the inability to have strong storytelling while the player is controlling a character?

Cutscenes are one option -- simply remove player control for the duration of the strong storytelling. These may allow for basic interactivity, in the form of QTEs and choices.

There are some middle grounds, where the storytelling isn't as strong as in a movie scene, but where there is some interactivity.

There are various kinds of environmental storytelling. Like if in a post apocalyptic story the player explores an empty house and finds two skeletons in an embrace in bed, with an empty bottle of pills next to them. Or letters the player can pick up and read, audiologs, and what I've called audiovisual-logs (like in Tacoma).

In a typical movie scene the viewer sees the story details unfold. This is 'direct storytelling'. Elsewhere I've called this 'visual action'. Most environmental storytelling, on the other hand, involves 'indirect storytelling'. It's indirect because the player doesn't see the story details unfold. Rather they're given information that tells them about some story details that (usually) happened earlier (and possibly elsewhere) -- like with audiologs. (The one exception is audiovisual logs). This gets around the issue of control and acting performance details, because in indirect storytelling there are no acting performance details. I've argued elsewhere that such storytelling is a weaker form of narrative.

There's also during-gameplay cutscenes, such as found in Half Life 2, the Metro games, and Dishonored. In these, NPCs act out a scene around the player.

With both environmental storytelling and during-gameplay cutscenes, the player still has control. With environmental storytelling, the player explores the environment to find things like audiologs, and they can continue moving around while listening to an audiolog. Or with like the audiovisual logs in Tacoma, the player can play, pause, rewind and fast-forward the audiovisual logs, and they can move around while viewing them to get a better angle on the details of interest. While  during-gameplay cutscenes are playing out, the player may still have control over their character's movements, or at least over where their character is looking.

However, in both environmental storytelling and during-gameplay cutscenes, while the player has some control, they can't use it to participate in the storytelling. Audiologs, for example, just play out a recording of something that happened earlier. In during-gameplay cutscenes, the player is effectively an audience member watching the details play out. The player character can't have an acting role within those story details playing out. If the acting details of the PC matter at any point in that during-gameplay cutscene, the game has to switch to a non-interactive cutscene for the required details.

Another means of addressing the issue of control and storytelling is through structuring the game as an Activity & Choice game. The choices part is that the player is given dialog choices and other explicit choices about what to do. Sometimes these can be difficult choices. Regarding the activities, when the player has control during gameplay, it is within a specific part of, or scene within, the story. Life is Strange 2 and Pentiment are examples of such games. In the former, in an early part of the game the player's character is preparing to go out to a party that evening. Within this scene, the player is given control to explore around the house to get some items to bring to the party (like some snacks, drinks, and money). The player can control their character to move them around, to look at various objects, and to pick up the ones they want to take to the party. So the player can't control an acting performance, but they can play a 'scene' to some extent.

Since it's a particular story scene being represented, the details can all be focused on being relevant to that scene. It could be that, when getting party supplies, the player character walks around in a carefree way. They might think certain thoughts relevant to that situation, as they go about. What they say when the player looks at different things could be suited to what's going on at that particular point in the story.

It allows more of a story focus than in something like, say, a typical point-and-click adventure game, where the player usually has several puzzles open and thus are going about in what is not a very specific story scene, but still most scenes that you'd find in a movie can't be recreated during the interactive parts of Activity & Choice games, as that'd require the player to have control over the acting performance details.

Overview of my posts on interactive storytelling

Here's an overview of all the posts I've written on interactive storytelling, to help organize my thoughts on it.

 

On the sort of interactive storytelling I'm interested in. 

Using interaction in a narrative media to enhance the focus on character perspective - sketching a high-level vision (Feb 2018)

My idea for a form of interactive storytelling involves using interaction to help put the player in the character's shoes. Imagine being able to understand a different culture by 'living it'.

Some thoughts on how this could be done. 1st person or 3rd person over-the-shoulder point of view. QTE-like interactions for immersion rather than challenge. To initiate elements of an activity, and to shift where the character's looking at or focusing on. Potentially text labels on the QTE targets, indicating what the action the player will do, to help them understand what the character's about to do.

Accurate 1st person visuals would be confusing - the interaction could make it more comprehensible. Lots of 'day in the life' details would be tedious in a non-interactive format. Interactivity could make it more engaging. 

Potentially narration from the character, about what they're doing and thinking. 

Possibly engage the player by having them predict or anticipate what the character's going to do, and in some small way reward them for correctly doing so.

Fictional realism (Jan 2024) 

Related to the previous post, I think interactivity could be useful for presenting 'fictional realism' - fiction that is trying to accurately portray real details (like a real time and place). It may have a strong plot, or it may sacrifice that to some extent in order to provide more 'slice of life' details. Interaction to increase immersion. To put the player in the character's shoes. To let the player experience what it's like to be that character. So to go beyond 'show, don't tell' to 'experience, don't show or tell'.

 

Some useful concepts for talking about interactive storytelling. 

"Strong" vs "loose" narratives (Aug 2022)

There's a spectrum between strong and loose narratives. In a strong narrative, everything that happens is there to contribute to the overall details of the story (like its dramatic arc, and themes). A coherent whole. Events build on what came earlier. Details that don't contribute to the overall story are cut. Loose narrative details don't contribute much to a clear overall story. Real life is a loose narrative. I argue that strong narratives generally mean better storytelling.

Behavior-psychology congruence (Jan 2024) 

Behavior-psychology congruence is important for storytelling. A character's "psychology" is their character and mental state. Their behavior should be congruent with their psychology -- it should reflect their psychology (as well as their circumstances). Control over a character, in a game, can lead to a lack of such congruence. The character's behavior will tend to reflect, to some extent, the player's psychology. 

 

Choice & narratives, and separating narratives from interaction (Aug 2022)

Choices cause a looser narrative. Interactivity in story-focused games is often used to provide the player with choices. The game may provide tens or hundreds of potential (interrelated) stories, with the player's choices determining which one they experience. It's difficult to construct that many different strong narratives. And pretty much impossible to do so if they're overlapping, as strong narratives don't "live close together" in "narrative space".

It's not obvious when a narrative is loose. In a loose narrative, the moment-to-moment storytelling may look like what could appear in a non-interactive work with strong storytelling (like a novel or movie). But a strong narrative is about how all the moment to moment contribute to the overall story, so it's easy to overlook the lack of a strong narrative. It concerns larger-scale story details.

Why the narrative is separate from the interaction. Choices give the player some agency. There can be a puzzle-like element to choose well to get better outcomes. The overall "interactive storytelling" package is narrative plus choices. It's easy to assume if the overall package is great, the storytelling must be great. But interactivity (gameplay) is separate from narrative. The choices determine what happens, and thus what specific narrative the player experiences. The narrative is the result of the interactions, which aren't part of it.

Interactive storytelling containing interactive gameplay segments and separate non-interactive, story-focused segments (Aug 2022) 

Many games alternate between interactive gameplay segments with loose narratives, and non-interactive (or with limited interaction) cutscenes with strong narratives. How the gameplay affects the storytelling. The gameplay puts large gaps between each bit of strong storytelling, effectively diluting the storytelling. The gameplay segments also constrain the types of narratives that can work in the story-focused segments. The overall plot has to be compatible with being repeatedly put on hold, for long periods of time, during the gameplay segments. And the plot has to be compatible with what the player character undertakes during the gameplay segments. In such games, the gameplay's the main focus, and the storytelling is used to enhance the overall experience. Not where the storytelling's the main focus, and the gameplay enhances the overall experience.

Control over a character means an abstracted existence in the game world, which makes storytelling difficult (Aug 2022)

At each moment in a movie, there are the "performance details" (the acting performances, involving their posture, body language, movements, facial expressions). These reflect the characters' personality, intentions, and mental state. These are abstracted away in games when the player has control over a character. The player couldn't have control over them. The player can say what action to do (e.g. 'walk forwards') but can't control the acting performance details of the action. So the posture and movement animations (like for walking) tend to be always exactly the same. Abstracted and generic. This makes it difficult to have strong storytelling while the player has such control.

Action-mirroring control schemes (Jan 2024) 

Action-mirroring control schemes are found in several story-focused games. To increase immersion, the input approximates something of the nature of the action to be performed. E.g. a window that can be slid to the left to open may be opened by pushing left on the joystick. Or a cup of coffee may be stirred by making a circling motion with the joystick. Or mashing a button to try to escape the grasp of a bad guy. This post explores all the different kinds of such control schemes, and considers how successful they are at increasing immersion. We argue they don't provide much immersion, and the immersion they do provide is for doing pretty basic tasks, so aren't very meaningful. They don't feel like doing the actual task. When we do these basic tasks in reality we tend to do them on autopilot (without thinking about how we're doing it).

We want a control scheme that more closely matches how we perceive and undertake actions. I argue that the one used in Beyond: Two Souls is more effective in this regard.

 

Audio and audiovisual logs (Aug 2022)

What they are and how they work. "Audiovisual logs", like found in "Everyone's Gone To The Rapture" and "Tacoma", are where the player sees a past scene unfolding, as if it was presently happening in the location the player is at. The role of these two kinds of logs in gameplay and storytelling.

Environment and storytelling (Jan 2024) 

Looking beyond just audio and audiovisual logs to all the ways a video game's environment can contribute to the storytelling in the game, and how effectively each of these contribute to the storytelling. Also, why environmental storytelling (e.g. letters and audio logs) contributing to backstory is more heavily used in games. And why games aren't suited to the strongest form of visual storytelling, 'visual action'. And some ways of using environmental storytelling that haven't been exploited.

 

Activity-and-Choice Story-Games (Jan 2024) 

A newer genre of story focused games that evolved out of point-and-click adventure games. Comparing the storytelling in these two types of games. In these games the puzzles are replaced by activities and choices.

Having a puzzle-focus restricts the storytelling. It has to be a story in which the main character solves puzzles. In which the plot can only be progressed by solving puzzles, which slows pacing. The gameplay is more part of solving the open puzzles rather than part of a particular story beat. Interactions with (and looking at) objects and other characters are primarily in service of the player solving puzzles.

Activities - character has a task or goal. A specific narrative situation. Player interacts with objects to complete it. Usually there's minimal challenge. It's there to provide some interaction that progresses the story. E.g. getting all the items from home to bring to a party (eg snacks, drinks, a blanket, money). Choices - explicit choices of what to do or say. Choice may need to be made within a time limit. These enable interactions with (and looking at) objects and other characters to be story-focused.

 

Cutscenes with interactivity (Jan 2024) 

The kinds of cutscenes in video games, and how interactivity in them affects the strength of their storytelling. The benchmark is 'visual-action storytelling'. There's the basic non-interactive cutscenes. QTE-and-choice cutscenes introduce basic interactivity. During-Gameplay Cutscenes enable more player control, at the expense of weaker storytelling.

Wolf Tivy and Christopher Sommer on what it takes to do novel work

Two pieces that resonate with my approach to life and to what I'm working on.

The first is Quit Your Job, by Wolf Tivy (Jan 2022). I came across it via this tweet from Kevin Kelly.

In society, there's always new problems to address and new things to create to make things better. You can live your life within other people's existing visions of problems to address and things to create. You can chase money and use that money to consume. But there's also the important task of creating a vision -- defining what are the new problems to address and things to create. And then going after that vision.

This essay is about what it takes to create and execute on such a vision, and how that really can't be done within the confines of a job. You need extended periods of freedom to explore. Thus, "Quit Your Job".

(Doing so is a good way for a person to use wealth they've obtained. It may be difficult for people with a family, unless they're rich. But it's easier than most people think, to live cheaply for an extended period of time, while working things out. And it's definitely possible to do if you're willing to be poor for an extended period, as I have been). 

Here's a sample of it:

The key implication is that while you have not yet found the unique opportunity that will be the engine and purpose of your empire, you have to adjust your sense of value. Value is very legible within a clear plan to reach a clear objective. But you cannot pursue interesting novelty—things that no one else is doing or which you have never seen before, or the little threads of nagging curiosity or doubt—by chasing along known direct value gradients. But that’s where the treasure is. That’s how you will find the place where you need to build. To get the biggest and most interesting payoffs, you have to start by chasing merely interesting novelty in an open-ended way.

Working even a good job cramps your sense of possibility, imposes narrow objectives, and eats away at the little things that could grow into big things if they weren’t so oppressed by the rigors of existing structure. I’ve seen this with my friends, in how they are full of ideas and adventurous spirit a few months after I convince them to quit their jobs. The world is full of ideas and opportunities to explore, but it takes time outside of structure to even adjust your eyes to the landscape of possibility. You are cramped by your job, unable to make the class of investments that is necessary for a life beyond the existing tracks.

If your role in the universe is structured work within order found and built by someone else, those off-road investments are pointless. This conventional work is usually more immediately valuable than anything you could do on your own and it does not require much open-ended exploratory leisure. This efficient pursuit of predictable value is the quiet dignity of the mass of working people. But if we are to solve the bigger structural, spiritual, and intellectual problems which aren’t addressed by existing institutions, someone needs to be exploring off of the established road, where there is a high probability of failing to accomplish anything at all, and a significant probability of discovering and exploiting the next big breakthroughs.

[...]

If you have the resources to spend some time exploring, if you are on to interesting threads of novelty that few other people have, and if you have the spirit to tighten your belt, throw out your map, and explore off-road, then your real job is to do so. It is a grave sin to neglect that kind of cosmic duty. But many more people have the means and privilege to quit their tracked careers than ever realize it and act on it. You need far less than you think to live in monk mode and pursue this kind of exploration. What this means in practice is that at some point far before you are or feel ready, you need to quit your job.

[...]

True ascent beyond the kept life comes only from taking bold, determinate leaps of faith on real constructive projects. [...]

To make such bets you must be indifferent at some level to whether you end up a king or a monk, or even dead. The indeterminate hedge-trader with his logarithmic utility function assigns infinite negative utility to ruin. The man of action serenely regards ruin as the most likely possible outcome, mitigates it where he can, and leaps anyway. He rejects the comfortable half-existence of drifting with the indeterminate human tide and manifests his bold vision into the world. Ruin is largely an illusion in the modern world anyway. If you lose everything you own, you generally still have your network and skills. Even a nominally risk-loving financial utility function is overly conservative in practice because it’s hard to lose these intangible assets.

Life necessarily involves these fatal leaps of faith—bets which you have no certain way of knowing will work out but which define your whole existence and require your intense effort. [...] The highest returns of life and glory come from taking hard bets on your best visions of the future and being able to make them work through dedicated struggle.

[...]

No one can or should be the lone overman who defines all value for himself. We need to cooperate with and defer to each other to make society possible. But even if we individually can only bite off a small piece of the overall purpose structure of our society to manage ourselves, we need to actually do that far more than we do now. For any given question of ends, someone, somewhere, must be taking responsibility. Someone must make that leap of faith to define ends for the rest of us to work towards. No one else is going to do it. Why not you?

[...]

to actually accomplish at your full potential, you have to start doubling down on particular bets long before you know that you can follow through. You won’t see the whole path when you begin. You will have no way of knowing whether it exists, or whether what you are pursuing is even possible. If you have more certainty than that, you aren’t aiming high enough. You have to bet your life on faith that the universe will provide if your vision is good enough.

[...]

Perhaps this is why our society has been so stagnant and uncreative in some ways for the past 50 years. We chose the path of comfort, certainty, measurable progress, and indeterminate hedging of bets. In our cowardice, we turned away from the uncertain leaps of faith of collective struggle after fatal ends that would have demanded us to truly live.

Read the full essay here.

 

The second piece is by former US National Team gymnastics coach, Christopher Sommer. It's from an email he sent to Tim Ferriss that was included in Tim Ferriss's 5-Bullet Friday email of 28 Sep 2024.

We live in a world where people try to manage everything, to make everything productive and efficient. There's deadlines for most things. Things should be done within certain time frames. To not be working to deadlines is seen as to be lazy, unfocused, or sloppy.

Yet this is not true for everything. Developing skills and undertaking creative tasks don't work this way. Trying to fit them into time frames will tend to lead to failure. They take how long they take. This especially applies for the tasks of creating a vision -- defining new problems to address and things to create -- and chasing after that vision, that the first, Wolf Tivy, piece was about.

The approach Sommer describes really reasonates with my own approach to research.

Here's the full text of it:

In fact, this impatience in dealing with frustration is the primary reason that most people fail to achieve their goals. Unreasonable expectations timewise, resulting in unnecessary frustration, due to a perceived feeling of failure. Achieving the extraordinary is not a linear process.

The secret is to show up, do the work, and go home.

A blue collar work ethic married to indomitable will. It is literally that simple. Nothing interferes. Nothing can sway you from your purpose. Once the decision is made, simply refuse to budge. Refuse to compromise.

And accept that quality long-term results require quality long-term focus. No emotion. No drama. No beating yourself up over small bumps in the road. Learn to enjoy and appreciate the process. This is especially important because you are going to spend far more time on the actual journey than with those all too brief moments of triumph at the end.

Certainly celebrate the moments of triumph when they occur. More importantly, learn from defeats when they happen. In fact, if you are not encountering defeat on a fairly regular basis, you are not trying hard enough. And absolutely refuse to accept less than your best.

Throw out a timeline. It will take what it takes.

If the commitment is to a long-term goal and not to a series of smaller intermediate goals, then only one decision needs to be made and adhered to. Clear, simple, straightforward. Much easier to maintain than having to make small decision after small decision to stay the course when dealing with each step along the way. This provides far too many opportunities to inadvertently drift from your chosen goal. The single decision is one of the most powerful tools in the toolbox.

Broadening the notion of affordances

In the design of physical objects and user interfaces, an object’s “affordances” are how the object’s appearances suggest to the user how the object should be interacted with.

Different door handles have different affordances.

Image source: What are affordances, and why do they matter?

In this post, I want to broaden the meaning of “affordances”.

Multiple tools may be all used for the same kind of task. Eg for recording textual information, there’s pen and paper, a word processor, a whiteboard, and voice notes. 

While the standard notion of affordances in design is “how the object suggests it may be used, before it is used”, the notion we’re describing here concerns how the object, while it is being used, shapes how the user does the task.

A whiteboard suits getting info down fairly quickly, in bullet points, and for drawing arrows between items to show their relationships.

A word processor encourages writing in full sentences. Because we can easily see what we’ve already noted down, they also encourage us to write in sequences of sentences and paragraphs.

Pencil and paper seems to be part way between the free-form nature of the whiteboard and the more regimented form of the word processor.

Voice notes are more focused on the present moment. You can’t see what you’ve already said, and it’s more effort to go back to hear the early part of the note. They’re good for brainstorming.

The traditional notion of affordances covers how the design of an object affects a user’s expectations about how to use that object, before they actually use it. We’re expanding this notion to also include how an object’s design shapes the way it is used by a user.

Interactive storytelling: Fictional realism

This post is about the notion of "fictional realism", which I am using to mean a fictional account that is nonetheless meant to 1) accurately portray the time and place where it is set, and, optionally, to 2) focus more on this portrayal, than on presenting a story to the reader/viewer.

Examples of fictional realism include the TV shows "The Wire" and "The Sopranos", the movie "Casino", the novel "One Day in the Life of Ivan Denisovich", and the game "Attentat 1942" (Steam page). Most of these examples have a fairly strong storytelling focus, so don't fit the optional second criteria.

A work of fictional realism may be intended to convey the same kinds of details that a non-fiction work may convey, but to do so using fictional characters (or fictionalised versions of real people), in fictional situations (or fictionalised versions of real situations).

 

"Interactive storytelling" can be used for fictional realism. One of my interests is in using interactive storytelling for exploring a strong form of fictional realism, meeting the second criteria described above (focus more on the portrayal of what life was like, than on presenting a story to the reader/viewer), that presents what it'd be like to be a specific character in a specific situation. So that the player can learn about the player character, about what they do, how they do it, and how they react to situations. Like worker on a sailing ship in the 17th century spice trade. What was their work like? How did they perceive their job (exciting? a journey of exploration?). What were their relationships with the various other sorts of people on the ship?

Interaction could help place the player in the character's shoes, to help immerse them in the character's world. I'd like to use interaction to let the player experience what it's like to be that character. There is the dictum 'show, don't tell'. I want 'experience, don't show or tell'.

I have some ideas about how the interactive storytelling could work, such as to achieve this, though I won't get into such details in this post.

Fictional realism can be used in an educational context. Or be an enriching kind of entertainment. By giving the player interactivity, and letting them experience what it's like to be that character, we hope we can make a compelling way to experience fictional realism.

 

I think that behavior-psychology congruence is a core requirement for fictional realism, and I'll explore this in a future post. In brief, I want the player to control the character such that the character acts in a realistic way.

 

I've also written about the notion of 'strong storytelling', which we can think of as effective or good storytelling. Strong storytelling has a strong focus on plot, and moving the plot forwards. Thus it will tend to cut-out details that aren't relevant to the plot. Thus it would cut out the sorts of details I'm interested in, in fictional realism. The sort of 'day in the life' details.

Compared to strong storytelling, fictional realism is more like real-life. Real-life tends not to be like a story. In stories, all the details are there to serve the overall goals of the story, like its climax, conclusion, and themes. In real life, things happen, but it's just one thing after the other, and they aren't there to result in some climax and conclusion.  

There is, however, no reason why a work of fictional realism couldn't have a plot. It could. It's just that the fictional realism details will dilute the story details, thus making it a weaker form of storytelling.

Strong storytelling and fictional realism are just different forms, each with their own pros and cons.

Interactive storytelling: Behavior-psychology congruence

We wish to introduce the notion of a fictional-character's behavior being congruent, or not, with their psychology. This will help us to, in subsequent posts, look at how, in interactive storytelling, the player having control affects the storytelling.

We use 'psychology' to mean two things: the character's makeup and circumstances.

A character's makeup, is their nature, their personality, their character, and how they think about situations. Such details are a result of their nature, and nurture. How their character is shaped by their life experiences. It includes how their personality might be changed by brain damage or a brain tumor. Or how medications they are taking affect their personality.

By a character's circumstances, we mean what has been going on in their life. Perhaps they have had a stressful few weeks at work. Maybe a loved one died a few months ago, and they are going through grief. Or maybe they started a new relationship and they are happy as a result.

I don't think there's a hard-and-fast distinction between a character's makeup and their circumstances. These are just rough categories.   

In real life, a person's behavior is always congruent with their psychology (their makeup and circumstances). In fictional works, we almost always strive to make a character's behavior congruent with their makeup and circumstances, though we may fail to achieve this. So there can be a lack of congruence.

Behavior-psychology congruence doesn't just apply to "realistic" characters. It applies to all characters, even wacky and "out there" cartoon characters. Wile E. Coyote from the Warner Brothers cartoons wants to catch the Roadrunner, and sets up traps for this purpose. Despite many failures, he's never one to give up trying. The Roadrunner, in turn, likes running fast along roads, and seems to take joy in making the Wile E.'s traps backfire on him. These characters are not realistic, they're not at all like real coyotes and roadrunners. But Wile E.'s psychology is to want to capture the Roadrunner, and to setup traps to do so, and so Wylie's behavior is congruent with that.

If a character has a cartoony makeup then their behavior should be cartoony as well, and good writers will make sure their behavior is congruent with their makeup and circumstances.

If there were scenes where Wile E. was sincerely explaining to other characters that he has been vegan since he became an adult, because he believes no animals should be harmed, then this would not be congruent with his established psychology.

 

In fiction, a character's behavior may be incongruent with their psychology, because of poor writing, poor acting, or poor directing. We can imagine that a very inexperienced writer setting out to write a novel. Earlier in their draft they gave the main character a gentle personality, whereas later on in the draft they gave them an aggressive personality, where the author didn't realise this change had happened. Which leads to inconsistencies in how that character is portrayed, with no explanation in the novel of why the character is different.

We may have a philosophical objection to this talk of incongruence between a character's psychology and their behavior. If all we as viewers or readers see is the character's behavior, and we infer their personality from that, then it would seem to be impossible for there to be such incongruence. Any apparent incongruence would just seem to be incongruence because we didn't yet know enough about the character's psychology. The philosophical objection is that we can only know behavior, so behavior is what defines our picture of the character's psychology, thus /by definition/ there can never be incongruence between them. Any /apparent/ incongruence is simply because we have formed an incorrect picture of their pscyhology, by jumping to incorrect conclusions about it based on the prior behavior of theirs that we've observed.

If a real person's behavior seems incongruous with their psychology, then it is our understanding of their psychology that is wrong (or incomplete). But here we are not talking about real people, but characters in fiction, fiction that may be written by a beginner, or an untalented, author.

In fiction (novels especially), the author may explicitly describe aspects of a character's personality. This way, that character's behavior can be incongruous with the stated aspects of the character's personality, if the writer is inexperienced or otherwise not very good.

But even when the character's personality is only inferrable from behavior, it is still possible for the two to be incongruous. It can be possible to infer psychological traits from behavior, and so we may have two sets of behavior B1 and B1, which reflect psychological traits P1 and P2 -- and P1 and P2 may conflict. A character may be terrified by speaking in front of their class, one point, and yet later inexplicably be supremely confident speaking in front of a large group. We're not saying it's impossible for there to be such a transformation; we're talking about the case of a story that has not included any details explaining such a transformation. At least one of those two behaviors is therefore incongruous with part of the person's psychology.

 

We usually expect that if the character seems to be acting incongruently with their established psychology, that the story will provide an explanation of why. But it may not, and the incongruence may be a result of poor writing (or acting or directing).

 

In subsequent posts, we'll look at how interactivity, in interactive storytelling, can lead to incongruence between a character's behavior and their psychology. The basic idea is that if the player can control a character's actions, then those actions will tend to reflect the player's psychology, not the character's. Using the terminology of those future posts, we'll explain why behavior-psychology congruence is necessary for strong storytelling and fictional realism.

Interactive storytelling: Environment and storytelling

In a video game, the environment includes the locations the player sees as they traverse the game world. The environment includes the objects they can interact with, such as objects they can pick up and look at, doors they can open, lights they can switch on, etc.

We'll look at the ways a game's environment can contribute to the storytelling in interactive storytelling like video games. Then we'll look at how effectively each of these ways can contribute to storytelling.

 

The setting

The game environment provides the setting for the story. A story about an ad executive living in New York City obviously is set in New York City. A story may lean heavily into its setting, and concern the nature of that setting -- like one that concerns the culture in New York City. Or the story's setting can be more of just a backdrop, where the same story could potentially be set in a number of different places, without changing much about it.

 

The stage necessary for story events

The game's environment may contain details necessary for enabling certain story events. A dense town or city, with suitably-small gaps between the rooftops enables a story event in which the protagonist is chased by several bad guys over rooftops.

 

Atmosphere and world building

An environment of moss and plant-covered ruins could contribute to a post-apocalyptic atmosphere. Posters plastered around a city, instructing the populace on how they should behave, could contribute to the world-building and atmosphere of story set in a fascist country. Dim lighting, with a yellowish hue, along with strange sounds, could give an alleyway an eerie atmosphere.


Characterisation

The environment can contribute to characterisation. If a character's house is neat and tidy (or very dirty, and messy) that will convey something about their personality. As could the paintings we find in their house, or the entries in the diary hidden under their pillow.


The main plot

The environment can contribute to the main plot. The space under a bed may contain a piece of evidence that conclusively shows who's guilty of a murder. A character finding it will be a major plot point.

 

Backstory

The environment can contribute to backstory. Backstory concerns past events, prior to the events the player is currently experiencing in the main story-thread. Those events could have occurred before the start of the main story thread. They could have happened a long time ago. Such distant backstory includes 'lore', that concerns historical details of the setting. Lore may be found in tomes that the player finds, or some runes written on some ruins.

We'll also take 'backstory' to include details that may have only recently happened. For example, halfway through the story the player might receive details about something that had happened a hour prior -- that is, an event that occurred well after the point in time where the story began. We'll call this backstory, too.

Environmental details that contribute to backstory include artefacts like diary entries, letters, memos, and books.

Audio-logs are a means to fill the player in on backstory that's used in video games. Audio-logs are sound recordings that the player can listen to. They may be physical objects, like a tape recorder, that the player can find and play. Such might be a sound recording of a diary entry or voice note. Or an audio-log recording might automatically start playing when it is 'triggered' by the player's actions -- like if the player enters a particular room, or perhaps opens a particular person's locker.

Audio-logs were popularised in the first-person shooter BioShock (2007), which used them as the main means to tell its story. Dear Esther (2012) and Gone Home (2013) are games entirely focused on exploring an environment and finding audio-logs, in order to piece together their stories. Those latter two games are key examples of the genre that came to be known as "walking simulators".

There are also what I've termed audiovisual-logs, used in games like The Vanishing of Ethan Carter (2014), where they only play a small role, Everybody's Gone to the Rapture (2015), and Tacoma (2017). Audiovisual-logs allow the player to see an audiovisual recreation of some past event, in the location where that event occurred. The player can walk around and through this recreation while it is playing.

 

(an audiovisual-log from Everybody's Gone to the Rapture)

(an audiovisual log from Tacoma)

The following links are to the original videos those gifs were created from. Everybody's Gone to the Rapture, and Tacoma.

Audiovisual-logs are the core means of storytelling in Everybody's Gone to the Rapture and Tacoma. In The Vanishing of Ethan Carter, they only play a relatively-small role[1].

[1] In that game, there are certain puzzles, each of which concerns finding out how a particular character died. At the completion of each one of these puzzles, the player is shown a cut-scene showing the full details of how the character died, and then after that the player can follow a floating light. When they get to the light's resting place, they can see a short audiovisual-log of what came next, after the overall happening they've just seen the cutscene of.

For more details on audiovisual-logs, see the audio-logs link, above.

There's what I've termed "frozen-moment-logs". The only game I know of that uses these is The Vanishing of Ethan Carter (2014). The player will come across places where they can unlock a frozen-moment-log, consisting of a static 3D image of some past occurrence (backstory) there, that the player can walk around and view from different angles. In that game, these logs appear as part of larger puzzles.
 

(a frozen-moment-log from The Vanishing of Ethan Carter)

 

Another technique, where the environment contributes to backstory, is what I'm calling "Inferred Backstory". This is where environmental details enable the player to infer some past events.

In a post-apocalyptic game, the player may come across a dilapidated house, and in one of its bedrooms find two mummified corpses in the bed, frozen in an embrace. On the bedside table may sit a framed photo of a happy couple, along with an empty bottle of sleeping-pills.

From these details, the player may infer that the couple was once happy, and loved each other, but ended up finding their circumstances untenable. They may imagine the couple coming to this realisation, taking the sleeping pills, and tearfully embracing each other in the bed as they awaited their fate.

Inferred backstory is often like a little puzzle, where there are clues and the player infers the backstory from them. Usually it's a very simple puzzle.

'Inferred backstory', as we are using the term, concerns any cases of where the player infers prior details from present-moment details. That can include when a player immediately and effortlessly infers some prior details, including details that recently occurred. E.g. they're in a forest and come across some fresh large-animal droppings. They'll immediately infer there was recently some kind of large animal in this place. Some other examples where the player will make immediate and effortless inference: if the player came across the charred remains of a fire, or doors that been broken open.

And of course, by inferring details that have recently happened (e.g. a large animal being here) we may also infer present-moment details -- e.g. that the large animal may be nearby us right now. 

Carson[2] calls these cases of inferred backstory "cause and effect" vignettes, and the above description of them is based on his article.

[2] "Environmental Storytelling: Creating Immersive 3D Worlds Using Lessons Learned from the Theme Park Industry", by Don Carson, March 2000

To effectively communicate story-relevant details

This section draws a lot from Carson.
 
The environment should be designed to effectively communicate important details to the player. Details like where the player is, what sort of place it is, and where they should go next. Usually, we want the player to be able immediately determine such details.

These kinds of concerns don't, of course, only apply to the environments in games. They also apply in movies, TV shows, theatre, and theme park rides. Think set design.

Here are some details that we want the environment to communicate to the player. Which details in a location are the most important story-relevant ones. The atmosphere of a location, and the kind of place it is. What objects and features are important for the player to be aware of. And where the player should go next.

The following can aid in that communication. We can draw attention to important details by how we arrange the objects and features in that location, and by how those objects and features are lit. And by not including too much detail in the location, especially detail that's of little relevance to the story. We don't want to confuse or overwhelm the player.

Contrast can be used to heighten qualities. For example, if we make the player crawl through a narrow passageway to reach a cave chamber, that can heighten the feeling of how large that chamber is. Or, if we want the player to feel that the temple in the forest is a pristine place, we can make them experience a disordered space (like thick jungle) before they find it.

The environment can be designed to guide players as to where they need to go next. For example, in a dark area, having a well-lit large object in one corner, that the player will want to investigate, where this will take them to the exit from this location, to the next place they need to go to.

Many of the ways that the environment can contribute to storytelling are showing rather than telling. Rather than explicitly describing the atmosphere and world building of an environment (perhaps by a character commenting on them), environmental details can show them. With inferred backstory, the player draws their own conclusion about what happened.

And, as Carson points out, the player being able to discover details like artefacts (letters, memos, audio-logs), and inferred backstory, themselves, can be a more enjoyable experience for them than them simply being told those details. (That discovery is part of the gameplay, so the enjoyment comes from the gameplay. It doesn't come from the storytelling).

Summary of how the environment may contribute to storytelling

To summarise, the environment may contribute to the following elements of a story:

• The setting (e.g. NYC)
• The stage enabling certain events (e.g. rooftops for a chase sequence)
• Atmosphere and world building (e.g. moss-covered ruins, strange sounds)
• Characterisation (e.g. a character's messy room)
• The main plot (e.g. evidence of who the murderer is).
• Backstory (e.g. diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed).

And there are techniques that can be used to effectively communicate such elements of the story. For example, to highlight the size of large cave chamber, make the player crawl though a small space to get to it.

 

Environmental storytelling

The reader may have noticed that we started this post with the heading "Environment and Storytelling" not "Environmental Storytelling". For these are two different things. "Environment and storytelling" refers to all of the ways that the environment contributes to storytelling. Whereas "Environmental storytelling" refers only to a subset of them.

"Environmental storytelling" is a commonly used term to describe storytelling in games. It is used to refer to only cases where the environment contributes to backstory.

For some people, "environmental storytelling" refers to all of the ways that environmental details can contribute to backstory: diary entries, letters, memos, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory (like the corpses in the bed).

For other people, "environmental storytelling" has an even narrower meaning. For them, it only refers to cases of inferred backstory, and does not refer to cases like diary entries, letters, memos, audio-logs, audiovisual-logs, and frozen-moment-logs.

Both definitions of "environmental storytelling" involve environmental details that are leftover from, or reflections of, the past (in the broad sense of "prior to the current moment", not just things in the more distant past). Obviously that's the case for Inferred Backstory like the corpses in the bed. It's also the case for things like diary entries, letters, memos, audio-logs, audiovisual-logs, and frozen-moment-logs.

In this post, we'll use "environmental storytelling" in its broader sense, which includes all the ways that environmental details can contribute to backstory.

Environment and storytelling in movies and TV

Movies and TV use the story's environment (sets, and on-location shots) for storytelling purposes, and of course have done so since well before video games came onto the scene. Movies/TV and games mostly use their environments in similar ways for storytelling, except for some of the ways backstory is used.

(As well as movies/TV, theatre, theme parks, and theme park rides, all use the environment to contribute to storytelling. But this post will focus only on video games, movies, and TV).

Movies/TV can use environmental details to contribute to:

• The setting (e.g. NYC)
• The stage enabling certain events (e.g. rooftops for a chase sequence)
• Atmosphere and world building (e.g. moss-covered ruins, strange sounds)
• Characterisation (e.g. a character's messy room)
• The main plot (e.g. evidence of who the murderer is).
• Backstory ('environmental storytelling', e.g. diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed).

One difference, regarding the techniques that can be used to effectively communicate such elements of the story, is the following. In games, the environment can be designed to guide players as to where they need to go next, which obviously doesn't apply to movies/TV/cutscenes. Whereas in movies/TV/cutscenes, the cinematography, lighting and set-design can guide where the viewer looks during scenes.

In a moment we'll get into some differences in how backstory (environmental storytelling) is used in movies/TV compared to games.

Why is environmental storytelling more common in games?

The environmental storytelling techniques (contributing to backstory) are either less common or not found at all in movies and TV. This includes things like diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed.

There's a clear difference between games and movies/TV in this respect, as games often heavily rely on such environmental storytelling. In games, it's often the main form of storytelling that's used. In movies and TV, it tends to be a supplemental form of storytelling.

What is the reason for this difference?

 

Cost and ease

Movies and TV are focused on "cinematics" -- visual portrayals of actors and environments. Whereas game development companies have a primary focus on gameplay. In most cases, it requires additional resources for the game developer to be able to include "cinematics" in their games. And smaller game developers may not have the skills and/or budget for this.

Compared to cinematics, it's cheaper and quicker to add artefacts like diary entries and letters to a game. Audio-logs require hiring voice actor(s), but a game might have a total of less than 1 hour of audio-log audio, which can be recorded quickly and thus doesn't require the expense of hiring a voice-actor for a long period of time.

On the other hand, high-quality animated cutscenes take more time to develop, and require animator(s) and voice actor(s). FMV (full motion video) or motion-capture for cutscenes requires (real or virtual) sets, and hiring actors. Motion capture requires specialised equipment (either purchased or hired) and the skills to turn its output into animation. I imagine that the time and money required for cutscenes is similar to the time and money required for audiovisual-logs.

 

Visual mediums excel at visual-action storytelling

Here's a reason that environmental storytelling is used less in movies/TV. Movies and TV are primarily visual, and excel at visual-action storytelling. This is the visual depiction of action (what I referred to as 'cinematics', above). By 'action', I don't mean just things like fights and shootouts, as you'd find in action movies. I mean 'action' in a general sense, of the visual details that can be captured by a video camera. This could include characters simply talking to each other, or a tense scene where two characters are sitting in the same room, each silently trying to ignore the other.

In constructing visual action, all the tools of acting (performance), cinematography, editing, and so on, are brought to bear. Visual-action storytelling is a strong form of storytelling.

If the movie or TV show contains backstory, it's usually presented through visual action -- that is, through a flashback. Environmental storytelling also conveys backstory, but it mostly does /not/ do so through visual action. In a moment we'll examine this, and see why environmental storytelling tends to thus be a weaker form of storytelling[3].

[3] Before leaving this topic, we can note that one kind of use of inferred backstory in movies/TV is where the 'clues' are amongst the background details of scene(s), that might only briefly be in shot. Most people watching the movie/show wouldn't notice them, and they're there as interesting details or "easter eggs" for repeat or careful viewers. And/or as details designed for other viewers to subconsciously take in.

 

Games excel at visual-action gameplay, but not visual-action storytelling

Like movies and TV, graphical games are also a visual medium -- with the addition of gameplay. They excel at incorporating gameplay into visual-action. Consider first-person shooters, platform games, racing games, etc. However, games do not excel at incorporating gameplay into visual-action storytelling.

That's why, if a game is to include visual-action storytelling, that's done in a cutscene (effectively a little movie) that is separate to the main gameplay. Cutscenes are usually non-interactive, though they can include simple forms of interactivity like Quick-time Events (QTEs) and "Choose Your Own Adventure"-style choices. We don't have a way of integrating the player having control over a character and movie-like visual action.

In visual-action storytelling, all the tools of acting (performance), cinematography, editing, and so on, are brought to bear. But, during gameplay, when the player has moment-to-moment control over a character (like the character's movements), it's not possible to strongly exploit those tools of visual-action storytelling.

Here we'll turn our attention to cutscenes. These aren't a form of environmental storytelling, but looking at them will help convey the point that games are poor at integrating interaction/gameplay with strong visual-action storytelling.

In this earlier post, I looked at the types of cutscenes in games, and the ways interactivity and cutscenes can be mixed together.

The standard non-interactive cutscenes are strong visual-action storytelling, but the involve no interactions at all.

In QTE-and-Choice cutscenes, simple player inputs are incorporated into the strong visual-action storytelling. These are interactions like QTEs (Quick-time Events), and choices where the player can choose from a small menu of options, such as dialogue choices or choices about which course of action to take (save Billy or save Jenny, from the oncoming horde of zombies).

QTE-and-Choice cutscenes are strong forms of visual-action storytelling, however as far as interactivity goes, they contain weak forms of interactions/gameplay. The player has very limited control over a character.

During-gameplay cutscenes are how most of the cutscenes are handled in games like Half-Life 2 and the Dishonored games. The player still has some degree of control over their character, while some scripted events occur around them. The player may be able to freely turn their head to look around, or that plus the freedom to move around (within the constraints of their environment, like brick walls etc).

As that earlier post argued, During-Gameplay cutscenes have stronger forms of gameplay but weaker forms of storytelling than non-interactive cutscenes.

So none of the kinds of cutscenes involve strong visual-action storytelling along with strong interaction/gameplay.

Returning to environmental storytelling, audiovisual-logs, like in Everybody's Gone to the Rapture, present visual action, but since it is visual action that the player can walk around and through, the tools of cinematography and editing can't be brought to bear on it. Like with during-gameplay cutscenes, the player is reduced to a spectator of the events.

In summary, even though visual-action storytelling is the strongest form of storytelling in a visual medium, games are not very suited to it. To use visual-action storytelling, games have to either include non-interactive or Q&C cutscenes, which clash somewhat with the interactive nature of games, or D-G cutscenes which have more interaction but weaker storytelling.

 

Environmental storytelling is more compatible with gameplay

That it's difficult to insert gameplay into visual-action storytelling is a reason why environmental storytelling is so often used in games.

Environmental storytelling can form part of the gameplay.

In it, the player explores the environment, and finds artefacts (like diary entries, letters, memos, audio-logs, and audiovisual-logs). A player who is not looking carefully might miss some of the artefacts. That exploration and finding is part of the gameplay.

Audio-logs can be listened to while the player is still engaging in gameplay, where they're moving around and continuing to explore.

When an audiovisual-log is playing, the player can move around, and within, the recreated visuals. The player may move around to find a good view of all the action, to follow a particular character around, or to be closer to one particular conversation (if there are multiple happening at the same time). The audiovisual-logs in Tacoma allow the player to scrub back and forth in the log, to find pertinent details.

Environmental storytelling (e.g. diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed) can present a kind of puzzle to the player. Something unknown, in the past, has happened, and environmental storytelling provides some clues to its nature. It's like each bit of environmental storytelling is a puzzle piece and the player needs to figure out how they fit together, to see the overall picture of what happened. This is a very common pattern in games that heavily use environmental storytelling. For example, in BioShock, Gone Home, Everybody's Gone to the Rapture, and Tacoma. (Note that this can be done in other mediums, like movies/TV, and novels. But games more often make use of such).

So all these forms of environmental storytelling are more compatible with gameplay. Environmental storytelling brings storytelling into the main gameplay parts of the game, rather than having it be fairly distinct from gameplay, as with cutscenes.

 

Environmental storytelling is generally a weaker form of storytelling

While environmental storytelling allows storytelling to be better integrated with the gameplay, it unfortunately involves a weaker form of storytelling.

We've stated that visual-action storytelling is the strongest form of storytelling in a visual medium. This, and some subsequent sections, look at why environmental storytelling is weaker than visual-action storytelling.

To clarify, I'm saying they're generally weaker forms of storytelling, not that they're bad. They can be quite effective. However, when they are heavily relied upon, like they often are in games, that will tend to weaken the overall storytelling.

Environmental storytelling conveys backstory. I suggest that storytelling that focuses on the main story thread is, generally speaking, stronger storytelling than that focusing on backstory. This is debatable, but I think it's the reason that backstory tends to be used sparingly in most movies and TV shows.

And, if backstory is used, it's more strongly presented with flashbacks, which present the details through visual action. Environmental storytelling mostly does not convey details through visual-action. So these forms of environmental storytelling are generally weaker forms of storytelling.

Audiovisual-logs, such as used in Everybody's Gone to the Rapture, and Tacoma, are the only form of environmental storytelling that's via visual action (during-gameplay cutscenes also use visual action, but they are, like normal cutscenes, not a form of environmental storytelling). Though, as mentioned before, audiovisual-logs can't make use of cinematography or editing, as the player is still in control of a character. The camera needs to be suited to moving a character around. Editing involves cutting out parts of the action, showing only the details before and after it, and that doesn't mesh well with gameplay. Still, audiovisual-logs have the potential to be fairly strong forms of storytelling -- similar to flashbacks.

All the other forms of environmental storytelling are not presented through visual action. They may be textual artefacts like diary entries and letters. And auditory ones like audio-logs.

What about inferred backstory, like the corpses in the bed? Here the backstory is told using visual details (from which the player infers past events). But those are present-moment visual details; the backstory being conveyed is not shown visually. It doesn't, for example, show the visual action of the couple taking the sleeping pills and getting into the bed.

The player coming to their own realisation of what happened, through the environmental storytelling, is something that a number of players enjoy. This is part of the appeal of environmental storytelling. However, I don't consider this to add a lot to the strength of the storytelling.

 

We can draw a distinction between artefacts that provide narrative details, and those that don't, but which convey narrative-relevant information. We can call these narrative and non-narrative artefacts.

Artefacts like diary entries and letters can be either narrative or non-narrative. They can convey narrative backstory, like a diary entry recounting an event that occurred. A non-narrative example is a diary entry that said "Bought new clothes at shops" and then went on to list the items of clothing. This is just some information, though from it the player may infer some narrative-relevant details (that the person who wrote it cared a lot about their neighbour, who they bought a number of items for).

(To turn to textual mediums for a moment, epistolary novels tell a story through letters sent between characters. "Epistolary novels" is also often used in a broader sense, covering stories told through any kinds of artefacts, such as diary entries, newspaper clippings, or other kinds of documents. Some well known examples are "Carrie", by Stephen King (1974), "Posession", by A. S. Byatt (1990), and the Adrian Mole series, by Sue Townsend (1982-2009). Such novels show how something akin to environmental storytelling can be used quite successfully for storytelling. I believe that using artefacts in this way is much more suited to textual mediums, and much less so in visual mediums like visual video games. This is because they are textual artifacts, which means they fit in with the textual nature of novels. Whereas, showing textual artefacts on screen for the viewers to read, or having them read out by a character, is not as suited to the visual-action character of visual mediums.)

I contend that such non-narrative information is, generally speaking, a weaker way of conveying details that are part of the narrative.

Audio-logs may simply be recorded versions of artefacts like diary entries, which may or may not be focused on narrative details. Narrative audio-logs might, for example, contain a recording of when some bad guys stormed a character's office, and took them hostage. That is, an audio-log may contain a recording of an event that happened. Audiovisual-logs will usually convey narrative details.

Inferred backstory (like the corpses in the bed) also conveys narrative details.

Here's a summary of the ways the environment may directly contribute to the narrative and those that do so indirectly.

These environmental storytelling techniques directly contribute to the narrative. Those that contribute to:

• The main plot (e.g. evidence of who the murderer is).
• backstory (with narrative artefacts)
  (e.g. certain of: diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed).

Environmental storytelling techniques that only indirectly contribute to the narrative/plot:

• The setting (e.g. NYC)
• The stage enabling certain events (e.g. rooftops for a chase sequence)
• Atmosphere and world building (e.g. moss-covered ruins, strange sounds)
• Characterisation (e.g. a character's messy room)
• non-narrative artefacts conveying backstory (e.g. diary entries, audio-logs).

And where environmental details are used to effectively communicate story-relevant details (e.g. to highlight size of large cave chamber, make player crawl though a small space to get to it).

Audio-logs and inferred backstory that convey narrative details, don't do so through visual action, so they are generally weaker forms of (back)storytelling than flashbacks, which do.

To help reinforce these points, we can note that movies and TV could employ an equivalent of audio-logs. There could be scenes where a character is listening to a sound recording. Or where the audience hears a character reading out a diary entry or letter. The visuals might show the character driving in their car as they listen to the audio recording, or walking around their house while reading the diary entry or letter.

Because the focus of such scenes would be on the audio, the visuals would essentially serve as a background to the audio. The visuals couldn't convey any substantial narrative details, as that would distract the viewer from the audio. So thus it'd be weaker storytelling, because it's not focused on visual action.

That such scenes are, as far as I'm aware, rare in movies and TV is, I suggest, because they're a weaker form of storytelling.

 

Pacing

Another reason flashbacks are a stronger means of conveying backstory is that their visual-storytelling benefits pacing. Pacing is an important part of storytelling. A story may concern some events that take place over a week, but they -- as represented in a movie or TV show -- may do so through only a couple of hours of visual action. They condense the details. They filter out the narratively-irrelevant details, to leave a more narratively-concentrated end-product. If the narratively-interesting details are padded out with a lot of irrelevant details, it will slow down the pacing, and dilute the narrative.

A heavy focus on environmental storytelling, as is often found in games, negatively affects the pacing. It slows the pacing.

Imagine if, during a movie or TV show, there were several occasions where the main character read a full-page diary entry, letter, or memo. Where, each time, the shot of them reading it lasted long enough for the character to read the full text. (Their reading of it might be conveyed with a voice-over representing the character's inner voice, as they read the page). That would, I think, make the pacing feel quite strange. It'd be jarring, to go from the normal speed of the pacing in a movie or TV show, to these really slowed-down segments.

(And this is on top of the fact that gameplay itself, also slows the pacing of the storytelling. Because there'll be long segments of gameplay-focused action in between each narrative-focused segment, and the gameplay is usually not conveying much vis a vis the game's narrative).

 

Limitations to narrative complexity of audio-logs and inferred backstory

I mentioned earlier that audio-logs and inferred backstory can convey narrative details. They are, however, quite limited in their ability to do so.

An individual audio-log or inferred backstory instance can only present fairly short and simple narrative details. And there are fairly strong limits on how substantial/complex the overall narrative details from the totality of the audio-logs/inferred-backstory.

With inferred backstory (like the corpses in the bed) there's visual details from which the player can infer past events. But it's difficult to convey a lot of detail in this fashion. The player has to infer -- figure out -- the events from the clues. It would be too complex for the player to infer more than a simple set of details. For one thing, it would be very challenging to indicate the sequence of the events.

Audio-logs that convey narrative events (like a recording of a kidnapping) need to be relatively brief. Audio-logs are designed such as to not get in the way of gameplay, while the player is listening to them. While listening, the player can still continue to explore around, and possibly even take on some enemies. While listening, they'll be watching visuals (of their environment) that are likely pretty unrelated to the content of the audio-log. Which means they face distractions while listening to the audio-log. They won't usually be paying 100% attention to it. So the player would have trouble being able to properly take in longer audio-logs. Also, if audio-logs were lengthy, then while the player is still listening to one, they might get into a situation (like a major fight with multiple enemies) where 1) they can't focus at all on the audio-log and 2) the audio-log might distract them from the gameplay. (Though this latter point might be addressable by a means to pause audio-logs).

All of the narrative audio/audiovisual -logs, and all of the inferred backstory, in a game, could together contribute to the overall narrative. However, there might be an average of, say, 10-30 minutes of playtime between each audio-log or inferred backstory that the player comes across. During which time the player is undertaking gameplay. This places fairly heavy demands on the player's memory, and the game designer can't expect that the player will be able to recall a lot of the specifics presented in earlier audio-logs and inferred backstory instances. Therefore, individual audio/audiovisual -logs and inferred backstory have to be designed to be somewhat stand-alone. They can't be narratively connected together in intricate ways.

So there are limits on how substantial/complex the overall narrative, conveyed through multiple audio/audiovisual -logs and/or inferred backstory instances, can be. One other reason for this, that applies to a number of games, is that different players may come across the audio/audiovisual -logs and/or inferred-backstories in somewhat different orders.

In contrast, visual action can convey a lengthy sequence of events (like, a whole movie's worth).

 

Contrived nature of audio-logs

Audio-logs are also somewhat contrived. Why were these recordings made in the first place? It might make sense if they're like diary entries or voice notes that a character made. But it doesn't if they're a recording of a narratively-significant event that happened. Who thinks to switch on a recorder just before a significant event? And why are the audio-logs found in various different places in the environment? Often it doesn't make sense. So this contrivance is another reason why you wouldn't have characters in movies/TV finding such audio-logs.

 

Audio/audiovisual -logs are only compatible with certain story settings

And both audio-logs and audiovisual-logs are also only compatible with certain kinds of stories. Neither of them could appear in a realistic story set in the 1700s. Audiovisual-logs, further, need to be in a story that contains elements that are magical, supernatural, alien, or high-tech.

 

In summary, environmental storytelling consists of generally-weaker forms of storytelling, and the reason they're often used in games is that they fit better with gameplay than do other means of storytelling.

 

Where next?

Given the storytelling limitations of the environmental storytelling we've looked at, are there other ways environmental storytelling could be used, that might be better for storytelling?


Further exploring the use of audiovisual-logs

Audiovisual-logs have potential because they can present narrative details through visual-action. The player sees the characters and events being portrayed. Yet, I'm only aware of three games that use them. One (The Vanishing of Ethan Carter -- see below) barely uses them, and the other two (Everybody's Gone to the Rapture, and Tacoma) don't lean much into the visual action.

I expect further exploration of the use of audiovisual-logs in future works. Especially since it should become cheaper and easier for game developers to create them. Motion capture and animating the captured data, is only going to get cheaper and easier over time. AI will likely play a role in that. AI will likely make it quicker and easier to record and process the motion capture data, and to generate animated models from it.

 

Audiovisual-logs with full character-detail

In The Vanishing of Ethan Carter (2014), the audiovisual-logs have a quite minor role. Each one is quite short -- probably 5-10 seconds long -- and there's only a few of them in the game (probably 5-10). In these you see the representations of the characters like you would in a cutscene, except you are there in the scene and can move around and look around while you're watching it play out.

In Everybody's Gone to the Rapture (2015) and Tacoma (2017), the audiovisual-logs are the main source of the storytelling, but at the same time, those games don't lean much into the visual action in the audiovisual-logs, because they show highly abstracted representations of the characters.

In Everybody's Gone to the Rapture (2015), the characters are represented by glowing, dancing points of light. It makes it difficult to even get a clear view of the characters and their movements.

(A still from an audiovisual-log in Everybody's Gone to the Rapture. From this one-minute video. It's from early in the game.)

In Tacoma (2017), you can see the shapes of each of the characters, but those shapes are just filled in with a single colour, where each character has their own colour. These character representations are more 'readable' than the ones in Everybody's Gone to the Rapture. However, in both of these games, you can't see any details of the character's faces. Without facial details, the audiovisual-logs are missing an important part of visual action.

(A still from an audiovisual-log in Tacoma. From this trailer for the game)

Those two games may have used highly abstracted representations of the characters for technical reasons (e.g. for performance). But whatever the reasons were, it seems clear to me that audiovisual-logs showing full details of the characters are far superior. The characters might be shown as semi-transparent, to indicate that you're seeing the details of a past event. Current gaming hardware should be able to handle showing such details, without any problem.

We can call these 'audiovisual-logs with full character-detail'. It's something I expect to see explored more in the future.

 

Audiovisual-logs about ongoing or future situations

Existing audiovisual-logs convey backstory. In The Vanishing of Ethan Carter, Everybody's Gone to the Rapture, and Tacoma, the player enters a situation where a series of events has taken place in the past, before they arrived[4]. The player's goal is to try to understand what happened. 

[4] or not quite so, for one of these games. But to explain would be a spoiler.

Rather than conveying details that happened a while prior to the current moment, audiovisual-logs could be used to convey events that happened only a short while ago, events that are happening now but in a different location, or even future events that haven't happened yet.

So despite what I've said elsewhere in this post, environmental storytelling is not inherently restricted to conveying backstory. What it can't convey are the generally stronger, from a storytelling perspective, details that are happening here and now where the player is.

 

NPC-perspective audiovisual-logs

Audiovisual-logs present some past situation involving some NPCs. Instead of presenting them from the player's point of view (POV), as is normally done, they could be presented through the POV of one or more of those NPCs.

The player could freely switch between the POVs of the different NPCs in the situation. Or perhaps there could be a puzzle element to it, where the player has to do something to unlock each of the different NPCs' POVs.

 

Combining inferred backstory with audiovisual-logs

The Vanishing of Ethan Carter contains puzzles where the player has to discover clues about some past events, and then put those clues in the correct temporal order. This is the player inferring some backstory from the clues. At the end of this process the player is shown a brief audiovisual-log.

There are other possible ways of combining inferred backstory and audiovisual-logs, that are yet to be explored. For example, the following.

The player comes across some clues to some inferred backstory, and once they've seen them all, the game could play an audiovisual-log of the backstory details.

For this to work, the game has to know that the player has noticed those clues. The game could have a 'look' verb, that the player could use on objects. And if the player has 'looked' at each of the clues, the game could play the audiovisual-log.

Or, to make it more challenging for the player, them noticing the clues might require them to apply a separate 'is clue' verb to each of the items they think are clues. That way, a player who just looks at every available objects (as players tend to do in games) would not thereby automatically find all the clues.

In The Vanishing of Ethan Carter, the player is required to put the clues in the correct temporal order. But there could be alternatives to this. Like correctly linking each clue to the person that left it behind.

Interaction within visual action, to enhance the storytelling

My primary interest is in storytelling, and how gameplay, or interaction, could be used to enhance the storytelling.

In a visual medium, visual action is the strongest way to convey storytelling details, so I am interested in how interaction can be used within visual-action storytelling, to enhance that storytelling.

I mentioned earlier that it is difficult to include gameplay within visual-action storytelling. With most of the means of environmental storytelling, the gameplay sits outside of full visual-action storytelling. Currently the only options for interaction in visual-action storytelling are QTEs, choosing from a small menu of actions (e.g. which of the two people do I try to save?), and during-gameplay cutscenes (which are not a form of environmental storytelling, just like cutscenes are not a form of environmental storytelling).

During-gameplay cutscenes (see post about the different kinds of cutscenes) are one way of combining gameplay and cutscenes. So far they have been used in relatively-few games, and their use could be explored further.

I think there are a lot of unexplored options for using interaction within visual-action storytelling, and it is these that I am primarily interested in exploring. But that is a topic for another post.