Interactive storytelling: Environment and storytelling

In a video game, the environment includes the locations the player sees as they traverse the game world. The environment includes the objects they can interact with, such as objects they can pick up and look at, doors they can open, lights they can switch on, etc.

We'll look at the ways a game's environment can contribute to the storytelling in interactive storytelling like video games. Then we'll look at how effectively each of these ways can contribute to storytelling.

 

The setting

The game environment provides the setting for the story. A story about an ad executive living in New York City obviously is set in New York City. A story may lean heavily into its setting, and concern the nature of that setting -- like one that concerns the culture in New York City. Or the story's setting can be more of just a backdrop, where the same story could potentially be set in a number of different places, without changing much about it.

 

The stage necessary for story events

The game's environment may contain details necessary for enabling certain story events. A dense town or city, with suitably-small gaps between the rooftops enables a story event in which the protagonist is chased by several bad guys over rooftops.

 

Atmosphere and world building

An environment of moss and plant-covered ruins could contribute to a post-apocalyptic atmosphere. Posters plastered around a city, instructing the populace on how they should behave, could contribute to the world-building and atmosphere of story set in a fascist country. Dim lighting, with a yellowish hue, along with strange sounds, could give an alleyway an eerie atmosphere.


Characterisation

The environment can contribute to characterisation. If a character's house is neat and tidy (or very dirty, and messy) that will convey something about their personality. As could the paintings we find in their house, or the entries in the diary hidden under their pillow.


The main plot

The environment can contribute to the main plot. The space under a bed may contain a piece of evidence that conclusively shows who's guilty of a murder. A character finding it will be a major plot point.

 

Backstory

The environment can contribute to backstory. Backstory concerns past events, prior to the events the player is currently experiencing in the main story-thread. Those events could have occurred before the start of the main story thread. They could have happened a long time ago. Such distant backstory includes 'lore', that concerns historical details of the setting. Lore may be found in tomes that the player finds, or some runes written on some ruins.

We'll also take 'backstory' to include details that may have only recently happened. For example, halfway through the story the player might receive details about something that had happened a hour prior -- that is, an event that occurred well after the point in time where the story began. We'll call this backstory, too.

Environmental details that contribute to backstory include artefacts like diary entries, letters, memos, and books.

Audio-logs are a means to fill the player in on backstory that's used in video games. Audio-logs are sound recordings that the player can listen to. They may be physical objects, like a tape recorder, that the player can find and play. Such might be a sound recording of a diary entry or voice note. Or an audio-log recording might automatically start playing when it is 'triggered' by the player's actions -- like if the player enters a particular room, or perhaps opens a particular person's locker.

Audio-logs were popularised in the first-person shooter BioShock (2007), which used them as the main means to tell its story. Dear Esther (2012) and Gone Home (2013) are games entirely focused on exploring an environment and finding audio-logs, in order to piece together their stories. Those latter two games are key examples of the genre that came to be known as "walking simulators".

There are also what I've termed audiovisual-logs, used in games like The Vanishing of Ethan Carter (2014), where they only play a small role, Everybody's Gone to the Rapture (2015), and Tacoma (2017). Audiovisual-logs allow the player to see an audiovisual recreation of some past event, in the location where that event occurred. The player can walk around and through this recreation while it is playing.

 

(an audiovisual-log from Everybody's Gone to the Rapture)

(an audiovisual log from Tacoma)

The following links are to the original videos those gifs were created from. Everybody's Gone to the Rapture, and Tacoma.

Audiovisual-logs are the core means of storytelling in Everybody's Gone to the Rapture and Tacoma. In The Vanishing of Ethan Carter, they only play a relatively-small role[1].

[1] In that game, there are certain puzzles, each of which concerns finding out how a particular character died. At the completion of each one of these puzzles, the player is shown a cut-scene showing the full details of how the character died, and then after that the player can follow a floating light. When they get to the light's resting place, they can see a short audiovisual-log of what came next, after the overall happening they've just seen the cutscene of.

For more details on audiovisual-logs, see the audio-logs link, above.

There's what I've termed "frozen-moment-logs". The only game I know of that uses these is The Vanishing of Ethan Carter (2014). The player will come across places where they can unlock a frozen-moment-log, consisting of a static 3D image of some past occurrence (backstory) there, that the player can walk around and view from different angles. In that game, these logs appear as part of larger puzzles.
 

(a frozen-moment-log from The Vanishing of Ethan Carter)

 

Another technique, where the environment contributes to backstory, is what I'm calling "Inferred Backstory". This is where environmental details enable the player to infer some past events.

In a post-apocalyptic game, the player may come across a dilapidated house, and in one of its bedrooms find two mummified corpses in the bed, frozen in an embrace. On the bedside table may sit a framed photo of a happy couple, along with an empty bottle of sleeping-pills.

From these details, the player may infer that the couple was once happy, and loved each other, but ended up finding their circumstances untenable. They may imagine the couple coming to this realisation, taking the sleeping pills, and tearfully embracing each other in the bed as they awaited their fate.

Inferred backstory is often like a little puzzle, where there are clues and the player infers the backstory from them. Usually it's a very simple puzzle.

'Inferred backstory', as we are using the term, concerns any cases of where the player infers prior details from present-moment details. That can include when a player immediately and effortlessly infers some prior details, including details that recently occurred. E.g. they're in a forest and come across some fresh large-animal droppings. They'll immediately infer there was recently some kind of large animal in this place. Some other examples where the player will make immediate and effortless inference: if the player came across the charred remains of a fire, or doors that been broken open.

And of course, by inferring details that have recently happened (e.g. a large animal being here) we may also infer present-moment details -- e.g. that the large animal may be nearby us right now. 

Carson[2] calls these cases of inferred backstory "cause and effect" vignettes, and the above description of them is based on his article.

[2] "Environmental Storytelling: Creating Immersive 3D Worlds Using Lessons Learned from the Theme Park Industry", by Don Carson, March 2000

To effectively communicate story-relevant details

This section draws a lot from Carson.
 
The environment should be designed to effectively communicate important details to the player. Details like where the player is, what sort of place it is, and where they should go next. Usually, we want the player to be able immediately determine such details.

These kinds of concerns don't, of course, only apply to the environments in games. They also apply in movies, TV shows, theatre, and theme park rides. Think set design.

Here are some details that we want the environment to communicate to the player. Which details in a location are the most important story-relevant ones. The atmosphere of a location, and the kind of place it is. What objects and features are important for the player to be aware of. And where the player should go next.

The following can aid in that communication. We can draw attention to important details by how we arrange the objects and features in that location, and by how those objects and features are lit. And by not including too much detail in the location, especially detail that's of little relevance to the story. We don't want to confuse or overwhelm the player.

Contrast can be used to heighten qualities. For example, if we make the player crawl through a narrow passageway to reach a cave chamber, that can heighten the feeling of how large that chamber is. Or, if we want the player to feel that the temple in the forest is a pristine place, we can make them experience a disordered space (like thick jungle) before they find it.

The environment can be designed to guide players as to where they need to go next. For example, in a dark area, having a well-lit large object in one corner, that the player will want to investigate, where this will take them to the exit from this location, to the next place they need to go to.

Many of the ways that the environment can contribute to storytelling are showing rather than telling. Rather than explicitly describing the atmosphere and world building of an environment (perhaps by a character commenting on them), environmental details can show them. With inferred backstory, the player draws their own conclusion about what happened.

And, as Carson points out, the player being able to discover details like artefacts (letters, memos, audio-logs), and inferred backstory, themselves, can be a more enjoyable experience for them than them simply being told those details. (That discovery is part of the gameplay, so the enjoyment comes from the gameplay. It doesn't come from the storytelling).

Summary of how the environment may contribute to storytelling

To summarise, the environment may contribute to the following elements of a story:

• The setting (e.g. NYC)
• The stage enabling certain events (e.g. rooftops for a chase sequence)
• Atmosphere and world building (e.g. moss-covered ruins, strange sounds)
• Characterisation (e.g. a character's messy room)
• The main plot (e.g. evidence of who the murderer is).
• Backstory (e.g. diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed).

And there are techniques that can be used to effectively communicate such elements of the story. For example, to highlight the size of large cave chamber, make the player crawl though a small space to get to it.

 

Environmental storytelling

The reader may have noticed that we started this post with the heading "Environment and Storytelling" not "Environmental Storytelling". For these are two different things. "Environment and storytelling" refers to all of the ways that the environment contributes to storytelling. Whereas "Environmental storytelling" refers only to a subset of them.

"Environmental storytelling" is a commonly used term to describe storytelling in games. It is used to refer to only cases where the environment contributes to backstory.

For some people, "environmental storytelling" refers to all of the ways that environmental details can contribute to backstory: diary entries, letters, memos, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory (like the corpses in the bed).

For other people, "environmental storytelling" has an even narrower meaning. For them, it only refers to cases of inferred backstory, and does not refer to cases like diary entries, letters, memos, audio-logs, audiovisual-logs, and frozen-moment-logs.

Both definitions of "environmental storytelling" involve environmental details that are leftover from, or reflections of, the past (in the broad sense of "prior to the current moment", not just things in the more distant past). Obviously that's the case for Inferred Backstory like the corpses in the bed. It's also the case for things like diary entries, letters, memos, audio-logs, audiovisual-logs, and frozen-moment-logs.

In this post, we'll use "environmental storytelling" in its broader sense, which includes all the ways that environmental details can contribute to backstory.

Environment and storytelling in movies and TV

Movies and TV use the story's environment (sets, and on-location shots) for storytelling purposes, and of course have done so since well before video games came onto the scene. Movies/TV and games mostly use their environments in similar ways for storytelling, except for some of the ways backstory is used.

(As well as movies/TV, theatre, theme parks, and theme park rides, all use the environment to contribute to storytelling. But this post will focus only on video games, movies, and TV).

Movies/TV can use environmental details to contribute to:

• The setting (e.g. NYC)
• The stage enabling certain events (e.g. rooftops for a chase sequence)
• Atmosphere and world building (e.g. moss-covered ruins, strange sounds)
• Characterisation (e.g. a character's messy room)
• The main plot (e.g. evidence of who the murderer is).
• Backstory ('environmental storytelling', e.g. diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed).

One difference, regarding the techniques that can be used to effectively communicate such elements of the story, is the following. In games, the environment can be designed to guide players as to where they need to go next, which obviously doesn't apply to movies/TV/cutscenes. Whereas in movies/TV/cutscenes, the cinematography, lighting and set-design can guide where the viewer looks during scenes.

In a moment we'll get into some differences in how backstory (environmental storytelling) is used in movies/TV compared to games.

Why is environmental storytelling more common in games?

The environmental storytelling techniques (contributing to backstory) are either less common or not found at all in movies and TV. This includes things like diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed.

There's a clear difference between games and movies/TV in this respect, as games often heavily rely on such environmental storytelling. In games, it's often the main form of storytelling that's used. In movies and TV, it tends to be a supplemental form of storytelling.

What is the reason for this difference?

 

Cost and ease

Movies and TV are focused on "cinematics" -- visual portrayals of actors and environments. Whereas game development companies have a primary focus on gameplay. In most cases, it requires additional resources for the game developer to be able to include "cinematics" in their games. And smaller game developers may not have the skills and/or budget for this.

Compared to cinematics, it's cheaper and quicker to add artefacts like diary entries and letters to a game. Audio-logs require hiring voice actor(s), but a game might have a total of less than 1 hour of audio-log audio, which can be recorded quickly and thus doesn't require the expense of hiring a voice-actor for a long period of time.

On the other hand, high-quality animated cutscenes take more time to develop, and require animator(s) and voice actor(s). FMV (full motion video) or motion-capture for cutscenes requires (real or virtual) sets, and hiring actors. Motion capture requires specialised equipment (either purchased or hired) and the skills to turn it's output into animation. I imagine that the time and money required for cutscenes is similar to the time and money required for audiovisual-logs.

 

Visual mediums excel at visual-action storytelling

Here's a reason that environmental storytelling is used less in movies/TV. Movies and TV are primarily visual, and excel at visual-action storytelling. This is the visual depiction of action (what I referred to as 'cinematics', above). By 'action', I don't mean just things like fights and shootouts, as you'd find in action movies. I mean 'action' in a general sense, of the visual details that can be captured by a video camera. This could include characters simply talking to each other, or a tense scene where two characters are sitting in the same room, each silently trying to ignore the other.

In constructing visual action, all the tools of acting (performance), cinematography, editing, and so on, are brought to bear. Visual-action storytelling is a strong form of storytelling.

If the movie or TV show contains backstory, it's usually presented through visual action -- that is, through a flashback. Environmental storytelling also conveys backstory, but it mostly does /not/ do so through visual action. In a moment we'll examine this, and see why environmental storytelling tends to thus be a weaker form of storytelling[3].

[3] Before leaving this topic, we can note that one kind of use of inferred backstory in movies/TV is where the 'clues' are amongst the background details of scene(s), that might only briefly be in shot. Most people watching the movie/show wouldn't notice them, and they're there as interesting details or "easter eggs" for repeat or careful viewers. And/or as details designed for other viewers to subconsciously take in.

 

Games excel at visual-action gameplay, but not visual-action storytelling

Like movies and TV, graphical games are also a visual medium -- with the addition of gameplay. They excel at incorporating gameplay into visual-action. Consider first-person shooters, platform games, racing games, etc. However, games do not excel at incorporating gameplay into visual-action storytelling.

That's why, if a game is to include visual-action storytelling, that's done in a cutscene (effectively a little movie) that is separate to the main gameplay. Cutscenes are usually non-interactive, though they can include simple forms of interactivity like Quick-time Events (QTEs) and "Choose Your Own Adventure"-style choices. We don't have a way of integrating the player having control over a character and movie-like visual action.

In visual-action storytelling, all the tools of acting (performance), cinematography, editing, and so on, are brought to bear. But, during gameplay, when the player has moment-to-moment control over a character (like the character's movements), it's not possible to strongly exploit those tools of visual-action storytelling.

Here we'll turn our attention to cutscenes. These aren't a form of environmental storytelling, but looking at them will help convey the point that games are poor at integrating interaction/gameplay with strong visual-action storytelling.

In this earlier post, I looked at the types of cutscenes in games, and the ways interactivity and cutscenes can be mixed together.

The standard non-interactive cutscenes are strong visual-action storytelling, but the involve no interactions at all.

In QTE-and-Choice cutscenes, simple player inputs are incorporated into the strong visual-action storytelling. These are interactions like QTEs (Quick-time Events), and choices where the player can choose from a small menu of options, such as dialogue choices or choices about which course of action to take (save Billy or save Jenny, from the oncoming horde of zombies).

QTE-and-Choice cutscenes are strong forms of visual-action storytelling, however as far as interactivity goes, they contain weak forms of interactions/gameplay. The player has very limited control over a character.

During-gameplay cutscenes are how most of the cutscenes are handled in games like Half-Life 2 and the Dishonored games. The player still has some degree of control over their character, while some scripted events occur around them. The player may be able to freely turn their head to look around, or that plus the freedom to move around (within the constraints of their environment, like brick walls etc).

As that earlier post argued, During-Gameplay cutscenes have stronger forms of gameplay but weaker forms of storytelling than non-interactive cutscenes.

So none of the kinds of cutscenes involve strong visual-action storytelling along with strong interaction/gameplay.

Returning to environmental storytelling, audiovisual-logs, like in Everybody's Gone to the Rapture, present visual action, but since it is visual action that the player can walk around and through, the tools of cinematography and editing can't be brought to bear on it. Like with during-gameplay cutscenes, the player is reduced to a spectator of the events.

In summary, even though visual-action storytelling is the strongest form of storytelling in a visual medium, games are not very suited to it. To use visual-action storytelling, games have to either include non-interactive or Q&C cutscenes, which clash somewhat with the interactive nature of games, or D-G cutscenes which have more interaction but weaker storytelling.

 

Environmental storytelling is more compatible with gameplay

That it's difficult to insert gameplay into visual-action storytelling is a reason why environmental storytelling is so often used in games.

Environmental storytelling can form part of the gameplay.

In it, the player explores the environment, and finds artefacts (like diary entries, letters, memos, audio-logs, and audiovisual-logs). A player who is not looking carefully might miss some of the artefacts. That exploration and finding is part of the gameplay.

Audio-logs can be listened to while the player is still engaging in gameplay, where they're moving around and continuing to explore.

When an audiovisual-log is playing, the player can move around, and within, the recreated visuals. The player may move around to find a good view of all the action, to follow a particular character around, or to be closer to one particular conversation (if there are multiple happening at the same time). The audiovisual-logs in Tacoma allow the player to scrub back and forth in the log, to find pertinent details.

Environmental storytelling (e.g. diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed) can present a kind of puzzle to the player. Something unknown, in the past, has happened, and environmental storytelling provides some clues to its nature. It's like each bit of environmental storytelling is a puzzle piece and the player needs to figure out how they fit together, to see the overall picture of what happened. This is a very common pattern in games that heavily use environmental storytelling. For example, in BioShock, Gone Home, Everybody's Gone to the Rapture, and Tacoma. (Note that this can be done in other mediums, like movies/TV, and novels. But games more often make use of such).

So all these forms of environmental storytelling are more compatible with gameplay. Environmental storytelling brings storytelling into the main gameplay parts of the game, rather than having it be fairly distinct from gameplay, as with cutscenes.

 

Environmental storytelling is generally a weaker form of storytelling

While environmental storytelling allows storytelling to be better integrated with the gameplay, it unfortunately involves a weaker form of storytelling.

We've stated that visual-action storytelling is the strongest form of storytelling in a visual medium. This, and some subsequent sections, look at why environmental storytelling is weaker than visual-action storytelling.

To clarify, I'm saying they're generally weaker forms of storytelling, not that they're bad. They can be quite effective. However, when they are heavily relied upon, like they often are in games, that will tend to weaken the overall storytelling.

Environmental storytelling conveys backstory. I suggest that storytelling that focuses on the main story thread is, generally speaking, stronger storytelling than that focusing on backstory. This is debatable, but I think it's the reason that backstory tends to be used sparingly in most movies and TV shows.

And, if backstory is used, it's more strongly presented with flashbacks, which present the details through visual action. Environmental storytelling mostly does not convey details through visual-action. So these forms of environmental storytelling are generally weaker forms of storytelling.

Audiovisual-logs, such as used in Everybody's Gone to the Rapture, and Tacoma, are the only form of environmental storytelling that's via visual action (during-gameplay cutscenes also use visual action, but they are, like normal cutscenes, not a form of environmental storytelling). Though, as mentioned before, audiovisual-logs can't make use of cinematography or editing, as the player is still in control of a character. The camera needs to be suited to moving a character around. Editing involves cutting out parts of the action, showing only the details before and after it, and that doesn't mesh well with gameplay. Still, audiovisual-logs have the potential to be fairly strong forms of storytelling -- similar to flashbacks.

All the other forms of environmental storytelling are not presented through visual action. They may be textual artefacts like diary entries and letters. And auditory ones like audio-logs.

What about inferred backstory, like the corpses in the bed? Here the backstory is told using visual details (from which the player infers past events). But those are present-moment visual details; the backstory being conveyed is not shown visually. It doesn't, for example, show the visual action of the couple taking the sleeping pills and getting into the bed.

The player coming to their own realisation of what happened, through the environmental storytelling, is something that a number of players enjoy. This is part of the appeal of environmental storytelling. However, I don't consider this to add a lot to the strength of the storytelling.

 

We can draw a distinction between artefacts that provide narrative details, and those that don't, but which convey narrative-relevant information. We can call these narrative and non-narrative artefacts.

Artefacts like diary entries and letters can be either narrative or non-narrative. They can convey narrative backstory, like a diary entry recounting an event that occurred. A non-narrative example is a diary entry that said "Bought new clothes at shops" and then went on to list the items of clothing. This is just some information, though from it the player may infer some narrative-relevant details (that the person who wrote it cared a lot about their neighbour, who they bought a number of items for).

(To turn to textual mediums for a moment, epistolary novels tell a story through letters sent between characters. "Epistolary novels" is also often used in a broader sense, covering stories told through any kinds of artefacts, such as diary entries, newspaper clippings, or other kinds of documents. Some well known examples are "Carrie", by Stephen King (1974), "Posession", by A. S. Byatt (1990), and the Adrian Mole series, by Sue Townsend (1982-2009). Such novels show how something akin to environmental storytelling can be used quite successfully for storytelling. I believe that using artefacts in this way is much more suited to textual mediums, and much less so in visual mediums like visual video games. This is because they are textual artifacts, which means they fit in with the textual nature of novels. Whereas, showing textual artefacts on screen for the viewers to read, or having them read out by a character, is not as suited to the visual-action character of visual mediums.)

I contend that such non-narrative information is, generally speaking, a weaker way of conveying details that are part of the narrative.

Audio-logs may simply be recorded versions of artefacts like diary entries, which may or may not be focused on narrative details. Narrative audio-logs might, for example, contain a recording of when some bad guys stormed a character's office, and took them hostage. That is, an audio-log may contain a recording of an event that happened. Audiovisual-logs will usually convey narrative details.

Inferred backstory (like the corpses in the bed) also conveys narrative details.

Here's a summary of the ways the environment may directly contribute to the narrative and those that do so indirectly.

These environmental storytelling techniques directly contribute to the narrative. Those that contribute to:

• The main plot (e.g. evidence of who the murderer is).
• backstory (with narrative artefacts)
  (e.g. certain of: diary entries, audio-logs, audiovisual-logs, frozen-moment-logs, and inferred backstory like the corpses in the bed).

Environmental storytelling techniques that only indirectly contribute to the narrative/plot:

• The setting (e.g. NYC)
• The stage enabling certain events (e.g. rooftops for a chase sequence)
• Atmosphere and world building (e.g. moss-covered ruins, strange sounds)
• Characterisation (e.g. a character's messy room)
• non-narrative artefacts conveying backstory (e.g. diary entries, audio-logs).

And where environmental details are used to effectively communicate story-relevant details (e.g. to highlight size of large cave chamber, make player crawl though a small space to get to it).

Audio-logs and inferred backstory that convey narrative details, don't do so through visual action, so they are generally weaker forms of (back)storytelling than flashbacks, which do.

To help reinforce these points, we can note that movies and TV could employ an equivalent of audio-logs. There could be scenes where a character is listening to a sound recording. Or where the audience hears a character reading out a diary entry or letter. The visuals might show the character driving in their car as they listen to the audio recording, or walking around their house while reading the diary entry or letter.

Because the focus of such scenes would be on the audio, the visuals would essentially serve as a background to the audio. The visuals couldn't convey any substantial narrative details, as that would distract the viewer from the audio. So thus it'd be weaker storytelling, because it's not focused on visual action.

That such scenes are, as far as I'm aware, rare in movies and TV is, I suggest, because they're a weaker form of storytelling.

 

Pacing

Another reason flashbacks are a stronger means of conveying backstory is that their visual-storytelling benefits pacing. Pacing is an important part of storytelling. A story may concern some events that take place over a week, but they -- as represented in a movie or TV show -- may do so through only a couple of hours of visual action. They condense the details. They filter out the narratively-irrelevant details, to leave a more narratively-concentrated end-product. If the narratively-interesting details are padded out with a lot of irrelevant details, it will slow down the pacing, and dilute the narrative.

A heavy focus on environmental storytelling, as is often found in games, negatively affects the pacing. It slows the pacing.

Imagine if, during a movie or TV show, there were several occasions where the main character read a full-page diary entry, letter, or memo. Where, each time, the shot of them reading it lasted long enough for the character to read the full text. (Their reading of it might be conveyed with a voice-over representing the character's inner voice, as they read the page). That would, I think, make the pacing feel quite strange. It'd be jarring, to go from the normal speed of the pacing in a movie or TV show, to these really slowed-down segments.

(And this is on top of the fact that gameplay itself, also slows the pacing of the storytelling. Because there'll be long segments of gameplay-focused action in between each narrative-focused segment, and the gameplay is usually not conveying much vis a vis the game's narrative).

 

Limitations to narrative complexity of audio-logs and inferred backstory

I mentioned earlier that audio-logs and inferred backstory can convey narrative details. They are, however, quite limited in their ability to do so.

An individual audio-log or inferred backstory instance can only present fairly short and simple narrative details. And there are fairly strong limits on how substantial/complex the overall narrative details from the totality of the audio-logs/inferred-backstory.

With inferred backstory (like the corpses in the bed) there's visual details from which the player can infer past events. But it's difficult to convey a lot of detail in this fashion. The player has to infer -- figure out -- the events from the clues. It would be too complex for the player to infer more than a simple set of details. For one thing, it would be very challenging to indicate the sequence of the events.

Audio-logs that convey narrative events (like a recording of a kidnapping) need to be relatively brief. Audio-logs are designed such as to not get in the way of gameplay, while the player is listening to them. While listening, the player can still continue to explore around, and possibly even take on some enemies. While listening, they'll be watching visuals (of their environment) that are likely pretty unrelated to the content of the audio-log. Which means they face distractions while listening to the audio-log. They won't usually be paying 100% attention to it. So the player would have trouble being able to properly take in longer audio-logs. Also, if audio-logs were lengthy, then while the player is still listening to one, they might get into a situation (like a major fight with multiple enemies) where 1) they can't focus at all on the audio-log and 2) the audio-log might distract them from the gameplay. (Though this latter point might be addressable by a means to pause audio-logs).

All of the narrative audio/audiovisual -logs, and all of the inferred backstory, in a game, could together contribute to the overall narrative. However, there might be an average of, say, 10-30 minutes of playtime between each audio-log or inferred backstory that the player comes across. During which time the player is undertaking gameplay. This places fairly heavy demands on the player's memory, and the game designer can't expect that the player will be able to recall a lot of the specifics presented in earlier audio-logs and inferred backstory instances. Therefore, individual audio/audiovisual -logs and inferred backstory have to be designed to be somewhat stand-alone. They can't be narratively connected together in intricate ways.

So there are limits on how substantial/complex the overall narrative, conveyed through multiple audio/audiovisual -logs and/or inferred backstory instances, can be. One other reason for this, that applies to a number of games, is that different players may come across the audio/audiovisual -logs and/or inferred-backstories in somewhat different orders.

In contrast, visual action can convey a lengthy sequence of events (like, a whole movie's worth).

 

Contrived nature of audio-logs

Audio-logs are also somewhat contrived. Why were these recordings made in the first place? It might make sense if they're like diary entries or voice notes that a character made. But it doesn't if they're a recording of a narratively-significant event that happened. Who thinks to switch on a recorder just before a significant event? And why are the audio-logs found in various different places in the environment? Often it doesn't make sense. So this contrivance is another reason why you wouldn't have characters in movies/TV finding such audio-logs.

 

Audio/audiovisual -logs are only compatible with certain story settings

And both audio-logs and audiovisual-logs are also only compatible with certain kinds of stories. Neither of them could appear in a realistic story set in the 1700s. Audiovisual-logs, further, need to be in a story that contains elements that are magical, supernatural, alien, or high-tech.

 

In summary, environmental storytelling consists of generally-weaker forms of storytelling, and the reason they're often used in games is that they fit better with gameplay than do other means of storytelling.

 

Where next?

Given the storytelling limitations of the environmental storytelling we've looked at, are there other ways environmental storytelling could be used, that might be better for storytelling?


Further exploring the use of audiovisual-logs

Audiovisual-logs have potential because they can present narrative details through visual-action. The player sees the characters and events being portrayed. Yet, I'm only aware of three games that use them. One (The Vanishing of Ethan Carter -- see below) barely uses them, and the other two (Everybody's Gone to the Rapture, and Tacoma) don't lean much into the visual action.

I expect further exploration of the use of audiovisual-logs in future works. Especially since it should become cheaper and easier for game developers to create them. Motion capture and animating the captured data, is only going to get cheaper and easier over time. AI will likely play a role in that. AI will likely make it quicker and easier to record and process the motion capture data, and to generate animated models from it.

 

Audiovisual-logs with full character-detail

In The Vanishing of Ethan Carter (2014), the audiovisual-logs have a quite minor role. Each one is quite short -- probably 5-10 seconds long -- and there's only a few of them in the game (probably 5-10). In these you see the representations of the characters like you would in a cutscene, except you are there in the scene and can move around and look around while you're watching it play out.

In Everybody's Gone to the Rapture (2015) and Tacoma (2017), the audiovisual-logs are the main source of the storytelling, but at the same time, those games don't lean much into the visual action in the audiovisual-logs, because they show highly abstracted representations of the characters.

In Everybody's Gone to the Rapture (2015), the characters are represented by glowing, dancing points of light. It makes it difficult to even get a clear view of the characters and their movements.

(A still from an audiovisual-log in Everybody's Gone to the Rapture. From this one-minute video. It's from early in the game.)

In Tacoma (2017), you can see the shapes of each of the characters, but those shapes are just filled in with a single colour, where each character has their own colour. These character representations are more 'readable' than the ones in Everybody's Gone to the Rapture. However, in both of these games, you can't see any details of the character's faces. Without facial details, the audiovisual-logs are missing an important part of visual action.

(A still from an audiovisual-log in Tacoma. From this trailer for the game)

Those two games may have used highly abstracted representations of the characters for technical reasons (e.g. for performance). But whatever the reasons were, it seems clear to me that audiovisual-logs showing full details of the characters are far superior. The characters might be shown as semi-transparent, to indicate that you're seeing the details of a past event. Current gaming hardware should be able to handle showing such details, without any problem.

We can call these 'audiovisual-logs with full character-detail'. It's something I expect to see explored more in the future.

 

Audiovisual-logs about ongoing or future situations

Existing audiovisual-logs convey backstory. In The Vanishing of Ethan Carter, Everybody's Gone to the Rapture, and Tacoma, the player enters a situation where a series of events has taken place in the past, before they arrived[4]. The player's goal is to try to understand what happened. 

[4] or not quite so, for one of these games. But to explain would be a spoiler.

Rather than conveying details that happened a while prior to the current moment, audiovisual-logs could be used to convey events that happened only a short while ago, events that are happening now but in a different location, or even future events that haven't happened yet.

So despite what I've said elsewhere in this post, environmental storytelling is not inherently restricted to conveying backstory. What it can't convey are the generally stronger, from a storytelling perspective, details that are happening here and now where the player is.

 

NPC-perspective audiovisual-logs

Audiovisual-logs present some past situation involving some NPCs. Instead of presenting them from the player's point of view (POV), as is normally done, they could be presented through the POV of one or more of those NPCs.

The player could freely switch between the POVs of the different NPCs in the situation. Or perhaps there could be a puzzle element to it, where the player has to do something to unlock each of the different NPCs' POVs.

 

Combining inferred backstory with audiovisual-logs

The Vanishing of Ethan Carter contains puzzles where the player has to discover clues about some past events, and then put those clues in the correct temporal order. This is the player inferring some backstory from the clues. At the end of this process the player is shown a brief audiovisual-log.

There are other possible ways of combining inferred backstory and audiovisual-logs, that are yet to be explored. For example, the following.

The player comes across some clues to some inferred backstory, and once they've seen them all, the game could play an audiovisual-log of the backstory details.

For this to work, the game has to know that the player has noticed those clues. The game could have a 'look' verb, that the player could use on objects. And if the player has 'looked' at each of the clues, the game could play the audiovisual-log.

Or, to make it more challenging for the player, them noticing the clues might require them to apply a separate 'is clue' verb to each of the items they think are clues. That way, a player who just looks at every available objects (as players tend to do in games) would not thereby automatically find all the clues.

In The Vanishing of Ethan Carter, the player is required to put the clues in the correct temporal order. But there could be alternatives to this. Like correctly linking each clue to the person that left it behind.

Interaction within visual action, to enhance the storytelling

My primary interest is in storytelling, and how gameplay, or interaction, could be used to enhance the storytelling.

In a visual medium, visual action is the strongest way to convey storytelling details, so I am interested in how interaction can be used within visual-action storytelling, to enhance that storytelling.

I mentioned earlier that it is difficult to include gameplay within visual-action storytelling. With most of the means of environmental storytelling, the gameplay sits outside of full visual-action storytelling. Currently the only options for interaction in visual-action storytelling are QTEs, choosing from a small menu of actions (e.g. which of the two people do I try to save?), and during-gameplay cutscenes (which are not a form of environmental storytelling, just like cutscenes are not a form of environmental storytelling).

During-gameplay cutscenes (see post about the different kinds of cutscenes) are one way of combining gameplay and cutscenes. So far they have been used in relatively-few games, and their use could be explored further.

I think there are a lot of unexplored options for using interaction within visual-action storytelling, and it is these that I am primarily interested in exploring. But that is a topic for another post.