Dancing with the New Romancer

“It’s too bad most of the BDs we do here are only good for flogging the log.”
Judy at Lizzie’s Bar- Cyberpunk 2077

Sci-fi authors, designers, and artists like to show us blueprints for possible (and impossible) futures. William Gibson described one of the most titillating possibilities by imagining a world in which humans could directly interface with neuro-computational networks. This idea gave birth to other possibilities like the metaverse, the matrix, OASIS, and so on. Cyberspace, another term popularized by Gibson, became the staple of the cyberpunk genre and has appeared in various forms in literature, board games, movies and video games. More recently, the wizards at CD Projekt Red have presented us with their vision of the future which involves doing a little brain dance.

No matter what you may have seen on movies or video games, we do not yet understand how our brain represents information internally. For example, if I tell you to think of a white elephant, then what does that representation look like in your neural net? It could be a highly detailed image similar to a computer graphic. On the other hand, it could be something more abstract, indescribable like a Picasso painting involving not only visual input but also auditory, kinesthetic, sensory, and so forth. Perhaps it’s somewhere in between, a kind of Dalian constellation of dream-forms held together by the persistence of memory. We just don’t know.

In the Cyberpunk 2077 video game, there is a technology called a BrainDance (BD) that allows a user to relive the recorded experiences of other people. Brain implants are required to record the original experience, which can then be transferred into a file (the BrainDance) and replayed with the help of head-mounted brain devices, allowing the operator to relive the experiences of the other person, including sounds, sights, feelings, sensations, and even emotions. They are a form of entertainment that is mainly used for sex and violence. In the context of the game, BDs are used to systematically analyze a scene in order to gain clues that advance the plot. BDs could be considered their own mini-game, serving a similar function that Gwent served in the Witcher 3 game, also created by CD Projekt Red.

In the absence of real world scientific data about how our brain represents information, the game designers had to imagine what BD technology would look like, keeping in mind the limits of current human-computer interaction. It could very well be that the mental constructs we actually develop are virtually incomprehensible if you try to recreate them in a computer game. Fleeting sense impressions like ghost ships in the cognitive waves. Nevertheless, creating a model of these mental abstractions in a video game can lead us to think of novel ways of exploring and navigating our consciousness in real life.

I recorded a BD the other day, and I thought it would be a good idea to deconstruct the clip and dig deeper into this hypothetical technology. If you play this game, then you likely know everything there’s to know about BDs and probably more. But I know that some people don’t play this game, so they might be interested in learning more about the ideas and concepts underlying its cyberpunk narrative. Also keep in mind that this is not a tutorial but just a discussion of the concept and its implementation.

The embedded video clip shows the protagonist, V, accessing a BD from security personnel. It contains a scene from the perspective of a guard, who is about to meet a violent end. I inserted headings throughout outlining the different components of the BD interface.

In the clip, V appears to be talking to someone. That’s just Keanu Reeve’s engram, which I discussed in a previous post. This little cameo by the actor also brings furthers questions about the brain technology underlying these neural net dreams. How is it all actually wired up within the nervous system? But let’s ignore Keanu and focus on the actual BD, as envisioned by the designers at CD Projekt Red.

In normal playback, the BD scene looks just like any ordinary in-game scene. There’s nothing surreal or “psychological” about it. Therefore, in this design scheme, the contents of someone's recorded consciousness look just like a faithful copy of the real world- clean, crisp, and rich in detail. As seen in the video, you can “rewind” the scenes the way you would an old style video tape. They even included some of the tape jitter one sees in old videos. It’s rather clever of the designers to use this anachronistic trick to denote the passage of time in a futuristic game. It pays homage to the era that gave birth to the Cyberpunk genre. In reality, it is not clear how we recall memories, but it’s unlikely that we “rewind” through them like an old tape. Our attentional spotlight somehow scans the cognitive darkness to reveal the ephemeral forms of memory. These neural abstractions are likely more phantasmagoric than those envisioned by the game designers. We do have to remember that they had limits imposed by the interactive technology being used to tell the tale (e.g., PCs, consoles, controllers, keyboards, and screens). So using a realistic approach would've been more difficult and time-consuming, particularly in an industry notorious for it's crunched deadlines.

The surreal, or mental, bits show up when you start interacting with the BD, and its various layers in the timeline editor. The game allows the player to search for narrative clues in the BD. When you activate the editor, you’ll be able to interact with three layers of the scene: visual. thermal, and auditory. Along the top of the BD interface, you see three sets of bars, each denoting a layer for accessing visual (yellow), thermal (blue), auditory (green) information. When a segment of any of the three bars becomes brighter, then somewhere along the timeline (as highlighted along the top bar), there is a clue. You have to analyze the scene (usually by pixel-hunting) and find the clues that will reveal further information hidden in the scene. Once you have found (or think you have) all the clues, then you can end the BD.

When you access each layer in the editor, the graphical environment changes its visual design. Instead of a sharp and vivid scene, you see a dark scene constructed of points of light that mainly render the objects of interest and facilitates the search. In the video, the first layer that I accessed was the thermal layer. Any clue with a strong heat signature will show up as yellow and red. Imagine what it would take to be able to record someone’s cognitive experiences and extract heat signatures from artifacts in the recording. The game designers took it a step further. After analyzing a heat signature from a device like a security camera, the player can access the recorded footage from said camera. This means that at the time that a person is recording a BD, he/she is also extracting data from surrounding devices and networks. We’re talking some really far out (and far-fetched?) technology here.

In the auditory layer, the player can analyze sounds like voices or ambient noises. The important sounds are denoted by the green concentric rings, which I imagine is the representation of a sound wave. Analyzing the rings reveals audio clues important to narrative development. In this scene, the clue is a conversation between characters.

The visual layer shows information using a similar pointillist design as the thermal and auditory layers. Sufficient detail is shown for the player to get a sense of the environment and facilitate the location and analysis of clues.

In editor mode, the scene has a more psychological feel to it. It’s somewhat haunting and strange. When you first enter it, an eerie feeling comes over you. There is an initial sense of disorientation and spookiness. It’s like watching someone’s raw consciousness. I wish they had given this weird vibe from the get-go. Perhaps they could’ve warped the geometry of the scene like in the movie Inception. In any case, the editor gives one a sense that one is interacting with a neural construct and not just a normal video. Regardless of its feasibility, the end result is a game scene with a hallucinatory quality to it, and for the first time we get the impression that we are experiencing a

graphic representation of data abstracted from banks of every computer in the human system. Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding...
William Gibson, Neuromancer

As mentioned, the BD is like its own mini-game, the way Gwent was in the Witcher game. Unfortunately, the BD is not as engaging and fun as Gwent was. It’s a shame because there’s a wealth of research on how the brain processes information at the sensory, perceptual, and cognitive levels. Yet, it seems to me that few games are taking advantage of this publicly available information. Imagine playing a triple-A game that actually tests your mental skills and abilities in a systematic manner, records the results, and plays them back to you for further refinement. That’s one of the things that makes me excited about video games. They’re not games per se, but systems capable of helping you examine the contents of your consciousness by making that famous Apollonian aphorism a reality- know thyself.

I know that Elon Musk is working hard on Neuralink, but in my very humble opinion, the chances of BD technology being ready by 2077 are fairly low, but you never know. There’s just a lot that we don’t know about how our own consciousness work, and for some reason we seem reluctant to find out. Psychology in general has that spooky vibe about it, so I understand why people shy away from the topic. We just don’t like learning about our own plumbing. This is why sci-fi video games are important. They provide us with a playful and interactive blueprint of the future that prepares us for what is coming down the evolutionary pipe. Mutation shock is lessened when the species is prepared. Will we be able to share memories and experiences in the future? I’m absolutely sure of it. Will it look like a Cyberpunk 2077 BrainDance? Not likely. Whatever form it takes, it will be delightfully bizarre, and that is perhaps the scariest part of these unmapped territories. How ready are we to tango with the one behind the mirror? Will we use these technologies to understand ourselves at a meta level? Or are we forever destined to use neural tech as just as another way of facilitating the flogging of the proverbial log?