Into the Uncanny Valley

One Sunday morning I was flipping through the New York Times Magazine, entertaining myself with the advertisements for real estate for the One Percent, when a full-page spread caught my attention. The image offered an aspirational view into an upper-floor condominium in one of the starchitect-designed residences in Lower Manhattan. With a backdrop of impossibly clear floor-to-ceiling glass walls, we were offered tantalizing glimpses of bespoke kitchen cabinetry and pedigreed furniture of leather and chrome.

All was as expected until I noticed the fine print in the corner of the page: This is an actual photograph. Unlike the assorted lawyered-up annotations of contemporary life — professional driver, do not attempt; this page intentionally left blank — here was not so much a disclaimer as a claimer. The image was so perfect and pristine, so devoid of the blemishes of everyday life, that the creators of the advertisement feared it was not credible.

What are we to make of this admission, with its unwittingly Magrittean overtones? Are we troubled to learn that the image is an indicative photograph and not a subjunctive rendering? Or troubled because we cannot tell the difference? It is disconcerting to realize that we can’t quite sort out the relationship of an image to the world; like not understanding whether someone is telling you a true story or a tall tale. We might marvel at the skill of the 17th-century Dutch painter Willem Claesz Heda, whose rendering of glass in Still Life with Oysters, a Silver Tazza, and Glassware looks astonishingly real. But we are never in doubt as to whether it is real. ¹ In her seminal critique of photography, Susan Sontag connects the medium’s perennial struggle with veracity — its manipulation of the truth — to the development of techniques for retouching negatives. As she tartly puts it, in a chapter on the rise of photography in the mid-19th century, “The news that the camera could lie made getting photographed much more popular.” More seriously, she continues:

The consequences of lying have to be more central for photography than they ever can be for painting, because the flat, usually rectangular images which are photographs make a claim to be true that paintings can never make. A fake painting (one whose attribution is false) falsifies the history of art. A fake photograph (one which was been retouched or tampered with, or whose caption is false) falsifies reality. ²

We might argue too that an unreliable photograph threatens the credibility of the medium itself as well as the entire category of indicative image production. Would that Sontag were still with us — Walter Benjamin too — to train her analytical skills upon digital image-making.

But why does the distinction between the real and unreal, the true and fake, matter at all? Why does it matter to architecture? At the risk of radical simplification, it matters because there is a real world and there is a design world. In a design proposal, there is a necessary overlap between the two worlds, and, as with the intersecting sets of a Venn diagram, that overlap is negotiated at every stage of a project, getting ever thicker as the design moves from concept to construction. Certain elements of reality cannot be entirely excluded, of course. An architect and client may decide at the start to ignore a particular fact, such as the presence of a large oak in the middle of a site; but at some point the continuing existence of the oak might need to be weighed against the longing for a lap pool.

Why does the distinction between real and unreal, true and fake, matter at all? Why does it matter to architecture?

Often there is a great deal more at stake. For too long designers and clients have ignored irrefutable and inconvenient environmental facts. An unshaded south-facing glass façade testifies to a gap — the absence of an intersection — between the real world and the design world. (In the design world, the sun rises where the architect wishes.) Today the regulatory apparatus of construction specifies much of the contents of the intersection — the inviolability of emergency egress, for example — but even that has been culturally constructed over time. Before the passage of the Americans with Disabilities Act, countless people occupied a bubble in the real world that rarely intersected with the desires of the design world. Architecture demands an intersection; but not a total eclipse. When reality eclipses the imagination, the result is banality; when the imagination eclipses reality, then we have abandoned architecture for the untethered spheres of science fiction, or gaming, or art. It’s at this point that images become ends in themselves rather than representations of a plausible new reality. ³

“We experience an eerie sensation”

For the creator of any image there is always an implicit question: What relationship with the world do you intend with this work? Since its invention almost two centuries ago, photography has been understood as an indicative form of representation: It draws from the world and aims to match the world. “The picture may distort,” writes Sontag, “but there is always a presumption that something exists, or did exist, which is like what’s in the picture.” ⁴ Today that very presumption is in doubt. Contemporary technologies — digital drawing and modeling, digital collage, and photo-editing software such as Photoshop — are blurring the old boundaries in ways that would have been impossible in the era of mechanical mockups. Photographs look like digital renderings; digital renderings look like photographs. This representational chiasmus leaves a viewer puzzled about the very intentions of an image. We do not know what we are looking at, or how to evaluate it.

An image that requires annotation to establish its relationship to the world — this is an actual photograph — is an image slipping down-slope into what the roboticist Masahiro Mori calls the “uncanny valley.”

Mori introduced the concept of the uncanny valley in a 1970 article in Energy, the in-house magazine of Esso (now Exxon). ⁵ Mori described his research into the ways in which humans respond to robots, and he identified a curious and complicated phenomenon. To a point, he found, anthropomorphic qualities in robots provoked positive responses: as the robots appeared more human-like, people found them more appealing. But at some tipping point of anthropomorphism — say, somewhere between C-3PO and Ava from Ex Machina — the robots become disconcerting. Mori offers as an example the invention of increasingly sophisticated prosthetics.

Recently, owing to great advances in fabrication technology, we cannot distinguish at a glance a prosthetic hand from a real one. Some models simulate wrinkles, veins, fingernails, and even fingerprints. Though similar to a real hand, the prosthetic hand’s color is pinker, as if it had just come out of the bath. One might say that the prosthetic hand has achieved a degree of resemblance to the human form, perhaps on a par with false teeth. However, when we realize the hand, which at first site looked real, is in fact artificial, we experience an eerie sensation. For example, we could be startled during a handshake by its limp boneless grip together with its texture and coldness. When this happens, we lose our sense of affinity, and the hand becomes uncanny. ⁶

The metaphoric “valley” is that point at which human-like robots lose their charm and become repellent.

Mori acknowledged in a 2011 interview with Wired that his theory originated in his own quirky views on robots, toys, and life-like figures. He recalled his uncomfortable reaction as a child to wax figures in exhibition dioramas, and how years later, when he was a professor at the Tokyo Institute of Technology, those memories informed his work with robots.

You know those wax dolls at exhibition halls? Like Columbus discovering America? I never really liked those. When I started working with robots, I remembered those dolls, and I thought, wouldn’t it be creepy if there was a human who didn’t blink? If their eyes just stared and stared at you. ⁷

Mori’s uncanny valley steepens and deepens when there is the possibility of movement. He draws a distinction between industrial robots, which are mere machines with no intended life-like qualities, and toys, the success of which often depends upon anthropomorphism. A machine that moves and then stops is simply a stopped machine; its limited movements mean nothing. But in a life-like robot, movement and stillness are both problematic: either its movements are insufficiently human and therefore arouse unease, or its stillness seems a kind of death. In his Energy essay, Mori uses as an example a “highly sophisticated” robot that was constructed for the 1970 World Exposition in Osaka. To enable it to smile, the robot was designed with “29 pairs of artificial facial muscles in the face (the same number as a human being).” The robot, however, could not smile at the same speed as humans, making it seem not happy but “creepy.” “This shows how,” Mori continues, “because of a variation in movement, something that has come to appear very close to human — like a robot, puppet, or prosthetic hand —could easily tumble down into the uncanny valley.” ⁸

The metaphoric ‘valley’ is that point at which human-like robots lose their charm and become repellent.

For many years Mori’s research remained obscure; only recently, as new technologies in the film and gaming industries began eroding the bounds between live-action and cartoon animation, did his insights begin to attract attention. To animate is to give spirit or life. When bestowed upon something inanimate, the results can be charming. Pixar has made a fortune by animating things, such as the implausibly charming trash compactor that is the hero of WALL-E.  But when what is being animated is already animate, the results can range from awkward to grotesque and even frightening. In Pixar’s Toy Story, the children and adult characters seem de-humanized through animation. In particular their movements — especially gravity-related movement such as running or jumping — seem inauthentic and disturbing; creepy, as Mori might put it. We have no way of assessing the realism of Mr. Potato Head’s gait, but we do know what a running child should look like, and so we are unsettled when we see that the animation is out of phase with real motion. ⁹

“Near enough to the truth to deceive you for a moment”

Because the experience of the uncanny arises in that perceptual space between a thing and a mind, it was of keen interest to Sigmund Freud. In his influential essay on the subject, Freud describes “the uncanny” as the province of “all that arouses dread and creeping horror”; as “that class of the terrifying which leads back to something long known to us, once very familiar.” ¹⁰ He uses the German terms heimlich and unheimlich, first emphasizing their opposite meanings of familiar and unfamiliar; but then, as if entering an infinity of mirrors, he argues that the differences start to collapse as what is known begins to seem strange, and as what is unfamiliar appears to be known. ¹¹ Architectural representations often embody this tension between familiar and unfamiliar. In an effective rendering, the new buildings or landscapes share the same illusionistic space with images of existing buildings or landscapes; adept users of Photoshop can produce drawings in which there is an almost exquisite confusion between real and unreal, heimlich and unheimlich. ¹² Haven’t we seen this place before?

Architectural representations often embody the tension between familiar and unfamiliar, an almost exquisite confusion between real and unreal.

Here it is useful to remember that the primary purpose of making an architectural drawing is not to induce an aesthetic experience; not to create a composition to be evaluated relative to a framed picture-space. In Languages of Art, the philosopher Nelson Goodman provides a succinct list of characteristics that distinguish the aesthetic from the non-aesthetic: “Density, repleteness, and exemplificationality, then, are earmarks of the aesthetic; articulateness, attenuation, and denotationality, earmarks of the nonaesthetic.” ¹³ Members of the viewing public — paying clients or impacted citizens — do not contemplate architectural representations with aesthetic detachment. We look into them — through them, even — to divine possible futures. Yet often the experience feels aesthetic. What is problematic is our confusion as to the object of that experience: is it the building or the image of building? “An uncanny effect,” writes Freud, “is often and easily produced by effacing the distinction between imagination and reality, such as when something that we have hitherto regarded as imaginary appears before us in reality, or when a symbol takes over the full functions and significance of the thing it symbolizes.” ¹⁴

What are some specific tells of uncanniness in architectural drawings?

One is the deployment of perspective. In That Hideous Strength, the final book of his dystopian Space Trilogy, C.S. Lewis describes hell as a place where perspective space breaks down. Toward the end of the novel, Lewis recounts the chilling experience of his protagonist, Mark, in a kind of futuristic captivity:

Sitting staring about him he next noticed the door — and thought at first that he was the victim of some optical illusion. It took him quite a long time to prove to himself that he was not. The point of the arch was not in the center: the whole thing was lopsided. Once again, the error was not gross. The thing was near enough to the truth to deceive you for a moment and to go on teasing the mind even after the deception had been unmasked. Involuntarily one kept shifting the head to find positions from which it would look right after all. ¹⁵

For Lewis’s hero, the experience of heimlich and unheimlich in such close correspondence constitutes a kind of damnation.

Is it because we are awash in utterly convincing perspectival images through the ubiquity of the photograph — magnified logarithmically since the rise of camera capabilities in mobile phones — that we continue to accept geometric perspective as the way the world looks? No matter the massive democratization of the means of image production, it is that old, privileged view that remains the dominant mode, or at least the default. Think of how the holodeck in Star Trek is represented before the program is activated: a fully Cartesian grid, the Federation Starship as if envisioned by Alberti.

Nonetheless art historians have long debated whether linear perspective was a discovery or an invention — a depiction of the truth of the world or an ingenious visual convention. ¹⁶ For centuries Western artists adhered to the laws of linear perspective, yet architectural photographers have regularly employed lenses to “correct” it. Three-dimensional modeling software allows us to choose the degree of distortion that most flatters our buildings. Paul Cezanne flouted the rules, yet his paintings of landscapes present the world as experienced, with multiple, non-intersecting planes, softened edges and blue distances. “It is Cezanne’s genius,” wrote Maurice Merleau-Ponty, “that when the overall composition of the picture is seen globally, perspectival distortions are no longer visible in their own right but rather contribute, as they do in natural vision, to impressions of an emerging order, an object in the act of appearing, organizing itself before our eyes.” ¹⁷

The conflation of painting and photography suggests one source of tension — of the uncanny — in hyper-realistic digital renderings.

The conflation of painting and photography, as ways of representing the world, suggests one of the sources of tension — of the uncanny — in hyper-realistic digital renderings. To put it another way: is an architectural rendering more akin to a painting or a photograph? Again, as Sontag reminds us, we understand a photograph to be of something that did or does exist, no matter what lenses, filters, or films were deployed to transform it. A painting has no necessary connection to the real. In earlier eras, architectural renderings were more akin to paintings; and often they were paintings. It was common practice for architects to commission a watercolor rendering at that ripe moment when it was time to reveal what a design would look like when built. The perspectivist would labor to make the rendering persuasive, appealing, and sufficiently realistic to inform decisions. This image would be carefully carried around to client presentations; sometimes an enlarged version would be displayed on the construction fence around the site. And of course, even the most naïve viewer knew it was an imaginative projection and not an image of an actual building.

But again, photography complicates the old certainties; in both film and digital photographs we can detect the uncanny in representations of perspectival space. Consider the images of the Innie Outie House created by the virtuoso digital renderer Peter Guthrie, from a design by the firm WOJR. We see in these moody images the familiar adherence to linear perspective along the x and y axes, and at the same time a refusal to comply with a vanishing point in the z direction. Thus the image borrows the conventions of contemporary architectural photography, which achieves similar effects with perspective-correcting lenses. This is hardly surprising; Guthrie has a background in architectural photography. Yet it is an intriguing continuation of a circle of influence that connects painting, photography, and now digital representation.

Guthrie, in an interview with Dezeen, argues that the public is comfortable “reading” architectural renderings. “Most people these days are incredibly familiar with computer-generated images (although they are usually in the form of feature films or computer games) but would find it harder to interpret a line drawing or watercolor of a proposed building.” ¹⁸ Incredibly familiar … with the unfamiliar. In Guthrie’s rendering the viewer is close to the house, standing where only a “familiar” or a trespasser would be. From this position the vertical edges and surfaces should be converging; but they are not because we are in an impossible and yet intimate place. We are looking through the window — should we even be here? — into an empty and minimally furnished bedroom. The unraked leaves, the unfurnished room … what is the story? The design is arresting, yet the image seems to tell a story of abandonment, akin to the loneliness of masterpiece house-museums where no one lives anymore.

Decades earlier, the architect and delineator Otto Eggers achieved equally uncanny effects in his perspectival drawings for the National Gallery of Art in Washington. (Eggers worked in the office of John Russell Pope, and completed the design after Pope’s death.) Eggers rendered the proposed museum with icy clarity, under a flat gray lightless sky, complying with only two of the three commonly available vanishing points. We are looking south, so there are just a few hints of light — notice the sparkling water of the fountains — from what seems a high sun behind the building. A few brave souls approach the entrance. The trees are in full leaf, casting short shadows, but in the foreground odd fallen leaves are scattered in the street. Perhaps they fell from some large but unseen tree which is just beyond the frame? What time is it? What season? What planet?

Eggers’s drawing would never be confused for a photograph; it is utterly atemporal, the very opposite of the camera’s captured moment. Eggers’s uncanniness is different from Guthrie’s, with its painterly fog and photo(un)realistic perspective. It is interesting, though, that both images depict fallen leaves around the buildings. In the real world, fallen leaves on the ground signify both a season and a sensibility; they speak of carelessness, abandonment, perhaps a willing concession to the invasion of nature, the inevitability of decay. But in the design world, those leaves are as carefully constructed and located as every line of the building they surround; they are indeed beautifully irrelevant tokens of the uncanny, softening the boundary between reality and imagination.

Architectural renderings are not, then, photographs; or are they? Visit the website of almost any architecture firm and try to discern among the elegant project thumbnails which are photographs and which are digital renderings. The very definition of the uncanny lies in this unsettling inability to determine exactly why what we are looking at seems at once familiar and strange.

Credible people make imaginary spaces credible.

There is another tell of the uncanny in architectural renderings. ¹⁹ Credible people make imaginary spaces credible. Architects put figures of humans in their drawings in order to signal the relative size of things without recourse to quantitative measure, as well as wordlessly to demonstrate what should happen in a space. A rendering is more than a pleasing image; it is a tool intended to persuade, provoke, inspire, and excite; all in preparation for the lengthy, costly, and often maddening process of fulfillment otherwise known as construction. Inserting images of people who look like us, and who are doing the things we do, can be vital to the very function of a perspective rendering.

I would argue that the inclusion of people in a drawing constitutes what in linguistics is called a “performative” — a type of utterance which in essence does what it says. “I promise” is a performative: by uttering those words, I make a promise. In architectural renderings, the presence of human figures asserts the reality of the project: its commitment to being constructed. That is their only role because they are not actually part of the project. They can walk out of the drawing and the imagined building will be unchanged; but what would be lost is the imagined world.

The thoughtfully populated rendering of a new building is nothing less than the promise of a felicitous new world.

J.L. Austin, the philosopher of language, has argued that performatives are neither true nor false; they can only be felicitous or infelicitous, happy or unhappy. A happy performative, Austin asserts, must meet certain conditions. It must follow a conventional model: you cannot simply invent the terms of the promise. Those who make a promise must also have the capacity to fulfill it. A happy performative must be done correctly and completely; the promisers must be sincere and in good faith. ²⁰ The thoughtfully populated rendering of a new building is nothing less than the promise of a felicitous new world.

What is Eggers’s promise in his drawings of the National Gallery? In the painting galleries, we see dignified figures; all appear to be men in suits, hardly distinguishable as individuals. A few appear slightly translucent; notice how the bench in the foreground seems to pass through the man’s topcoat. None of these gallery goers is striking a pose, striding purposefully, or betraying any level of art-induced excitement. There is not a single child. The promise of this picture is that art is serious business, viewed by serious adults.

BIG’s people seem deliberately to attract our attention. Why is that young woman wearing sunglasses inside an art gallery?

It is revealing to compare Eggers’s performatives with those of a contemporary rendering of another museum in Washington. In drawings of the proposed redesign of the south campus of the Smithsonian Institution, the Bjarke Ingels Group portrays people who, like Eggers’s people, are gazing at art in a high-ceilinged space. Yet the effect is strikingly different. BIG’s people — curiously, most women — seem deliberately to attract our attention. Why is that young woman wearing sunglasses inside an art gallery? Is it because the space is filled with glaring, white-hot sunlight? Is that why the women are wearing sleeveless dresses? In this newer rendering, we are distracted; we can’t help but look at the people qua people as we realize they are not drawn figures but rather images of real people inserted seamlessly into this imagined space.

Photomontage is an immensely useful rendering technique; but it turns out there are consequences to populating imagined worlds with realistic people. In contemporary renderings, every figure has a face, a race, a gender, a mood; the architect has to consider what they are wearing and whether the clothes are appropriate to the setting and the hypothetical season. Above all render people must always appear to be happy people; even if what they are viewing on the walls of a gallery is not art that would inspire happiness. All of which is to say that the BIG rendering has taken us back to the valley; not exactly the valley of Mori’s spooky robots, but uncanny nevertheless. ²¹

It’s hard not to conclude that the uncanny is everywhere in architectural drawings: their fundamental responsibility is to combine depictions of what is real — indicative representations — with depictions of what is possible or likely — volitional subjunctive representations. The uncanny reveals itself in the precision that is too precise, and the uncanny accuracy of a digital drawing can act as an unhappy performative. For all too often the rendering is making a promise that the building cannot redeem; that visitors will be happy, that surfaces will be clean and shiny, that the sky will be cloudless and clear.