Imagine walking down the gray granite steps of a weathering castle. A few flickering stars just make their way through the dense, smoky fog surrounding you. In the distance, you hear a deep, towering voice speaking with assertion. You trace the sound, moving about in this surreal world until you see him. A 30-foot-tall giant hovers, clad in steel armor — the ghost of Hamlet’s father.

While this may sound like a theater buff’s fever dream, the above depiction is real. Or, at least, virtually real.

Two years ago, Tandon professor and alumnus Javier Molina and director David Gochfeld had an idea to merge the world of art with interactive technology. The two imagined a live performance where actors and audience members on separate sides of Earth could come together for a theatrical performance in real time. In time, this vision would become “To Be with Hamlet,” a theatrical performance of Shakespeare’s “Hamlet,” viewed entirely through virtual reality.

Utilizing an array of advanced game design technology at NYU’s Media and Games Network — MAGNET — at Tandon, the team behind “To Be with Hamlet” created a world which would serve as a virtual theater unrestricted by real-world physical limitations.

“Theater has always been a social experiment first and foremost,” Molina said. “But in this space you can build entire worlds around you and build characters around you that you can’t in traditional theater.”

“To Be with Hamlet” had its first test run in April at the Tribeca Film Festival. The project had its own private room as part of a full floor of VR experiences.

Tandon senior Alexis Trevizo experienced the virtual performance first hand and came away amazed.

“Overall, I thought the experience was inspirational,” Trevino said. “‘To Be With Hamlet ‘has inspired me to learn more about the production of VR experiences.”

Though the audience members felt they were merely steps away from the performers in the virtual world, the two actors were, in reality, an island away in Brooklyn, reciting Shakespearean prose while wearing skintight motion capture suits.

“It’s sort of like diving into the machine’s head,” said Roger Casey, the actor who played the Ghost of Hamlet’s father. “It’s a lot like puppetry.”

Originally versed in theater, film and combat choreography, the live virtual performance of “To Be with Hamlet” proved challenging, both conceptually and physically for Casey. Unlike traditional theater, the VR setup of “To Be with Hamlet” meant that Casey had to mime the movements of a floating 30 foot Paladin, all while maintaining complex lines.

“You have to figure out how to split your brain,” Casey said.  

Christian Grewell, an adjunct arts professor at NYU Shanghai, has been interested in virtual reality since the technology’s inception. Grewell, one of the first backers of the original Oculus Rift system was approached early on by Molina to create a social VR technology. This new social software would eventually be called M3diate.

Using M3ditate, viewers from as far away as Abu Dhabi were able to watch the performance and interact with viewers from the Tribeca Film Festival in real time.

The minds behind “To Be with Hamlet” hope to expand the project beyond its current 12-minute performance. Molina said he wants to continue to add more complex elements to the project.

Molina hopes the experience will feature mixed reality and augmented reality features within the next two years. By using all of these different technologies, Molina seeks to create an experience that goes far beyond traditional immersive theater. Both Molina and Grewell see the Hamlet project as catalyst for future VR projects involving the arts and educational development.

Despite the multitude of possibilities VR technologies may offer to art, there are some potential roadblocks. One possible barrier is the idea of the “uncanny valley” — the theory that states that humans tend to feel uncomfortable or freaked out when a non-human figure appears close to but not quite human. Grewell says that overcoming the uncanny valley is a constant challenge but feels the team has largely succeeded.

The second and ultimately greatest barrier for VR performances comes down to costs. For less than 15 minutes of content, Grewell estimates the costs for “To Be With Hamet” to be over $32,000.

Despite these factors, both Molina and Grewell are optimistic for the future integration of virtual reality and art. Molina pointed to the reactions at the film festival as a testament to the technology’s benefits.

“We had a kid with very little knowledge of Shakespeare who got immersed in the environment and walked out with a new knowledge.” Molina said.

Molina went on to say that now, while the technology is still in its infancy, is the prime time for art and education to take full advantage of the potential of virtual reality. Ironically, while NYU’s engineering department has programs catering to the platform, the school’s theater department currently does not offer any classes teaching students how to use motion capture technologies.

CORRECTION Sept. 18: A previous version of this article incorrectly stated the name of one of the creators of the VR production as Jose Molina, when his name is in fact Javier Molina.