Marvellous Reality - 360 CGI+Real-life footage research project

From March 2018, I started leading a research project funded by Debora Arango Higher Education School. The project aimed to create a bespoke process for the creation of an immersive story.

For six months, I led and worked side by side with a team of professionals from the audiovisual industry i.e. audio specialist, a camera crew, a producer and actors.

The first plan was to create a 3D environment and to place real-life footage of performing actors. So we started shooting actors against a green screen and testing the resulting video inside of very basic 3D environments.

After testing the chromed characters inside of both Unity and Blender, I decided that the most efficient course of action due to the complexity of this project was to produce a Monoscopic 360 video with ambisonic sound instead of building a VR app.

We spent a good amount of working hours trying to figure out the best way to place characters in front of a moving camera. One of the biggest challenges was consistently matching lighting schemes on both ends (physical and digital), Another big challenge was to figure out how to place and reveal characters in the 3D environment, preventing them from revealing their flatness once the viewer’s point of view changed.

We managed to overcome all these obstacles and ended up introducing a total of 5 characters, including two that directly interacted with the viewer at a very close range.

Regarding the 3D environment, we discovered that purchasing an existing 3D environment was not an option, as we wanted it to reflect specific architectural features from our culture. However, the models available online have very different ones, suggesting that depicting a typical Colombian city hasn’t been of interest to 3D artists.

So I decided to start building the entire fragment of the city from scratch.

The first models were quite basic, and this was actually pretty useful as the script evolved a lot and we ended up changing the story and characters more than 3 times.

I also started testing two different approaches for building creation, the first one explored the process of building an entire facade by extruding some features of a frontal photo of an existing building.

The process proved to add a quite realistic look to the scene but, at the same time, added a lot of size to the textures used in the project and demanded a lot of retouching, e.g removing cables from the facades or getting rid of shiny materials on doors and windows.

The second option was to build all the locations from scratch using conventional 3D modelling and texturing techniques.

In the end, I decided to mostly do every asset from scratch in order to have enough flexibility to design the 3D environment. In total, it took me over three months to build a fragment of a typical Colombian city spread across a 5 by 5 array of blocks.

It is worth mentioning that I did use Google Maps to navigate the cities of Medellín and Envigado to get inspired and also to get the footage I needed to replicate architectural features of two emblematic buildings.

Finally, to create the ambisonic sound for the video, I learned how to integrate Reaper a (Digital audio production application) and the Facebook 360 workstation, which allowed me to spatialize both sound effects and dialogues.

The following is a 2K version of the project; it took 8 days of continuous rendering using 16 27-inch iMacs to produce a total of 8800 frames.

This is an ongoing project. At the moment, I am looking to get more funding to finish adding more characters and 3D animations, enhancing the sound FX and rendering it in at least 4k quality.