Introduction
Fractura is a room where we explored extensive use of Generative AI (GenAI) tools in the Rec Room creation workflow.
The result was the creation of an immersive alien world brimming with ancient mysteries!
Rec Room’s in-house Creative Team took a broad and experimental approach, evaluating over 20 different GenAI tools to identify which had the most potential to benefit Rec Room’s creative community. Since Rec Room Studio is built on top of Unity 3D, all of the GenAI tools mentioned in this doc and others can be used by any Rec Room creator today.
Concept Phase
The use of GenAI started in the concept phase - the team used ChatGPT to develop the ideas behind Fractura, iterating through many prompts. The team explored ideas for landscapes, flora, and lore to flesh out the concept.
These ideas were then visualized using Midjourney and Dall-E. Fed by the text output of ChatGPT, these AI image generators created striking visual concepts.
This process was iterative, with the team using each round as inspiration to refine the ideas and shape Fractura’s identity and aesthetic.
If you want to see “behind the scenes”, here are detailed logs of what the concepting phase looked like:
Midjourney and Dal-E Generated Concepts
Skyboxes
In video games, a “skybox” is an immersive texture that makes the environment seem larger than it really is. Creating convincing skyboxes is a difficult task that requires artists to create tricky “spherical” textures.
For Fractura, the team used Skybox AI by Blockade Labs. Skybox AI allowed the team to quickly generate skyboxes with the right kind of ethereal ambiance by entering text prompts based on the output of the concept phase.
The ease of generation allowed the team to create multiple skyboxes that players can choose from dynamically in the world of Fractura.
3D Asset Generation
GenAI was also used to generate 3D assets. Tools like CSM, Shap-E, and others had pros and cons. These tools were used to create trees, ancient stone robots, floating land masses, and distant buildings, instilling Fractura with an otherworldly allure.
However, optimization and performance challenges were commonplace. The assets generated often had very high polygon counts, requiring manual simplification and optimization so they could be used successfully in Rec Room Studio. Generation times were also often quite lengthy.
Despite some challenges, the future for 3D GenAI looks bright, and the team looks forward to further explorations!
Texture Creation and Soundscape
The team used Withpoly.com for texture creation and MusicGen for the soundscape.
Withpoly.com facilitated the rapid generation of textures and normal maps, allowing the team to integrate detailed textures into the game's environment and architectural elements. Since it was quick and easy to iterate on textures and normal maps, the team found they could use this tool to get just the kind of “otherworldly allure” they were going for.
When it came to the soundscape, MusicGen really delivered. This AI-powered music generator helped create a soundscape tailored to Fractura's alien landscape. The tool's ability to synthesize otherworldly tones, evocative melodies, and ambient soundscapes helped instill Fractura with captivating and immersive audio.
Want to listen to Fractura’s soundscape? Fractura’s Soundscape
Ideation and Interactive Elements
ChatGPT, Midjourney, and CSM helped the team create interactive experiences within the alien landscape. The incorporation of AI-generated components led to the development activities, such as a target shooting game and an interactive fire spirit.
While these interactive elements weren’t generated “whole cloth” by GenAI tools, the GenAI applications made the creative process more efficient and enabled a smooth flow from conceptualizing ideas to implementing tangible in-game elements.
Envisioning the Future of GenAI
The final exploration zone, named the Holodeck, symbolizes the future potential of GenAI! Using the GenAI tools from Blockade Labs, the team created an experience where players can choose from a variety of generated 3D environments - a glimpse of the future where immersive 3D environments can be generated on the fly in response to player input.
Conclusion
Fractura is a milestone in Rec Room’s exploration of GenAI, highlighting both its transformative potential and the challenges in integrating it. Fractura exemplifies how Rec Room creators can leverage GenAI and Rec Room Studio to iterate and craft engaging rooms efficiently based on their imagination, creativity, and vision.