A Look Under the Hoodie - A Technical Deep Dive on Full Body Avatars

The full body avatar beta is now available for everyone! To celebrate, we wanted to share some of the nerdy details on how we made the full body avatar work and a refresher on why we did it. Jump to the end if you’re curious about what’s next…

Why we decided to make full body avatars

More self-expression - One of the main reasons for introducing full body avatars in Rec Room is to create additional opportunities for players to express themselves. The classic bean avatars, while iconic, can sometimes be limited in conveying the widest possible range of emotions and actions.
Easier to understand what’s going on in the game - Our beloved beans are amazing, but sometimes it's a little hard to visually understand what they're doing outside the context of Rec Room. Full human silhouettes with connected limbs make it easier to understand visually what’s going on.
Easier for creation - Soon, creators will be able to make their own avatar items to wear and share in Rec Room. Doing the work for the full body avatars allowed us to update the skeleton shared between beans and full body avatars to better match the industry standard so it's easier for our creators to make custom shirts, pants, hats, gloves, shoes, etc.
Third-Person is better with full bodies - While controlling and watching your avatar in this improved third-person mode, gameplay and social interactions are just more fun with the full body.
Adding new features we’ve wanted in our avatars - Adding new features like noses, adjustable glasses, adjustable body shapes, and eye gleams help breathe new expression into our avatars. These small details contribute to the overall relatability and expressiveness of the avatars, giving our players more options for how they want to look.
Adds more ways for all of us to make money - By adding noses, eyebrows, fingers, arms, legs, and feet, we are expanding the kinds of things we and eventually our creators can make and sell to our community.

This was the initial concept art we created as a target for what full body avatars could look like. We’ve made a lot of tweaks and improvements since then, but we feel like the original charm remains.

Fitting more limbs into the same amount of memory

While we were developing the full body avatar, we recognized that our crash rate due to memory for all of Rec Room, especially on mobile devices, had gotten really bad. Every developer at the company was tasked with making memory optimization their top priority. One of the challenges that made it difficult to ship full body avatars within our original timeline is that we decided we needed to get to a place where the full body avatar, with all of their extra limbs, animation systems, etc., had to use the same (or less) memory as the floating bean. We called this “getting to memory neutral.”

Here’s where we were when we started the optimization work.

Our benchmark floating bean avatar had 6,095 vertices (which define the geometry), and cost about 4 MB per avatar (which includes all 3 levels of detail meshes). The floating bean also had a total of 18 bones and 4 weights per vertex.

*This is the benchmark avatar we used for comparison of our bean geometry cost vs our full body cost*

The equivalent full body avatar (which added default pants and shoes) had 11,384 vertices, and cost about 8 MB per avatar before optimization. The full body skeleton currently contains 102 active bones with 4 bone weights per vertex.

We needed to find a way to cut our full body avatar cost in half in order to hit our goal of being “memory neutral.”

We got there with 3 key innovations.

Memory Optimization Innovation #1: Skin Culling Masks

For our first innovation, we built skin culling masks. This technique allows us to throw away some of those expensive vertices in memory by not building parts of the avatar’s skin that are not visible based on what they’re wearing. This means that for every clothing item our artists and creators will author, they can select from a set of existing “skin culling masks” to inform the avatar system which skin faces are hidden when an item is worn.

Whenever you change what you’re wearing, we look through the set of items you’ve equipped and determine which geometry faces can be completely discarded. This comes with the added benefit of avoiding clipping bugs by just removing the problematic underlying skin!

You’ve probably noticed during our beta that when authoring our huge catalog of avatar items, some culling masks weren’t set correctly, or needed to be adjusted. While we fixed many neck and wrist gaps, missing fingers and hands there’s still work to do. We are continuing to clean this up during our polish work.

You can see here in the right image that there are fewer skin polygons under the clothes compared to the left image that doesn’t have skin culling turned on. The Skin Culling technique allowed us to throw away between 1k to 2k vertices on average.

Memory Optimization Innovation #2: MeshData, and Vertex Data Compression

For our second innovation, we re-wrote how the avatar mesh was built using a new primitive called the MeshData feature. This dramatically sped up how our avatars are built, and removed an annoying frame hitch that sometimes happened.

Before, you’d probably notice when a player joins a room there can be a few frames of lag, especially on lower-powered devices like the Quest 2. That’s no longer the case with full body avatars.

Another nice thing about the MeshData feature is that it allowed us to compress the data per vertex so we could manipulate how much memory that bone weights and UVs took up. With this optimization, we saved about 30% of our total memory.

This is a snapshot from deep within our code of how we changed the layout of our vertex data to trade precision for memory efficiency (using half and byte data types instead of floats and integers saved us 60% of our unvarying data)

Memory Optimization Innovation #3: Just-in-Time Level of Detail

For our third and final innovation, we developed a technique we call just-in-time level of detail. The classic floating bean system builds all three level of detail meshes whenever a new avatar appears and holds on to them in memory. We then only show one of these meshes at a time based on how far that player is from your camera.

*Showing the different level of details that an avatar may have*

Because of the work we did to use the MeshData type above, we could explore building a dynamic avatar that builds the avatar when the level of detail should change without experiencing those frame lags. So now we only pay the memory for what we display. This saved us another 40% of memory on average.

This trades off some compute time for memory savings, but we found across all of our platforms that thanks to Unity’s MeshData and multithreaded Jobs, it does not impact FPS!

So in total, we were able to reduce the cost of the full body avatar by around 60~70%, more than the 50% we needed. This is why we say that the full body avatar is more optimized than the floating bean.

Memory problem solved! What about the frame rate?

Thanks to Unity’s animation system, we found that all of our platforms could pretty smoothly handle the complexity of the extra limbs. Where we did add some considerable computational expense is in the full body inverse kinematic (IK) solver we use.

Full body IK helps with the believable movements of the body, arms, and legs on screens and in VR while only tracking or animating your head and hands. With enough players in a room, lower-end platforms started to show bad FPS. So instead of updating the full body IK solver for every player on every frame, we built a system to dynamically update player animations up to a certain amount per frame, prioritizing those avatars closest to the player or those that haven't been updated in a few frames.

This priority-sorted technique ensures that you always get a good FPS, but at the cost of some players’ movement stuttering a bit, especially when they are far away from your camera. This felt like the right trade-off for us.

Interaction Improvements: Showing your hands while grabbing stuff

One of the big challenges for full body avatars was coming up with a way to show you grabbing objects. With the floating bean, we got away with this by hiding your hand whenever you were holding something. But since one of our major goals was to make it easier for others to understand what you were doing, we knew we needed to find a good solution for making sure holding items looked good.

We tried a bunch of different approaches, including snapping your virtual hand to a spot on the grabbed item. But that felt like the game was physically pulling you around and rotating your hand. We decided that we should always honor where your hand is in real space and have the grabbed objects come to them, so your virtual hand feels as close as possible to your real one. You can especially feel how this improves the stabilizing handles for laser weapons while in VR.

We did find one exception to the rule that felt nice. Instead of allowing your avatar arms to stretch as long as they need to match the location of your real hand, there’s a point where if you move a held item too far, your hand will look like it let go and the tool hovers where you’re holding it in real space…kind of like you’re still controlling it with magical powers. We call it the “Telekinesis effect.”

Then there’s the visual gripping effect for (many) objects. We wanted the visual to look cohesive for Rec Room objects. In order to give our artists control over how hands are posed when grabbing the object, we created an internal tool called the Hand Grip Placer. Our Artists can then determine how objects should sit in the hand and what hand pose they should take. During the beta period of full body avatars, we are continuing to work through all of the 500+ items you can grab in Rec Room to make sure that you have great-looking grips.

*This is an internal tool we wrote to give our Grip Artists a way to place and pose how the hand should grab each one of our 500+ items in the game.*

Improved movement & animation

Thanks to the work of @joedanimation , we have enhanced and added more life to our avatar animations. This includes improvements in walking, running, jumping, climbing, clambering, wall running, and sliding. Our goal is that Rec Room feels immersive and expressive, and these improved animations do a lot to ground the avatars in the reality of the game space.

We are still working on polishing all of the animations in the game as part of our beta work.

Full body avatars is still in beta, so what are we working on to get it out of beta?

We are continuously iterating on the full body avatar, and now that every player can use them, we are interested in hearing and working on feedback from everyone.

We are continuing to work on how the full body avatar moves in VR and screens (especially with dancing and idle animations), fixing holotars to work with full body, making it so a creator can determine which avatar types are used in their costumes, fixing the ability to adjust how hats sit on your head, fixing more screen-based animations (have you noticed how full body avatars drink water bottles through their eye? :P), finishing up all of the grip poses for the 500+ items, and exposing the underlying sliders that shape your body and head so you can have more control in customizing your form.

We are also continuing to polish some of our legacy avatar items that don’t look great yet on the full body avatars (we know some gloves aren’t where we’d like them yet). Finally, we haven’t forgotten about full body tracking and finger tracking for our VR players.

Only when this list of things are done (and we address additional feedback we receive) will we consider dropping the beta status.

Our commitment to the floating bean avatars

We know a lot of you are concerned that we may deprecate the floating bean avatar when we determine full body avatars are ready to be out of beta. Don’t worry! We are committed to supporting and upgrading floating beans that have been here since the beginning of Rec Room.

One of our big tasks right now is to take the optimizations that you read about above— such as skin culling masks and just-in-time level of detail— and bring them to the bean so we can reduce their cost by 60%. This will be a big help in our fight to improve performance on all platforms.

Additionally, we plan to bring over some new features to the bean like being able to see your customized hands when grabbing objects, being able to see your torso when you look down, and the option to have a nose, the option to have eyebrows, and the option for eye gleam. Whether you want to preserve your classic bean as is, selectively add some of the improvements (such as eye gleam, or eyebrows), or move all the way to a full body avatar - we will fully support you.

Look out for an upcoming dev blog where we will dive deeper into our plans for preserving and enhancing the floating bean avatars.