Smooth DK clip playback on Quest 2 using Unity

Hi everyone,

has anyone got advise for smooth playback for DK clips on the Quest 2 using Unity? I’m aware there’s the post from Jillian here (Depthkit for the Oculus Quest).

As far as I know I have most of it set up as directed by DK, Oculus etc. and I’ve found using Vulkan helped my playback a lot. I’ve got it working and my clips run smoothly to a point, but then freeze for a moment, and then continue. I need to get rid of this staggered playback.

I’ve tried a few Garbage Collection tweaks (forcing collection every 30 frames in Update) which made no difference, after a look in the Unity profiler I think it may be due to the Video rendering needing the next frame…?

The clips were shot on a Kinect Azure. I’ve crunched down the clips as modest 720x1200 mp4s using AVC x264 and the Depthkit quality is set to “low” in the Editor but I’m running out of ideas for getting this working.

Anyone been able to get DK clips running smoothly on Quest?

Thanks

I’m wondering if loading DK clips from StreamingAssets using WebRequest to set the URL would help? Maybe I’ll try that…

Hi @Psicon_Lab, thanks for sharing these tests. If using Depthkit Plugin v0.2.7 you are doing all the right things with reduced mesh density and video resolution. We have not yet tested if loading assets from StreamingAssets will improve performance specifically for the Quest 2.

I’m curious, are you trying to run many Depthkit clips at one time?

We are committed to optimizing performance in future developments for the Quest and we look forward to keeping you posted!

Hi @jillianmorrow thanks for your support. No, this is a single depth kit clip in an empty environment.

Actually, I have had DK clips running smoothly on Quest before but only single clips, one at a time shot with the older, lower res Kinect V2 so seems linked to the slight increase in quality of the clips from the Azure.

Would a change in Render Pipeline in Unity help maybe?

I’m starting to think this is all about video playback performance on the Quest 2…?

Ok, for the record, I’ve just built for Quest2 with 6x DK clips running simultaneously and playback was smooth. The framerate was low but I did not experience the staggered playback I’d seen before (which is bizarre as I used an almost identical project, but anyway it’s clearly something in my project that’s causing issues).

I’ve found that tidying up the clips in post and removing some unwanted background has had big performance benefits. I’m guessing the mesh is more cohesive and spans a lot less physical space if there are no random artefacts pulling it out of shape here and there.

@Psicon_Lab ensuring you are applying a crop within the Depthkit Refinement workflow should help, in addition to reducing the clip resolution. For additional support, let me know if jumping on a screen share or call would be helpful. That way we can discuss your asset specifics.

I have 12 different 45 minute performances. In order to get them working on the quest 2 I had to trim them down to 5 minutes each. 1/2 of the depth kit performances would only work at first but then I managed 3 total. I had to use the scripting buttons of AV pro in order to close the other Depth kit footage that was not playing.

Bumping this thread as some users have recently been asking about how to get performant playback on Quest 2 with the new Unity Core Expansion Packages.

The following best practices apply generally to both Single-perspective capture from Depthkit Core and Depthkit Cinema rendered in the Core Expansion.

For Depthkit Studio captures with multiple-perspectives, please use the Studio Lite renderer and refer to to our platform support matrix.

Here’s how we have been able to achieve performant playback of Core Expansion Quest:

• Use AVPro for playback
• Encode as HEVC (H.265) MP4
• Do not exceed 2048x2048 in video dimensions
• Encode bitrate around 8 Mbps
• When configuring the clip, set Mesh Density to Medium.
• Enable Edge Smoothing, using the Radius and Strengthen to improve edge quality with the lower mesh density recommend above
• Decrease the size of the Edge Mask

Hope this helps folks out there unlock Quest with this workflow

@Psicon_Lab & @dustygonzales - A quick update for this thread: One of the new features found in Depthkit v0.7.0 is an export settings preset for Meta Quest, which leverages the HEVC (H.265) codec, TV color range, and other settings to generate Combined Per Pixel videos optimized for playback on the Meta Quest platform.

You can read more about some more of the new features in our latest blog post, and find more info in the announcement for the new update.