HoloLens 2 Depthkit Render Not Appearing in Device

I am asking for a college development project about the DepthKit rendering a one-camera video on the HoloLens2 device.

The integration to Unity is fully functional, however, the DepthKit render does not appear when in the HoloLens device. We can run in the editor all of the animations and even interact with them when given a box collider. Then when deploying the Unity build to the HoloLens 2 device, we see no video appearing as we do in our Oculus environment.

I have the assumption that maybe certain settings are not loaded properly, or possibly the HoloLens 2 is having issues with rendering DepthKit, but I thought I would ask for some assistance. We should be seeing, in the image below, a render from our DepthKit integration on Unity. The render would be beside the breathing ring and would show the upper half of an individual but it seems to either not activate or doesn’t load into the HoloLens 2 device. I will note again that it does load and can be interacted with in the Unity editor but not the device.

image867×621 41.1 KB

Hi, @HILABMSU. Can you share more information about your project?

• Which type of Depthkit assets are you trying to render? (Studio? Core/Cinema?)
• Are you getting any errors, warnings, or other messages in the Unity console when running the scene in Play Mode on the computer?
• Which version of Unity are you using?
• Which render pipeline are you using?
• Which Look component is applied to your Depthkit asset?
• Which version of AVPro are you using as the video player for the Depthkit asset?
• Are you able to get the Depthkit video to play as a 2D quad in the scene (bypassing the Depthkit renderer), or does that also not show up in the build?
• For this testing, are you able to use Microsoft’s Holographic Remoting to render on the PC, and pixel stream to the Hololens? We have seen other teams succesfully integrate this into their Depthkit projects to performantly render Depthkit Studio assets on a Hololens.

Let me know the answers to the above, and we’ll see if we can narrow down the issue.

1. We are using the DepthKit Core render.
2. We receive one error but that is from the mixed reality toolkit and is due to a pathway that is too long for a cached item.
3. We are working with Unity 2020.3.21f1
4. We are using the 3D render pipeline, no URP or HDRP.
5. I believe it is Core Photo Look Built-in RP
6. I do not know which version as of now, I do not know entirely how I would verify what version we are using for the project.
7. I did not know we could render it as a 2D quad, but when trying the direct mp4 file it does not properly setup for our project. I did go for another way of setting up the DepthKit clip system and tested it in the hololens, but they do not seem to be appearing.
8. I did the debug mode with Holographic Remoting and still did not see anything appear, even if it did I feel that remote linking isn’t going to help since we do want the system on the HoloLens 2.

I tried some of the things you mentioned but we are still not seeing results of our Depthkit render appearing. I hope the info I provided helps.

• HILAB Team

@HILABMSU

Can you confirm which video player component is in your Depthkit object? Is it the standard Unity Video Player? If so, have you tried installing our depthkit.avpro package, and separately, AVPro (free trial and full verison available in the Unity Store) and using that as the Depthkit object’s video player type instead?

The goal of getting the Depthkit video play on a quad is to lets us know if the issue is occurring within video playback, or if it’s within the renderer. If we don’t know whether or not the video plays in your scene on a quad (bypassing the Depthkit renderer), then we cannot narrow down where the issue is occurring.

Also, do you have any more details about the Depthkit video? What is the resolution? Is it an h264 MP4 that was directly exported from Depthkit, or have you encoded/transcoded it to another file type and/or codec?

@HILABMSU - Just following up on this. If you’re still experiencing this issue, please provide information for the following questions:

• Can you confirm which video player component is in your Depthkit object?
• Does the video play as a quad?
• What are the resolution, codec, and other spec’s of your Depthkit video?

If this is no longer an issue, let us know so we can close the ticket. Thanks!

I do not know where to verify if the video is played as quad at this point and I do not know where to verify the resolution and codec of the DepthKit video. If that is in the editor I will relay to my team where to verify that information.

Do you guys have tutorials for the components of the video player and where these components are so I can get the right information to your side?

The team was recently on a break due to the college research atmosphere, sorry for the very late response.

@HILABMSU, no worries on the delay - We simply want to get you unstuck as quickly as is convenient for you.

The Depthkit video can used to texture a Unity object by assigning it to a render texture and material, and then connecting that material to the object, as in this tutorial. We recommend following this guide to assign the video as the texture for a quad. Doing so verifies that the video player is either working, or needs troubleshooting.

To know which video player component you are using, which Player is specified in your Depthkit Clip component? The default is Video Player (Unity) - see below.

If the Depthkit video was produced as a result of exporting from Depthkit, it is an MP4 container with an h.264 codec. If it was encoded any other way, you will have to have specified the container and codec format.

Other details about the video, like resolution, can be found by selecting it in Windows’ File Explorer and looking at the Details Pane:

image1512×1015 89 KB

Let us know if you need any more guidance to produce the details we have requested, or if you’re still getting stuck.

I am going to try an use the AVPro Video player, but I can say that we are using the Unity Video Player and when loading onto the device the Quad View functions. We can see the MP4 play with the guided instructions you provided.

I also re-imported all the files and then checked the video MP4 we are using, our video sample is an MP4 file that comes with codec and everything, which means it is exported from DepthKit properly by the examples you provided.

Update Each time we try to use the AVPro Video we get grayed out selections in the upper window options, and Unity does not give AVPro as a video player options. We are trying the trial version with the packages provided and it should be working but seems to not be functional even following the steps from the DepthKit documentation for a re-imported setup.

Hello Cory Allen, I just wanted to follow-up and say we are still having issues with the HoloLens 2 environment of the DepthKit. I sent an update that detailed the information that you needed from our team and I also commented on how AVPro was not working. I hope we can continue to collaborate and if meeting virtually is better I would appreciate that as well.

@HILABMSU Thanks for the information you provided.

It sounds like the Unity video player is working when playing back the video on a quad - Is that correct? If that is the case, then this sounds like an issue with the renderer, not the video player, so testing with AVPro is unnecessary.

Can you describe the step-by-step process you are using to set up Unity to build to the HoloLens, and the steps you are taking to incorporate Depthkit? Please include specific details like the version number of any plugins/packages used. For example:

• Create a new project in Unity 2020.3.21f1 using the 3D Template (Built-in Render Pipeline)
• Switch build platform to Universal Windows Platform
• In Package Manager, add Windows XR Plugin version 4.6.5 from the Unity Registry
• In Package Manager, add the Depthkit Core package version 0.12.2, and importing the Depthkit Clip + Core Built-in Look Prefab
• Convert Main Camera to XR Rig
• Adding the Depthkit prefab to the scene, and linking the combined per pixel video, poster, and metadata to the object.
• Enter Play Mode - Confirm that the Depthkit asset appears in the Game View, and that there are no errors in the Console
• Build to HoloLens - Confirm that there are no errors

If you are constructing your project differently, or seeing errors or other feedback at any point, please describe this in your sequence of steps. This may reveal where the error is occurring, or help us recreate the issue on our end.

@HILABMSU As an alternative, making a screen recording which takes us through your process of setting up the project, and describing the steps you take as you show them, may be a quicker and more natural way to show us how your project is constructed, and may reveal where the issue is. Then share the screen recording video with us here.

The creation of a HoloLens project that we use is based off of the MRTK Toolkit that is provided with HoloLens 2. The package is built for Unity and is setup by following the exact steps while creating a Unity environment. The steps to setup are as followed: (Based on : https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/new-openxr-project-with-mrtk --And-- DepthKit Videos on Youtube)

1. Build a new Unity Project, Unity 2021.3.18f1, using the 3D Template with the Built-In Render Pipeline. No other Pipeline can be used in the project for consistency on platforms.

2. Set Build Settings to the following: Universal Windows Platform, Architecture: ARM 64-Bit, Build Type: D3D Project, Target SDK: Latest, Minimum Platform Version: 10.0.10240.0, Visual Studio 2019 (Required), Build and Run On: Local Machine, and all other settings are not configured.

3. Switch Platforms and Return to the Editor Once Swapped.

4. We will now install the most recent and current packages for DepthKit and MRTK Toolkit. These packages are two of three that we will have on the HoloLens 2. The final packages is built by the lab and features one of the objects we have but will have no conflicts with any other items.

5. After installing the DepthKit Unity Expansion Package, and the Mixed Reality Feature Tool from Microsoft. You will run the .exe file for the Mixed Reality Feature Tool and proceed into the selection section. In here you will, select your Unity Project file path and go into discover features, which is where you will select Mixed Reality Toolkit Foundations (Latest In Mixed Reality Toolkit 0 out of 10 Section) and the Mixed Reality OpenXR Plugin (Latest in Platform Support 0 out of 5), then select Get Features.

6. After getting features, you will validate, and if no errors occur, then import. Follow the prompts until it says the Unity Project has been updated, then return to Unity. If a warning about backend items appears then select yes and wait for all updates to occur.

7. After Entering Unity a Prompt for MRTK will appear, you will do the following to automatically integrate MRTK 2 into the new Unity Project.
   a. First, select the OpenXR Plugin (recomended), then click on the OpenXR box that appears in a new settings menu.
   b. Next, Return back to the initial window that has MRTK setup and select apply settings, then click the fix problems in the upper right of the original settings window. You may need to do this multiple times until the fix prompt disappears, then return back to MRTK setup.
   c. Then, you will click the Next button, to the right of Apply, this will take you to a new area where you will go down and select apply. You will click the Next button again to proceed to the next step.
   d. Finally, you will import TMP essentials to provide tools for HoloLens 2 development. You will select Done in the MRTK setup panel and then close the settings window. This setup is complete and MRTK 2 is fully imported into your Unity Project.

8. Then we will go to the package manager under the Window tab. You will select the plus sign in the upper left and add a package from disk. You will navigate to the folder that has your DepthKit packages and select the following folders in the following order to add: depthkit.core, depthkit.core.shadergraph, and depthkit.studio (No Shadergraph for Studio with Unity 3D).

9. We will then navigate to the package, in our package manager, that is labeled DepthKit Core, which will have imports for prefabs that we are going to use. Select the import option for the Prefab and once it is installed navigate to the Assets folders. Here we are going to add a DepthKit Clip Folder that will hold our DepthKit Footage and Recording. (We should add the three files recieved from the DepthKit application, which will be a .txt file, .png poster image, and the .mp4 depth render)

10. We will want to build a new scene at this point and have no assets loaded into it. This will allow us to have a clean slate for development. (There will be a camera and light for now in the new scene, named appropriately for the task)

11. Then navigate to the DepthKit Core Asset Folder, which will have the prefabs that we want to create a variant of. We will create a variant of the clip plus core look prefab, which will be brought over to our created folder of DepthKit Clips. We will then drag the .txt, .png, and .mp4 files to the slots that present on the right side, while our prefab variant is selected.

12. This will prepare our DepthKit Object to the fullest extent needed in our project. We will drag the Prefab into the scene once we have setup the HoloLens 2 functionality.

13. If we look at our tabs we will see a Mixed Reality Tab at this point, this tab can be dropped down and toolkit selected. In Toolkit we will select add to scene and configure, which will add the MRTK2 assets we need and open a panel to the right view of Unity.

14. We could modify the panel to the right but it is already configured for HoloLens 2 to start, we will now add the prefab variant and position in front of the camera view of the Unity MRTK camera. This will be a running DepthKit object in our HoloLens 2 build. We will be able to build our project now with the current settings and DepthKit object and retrieve the .sln file from the build.

The following instructions allow us to create a build file that will be loaded into Visual Studio 2019, when getting Visual Studio it must be 2019 with UWP and Unity integration selected in the installation. This will then run the solution and will deploy the solution into the HoloLens 2. The loading of the device should cause no issue since it is the only way to deploy the device besides holographic remoting. However, there were no results on our end for Holographic remoting. I will upload a video soon as to have a more visual form of our end of the project.

However, the steps we have here will be exactly what is in the video, all the steps are derived from the DepthKit documentation and the Microsoft documentation for MRTK2 HoloLens2.

I hope this helps and look forward to hearing from your team again.

Thanks for the detailed description. I noticed a couple of details which may be contributing to the issue:

• Our packages currently tested and validated against Unity v2020.3 (LTS) - You are using Unity 2021.3.18f1, which may be introducing issues, so try testing in Unity v2020.3 first, and then experimenting with other versions if that’s successful.
• Depthkit Core is the only package required for single-sensor captures in the Built-in render pipeline. I noticed you are also adding the Depthkit Core Shader Graph package (which is not used in the Built-in render pipeline, and may even cause compiling errors). I recommend testing a Depthkit Core capture (single-sensor) with only the Depthkit Core package installed, and only when that’s successfully running on the HoloLens, adding the Depthkit Studio package and import the Depthkit Studio Clip + Built-in Look prefab to render Studio (multi-sensor) captures.

Let me know if addressing these changes anything - Otherwise, I’ll keep an eye out for your screen recording, and go from there.

Here is the video capture, sorry for the delay, some conflicts came up in the project for a short period. The setup described did not work on the HoloLens 2 device also, even with the recommendations that were provided from your end of the project.

@CoryAllen

I just wanted to follow-up and see how progress is going to replicate the issues on your end. We still are not seeing results and are now splitting up to try multiple types of settings to see success on the HoloLens 2 device.

I wanted to ask though if the DepthKit software has ran on a HoloLens 2 device independently before with other teams and developers? I just wanna know if there was past success and if they did something specific.

Finally, I was wondering if a virtual meeting would be more preferable to walk through many possibilities, the IPal team needs to make progress on the HoloLens 2 with the DepthKit software. Sorry if any of this is too pressing, but there are deadlines that must be met and we just want to ask for updates on your teams end?

-IPal

@HILABMSU Thanks for sharing video - This should help anyone who is trying to reproduce the issue. From what I can see, it looks like Depthkit assets are all configured properly, so it’s unclear which part of the build or pipeline is causing the Depthkit asset to fail to render in the HoloLens itself.

Because such a small segment of our customers publish to HoloLens, it is not part of our hardware testbed and therefore we can’t guarantee support (as noted in our Unity Platform Support documentation). I have spent some time trying to get familiar with the MRTK, but without a HoloLens 2 in-hand, have not been able to get very far.

That said, we have seen other customers successfully publish directly to HoloLens (see Jessica from @HS-RheinMain’s work here: Towards Mobile Holographic Storytelling at Historic Sites - YouTube).

For one-on-one remote support from someone on our technical team, we provide Priority Support at $750 per hour. If you are interested in this service, please send an email to support@depthkit.tv.

@CoryAllen Thank you for supporting our efforts thus far and I hope you can maybe still have a chance, or have, recreated some form of the issues. We will reach out if we would like to commit to remote support, and I was wondering if you have any other users who have successfully worked with the HoloLens.

I already know you have helped us a ton to break down the issue but I just wanna see if you have a few more resources to see if we can rectify this issue by networking and branching out into the DepthKit Forum community.

@HILABMSU Dusting off this thread to let you know that I just sent you an email with the pre-release patch of the Depthkit Expansion packages for Unity which we think will address some of the issues you have been experiencing when attempting to build to HoloLens. Let us know if this resolves your issue - Thanks!