Empty Multi Cam Display in DepthKit 0.6.2

Hello all,

Our lab’s computer got a refresh from Win 10 to Win 11. I’ve installed DepthKit 0.6.2, updated the 5-kinect system to latest firmware, and followed the docs to get to the calibration step. The single camera view is displaying. The multi cam view is blank grey. Would appreciate any insight or suggestions. Happy to provide more details as well.

Thanks,
William

image1028×854 46 KB

The console log, when clicking calibration, fills up with

[2023-03-14 17:24:33.198] [error] [t=2636] ..\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 178833833 type:Depth
[2023-03-14 17:24:33.200] [error] [t=14316] ..\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 178834777 type:Depth
[2023-03-14 17:24:33.232] [error] [t=2636] ..\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 178867155 type:Depth
[2023-03-14 17:24:33.233] [error] [t=14316] ..\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 178868100 type:Depth
[2023-03-14 17:24:33.262] [error] [t=2636] ..\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 178900500 type:Depth
[2023-03-14 17:24:33.265] [error] [t=14316] ..\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 178901444 type:Depth
2023-03-14 17:24:33.266 ERROR [572] [Depthkit::DepthSensorStage::onSensorEvent@307] Depth frame dropped. Delta: 66667
2023-03-14 17:24:33.266 ERROR [12848] [Depthkit::DepthSensorStage::onSensorEvent@323] Infrared frame dropped. Delta: 66667
2023-03-14 17:24:33.267 ERROR [12104] [Depthkit::DepthSensorStage::onSensorEvent@323] Infrared frame dropped. Delta: 66667
2023-03-14 17:24:33.267 ERROR [2548] [Depthkit::DepthSensorStage::onSensorEvent@307] Depth frame dropped. Delta: 66667
[2023-03-14 17:24:33.296] [error] [t=2636] ..\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 178933822 type:Depth

Computer Details Below

Depthkit v0.6.2 build 0-#3ba03
2023-03-14 17:10:35.733 INFO  [15860] [Depthkit::LogSystemInfo@14] ====================================================
2023-03-14 17:10:35.733 INFO  [15860] [Depthkit::LogSystemInfo@15]                 Depthkit System Information
2023-03-14 17:10:35.733 INFO  [15860] [Depthkit::LogSystemInfo@16] ====================================================
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@29]  CPU Manufacturer: Intel
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@30]  CPU Model: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@31]  Number of logical CPU cores: 36
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@32]  Total System memory (MB): 97997
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@34]
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@52]  GPU Manufacturer: Nvidia
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@53]  GPU Model: Quadro RTX 4000/PCIe/SSE2
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@54]  GPU Driver version: NVIDIA 528.49
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@55]  Total Dedicated GPU memory (MB): 8192
2023-03-14 17:10:35.734 INFO  [15860] [Depthkit::LogSystemInfo@57] ====================================================

Hi, @CornellTech. From the console errors, it seems like the hardware configuration of the sensors needs to be adjusted. If they are each successfully streaming in the single-sensor Record context, then perhaps it’s an issue with the sync cable configuration.

What does the Sensor Configuration panel look like? (To see a full view of this, collapse the Calibration Grade panel to make room in the interface.) Does the number of sensors reported there match the number of sensors you have plugged in? Does the sensor with only a sync cable in it’s Sync Out port appear in that list as being assigned the Controller role (as indicated by the star icon next to it)? Be sure to reference our Depthkit Studio Interface documentation and Sensor Sync documentation as you troubleshoot.

• The number of sensors shown on the sensor configuration panel does match the number of sensors connected to the computer
• The sensor with a single cable in its Sync Out port appears as being assigned the controller role.

The physical sensor setup, cables, and connections are functional on a Win 10 boot drive with DepthKit Studio v0.6.0 (i.e. I can see the all 5 sensors visualized in the multicam view). Still unsure what could be causing the failed rendering in the multicam view on this system. Thanks again for the help.

@CornellTech to confirm sync is not the issue, are you seeing this icon of the interconnected chain links once you start streaming?

Regarding the operating systems, are you saying that with the hardware untouched, booting the same PC into Windows 11 produces the issue, but booting into Windows 10 does not? If that is the case, this could be a driver issue:

• While booted in Windows 11, unplug and reconnect the USB cable from the computer for each sensor, one at a time leaving a few seconds between each one, to ensure the driver for the sensor is loaded properly. If this doesn’t work, you can also try resetting the drivers for the sensors.
• If using a StarTech PCIe card, ensure that a recent Microsoft driver is installed on the Renesas USB host controllers.
• Be sure you’re GPU driver is also the latest available for the RTX 4000 (currently 528.49). The most recent version is confirmed in your Depthkit console.

@CornellTech My team pointed out that if all sensors are streaming in the single-sensor context, that the sensor and PCIe card drivers are likely not the issue, so I have updated my last post.

Also, which color and depth resolution are the sensors set to? Have you tried lower resolutions?

• The icon of the interconnected chain links is blinking Red (not solid red)
• Correct, booting the same PC into Windows 11 produces the issue, but booting into Windows 10 does not.
• (before unplug/replug of USB connections) I tried lower resolutions, still getting the blank view
• (after unplug/replug of USB connections) I selected “start calibration”, I now see an image! However, the view is frozen and unresponsive. The DepthKit Studio buttons and UI are still responsive.

Any thoughts on other drivers that I might try installing/reinstalling?

image1723×1020 96.4 KB

Hi @CornellTech
Playing tag with @CoryAllen here to get you unblocked.

First I want to acknowledge this is a peculiar issue, we have not yet encountered a system that behaves differently on Windows 10 and Windows 11. Thank you for your patience in working through it with us.

Given the type of behavior you are seeing where dropped frames are streaming into the console, and yet the application is reporting all cameras are online and sync is working, it’s very likely that there is one or maybe a few cameras that are plugged into USB ports that are not keeping pace with the required data rates.

To narrow this down the offending cameras, there is a very specific debugging procedure I’d like for you to follow:

• Start by unplugging all but the first two cameras in your sync chain from the computer USB. In your case, this is “sensor 1” which is the sensors that only has a Sync Out cable connected, and then whichever sensor is directly connected to sensor 1 in the chain.
• You should verify that the other 3 sensors then have empty circles (as opposed to dots) in the sensor panels.
• Configure those sensors to 720p color and Narrow Raw depth. This is in order to avoid any performance bottlenecks.
• Start streaming with just those two sensors. For this test, do not “start calibrating” rather just test if the start streaming function works. There is additional overhead in calibration streaming that we want to avoid here at first.
• Observe whether just those two sensors stream effectively.
• If they do, then stop streaming and plug in the USB from the 3rd sensors in the sync chain, ensure it’s circle fills to show it’s online, configure it to 720p/narrow raw, then start streaming again. Then repeat for 4 and then 5 sensors, assuming each streaming session works
• As soon as you find that streaming does not work after adding a sensor, the offending sensor will be identified as the one you just added. We can then diagnose if there is an issue with that USB port or driver, or otherwise something that’s only occurring with Windows 11.
• If they all start streaming successfully. then try going to “start calibration” mode.
• From there if streaming effectively, you can try increasing the resolution to 1080p to see if that triggers the issue.

Please let us know if this diagnosis process is clear, and let us know your results as you go through it.

Hi @James , thanks for the guide. Didn’t get as far into the procedure as I hoped, but it seems that it is a USB port issue (see console output below).

• I unplugged all but the first two sensors in the chain
• Only one sensor shows a solid red dot (this is the second sensor in the chain)
• The other, sensor 1 (first in the chain) is an empty circle
• Streaming button is disabled (I assume since it only detects one sensor)

Console output below

Depthkit v0.6.2 build 0-#3ba03
2023-03-15 15:38:39.631 INFO  [5376] [Depthkit::LogSystemInfo@14] ====================================================
2023-03-15 15:38:39.632 INFO  [5376] [Depthkit::LogSystemInfo@15]                 Depthkit System Information
2023-03-15 15:38:39.632 INFO  [5376] [Depthkit::LogSystemInfo@16] ====================================================
2023-03-15 15:38:39.632 INFO  [5376] [Depthkit::LogSystemInfo@29]  CPU Manufacturer: Intel
2023-03-15 15:38:39.632 INFO  [5376] [Depthkit::LogSystemInfo@30]  CPU Model: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
2023-03-15 15:38:39.632 INFO  [5376] [Depthkit::LogSystemInfo@31]  Number of logical CPU cores: 36
2023-03-15 15:38:39.632 INFO  [5376] [Depthkit::LogSystemInfo@32]  Total System memory (MB): 97997
2023-03-15 15:38:39.632 INFO  [5376] [Depthkit::LogSystemInfo@34]
2023-03-15 15:38:39.632 INFO  [5376] [Depthkit::LogSystemInfo@52]  GPU Manufacturer: Nvidia
2023-03-15 15:38:39.633 INFO  [5376] [Depthkit::LogSystemInfo@53]  GPU Model: Quadro RTX 4000/PCIe/SSE2
2023-03-15 15:38:39.633 INFO  [5376] [Depthkit::LogSystemInfo@54]  GPU Driver version: NVIDIA 528.49
2023-03-15 15:38:39.633 INFO  [5376] [Depthkit::LogSystemInfo@55]  Total Dedicated GPU memory (MB): 8192
2023-03-15 15:38:39.633 INFO  [5376] [Depthkit::LogSystemInfo@57] ====================================================
[2023-03-15 15:38:43.875] [error] [t=5376] ..\src\usbcommand\usbcommand.c (30): TraceLibUsbError(). ..\src\usbcommand\usbcommand.c (155): libusb_get_bos_descriptor(usbcmd->libusb, &bos_desc) returned LIBUSB_ERROR_PIPE in populate_container_id
[2023-03-15 15:38:43.876] [error] [t=5376] ..\src\usbcommand\usbcommand.c (370): find_libusb_device(). Unable to open LIBUSB at index 0
[2023-03-15 15:38:43.876] [error] [t=5376] ..\src\depth_mcu\depth_mcu.c (68): usb_cmd_create(USB_DEVICE_DEPTH_PROCESSOR, device_index, NULL, &depthmcu->usb_cmd) returned failure in depthmcu_create()
[2023-03-15 15:38:43.877] [error] [t=5376] ..\src\sdk\k4a.c (133): depthmcu_create(index, &device->depthmcu) returned failure in k4a_device_open()
2023-03-15 15:38:43.877 WARN  [5376] [Depthkit::IO::DepthSensor::DepthSensorAzureKinect::getConnectedDevices@267] : Failed to open device

Hey @CornellTech
Making progress. Given what you are describing, I would try the following to get the second USB port to recognize:

• Unplug the second sensor USB
• Restart the PC
• Plug it in again once fully rebooted
• Restart Depthkit

If that sequence does not re-recognize the sensor, I would abandon that particular USB port and try another.

As we diagnose USB, can you confirm if you are using this exact StarTech USB PCIe expansion card?

Thanks again for all the help - after tearing through the old set up it turns out our issues were a combination of incorrect driver configurations and unsupported USB extension cables (which we’ve fully replaced now). Our 5 camera setup is finally rebuilt and running smoothly both on our older Win10 OS and the new Win11.