Masking some areas of Depthkit Studio volumetric clip in Unity

Hi Dephtkit,

It’s pretty often we need to mask some objects inside Studio clips - let’s say some large chunks of noise that can’t be removed with refinement or some minor objects that we don’t like. It cannot be cut with boundaries because it’s not on the edge of the clip. So there’s an idea to mask it with a primitive 3D object and apply a shader to it so it would hide a part of Studio clip that is inside this primitive and make it overall transparent. So that all objects behind are visible.

Please advise what shader and settings would you recommend applying to this masking primitive? Currently, it’s needed for URP project but in the future may be for built-in RP as well.

Thanks a lot.

Hi Nikita,

This would be a great feature, unfortunately it is not currently supported out of the box, and would require some shader modifications in order to achieve this.

At a high level, I would recommend modifying the GetSDFCamContribution function within the depthkit.studio\Runtime\Resources\Shaders\DataSource\GenerateVolume.compute file. You should be able check if the position being processed is within a masking primitive there and return from the function early.

If you’re up for it let me know and I can advise further.

Hi Tim,
Yes, I’d love to try this method. Would you like to arrange a call to implement it in real time or provide step-by-step written guidance?

Hi Nikita,
For bespoke 1-on-1 support requests like this which involve new features or custom coding, we offer hourly paid support. If that’s something you’re interested in pursuing please email support@depthkit.tv and we can take it from there.

Hi Tim,
I’m returning again to this highly demanded issue.
What would you recommend to modify here to “check if the position being processed is within a masking primitive there and return from the function early”? For masking, I’ll use a cube from a scene. Would it be easier to assign it in this script as a public GameObject, Collider, Vector3, or something else? What components does this object have to have for masking in this script? I appreciate your help a lot.

void  GetSDFCamContribution(in uint perspectiveIndex, in float3 pos3d, inout float accumulatedWeight, inout float accumulatedSDF)
{
    int flag = VOX_UNKNOWN;
    float sdf; 
    float newWeight = 0.0;
    float normalWeight = 0.0f;
    
    PerspectiveGeometry data = _PerspectiveGeometryData[perspectiveIndex];
    
    if (data.enabled == 0)
        return; //this perspective is disabled

    if (GetSDFContribution(perspectiveIndex, pos3d, flag, sdf, normalWeight))
    {
        if (!CalculateSDFWeight(perspectiveIndex, pos3d, sdf, flag, data, normalWeight, newWeight))
            return;
    }
    else
    {
        // GetSDFContribution return false and the flag values are either VOX_UNKNOWN or VOX_INVALID
        // weight unknown voxels are updated to lie at the _sdfSensitivity on the inside of the surface
        float weightUnknown = lerp(_WeightUnknown, data.weightUnknown, data.overrideWeightUnknown);
        // weight unknown values larger than epsilon are used to keep the surface from getting noisy
        if (flag == VOX_UNKNOWN && weightUnknown >= epsilon)
        {
            sdf = -_SdfSensitivity;
            newWeight = weightUnknown;
        }
        else
        {
            // for voxels flagged invalid or unknown voxels with very low weight known values, the sdf and weight is not updated. 
            // This is so the accumulatedSDF remains at the InvalidSDF value so they can be filtered out
            // return without updating the accumulatedSDF and accumulatedWeight value.
            return;
        }
    }
    // accumulatedSDF is set to InvalidSDF so the first perspective that has a valid update to this must set it
    if (accumulatedSDF >= Invalid_Sdf_compare)
    {
        accumulatedSDF = newWeight * sdf;
        accumulatedWeight = newWeight;
    }
    else
    {
        // subsequent perspectives can accumulate the sdf and weight values
        accumulatedSDF += newWeight * sdf;
        accumulatedWeight += newWeight;
    }
}

Hi NIkita,

This function is given a variable pos3d which represents the world space position that the voxel grid is being calculated for. The simplest way to mask something out would be in the form of an if statement at the top of the function like:

if (pos3d.y <= 0.0f) {
    return;
}

In this case, the statement above would mask out anything that is below the plane y=0 in world space.

A simple extension to this would be to use an Axis Aligned Bounding Box, and return early if the pos3d variable is within the bounds. Unity provides this via the Bounds interface.

The nice thing about AABBs is that they can be represneted by two 3 component vectors; one for the min value and one for the max value. Assuming we have sent some AABB into the shader as minAABB and maxAABB, to check if a point is within the AABB we could rewrite our code as follows:

if (pos3d.x > minAABB.x && pos3d.x < maxAABB.x &&
    pos3d.y > minAABB.y && pos3d.y < maxAABB.y &&
    pos3d.z > minAABB.z && pos3d.z < maxAABB.z) {
    return;
}

I hope this helps!

Hi Tim,
Thank you very much for your further explanation.
So I created four masking cubes, and they have box colliders:

Screen Shot 2022-10-11 at 12.57.43 PM1756×752 126 KB

And I added your code into the script like this:

void  GetSDFCamContribution(in uint perspectiveIndex, in float3 pos3d, inout float accumulatedWeight, inout float accumulatedSDF)
{
    int flag = VOX_UNKNOWN;
    float sdf; 
    float newWeight = 0.0;
    float normalWeight = 0.0f;
    
    PerspectiveGeometry data = _PerspectiveGeometryData[perspectiveIndex];

    if (pos3d.y <= 0.0f) {
        return;
    }

    if (pos3d.x > minAABB.x && pos3d.x < maxAABB.x &&
    pos3d.y > minAABB.y && pos3d.y < maxAABB.y &&
    pos3d.z > minAABB.z && pos3d.z < maxAABB.z) {
    return;
    }

    if (data.enabled == 0)
        return; //this perspective is disabled

    if (GetSDFContribution(perspectiveIndex, pos3d, flag, sdf, normalWeight))
    {
        if (!CalculateSDFWeight(perspectiveIndex, pos3d, sdf, flag, data, normalWeight, newWeight))
            return;
    }
    else
    {
        // GetSDFContribution return false and the flag values are either VOX_UNKNOWN or VOX_INVALID
        // weight unknown voxels are updated to lie at the _sdfSensitivity on the inside of the surface
        float weightUnknown = lerp(_WeightUnknown, data.weightUnknown, data.overrideWeightUnknown);
        // weight unknown values larger than epsilon are used to keep the surface from getting noisy
        if (flag == VOX_UNKNOWN && weightUnknown >= epsilon)
        {
            sdf = -_SdfSensitivity;
            newWeight = weightUnknown;
        }
        else
        {
            // for voxels flagged invalid or unknown voxels with very low weight known values, the sdf and weight is not updated. 
            // This is so the accumulatedSDF remains at the InvalidSDF value so they can be filtered out
            // return without updating the accumulatedSDF and accumulatedWeight value.
            return;
        }
    }
    // accumulatedSDF is set to InvalidSDF so the first perspective that has a valid update to this must set it
    if (accumulatedSDF >= Invalid_Sdf_compare)
    {
        accumulatedSDF = newWeight * sdf;
        accumulatedWeight = newWeight;
    }
    else
    {
        // subsequent perspectives can accumulate the sdf and weight values
        accumulatedSDF += newWeight * sdf;
        accumulatedWeight += newWeight;
    }
}

• Then how to best reference these masking cubes in this script since it’s not MonoBehaviour?
• And how to reference them as pos3d xyz?
  I’ve never worked in C# on this level, so I have no idea about implementing this connection.
  I guess it would be best to set a list of 3D cubes, so I could add whatever amount I’ll need in the future, and they all would be counted by your script as masks. What do you think about this?

Thank you.

Hi Nikita,

The method I described will only work with Axis-Aligned Bounding Boxes, meaning they cannot have any rotation applied to them. This is what allows them to be stored as two 3D points, a min and max. Once rotation is added into the mix, things get more complicated in the shader, and the rotation data would also need to be sent to the shader as well.

In order to send data to the shader, in this case the volume generation compute shader, you’d need to modify StudioMeshSource::SetVolumeGenerationPassProperties, as well as define the values to be sent into the shader within the GenerateVolume.compute file. You can see how this works with the other properties that are being set in SetVolumeGenerationPassProperties.

In order to send in a list of cubes, you would need to define it as a structured buffer, similar to how _PerspectiveGeometryData is done.

At this point, the problem is not Depthkit specific, but Unity in general, so you should be able to get better help on this from Unity tutorials and from Unity’s documentation.