FukaFuka Seika

Obscure Unity Knowledge

Wed, 21 May 2025 00:00:00 GMT

Recently I've come across some weird things about Unity that you might never ever need to know. But I think I'd still put them here in case one day I might need to recall them. These knowledges are small enough to not be worth separate posts, so, well, let them grow in here then.

1. Graphics Buffers are not mapped to persistent physical memory

Whenever you create a ComputeBuffer or GraphicsBuffer, what you have in hand is not always the same chunk of memory. Unity may reallocate the underlying physical memory, but keeps the managed buffer reference intact.

In other words, the managed reference of ComputeBuffer or GraphicsBuffer is always the same, but the underlying physical memory address may change without Unity telling you this. Calling GetNativePtr might give you different results even on the same ComputeBuffer.

I discovered this when I tried to develop a tool that tracks buffer allocation and deallocation. Back then, I thought the physical memory address is persistent (via GetNativePtr), but then I noticed a significant amount of non-released buffers in my tool, despite walking through the debugger it shows that all these buffers' Dispose methods are properly called. Well, when I match a dispose call with a creation call, I checked two things - managed reference address and the native pointer. And this is the moment I found that despite the managed reference stays the same, the native pointer had changed.

Well, what does it mean? It means you should never cache the native pointer. In day-to-day Unity development, this is almost irrelevant. But this is crucial when it comes to Native Render Plugins, where you pass this pointer to your plugin. You must always call GetNativePtr just before calling your plugin code.

... Which means there might be another artifact - two different managed references pointing to the same piece of physical memory. This is fairly common in render graph implementations - if two logical resources have compatible descriptors and their lifespans do not overlap, then we could alias them to the same physical memory. But this is just my educated guess - I haven't encountered a case where this knowledge becomes significant when using ComputeBuffer or GraphicsBuffer.

2. GPU Flush to Zero

When a floating point number is too small, it gets flushed to 0. This is a logical way to combat precision issues related to floats, but it would also induce subtle bugs that could be hard to track if you are not aware of it.

I noticed this behavior when trying to pack data on the CPU side and send the data to the GPU. Essentially, I have 2 halves, and I pack them to float. When the first half is 0, obviously, the packed float becomes extraordinarily small. While this number is preserved when passed to the GPU, the GPU simply thinks it is noise and flushes the entire float to 0, making the second half erroneously 0 as well. In RenderDoc, the symptom would be that the value becomes 0 after stepping through any arithmetic on the said value.

If you want finer technical details, here's the NVIDIA article on denormal flushing.

Simply put, do not use floats to pack values; always use unsigned integers.

Grass Interaction - Part 4

Wed, 22 Jan 2025 00:00:00 GMT

In this fourth part, we will finally add vertex displacement to the grass's vertex shader. We will use shader graph - which should make things a touch easier. But most important logic will be put inside a custom function node which should make it easier to include for readers that prefer a pure code approach.

Pendulum Library

Before diving into the vertex shader straight away, let's create a pendulum library. Recall what we have established in part one:

The 3 cases for grass motion $\theta(t)$ are:

Underdamped : $\theta(t)=\theta_0*e^{-\gamma t}cos(\omega t)$
Overdamped: $\theta(t)= e^{-\gamma t}(A_1e^{\omega t}+A_2e^{-\omega t})$ where $A_1 = \frac{\theta_0}{2} + \frac{\gamma\theta_0}{2\omega}$ and $A_2 = \frac{\theta_0}{2} - \frac{\gamma\theta_0}{2\omega}$
Critically damped: $\theta(t) = \theta_0(1+\gamma t)e^{-\gamma t}$

With:

$\gamma=\frac{b}{2*l_0}$
$\omega_0 = \sqrt{\frac{g-k}{l_0}}$
$\omega = \sqrt{abs(\gamma^2-\omega_0^2)}$

The implementation is straightforward, with some cached computations here and there. We are using half precision because we don't really need single precision for this computation (I've tested both; they aren't noticeably different).

Create a Pendulum.hlsl file and type/copy-paste the following code:

#pragma once  
  
/* Physical simulation of damped pendulums.  
 * * All formulae use "small angle approximation" ( sin(x) = x ) even for big angles.  
 * Which means it is not strictly correct but should be visually correct. */  
#define PENDULUM_GRAVITY 9.8  
  
half pendulum_omega_zero_sqr(half pendulumLen, half elasticity)  
{  
    return (elasticity + PENDULUM_GRAVITY)/pendulumLen;  
}  
  
half pendulum_gamma(half dampingCoef, half pendulumLen)  
{  
    return 0.5*dampingCoef/pendulumLen;  
}  
  
half pendulum_omega(half elasticity, half pendulumLen, half gamma)  
{  
    return sqrt(max(0,pendulum_omega_zero_sqr(pendulumLen, elasticity) - gamma*gamma));  
}  
  
/* In all cases, we assume initial conditions for f(t) -> θ:  
 *  -  f(0) = θ0 (initial angle) *  -  f'(0) = 0 (no initial angular velocity) */  
half pendulum_underdamped(half initialAngle, half t, half elasticity, half pendulumLen, half dampingCoef)  
{  
    half gamma = pendulum_gamma(dampingCoef, pendulumLen);  
    return initialAngle*exp(-t*gamma)*cos(pendulum_omega(elasticity,pendulumLen,gamma)*t);  
}  
  
  
half pendulum_critically_damped(half initialAngle, half t, half elasticity, half pendulumLen, half dampingCoef)  
{  
    half gamma = pendulum_gamma(dampingCoef, pendulumLen);  
    return initialAngle * (1 + gamma * t) * exp(-t*gamma);  
}  
  
  
half pendulum_overdamped(half initialAngle, half t, half elasticity, half pendulumLen, half dampingCoef)  
{  
    half gamma = pendulum_gamma(dampingCoef, pendulumLen);  
    half omega0 = pendulum_omega_zero_sqr(pendulumLen, elasticity);  
    half r = sqrt( abs(gamma * gamma - omega0 * omega0) ) ;  
    half r1 = - gamma + r;  
    half r2 = - gamma - r;  
    half rdiff = r1 - r2;  
    half a = -initialAngle * r2 / rdiff;  
    half b = initialAngle * (1 + r1 / rdiff);  
    return a*exp(r1*t) + b*exp(r2*t);  
}

Current Angle and Displacement

We will now sample the texture we filled in the last part. The texture provides initial angle and elapsed time for our motion equation. We still need elasticity, pendulum length, and damping factor. These will be passed into our vertex displace function and are exposed in the inspector.

Create PlantVertexDisplace.hlsl and start with the following:

#include "Pendulum.hlsl"  
  
/* osParams layout  
 * x: min os y pos (anchor) * y: max os y pos * z: * w:  
 */
void PlantVertexDisplace_float(  
    in float3 osPos,  
    in float3 wsPos,  
    in float4 osParams,  
    in float dampingCoef,  
    in float elasticity,  
    in float4 influenceTex,  
    in float trample,  
    out float3 osDisplacement,  
    out float trampleStr  
)  
{  
    half angle = 0;  
    half initialAngle = length(influenceTex.xy);  
    half t = influenceTex.w;  
    half pendulumLen = osParams.y - osParams.x;  
    #if _PENDULUM_CRITICALLY_DAMPED  
    angle = pendulum_critically_damped(initialAngle, t, elasticity, pendulumLen, dampingCoef);    
    #elif _PENDULUM_OVERDAMPED    
    angle = pendulum_overdamped(initialAngle, t, elasticity, pendulumLen, dampingCoef);    
    #elif _PENDULUM_UNDERDAMPED    
    angle = pendulum_underdamped(initialAngle, t, elasticity, pendulumLen, dampingCoef);  
    #endif
}

Some explanations of the parameters:

osPos is the vertex's Object Space position.
wsPos is the object's World Space position - not the world space position of the vertex. I have tried the latter, and it looked horrible. (You could try it yourself)
osParam, the x component is the object's root in object space, and y is the object's highest point in object space. (float4 is actually an overshot, you could just use float2)
influenceTex is the sampled influence texture's color value.
trample is the trample strength multiplier. We will discuss this later.
osDisplacement is the object space displacement vector.
trampleStr is the resulting trample strength, again, discussed later.

Physically speaking, the equation we should use is determined by the constants, but here we leave the choice to the artist instead - as they probably just want the grass to behave in a particular way by clicking a button, without thinking about how the constants should be adjusted to achieve that look. And to be honest, most of the time you would just want the under-damped case. If that's true, then you could remove the two other cases.

Optionally, you could make pendulum length a standalone exposed constant for even more artistic control (while being less physically correct).

We have the current angle now. The next thing is to turn it into a displacement vector. To do this, we need to rotate the grass blade by the current angle. The axis of this rotation is perpendicular to the plane containing the grass blade and the direction vector (the normalized rg channel of the influence texture).

Here's a diagram illustrating this rotation:

Note that we are not rotating osPos directly, as the root of the object may not be where the object space position is zero. Therefore, we account for the plannar displacement of the grass blade (displacement in xz plane relative to object space origin), and the height of the root configured via osParams.x.

Let's prepare for the rotation:

float3 directionVect = normalize(float3(dir.x, 0.001, dir.y));  
float3 rotationAxis = cross(directionVect, float3(0,1,0));  
rotationAxis.y = max(rotationAxis.y, 0.001);  
rotationAxis = normalize(rotationAxis);  
float3 planarDisplacement = float3(osPos.x, 0, osPos.z);  
float3 grassBlade = osPos - planarDisplacement;

There are plenty of small values here, because we don't want normalization to end up with a zero division. And when they do, the grass simply won't render.

For the rotation itself, we will use Rodrigue's Rotation Formula:

The formula is given by:

$$\mathbf{v}_{\text{rot}} = \mathbf{v} \cos \theta + (\mathbf{k} \times \mathbf{v}) \sin \theta + \mathbf{k} (\mathbf{k} \cdot \mathbf{v}) (1 - \cos \theta)$$

Where:

$\mathbf{v}$ is the original vector to be rotated.
$\mathbf{k}$ is the unit vector along the axis of rotation.
$\theta$ is the angle of rotation.
$\mathbf{v}_{\text{rot}}$ is the resulting rotated vector.

// Rodrigue rotation  
float3 afterRot = grassBlade * cos(angle) + cross(rotationAxis, grassBlade)* sin(angle) + rotationAxis*dot(rotationAxis,grassBlade)*(1-cos(angle));

The osDisplacement is simply:

osDisplacement = grassBlade - afterRot;

// for now, set trample strength to 0
trampleStr = 0;

NOTE: In "standard" implementation of interactive grass, you would typically find multiplying the displacement by either uv.y or normalized height of the vertex. This is unnecessary in this implementation, as the rotation itself already accounts for the "closer to root, less displacement" effect.

This should be enough for setting up our interactive grass. Let's create a grass shader to see it in action!

The Grass Shader

Properties

Create a Lit Shader Graph, and add these properties to the blackboard:

The Scope of _InfluenceTexture must be set to Global so that it correctly references the global texture we set in EIR custom pass.

The _Pendulum enum is set as follows:

NOTE: Some values are not yet exposed since this is a test shader - particularly osParams. You should definitely expose it later.

Plant Vertex Displace custom function

To use the function we created earlier, we will need to use a Custom Function node. But it is usually better to create a Sub Shader Graph for it so that you could drop it in another shader with no need to create all the ins and outs. (You could always copy-paste it, though, but I find creating sub shader graphs easier to manage.)

The process of creating this sub shader graph is tedious, but here it is...

It's literally just a wrapper around a custom function node.

There's a warning about not having a half precision method. You could ignore it since we will use a single precision shader graph, anyway.

Sample Influence Texture

Back to the lit shader graph. The first thing we need to do is to sample the influence texture. Here we need to convert a world position to uv, and the correct formula for it is:

$$uv = (worldPos.xz - worldOrigin.xz)/worldSize$$

worldSize and worldOrigin should actually be read from global floats. I'm cutting some corners here by hard-coding the world size into the shader graph. If you have multiple maps that vary in size, then don't copy me!

The Position node here is a legacy of trial-and-error. I used to take vertex position as wsPos but it turned out horribly by having crazy deformations, therefore I changed it to an Object node so that the whole object samples with the same uv, giving a more consistent result.

Wire Up Plant Vertex Displace

There's not much to explain here, except for the hard-coded osParams.

NOTE: Well, I notice I named a parameter InfluenceTexA here. This is again a legacy naming. I used to have 2 influence textures before I figured out how to pack everything into one texture.

Final Vertex Position

Finally, we could compute the vertex position (object space). This should simply be vertex's object space position plus the displacement. But here, we limit the height of the resulting position so that it never exceeds the original height (more of a precaution).

Testing

Alright, now you could create a material and put it on your grass. If you don't have any... you could create a Tree with 3D Object / Tree.

Next, create an object with EntityInfluenceProvide, set its radius and height accordingly. The following example shows an object with radius 25 and height 5.

Drag around the object and play with the constants to see the interaction in action. You might need to enter play mode to see it smoothly.

Entity Elevation

The previous test shows we are ignoring the elevation of the entity:

And indeed... we haven't used the elevation we calculated in EIR custom pass yet. Let's fix it now.

The idea is to convert the entity's elevation to something comparable to the grass blade's height. The following lines show an normalization approach:

float entityElevation = influenceTex.z;  
// determine press strength (reduce strength when the object is above the grass)  
float rawStr = 1 - (entityElevation - wsPos.y)/max(0.1,osParams.y - osParams.x);  
float str = smoothstep(0.5, 1, rawStr);

And then, after calculating the rotation, we multiply the result by str:

osDisplacement *= str;

This solution is still simplistic and doesn't account for other cases, for example, when the entity is below the grass. But this is only relevant if you have underground sections.

Trample

Trample effect primarily considers the distance between the grass blade and the entity - the closer the entity is, the stronger our trample effect. However, we do not have this distance yet, and we do not have extra channels to store the distance in our texture. We could, of course, add another texture, but performance-wise that would be terrible. So instead, we could guess this distance from initialAngle.

We know initialAngle is clamped between 0 and 90 degrees, and when it is 0, the grass is upright, unaffected by any entity. Also, the function that gives initialAngle is a strictly decreasing function between 0 and entity radius, and for a given entity, it only depends on the distance between the entity and the pixel, therefore we could safely use it to estimate the distance:

float dist = initialAngle/HALF_PI;  
trampleStr = saturate(saturate(str) *  saturate(1 - dist) * trample * (1 - saturate(t)));

OK there's actually a lot more going on here.

str: this is the same as entity elevation's impact. If the entity is high-up then we do not apply any trample.
1 - dist: distance between the entity and the grass blade.
trample: exposed constant for fine tuning.
1 - saturate(t): reduce trample over time.

The added lines produce a result like this:

Rotation and Scale

One final problem we need to solve is that we have ignored the grass blade's transform rotation. For example, this is when the grass has 180 degrees rotation around the y axis:

It is defying the rules of physics by leaning towards the entity.

The solution is simple yet slightly weird in shader graph: multiplying the final vertex position by a rotation matrix. This rotation matrix is part of the model matrix of the grass blade, and we will take the upper-left 3x3 of it, which actually gives rotation and scale transformation. After that, we divide the result by the object's scale or else the scale is applied twice:

Recap

This should provide a good enough base for everything. Feel free to adjust any of the calculations.

Here's the vertex displacement function so far:

#include "Pendulum.hlsl"  
#define HALF_PI 1.5707963  
  
void PlantVertexDisplace_float(  
    in float3 osPos,  
    in float3 wsPos,  
    in float4 osParams,  
    in float dampingCoef,  
    in float elasticity,  
    in float4 influenceTex,  
    in float trample,  
    out float3 osDisplacement,  
    out float trampleStr  
)  
{  
    half angle = 0;  
    half initialAngle = length(influenceTex.xy);  
    half t = influenceTex.w;  
    half pendulumLen = osParams.y - osParams.x;  
    #if _PENDULUM_CRITICALLY_DAMPED  
    angle = pendulum_critically_damped(initialAngle, t, elasticity, pendulumLen, dampingCoef);    
    #elif _PENDULUM_OVERDAMPED    
    angle = pendulum_overdamped(initialAngle, t, elasticity, pendulumLen, dampingCoef);    
    #elif _PENDULUM_UNDERDAMPED    
    angle = pendulum_underdamped(initialAngle, t, elasticity, pendulumLen, dampingCoef);  
    #endif   
    osDisplacement = 0;  
    trampleStr = 0;  
    osPos.y -= osParams.x;  
    half2 dir = initialAngle > 0.0001 ? influenceTex.xy/initialAngle : half2(0,0);  
    float entityElevation = influenceTex.z;  
    // determine press strength (reduce strength when the object is above the grass)  
    float rawStr = 1 - (entityElevation - wsPos.y)/max(0.1,osParams.y - osParams.x);  
    float str = smoothstep(0.5, 1, rawStr);  
      
    float3 directionVect = normalize(float3(dir.x, 0.001, dir.y));  
    float3 rotationAxis = cross(directionVect, float3(0,1,0));  
    rotationAxis.y = max(rotationAxis.y, 0.001);  
    rotationAxis = normalize(rotationAxis);  
    float3 planarDisplacement = float3(osPos.x, 0, osPos.z);  
    float3 grassBlade = osPos - planarDisplacement;  
    /* Instead of lateral displacement, we use rotation.  
     * Rotation can better preserve the shape of the grass.     */    // Rodrigue rotation    float3 afterRot = grassBlade * cos(angle) + cross(rotationAxis, grassBlade)* sin(angle) + rotationAxis*dot(rotationAxis,grassBlade)*(1-cos(angle));  
  
    osDisplacement = (grassBlade - afterRot);  
    osDisplacement *= str;  
      
    float dist = initialAngle/HALF_PI;  
    trampleStr = saturate(saturate(str) *  saturate(1 - dist) * trample * (1 - saturate(t)));  
}

In the next and final part we will address some other issue and performance concerns.

Grass Interaction - Part 3

Thu, 16 Jan 2025 00:00:00 GMT

In this third part, we will set up the Entity Influence system, calculating non-grass-specific data required by the motion equation.

NOTE: All using declarations are hidden for clarity. You should be able to auto-complete them.

Entities and Entity Influence Registry

We start by defining the interface of our entities:

public interface IEntityInfluenceInfo{
	public Vector3 WorldPos {get;}
	public float Radius {get;}
	public Vector3 WorldVel {get;}
	public float Height {get;}
}

C# doesn't have multiple inheritance, thus using interface is the most flexible solution. The interface matches closely what we have defined in the compute shader InfluenceInfo. However, since we need to push the data into a buffer, and an interface is not blittable, we need a struct:

public struct EntityInfluenceInfo  
{  
    public Vector3 worldPos;  
    public float radius;  
    public Vector3 worldVel;  
    public float height;  
    public void CopyFrom(IEntityInfluenceInfo info)  
    {  
        worldPos = info.WorldPos;  
        worldVel = info.WorldVel;  
        radius = info.Radius;  
        height = info.Height;  
    }  
}

NOTE: The order of the fields in EntityInfluenceInfo must match InfluenceInfo declared in HLSL. When transmitting data from CPU to GPU, there's no way to remap fields as the data is essentially raw (void*). Furthermore, the computer does not give you any error message when you get the order wrong except when the ordering changed the size of the struct.

This would seem redundant, having both an interface and a struct, why not just having a struct and store it as a member field? The reason is we want our system to take care of data updating, instead of letting the containing class to call update per-frame (if we only retain EntityInfluenceInfo struct, the containing class will also need to set its values per-frame). This would make the API more friendly.

Next, we will create a class that gathers all the info. For simplicity, we will make it a singleton.

public class EntityInfluenceRegistry  
{  
	private EntityInfluenceRegistry(){}
	private static EntityInfluenceRegistry _instance;  
	public static EntityInfluenceRegistry Instance  
	{  
		get  
		{  
			if (_instance == null)  
		    {  
		        _instance = new EntityInfluenceRegistry();  
			}  
			return _instance;  
		}  
	}
	
}

We need a way to let other components to register entity influence info, as well as removal. Therefore, let's first add an Add and a Remove method. We are also tracking "large entities" so that we can determine the transcription method.

private List<IEntityInfluenceInfo> _influenceInfos = new();
private const float PerfThreshold = 2;  
private int _largeEntityCount = 0;

public int Size => _influenceInfos.Count;

public void Add(IEntityInfluenceInfo info)  
{  
	_influenceInfos.Add(info);  
	if(info.Radius > PerfThreshold) _largeEntityCount++;  
}  
  
public void Remove(IEntityInfluenceInfo info)  
{  
	_influenceInfos.Remove(info);  
	if(info.Radius > PerfThreshold) _largeEntityCount--;  
}

NOTE: For simplicity, the PerfThreshold is a constant field, but you could definitely expose it as an adjustable field in the inspector.

Since we are here, let's solve the transcription method first. Create an enum EIRTranscribeMethod

public enum EIRTranscribeMethod  
{  
    /// small entities, large quantity  
    Method1,  
    /// large entities, small quantity  
    Method2  
}

NOTE: "EIR" is short for "Entity Influence Recording"

Then, if we have any entity that is considered "large", we will use the second method. However, it is totally possible to have one large entity and hundreds of small entities, then both methods would fall short. A possible way to solve it is to split large and small entities to their dedicated buffer, and use the appropriate methods to transcribe them in separated compute passes. The implementation of such approach is left to you. For now, we just naively check if there's any large entity:

public EIRTranscribeMethod Method => _largeEntityCount > 0 ? EIRTranscribeMethod.Method2 : EIRTranscribeMethod.Method1;

We now need a GraphicsBuffer to transfer the data collected. Unlike a C# List, GraphicsBuffer must have a predefined size. We will use 512 elements for now, but if you do actually have more than 512 entities, you would want to increase that.

private GraphicsBuffer _influenceBuffer;  
private EntityInfluenceInfo[] _stagingBuffer = new EntityInfluenceInfo[512];
private int InfluenceInfoSize => sizeof(float) * 8;

public GraphicsBuffer InfluenceInfo  
{  
	get  
	{  
		if (_influenceBuffer == null || !_influenceBuffer.IsValid())  
		{  
			_influenceBuffer?.Dispose();  
		    _influenceBuffer = new GraphicsBuffer(GraphicsBuffer.Target.Structured, GraphicsBuffer.UsageFlags.None, _infos.Length, InfluenceInfoSize);  
		}  
		return _influenceBuffer;  
	}  
}

NOTE: You may notice there are 3 copies of the same data: one in List, one in EntityInfluenceInfo[], and one in GraphicsBuffer. The GraphicsBuffer resides on the GPU hence is different from the other two that are on the CPU side. The EntityInfluenceInfo[] is like a staging buffer, which contains raw data (values are kept), and the List contains the providers of the raw data (therefore only references are kept).

Finally, we add an Update method to copy data from the list to the staging buffer, and bind it to the graphics buffer:

public void Update()  
{  
	for (int i = 0; i < _influenceInfos.Count; ++i)  
	{  
		_stagingBuffer[i].CopyFrom(_influenceInfos[i]);  
	}  
	InfluenceBuffer.SetData(_stagingBuffer);  
}

We will add a test script called EntityInfluenceProvider. This script has a velocity field that moves the object to the right, and whenever it reaches 4096 it resets its horizontal position to 0.

[ExecuteAlways]  
public class EntityInfluenceProvider : MonoBehaviour, IEntityInfluenceInfo  
{  
    public float radius = 5;  
    public float height = 1;  
    public float vel = 10;  
    private Vector3 _lastPos = Vector3.zero;  
    private Vector3 _velocity = Vector3.zero;  
    private void OnEnable()  
    {  
        EntityInfluenceRegistry.Instance.Add(this);  
    }  
  
    private void OnDisable()  
    {  
        EntityInfluenceRegistry.Instance.Remove(this);  
    }  
  
    private void Update()  
    {  
        _velocity = (transform.position - _lastPos) / Time.deltaTime;  
        _lastPos = transform.position;  
        transform.position += Vector3.right * (Time.deltaTime * vel);  
        if (transform.position.x > 4096)  
        {  
            var vector3 = transform.position;  
            vector3.x = 0;  
            transform.position = vector3;  
        }  
    }  
    public Vector3 WorldPos => transform.position;  
    public Vector3 WorldVel => _velocity;  
    public float Radius => radius;  
    public float Height => height;  
}

Custom Pass

This is the part where HDRP and URP part ways - they offer different API for doing this. Though in a nutshell, what we will be doing is setup some compute dispatch calls, and it should be fairly easy to do in either pipelines.

public class EntityInfluenceRecordingPass : CustomPass  
{  
    // Shader parameters 
    private static readonly int InfluenceTexture = Shader.PropertyToID("_InfluenceTexture");  
    private static readonly int InfluenceTextureParams = Shader.PropertyToID("_InfluenceTextureParams");  
    private static readonly int InfluenceInfo = Shader.PropertyToID("_InfluenceInfo");  
    private static readonly int InfluenceInfoCount = Shader.PropertyToID("_InfluenceInfoCount");  
    private static readonly int Time1 = Shader.PropertyToID("_TimeParams");  
    
    // Serialized properties
    public ComputeShader scriptoriumShader;  
    public float coverageSize;  
    public Vector2 offset;  
    public int textureSize;  
      
    // Under the hood stuff
    private RenderTexture _influenceTexture;  
    private Vector4 _influenceTextureParams;  
    private bool _initSucceeded = false;
    // Kernels  
    private int _updateTimeKernel;  
    private int _transcribeKernel1;  
    private int _transcribeKernel2;
}

Next, we will initialize all the required resources in the Setup method:

        protected override void Setup(ScriptableRenderContext renderContext, CommandBuffer cmd)  
        {  
            // sanity check
            Assert.IsNotNull(scriptoriumShader, "EIR: ScriptoriumShader is null");  
            Assert.IsTrue(coverageSize != 0, "EIR: CoverageSize is 0");  
            Assert.IsTrue(textureSize != 0, "EIR: TextureSize is 0");  
            base.Setup(renderContext, cmd);  
            // texture creation
            var descriptor = new RenderTextureDescriptor(textureSize, textureSize)  
            {  
                graphicsFormat = GraphicsFormat.R16G16B16A16_SFloat,  
                depthBufferBits = 0,  
                depthStencilFormat = GraphicsFormat.None,  
                enableRandomWrite = true,  
            };  
  
            _influenceTexture = new RenderTexture(descriptor);  
            _influenceTexture.Create();
            // finding kernels
            _updateTimeKernel = scriptoriumShader.FindKernel("UpdateTime");  
            _transcribeKernel1 = scriptoriumShader.FindKernel("TranscribeMethod1");  
            _transcribeKernel2 = scriptoriumShader.FindKernel("TranscribeMethod2");  
  
            // set params only once in build to reduce overhead  
#if !UNITY_EDITOR  
            SetParams(cmd);  
#endif  
            _initSucceeded = true;  
        }  
  
        private void SetKernelParams(CommandBuffer cmd, int kernelID)  
        {  
            cmd.SetComputeBufferParam(scriptoriumShader,kernelID, InfluenceInfo, EntityInfluenceRegistry.Instance.InfluenceBuffer);  
            cmd.SetComputeTextureParam(scriptoriumShader, kernelID, InfluenceTexture, _influenceTexture);  
        }  
  
        private void SetParams(CommandBuffer cmd)  
        {  
            _influenceTextureParams = new Vector4(coverageSize / textureSize, textureSize, offset.x, offset.y);  
            cmd.SetGlobalTexture(InfluenceTexture, _influenceTexture);  
            SetKernelParams(cmd, _transcribeKernel1);  
            SetKernelParams(cmd, _transcribeKernel2);  
            SetKernelParams(cmd, _updateTimeKernel);  
            cmd.SetComputeFloatParam(scriptoriumShader, LerpFactor, lerpFactor);  
            cmd.SetGlobalVector(InfluenceTextureParams, _influenceTextureParams);  
        }

NOTE: We use CommandBuffer.Set... methods because they generally have less overhead compared to Shader.Set... methods. The exact implementation is hidden from us, but it is possible that Shader.Set... methods need to create onetime-submit command buffers. Benchmarking showed CommandBuffer.Set... methods are 1.5x-2x faster than equivalent Shader.Set... methods.

NOTE: Resource bindings only need to be set once. This is why I put SetParams call in Setup rather than Execute, since they won't change in the built game. In Editor it is called per-frame since we might want to adjust constants in real-time.

And now in Execute, we trigger the update on EntityInfluenceRegistry and dispatch the two computes:

        protected override void Execute(CustomPassContext ctx)  
        {  
            if (!_initSucceeded) return;  
            base.Execute(ctx);  

			// trigger registry update
            EntityInfluenceRegistry.Instance.Update();  
            // set influence info count (array bound)
            scriptoriumShader.SetInt(InfluenceInfoCount, EntityInfluenceRegistry.Instance.Size);  
            // update time
            scriptoriumShader.SetVector(Time1, new Vector4(Time.deltaTime, 0, 0, 0));  
            var cmd = ctx.cmd;  
            // In Editor mode, set params every frame to support param adjustments.  
#if UNITY_EDITOR  
            SetParams(cmd);  
#endif  
            cmd.BeginSample("EntityInfluence Time Update");  
            cmd.DispatchCompute(scriptoriumShader, _updateTimeKernel,  
                textureSize / 8, textureSize / 8, 1);  
            cmd.EndSample("EntityInfluence Time Update"); 
             
            cmd.BeginSample("EntityInfluence Transcribe");  
            cmd.DispatchCompute(scriptoriumShader,  
                EntityInfluenceRegistry.Instance.Method == EIRTranscribeMethod.Method1  
                    ? _transcribeKernel1  
                    : _transcribeKernel2,  
                textureSize / 8, textureSize / 8, 1);  
            cmd.EndSample("EntityInfluence Transcribe");  
        }

Finally in Cleanup, release the texture:

protected override void Cleanup()  
{  
    _influenceTexture.Release();  
}

That concludes the custom pass. Create a Custom Pass Volume in the Editor and hook up the EIR pass:

Debugging

Let's setup a simple shader to visualize our influence texture:

It samples the global texture _InfluenceTexture, and that's it. Remember to set the scope of the influence texture parameter to Global or else it won't work.

Create a material from this shader, add an UI Image and assign the image with the material created.

Now create an empty object and add an EntityInfluenceProvider - the test script we created earlier. You should see a circle in the UI image:

Here I set the radius to 50 so that it could be seen clearly. If the velocity of the object is not zero then you should see it leaving a trail behind:

Even if you disable the object, the trail should stay. You could modify the visualizer so that we take time into account:

Play around with the setup and check if everything work as expected so far.

In the next part we will finish the final piece of the puzzle - vertex displacement in shader.

Grass Interaction - Part 2

Mon, 13 Jan 2025 00:00:00 GMT

In the second part, we are going to set up the shader part of the Entity Influence system. In this system, we will pre-compute some of the required parameters so that the vertex shader can later read them and use the appropriate motion equation to calculate the angle.

Since we might have different grass that react differently based on grass-specific parameters, the design philosophy is to separate entity influence and grass - we do not want to see any grass-specific constants in the entity influence system.

NOTE: This part expects you to know how Compute Shaders work, and some knowledge of the render pipeline you are working with. You should at least know how to create a CustomPass or ScriptableRenderPass in your render pipeline. We will work in HDRP and use CustomPass here. I will try my best to abstract away the render pipeline specific features (which also future-proofs API changes). Feel free to make necessary adjustments according to the API you are given.

Overview of the Entity Influence System

Each entity will have an EntityInfoProvider attached to them. The script will inform EntityInfoRegistry about the current position, radius, height, etc. of the entity.

Each frame, EntityInfoRegistry packs these infos into a GraphicsBuffer. A pass will bind this buffer to a compute shader as well as other globally tweakable parameters, and dispatch the compute shader. The compute shader will calculate the influence of each entity, and write the results to Entity Influence Texture(s), which will later be used in vertex shaders.

We are using textures because we have multiple entities that may influence grass. Sampling texture is going to be better than iterating through each entity in vertex shader. If you only have a very limited amount of entities, then it is possible to pass the buffer directly to the vertex shader and do all the calculations there - at the cost of vertex shader performance.

We defer part of the calculation to a compute shader because that provides us a way to harness the power of Async Compute. If we were to do everything in grass' vertex shader, then everything will be done on Graphics queue. Also, even if we don't use async compute, as the compute shader really doesn't depend on anything, the driver can schedule it more effectively. The drawback is that we need texture to store intermediate results, therefore it isn't an obvious gain, but a memory-speed trade-off.

Inputs and Outputs

Let's review the theories, and determine what data do we need to establish the systems.

$$\theta_0(d) = \frac{\pi}{2}-arctan(\frac{2h_0d}{R_0^2-d^2})$$

For initial angle calculation, we need the constant $c$ and distance $d$. For $c$, we need the height $h_0$ and radius $R_0$ of the instigator, which are simply 2 tweakable floats.

As for $d$, by definition, is the distance between the center of the instigator (therefore, instigator's transform position), and the root of the grass. Now, the root of the grass may seem like a grass-specific data, but we could simply convert pixel position to world position. Though if we do so, the resolution of the texture will determine the accuracy of our simulation.

As for the motion equation, we take the underdamped case as an example:

$$\theta(t)=\theta_0*e^{-\gamma t}cos(\omega t)$$

With:

$\gamma=\frac{b}{2*l_0}$
$\omega_0 = \sqrt{\frac{g-k}{l_0}}$
$\omega = \sqrt{abs(\gamma^2-\omega_0^2)}$

We need grass length $l_0$, elasticity constant $k$, damping coefficient $b$, and elapsed time $t$. The length, elasticity constant, and damping coefficient are all grass-specific, so we only need to store elapsed time. The definition of elapsed time, in our case, is the time when grass leaves the influence of an entity. There are 2 ways to store this elapsed time:

Store the timestamp t when the grass is last in an entity's influence, then in the vertex shader, we can calculate the elapsed time with _Time.y - t.
Store the elapsed time dt directly. Each frame we increase the elapsed time dt by unity_DeltaTime.x, and set dt = 0 if the grass is in any entity's influence.

To avoid potential overflow, we will adopt the second approach.

Finally, calculating the angle is not enough. We still need to know in which direction should the grass bend. That means we need a Vector2 to represent the direction. We also need the entity's bottom surface elevation, because if an entity is above the grass, then it should exert no influence on the grass. (remember the height of the grass is only available in vertex shader, so we can't pre-calculate it!)

To summarize, this will be the info that we must produce in our Entity Influence System:

Direction of the influence (Vector2, or 2 floats)
Elevation of the bottom surface of the entity (float)
Initial angle (float)
Elapsed time (float)

"Blast! That's 5 floats in total! That means we need 2 textures to fit them all in!" - Very perceptive! But here's a trick we can do... see, when we need the direction, we do not need the magnitude of the Vector2. Later in the vertex shader, we would still call normalize on it. That means the magnitude of the vector can store another float... let it be the initial angle then!

NOTE: The choice of storing initial angle in magnitude is actually deliberate. The bottom surface elevation can be negative, so we can't store it in magnitude. Elapsed time is positive, but if we store it in the magnitude, whenever we update influence, since time is reset to 0, it will set the direction to 0 as well. The initial angle does not have this side effect. Therefore, the value that is the most suitable for storing in magnitude is the initial angle.

Writing the Compute Shader

Let's start by declaring the inputs and outputs.

Influence Info

The buffer containing entity influence info will have 2 float3 and 2 float. We pack them into a InfluenceInfo struct and use a StructuredBuffer to communicate between CPU and GPU. The _InfluenceInfoCount tracks the effective length of the buffer, and is equal to the number of entities registered. We will be allocating a large enough buffer for _InfluenceInfo (e.g. 256 elements), and use the count to delimit valid data range.

struct InfluenceInfo  
{  
    float3 worldPosition;  
    float radius;  
    float3 worldVelocity;  
    float height;  
};  
  
// raw influence info data  
StructuredBuffer<InfluenceInfo> _InfluenceInfo;

// valid length of influence info  
int _InfluenceInfoCount;

NOTE: The order of the fields in InfluenceInfo is important due to alignment requirements. In GPU shaders, buffers must adhere to specific alignment rules, which means types like float3 may get padded to ensure they fit properly in memory. If you don't account for this, you may end up allocating more memory than necessary.

The struct above, with consecutive float3 and float packed into one 16-byte block, is equivalent to 32 bytes in total. However, if you order your struct like this:
struct InfluenceInfo  
{  
  float3 worldPosition;  
  float3 worldVelocity;  
  float radius;  
  float height;  
}; 
It will take up 48 bytes (3 float4), because the first float3 cannot be packed with the second float3 due to alignment rules, and as a result, it gets its own 16-byte block, with 4 bytes unused:

The first float3 occupies 16 bytes, but the remaining 4 bytes are unused (due to padding).

The second float3 and float fields fit into the second 16-byte block without wasting any space.

The final float gets its own 16-byte block, with 12 bytes wasted.

As a general rule, it's good practice to ensure that fields in your struct are ordered in such a way that they can fit neatly into 16-byte blocks without any fields "sitting between two blocks." This will help minimize wasted memory and ensure more efficient data storage.

Texture and Other Constants

We will be using a texture of format R16G16B16A16_SFloat which means 16-bit (half) per channel. Our calculation doesn't need to be extremely precise so half is enough. You could eventually use integer formats but then you need to write conversion code. For simplicity, we will stick with the float format.

RWTexture2D<half4> _InfluenceTexture;

The layout of a pixel looks like this:

R: direction vector x component
G: direction vector y component
B: the elevation of the bottom surface of the object
A: elapsed time

Also, don't forget that the magnitude of the direction vector is the initial angle!

Since we will be doing some conversion between pixels and world space, we need to know how much space does the texture cover, its world space offset, as well as the texture's size. We will pack all these info into a float4:

 /**    
 * x: 1 px = x units in world space (world size/texture size)  
 * y: texture px size (square texture) 
 * z: WS origin.x (origin is the lower left corner) 
 * w: WS origin.y 
 */
 float4 _InfluenceTextureParams;

Finally we add the _TimeParams:

  
 /**  
 * x: Time.deltaTime 
 */
 float4 _TimeParams;

And that's all of the ins and outs of the compute shader. Your shader should look like this:

struct InfluenceInfo  
{  
    float3 worldPosition;  
    float radius;  
    float3 worldVelocity;  
    float enttConstant;  
};  
  
// raw influence info data  
StructuredBuffer<InfluenceInfo> _InfluenceInfo;  
  
// valid length of influence info  
int _InfluenceInfoCount;  
  
/** Texture to transcribe influence info to.  
 * Data layout of the influence texture *  
 * R:  N.x  \  initial angle
 * G:  N.y  /   
 * B:  bottom surface elevation
 * A:  elapsed time since release 
 */
 RWTexture2D<half4> _InfluenceTexture;  
  
/**    
 * x: 1 px = x units in world space (world size/texture size)  
 * y: texture px size (square texture) 
 * z: WS origin.x (origin is the lower left corner) 
 * w: WS origin.y 
 */
float4 _InfluenceTextureParams;  
  
/**  
 * x: Time.deltaTime 
 */
 float4 _TimeParams;

Time Update

Let's start with the easy one. As said before, we need to update the elapsed time each frame. This is simply adding delta time to the time channel of the influence texture, nothing fancy.

#pragma kernel UpdateTime

[numthreads(8,8,1)]  
void UpdateTime(uint3 id: SV_DispatchThreadID)  
{  
    uint2 px = id.xy;  
    // ensure the pixel is within bounds
    if (px.x >= _InfluenceTextureParams.y || px.y >= _InfluenceTextureParams.y) return;  
    half4 pxData = _InfluenceTexture[px];  
    half t = pxData.w;  
    t += _TimeParams.x;  
    _InfluenceTexture[px] = half4(pxData.xyz, t);  
}

We are essentially updating every pixel here, regardless of whether the pixel is in an entity's influence. This is because we will later transcribe entity influence and that will overwrite relevant pixel's time.

Entity Influence Transcription

Now we need to take care of the pixels that fall into an entity's influence.

We will implement 2 different methods for Entity Influence Transcription, each with some performance concerns and we need to switch from one to the other depending on the entities.

Iterate Per Entity: For each compute thread we deal with one entity, calculate the position for every pixel that fall under the influence of that entity, and update them. This method is better when entities are small but numerous.
Iterate Per Pixel: For each compute thread we deal with one pixel, and we iterate through all the entities. If we find that the pixel falls in the influence of any of them, we update the pixel with corresponding calculations. This method is better when entities are large but not numerous.

The size of the entity depends on its pixel coverage. An entity having a radius 5 units on a texture with 1 px = 1 unit is equivalent to an entity having a radius 1 on a texture with 1 px = 0.2 unit.

Texture Update Logic

Let's tackle the part in common first: Given a pixel and an entity, does the pixel falls into the entity's influence?

The logic is simple, since pixel is in texture space, and entity position is in world space, we need to compare them in the same space.

If we compare in pixel, we need to convert entity's position and its radius to pixel.
If we compare in world units, we need to convert pixel's position to world space.

The latter requires one less conversion so we will compare them in world units:

$$PixelWorldPos = TextureOriginWS + PixelCoord*PixelToWorldRatio$$

And if the pixel is in the entity's influence, then its distance to the entity must be smaller than the entity's radius:

half2 PixelToWorldPos(in uint2 px){
	half ratio = _InfluenceTextureParams.x;
    half offset = _InfluenceTextureParams.zw;
	return offset + px * ratio;
}

bool IsPixelInInfluence(in half2 pxWorldPos, in InfluenceInfo entity){
	return distance(pxWorldPos , entity.worldPosition.xz) < entity.radius;
}

We separate the conversion and the check because later we will also need to use the pixel's world position. We could convert once and reuse the result in other function calls.

The next question is, if we found a pixel in an entity's influence, how do we update its color values?

For A channel, we need to reset the elapsed time to 0 as the entity's influence will cause the grass to be in a static state.
For RG (normalized direction), we calculate the displacement from the entity's center to the pixel, in world space.
For the B channel, we calculate via: $$elevation = entityY-h_0+h(d)$$ for the bottom surface elevation.
For the magnitude of RG (initial angle), we will simply apply the formula we saw in part 1 of the series to calculate the initial angle.

#define HALF_PI 1.5707963
  
half EllipticElevation(half x2, half a2, half b)  
{  
    return b *( sqrt(max(0,1 - x2 / a2 )) + 1 );  
}  

void UpdateInfluence(in uint2 px, in half2 pxWorldPos, in InfluenceInfo entity){  
    
    half2 displacement = pxWorldPos - entity.worldPosition.xz;  
    half d = length(displacement);  
    half angle = length(originalData.xy);  
      
    displacement = SafeNormalize(originalData.xy);
      
    half R2 = (entity.radius * entity.radius);  
    half h = entity.height;  
    half d2 = d * d;  
    
    // calculate initial angle  
    half initialAngle = HALF_PI - atan( 2 * entity.height * d / max( R2 - d2, 0.00001);
                            
    // calculate elevation  
    half entityElevation = entity.worldPosition.y - EllipticElevation(d2, R2, h);   
    _InfluenceTexture[px] = half4(displacement * max(0.01,initialAngle), entityElevation, 0);  
}

You might notice that the velocity of the entity is not used. We will be using it in a later chapter to adjust the simulation for moving entities.

NOTE: Strictly speaking, setting time to 0 will actually cause grass blade to "teleport" to the target position. You could try to lerp the time to 0, essentially "reverse" the motion. But that would look particularly weird when the grass is underdamped.

And that's it! We will now use them in the actual kernels.

Transcription Method 1: Iterate Per Entity

In this method, each thread will process an entity. We are using SV_GroupIndex as it directly gives the index in _InfluenceInfo buffer. Let's start by writing some boilerplate code:

#pragma kernel TranscribeMethod1

[numthreads(8,8,1)]  
void TranscribeMethod1(uint id: SV_GroupIndex)  
{  
    if (id >= _InfluenceInfoCount) return;  
    InfluenceInfo info = _InfluenceInfo[id];  
    //...
}

In the next step, we need to determine which pixels are in the entity's influence. To do this, we will convert the entity's position and radius into pixels.

#pragma kernel TranscribeMethod1

[numthreads(8,8,1)]  
void TranscribeMethod1(uint id: SV_GroupIndex)  
{  
    if (id >= _InfluenceInfoCount) return;  
    InfluenceInfo info = _InfluenceInfo[id];  

	half radius = info.radius;  
	int pxRadius = radius / _InfluenceTextureParams.x;  
	uint2 infoPxPos = (info.worldPosition.xz - _InfluenceTextureParams.zw) / _InfluenceTextureParams.x;  
	
	for (int x = -pxRadius; x <= pxRadius; x++)  
	{  
	    for (int y = -pxRadius; y <= pxRadius; y++)  
	    {
			int2 px = infoPxPos + int2(x, y); 
		    //...
	    }
	}
}

The two for loops cover a square instead of a circle, so we still need to check if a pixel is in the circle by calling IsPixelInInfluence. After that, if we find the pixel is indeed in influence, we call UpdateInfluence.

#pragma kernel TranscribeMethod1

[numthreads(8,8,1)]  
void TranscribeMethod1(uint id: SV_GroupIndex)  
{  
    if (id >= _InfluenceInfoCount) return;  
    InfluenceInfo info = _InfluenceInfo[id];  
    half radius = info.radius;  
    int pxRadius = radius / _InfluenceTextureParams.x;  
    uint2 infoPxPos = (info.worldPosition.xz - _InfluenceTextureParams.zw) / _InfluenceTextureParams.x;  
    for (int x = -pxRadius; x <= pxRadius; x++)  
    {  
        for (int y = -pxRadius; y <= pxRadius; y++)  
        {  
            int2 px = infoPxPos + int2(x, y);  
            half2 thisPxWsPos = PixelToWorldPos(px);  
            if (IsPixelInInfluence(thisPxWsPos, info))  
            {  
                UpdateInfluence(px, thisPxWsPos, info);  
            }  
        }  
    }  
}

Transcription Method 2: Iterate Per Pixel

The idea is similar, but this time we simply go through all the pixels in the texture and check if it is in any of the entity's influence. We are using SV_DispatchThreadID which gives us pixel coordinates.

#pragma kernel TranscribeMethod2

[numthreads(8,8,1)]  
void TranscribeMethod2(uint3 id: SV_DispatchThreadID)  
{  
    uint2 px = id.xy;  
    if (px.x >= _InfluenceTextureParams.y || px.y >= _InfluenceTextureParams.y) return;  
    // Iterate over each entity  
    for (int i = 0; i < _InfluenceInfoCount; i++)  
    {  
        InfluenceInfo info = _InfluenceInfo[i];  
        half2 thisPxWsPos = PixelToWorldPos(px);  
        if (IsPixelInInfluence(thisPxWsPos, info))  
        {  
            UpdateInfluence(px, thisPxWsPos, info);  
            return;  
        }  
    }  
}

And that concludes our compute shader (for now).

In the next part, we will setup the CPU side of the entity influence system.

Grass Interaction - Part 1

Sun, 12 Jan 2025 00:00:00 GMT

Grass Interaction, a staple in any game that has grass, and an inevitable when making open-world games (unless you are making a scorched land). However, most of the tutorials out there seem to be a little limited when it comes to make natural-looking grass interaction. Most would stop at the trample effect - which is ok for basic use cases. The problem becomes more complex when you add NPCs or making a bouncing effect.

To fill the gap, this series aim to provide a slightly overkill simulation of grass interaction - by overkill I mean it won't be frame-rate friendly since we will be doing lots of expensive calculations. You could simplify the calculations depending on your project need.

Overview

We will achieve the following effects:

Grass is trampled when very near an instigator.
Grass curves outwards when near an instigator.
Grass will gradually restore its position after leaving the influence of an instigator.
Grass may show some oscillation during the restoration process.
Each grass can have its own elasticity constant and damping coefficient.
Support multiple instigators.

We will improve grass interaction based on the most basic method - adding a displacement in the vertex shader. There are plenty of other methods that provide better simulations, for example, using bone animation, suitable for more detailed plants. The technique we are using assumes the following conditions:

Newtonian physics applies - we only deal with macro, low-velocity objects.
Grass width is significantly smaller than instigators - like grass vs human in real life. If your game is about grass vs cricket, then this technique is suboptimal.
No precise physical simulation needed - we will make lots of simplifications and compromises to achieve a visually correct result, but these simplifications will make the simulation not strictly physically correct.

The series is divided into 5 parts:

Part 1 (this post): physical abstraction of grass, instigators, and relevant theories.
Part 2: Calculating influence of instigators.
Part 3: Send influence data to the GPU.
Part 4: Applying vertex displacement.
Part 5: Adjustments and Refinements.

NOTE: The theoretical part requires you to have some college level calculus and Newtonian physics knowledge to understand completely. In fact, it would be very similar to a first year physics course. But it is OK if you don't understand everything - it is not required to achieve the effects mentioned above.

Now let's dive into the theory.

Physical Abstraction of Grass

Formulation of the System

We want to reduce the grass interaction problem to a well-studied physical system. What could it be, then?

Empirically, the movement of grass resembles a mass attached to an elastic rod. One end of the elastic rod is attached to the ground and is immobile, the other end is attached to the mass. The elastic rod can be bent but cannot be compressed. For simplicity, the mass of the elastic rod is ignored.

Though this simplification is still too complex for us, due to the rod being bendable and changing the straight distance between the mass and the root. What we can do is to assume the rod is non-bendable (rigid), but still providing elastic force $ F_s $. Imagine there is a hidden contraption at the root of the grass, driven by a spring, whenever the grass isn't upright, the contraption provides the elastic force to restore the upright position.

If you now flip your monitor upside down, you might recognize this diagram in other literature: This is very similar to a damped/driven pendulum system, except it is upside down, and that is indeed a well-studied physical system.

Force Analysis

Our goal is to obtain an analytical expression $\theta(t)$ that, given elapsed time $t$, we obtain the signed angle $\theta$.

Assume the signed angle between the up direction and the rigid rod is $\theta$:

$Fs$ is the elastic force perpendicular to the rigid rod and always pointing towards the upright position. Similar to Hooke's Law, this force is proportional to angle $\theta$. Hence, $$ Fs=mk\theta$$ where $k$ is the elasticity of the grass.
The gravity $G$ can be decomposed into a force that goes along the rigid rod $Gy$ and another that is perpendicular to the rigid rod, in the opposite direction of the elastic force $Gx$. $Gy$ is balanced out by the supportive force of the rigid rod, then it only leaves $Gx$ our point of interest. With a little trigonometry, we can find $$Gx = sin(\theta)mg$$
Another damping force $D$ proportional to the angular velocity of the mass: $$D = \frac{d\theta}{dt}bm$$ where $b$ is the damping coefficient. This force opposes the direction of the velocity.

NOTE: Usually we do not multiply damping force by mass. But since both $b$ and $m$ are invariant during the simulation, $b*m$ can be seen as just a constant of the grass that replaces the conventional damping coefficient.

Therefore, with the 2nd law:

$$ a = \frac{d^2x}{dt^2}=k*\theta- sin(\theta)*g - \frac{d\theta}{dt}*b $$

We know that linear displacement can be expressed as $dx = l_0*d\theta$ where $l_0$ is the length of the rod. Derive the expression 2 times, then we will have:

$$ \frac{d^2x}{dt^2} = l_0*\frac{d^2\theta}{dt^2} = k*\theta- sin(\theta)*g - \frac{d\theta}{dt}*b $$

Simplify the equation and we have a familiar form:

$$ l_0*\frac{d^2\theta}{dt^2} + b*\frac{d\theta}{dt} + gsin(\theta) - k\theta = 0 $$

This equation is nonlinear because of the $g*sin(\theta)$ term, and we have to do a classic trick to make it solvable analytically - Small Angle Approximation.

The following graph shows $f(x) = sin(x)$ (green), $f(x) = x$ (red) and $f(x)=x-\frac{x^3}{6}$ (blue).

Around 28 degrees (0.5 radians), we can see that $f(x) = x$ is very close to $f(x) = sin(x)$. The Taylor expansion with 2 terms $f(x)=x-\frac{x^3}{6}$ gives a better approximation, but it doesn't make the equation easier to solve.

However, remember that we do not need a strictly physical simulation, we just need it to be visually correct. So we will use $x$ to approximate $sin(x)$ in all cases. Our equation now becomes:

$$ \frac{d^2\theta}{dt^2} + \frac{b}{l_0}\frac{d\theta}{dt} + \frac{g-k}{l_0}\theta = 0 $$

We now assume:

$\gamma=\frac{b}{2*l_0}$
$\omega_0 = \sqrt{\frac{g-k}{l_0}}$

then we can simplify the equation to an even more familiar form:

$$ \frac{d^2\theta}{dt^2} + 2\gamma*\frac{d\theta}{dt} + \omega_0^2*\theta = 0 $$

Solving the Differential Equation

We will now use the Exponential Guess method:

Guess an exponential solution of the form $Ae^{\alpha t}$
Derive the condition $\alpha$ must satisfy in order for $Ae^{\alpha t}$ to be a solution of the equation.
Solve the condition for values of $α_1,...,α_n$
If the $α_i$ are unique, write general solution as $x(t) = A_1e^{\alpha_1 t}+A_2e^{\alpha_2t} + ··· + A_ne^{\alpha_nt}$.

If $\theta(t)$ is of form $Ae^{\alpha t}$ then:

$$ A(\alpha^2+2\gamma\alpha+\omega_0^2)e^{\alpha t}=0 $$

Which means:

$$ \alpha^2+2\gamma\alpha+\omega_0^2=0 $$

And the solution for $\alpha$ is

$$ \alpha = -2\gamma \pm \sqrt{\gamma^2-\omega_0^2} $$

Depending on the value of $\gamma$ and $\omega_0$, we can have 3 different cases:

$\omega_0>\gamma$ : weak damping or underdamped
$\omega_0=\gamma$ : critical damping or critically damped
$\omega_0<\gamma$: strong damping or overdamped

we also note:

$$ \omega = \sqrt{abs(\gamma^2-\omega_0^2)} $$

Now to properly solve the equations we need some initial conditions. In our grass simulation, we assume the grass has no initial velocity when it leaves an instigator's influence, and we can write the initial conditions as follows:

$\theta(0)=\theta_0$ (initial angle)
$\frac{d\theta(0)}{dt}=0$ (no initial velocity)

Underdamped

When underdamped, the solution to our equation is complex:

$$ \theta(t)=A_1e^{\alpha_1 t}+A_2e^{\alpha_2 t} = e^{-\gamma t}(A_1e^{i\omega t}+A_2e^{-i\omega t}) $$

But since we need a real solution, we can just take the real part (recall Euler's formula $e^{it}=cos(t)+isin(t)$ ):

$$ \theta(t)=e^{-\gamma t}(A_1cos(\omega t)+A_2sin(\omega t)) $$

since cos and sin are only different in phase, we can further rewrite the previous solution as

$$ \theta(t)=A_0*e^{-\gamma t}cos(\omega t-\phi) $$

We only need to use initial conditions to solve $A_0$ and $\phi$ now.

given $\frac{d\theta(0)}{dt}=0$ we have $\phi = 0$
given $\theta(0)=\theta_0$ we have $A_0 = \theta_0$

Hence, for underdamped case, we have

$$ \theta(t)=\theta_0*e^{-\gamma t}cos(\omega t) $$

Overdamped

Similar to underdamped, but without the imaginary:

$$ \theta(t)=A_1e^{\alpha_1 t}+A_2e^{\alpha_2 t} = e^{-\gamma t}(A_1e^{\omega t}+A_2e^{-\omega t}) $$

with the initial conditions:

given $\theta(0)=\theta_0$ we have $A_1+A_2 = \theta_0$
given $\frac{d\theta(0)}{dt}=0$ we have $A_1-A_2=\frac{\gamma\theta_0}{\omega}$

Solve them and we get

$$ A_1 = \frac{\theta_0}{2} + \frac{\gamma\theta_0}{2\omega} $$

and

$$ A_2 = \frac{\theta_0}{2} - \frac{\gamma\theta_0}{2\omega} $$

Critically Damped

In this case we have non-unique solution for $\alpha$ and we must write the solution in another form (note that we have $\omega_0 = \gamma$ then $\alpha = -\gamma$):

$$ \theta(t) = (A+Bt)e^{-\gamma t} $$ Again, applying our initial conditions:

given $\theta(0)=\theta_0$ we have $A = \theta_0$
given $\frac{d\theta(0)}{dt}=0$ we have $B=\gamma\theta_0$

Hence,

$$ \theta(t) = \theta_0(1+\gamma t)e^{-\gamma t} $$

Now we have everything we need to describe the movement of the grass when the instigator's influence disappears!

Physical Abstraction of Instigators

Now the problem is how we determine the initial condition $\theta_0$ mentioned above. And this initial angle is tied to the shape of the instigator.

We assume the height of the instigator can be defined as a function $h(x)$ where $x$ represents the distance between the center of the instigator and the root of the grass.

A simple approach would be to use linear interpolation:

$$ h(x) = cx $$ where c is an instigator-specific constant.

Second Order Polynomial

In our case, let us do something a little bit more unnecessary: use a curve instead.

$$ h(x) = cx^2=\frac{h_0}{R_0^2}x^2 $$

NOTE: Later, I found out that I overlooked a crucial aspect - $$\theta(R_0) = 0$$. This is not easily satisfied when using a second order polynomial with the tangent case simplification. Eventually I would end up using elliptic curve as it guarantees $$\frac{dh(R_0)}{d(d)}=tan(\pi/2)$$. But I still leave the polynomial reasoning here to demonstrate the thought process. If you don't care about it, jump to the ellipse section.

Where:

$h_0$ is the height of the instigator.
$R_0$ is the radius of the instigator.

The advantage of using a second order function is that it resembles placing a capsule collider which is exactly what a character controller looks like.

Grass has a fixed length of $l_0$, thus we can use a segment $g(x)=p(x-d)$ to express it, where $d$ is the distance between the center of the instigator and the root of the grass. It also means we may have 2 cases to deal with when trying to solve the initial angle:

Tangent Case: the grass blade is a tangent of the instigator curve.
Non-tangent Case: the grass blade's extremity touches the instigator curve.

Tangent Case

Assume the intersection between the grass segment and the curve is $x_0$. The derivative of the curve at $x_0$ provides the slope of the grass segment. Hence:

$$p=h'(x_0)=2cx_0$$

And since the line intersects with the curve at $x_0$, we have:

$$x_0^2*c=2cx_0(x_0-d)$$

after simplification:

$$x_0 = 2d$$

the grass equation becomes:

$$g(x)=4cd(x-d)$$

and we have:

$$\theta_0(d) = \frac{\pi}{2}-arctan(4cd)$$

Non-tangent Case

In this case, the intersection is $(x_0,cx_0^2)$ by evaluating the instigator curve at $x_0$. The displacement relative to $(d,0)$ is $(x_0-d,cx_0^2)$. This vector has length of $l_0$. Then:

$$(x_0-d)^2+c^2*x_0^4=l_0^2$$

after expansion:

$$c^2x_0^4+x_0^2-2dx_0+(d^2-l_0^2)=0$$

Unfortunately, this is not practical to solve. But remember we are not doing physically accurate simulation, so we will treat the non-tangent case as tangent case.

The issue

The polynomial approach has a major issue that I only discovered later - it can create discontinuity at $R_0$ where we expect 0 degrees:

This is a fairly noticeable discontinuity and causes grass blade to sort of "teleport" when entering an entity's influence.

Here's a better illustration of the problem:

The curve has $$h_0 = 2$$ and $$R_0 = 5$$, ideally; we wish the grass whose root is at 5 to be close to upright, because anything outside of 5 should be upright. However, in the above diagram, we can see that the tangent of the curve passing point (5,0) is not at all upright. This means any grass that touches the rim of the object will suddenly lean with that angle, resulting in visual pops.

To eradicate this problem, we need to change how the object is abstracted, And the curve that has a slope of 0 at origin and a slope of infinity (perpendicular to the x axis) at $R_0$, is part of an ellipse.

Ellipse

We now model the instigator as an ellipse, and its equation would look like this:

$$\frac{y^2}{h_0^2}+\frac{x^2}{R_0^2}=1$$

NOTE: $y$ is $h(x)$ seen in the polynomial approach.

This ellipse doesn't pass through $(0,0)$ and $(R_0,h_0)$, but $(R_0,0)$ and $(0,-h_0)$. For now, it is fine, because what we will calculate is the slope of the tangent, which doesn't change even if we move the ellipse.

To obtain the slope $\frac{dy}{dx}$, we derive by $\frac{d}{dx}$:

$$\frac{2}{h_0^2}y\frac{dy}{dx}+\frac{2}{R_0^2}*x = 0$$

After simplification, we obtain:

$$\frac{dy}{dx}=-\frac{h_0^2}{R_0^2}\frac{x}{y}$$ Now we need to find a point $(x_0,y_0)$ on the ellipse to calculate the value of the slope.

We already know that the tangent should pass through $(d,0)$, but since our ellipse is shifted down $h_0$ units, the point becomes $(d,-h_0)$. The tangent equation of an ellipse is given by:

$$\frac{x_0x}{R_0^2}+\frac{y_0y}{h_0^2} = 1$$

Substitute with $(d,-h_0)$, we obtain

$$\frac{x_0d}{R_0^2}-\frac{y_0h_0}{h_0^2} = 1$$ moving the terms around, we get

$$y_0=h_0(\frac{x_0d}{R_0^2}-1)$$ Now we plug it back to the ellipse equation:

$$\frac{(h_0(\frac{x_0d}{R_0^2}-1))^2}{h_0^2}+\frac{x_0^2}{R_0^2}=1$$

which simplifies to:

$$x_0((R_0^2+d^2)x_0-2R_0^2h_0)=0$$ we have 2 solutions, either $$x_0=0$$ (valid but not what we are interested in), and

$$x_0=\frac{2R_0^2h_0}{R_0^2+d^2}$$

then we plug it back to the $y_0$ equation we have

$$y_0 = h_0(\frac{2d^2}{R_0^2+d^2}-1)$$ with these we can express the slope as a function of $d$:

$$\frac{dy}{dx}=-\frac{h_0^2}{R_0^2}\frac{\frac{2R_0^2h_0}{R_0^2+d^2}}{h_0(\frac{2d^2}{R_0^2+d^2}-1)}=\frac{2h_0d}{R_0^2-d^2}$$ Finally, we can express the initial angle as

$$\theta_0(d) = \frac{\pi}{2}-arctan(\frac{2h_0d}{R_0^2-d^2})$$

Here's the initial angle curve for an instigator that has 10 radius and 3 height, we can see the curve approaches 0 when approaching 10 from the left:

We must take precautions in this case : when $$d=R_0$$, we have division by zero. and we must account for it in shader.

We will need to express $y$ with $x$ later, and this time we must move the ellipse up by $h_0$ units:

$$\frac{(y-h_0)^2}{h_0^2}+\frac{x^2}{R_0^2}=1$$

After simplification, we have

$$y=h(x)=h_0(\sqrt{1-\frac{x^2}{R_0^2}}+1)$$

Conclusion

We established the 3 cases for grass motion $\theta(t)$ :

Underdamped : $\theta(t)=\theta_0*e^{-\gamma t}cos(\omega t)$
Overdamped: $\theta(t)= e^{-\gamma t}(A_1e^{\omega t}+A_2e^{-\omega t})$ where $A_1 = \frac{\theta_0}{2} + \frac{\gamma\theta_0}{2\omega}$ and $A_2 = \frac{\theta_0}{2} - \frac{\gamma\theta_0}{2\omega}$
Critically damped: $\theta(t) = \theta_0(1+\gamma t)e^{-\gamma t}$

With:

$\gamma=\frac{b}{2*l_0}$
$\omega_0 = \sqrt{\frac{g-k}{l_0}}$
$\omega = \sqrt{abs(\gamma^2-\omega_0^2)}$

And a way to express initial angle $\theta_0$ with an elliptic instigator:

$$\theta_0(d) = \frac{\pi}{2}-arctan(\frac{2h_0d}{R_0^2-d^2})$$

The height of the lower surface at of the instigator at $d$ relative to its lowest point is expressed as

$$h(d)=h_0(\sqrt{1-\frac{d^2}{R_0^2}}+1)$$

In part 2 we will record the influence of the instigator (its position, velocity, and some calculated results) to fuel the eventual calculation in the vertex shader.

Unity Profiling: C++ Capture

Sun, 05 Jan 2025 00:00:00 GMT

When you start to keep an eye on the frame rate of your game, chances are you will start with the official solution: Unity Profiler. Indeed, most times the built-in profiler is more than enough to know what's causing frame rate drops. Though, the profiler is slightly limited when it comes to anything that's happening on the GPU - you only know vaguely in which pass the GPU is suffering, but you must use your experience or educated guess to pinpoint the exact problem.

Nsight Graphics and Render Doc are used for frame debugging (mostly), and the timer in the frame debugger is not much better than Unity's built-in profiler.

C++ Capture was a trick that I discovered myself to gain extra insight into performance issues. It still uses Nsight Graphics - usually we can't access the GPU counter when launching Nsight, because that requires running Unity in admin mode, which Unity directly says Don't do it.

The idea is simple: launch Nsight in frame debugging mode as usual - no admin privilege needed, and then play the game. Capture a frame, and then hit Export C++ Capture. Build the C++ capture, and launch the C++ capture with GPU Trace in admin mode. And there you can get all the analytics about shader bottlenecks and more accurate timer values.

There's a small catch: this method isn't guaranteed to work. You could fail to export a capture, or the capture crashes your Nsight when attempting to use GPU Trace on it.

Step-by-Step Guide

First, launch Nsight, and click Start Activity:

In the pop-up, fill Application Executable with the path to Unity Editor executable (if you are not sure where it is located, check out the Installs tab in your Unity Hub).

Then, fill Command Line Arguments with --projectPath {path to your project}. You can also add other arguments specified here.

Make sure in Activity, you are launching Frame Debugger.

The editor should launch, and when it finishes, play the game regularly till you meet a performance critical scenario. Either hit F11 to capture or click Capture for Live Analysis on the upper-right corner of your Nsight:

Sometimes it can be tricky to capture the window you want... to save yourself from such a hassle, maximize the game view before capturing.

Still on the upper-right corner, there should be an Export as C++ Capture button, click that.

A new window should pop-up below - it represents the C++ Capture of your frame. Navigate to the Build section, and click the Build button. Wait until the build finishes. (You can actually build the capture later, before launching GPU trace)

Now you can close the Nsight.

Relaunch the Nsight with Admin rights, navigate to the Continue section of the welcome screen. There you should find your C++ capture. Click it.

Scroll all the way down to the Run section and click Connect:

Make sure you select GPU Trace Profiler and click launch:

Click Collect GPU Trace after seeing your capture launched:

NOTE : The above screenshot failed to capture as it shows a dropdown box for window selection - it means my PC is using both an embedded GPU and a dedicated GPU. Nsight doesn't really like that. Force the PC to use discrete GPU via settings should solve the problem.

And now you have a complex interface to navigate... From this point on, I suggest looking for the dedicated documentation of the GPU Trace as I simply can't cover that much content in this post, and many of the concepts are fairly low level.

For some quick info, though, the hotspot section shows which shader caused the most performance issue:

Clicking on it will send you to the line where the problem happened:

Euh, but this isn't shader code I'm used to! - true, you will be looking at disassembly code, but usually the issue tells you the root cause and you should be able to pinpoint the line in the original shader code.

Dynamic Resolution - "Hardware" vs "Software"

Thu, 26 Dec 2024 00:00:00 GMT

In Unity' HDRP, dynamic resolution can be used to improve framerate without sacrificing too much of the game's visual qualities. Most of the tweakable parameters are documented, except for "Hardware" and "Software" dynamic resolution modes. OK, then, what are those?

I spent some time poking into the two different modes and it turned out "Hardware" and "Software" aren't that simple, they have serious impact on the whole render pipeline.

What's more, in my opinion, dynamic resolution is not a hardware-level thing. They just use the graphics API differently to achieve the downscaling effect.

"Software" dynamic resolution (Viewport)

Before tackling what "viewport implementation" is, let us review how is the viewport data specified:

In VkPipeline construction, we could specify vkViewports in VkPipelineViewportStateCreateInfo, or when using dynamic viewports, it is set with vkCmdSetViewport.

Non-dynamic viewports are preferable when viewport does not change. If you do change it however, you need to recreate the whole VkPipeline which would result in significant stutter.
Dynamic viewports are preferable when viewport does change. This does induce small runtime performance costs due to the extra command.

Unity uses dynamic viewport by default: (capture taken when dynamic resolution is disabled)

Therefore, dynamic resolution does not induce extra overhead of set viewport commands.

Now, the viewport implementation.

Essentially, viewport implementation creates one single VkImage with the size of the swapchain extents and draws on a limited area specified by the viewport:

Note the Dimension here: it is exactly the size of the swapchain extents:

Viewport affects clip space coordinates translation, mapping the range of the clip space to the area of the viewport. When the driver knows the viewport beforehand, it doesn't need to run the fragment program on pixels outside of the viewport, and that's how we gain the performance boost.

"Hardware" dynamic resolution (Memory Aliasing)

Hardware implementation is only available in DX12 and Vulkan for good reason: Memory Aliasing can only be achieved with lower-level graphics API.

In a nutshell, Memory Aliasing is creating multiple VkImage but map them to the same chunk of memory (the largest chunk required by all these images).

Here's a capture taken when memory aliasing implementation is enabled:

Note here the Dimension is no longer the extents of the swapchain:

Unity still issues vkCmdSetViewport call, but now the viewport covers the entire render target rather than just a portion.

In Unity, render textures that have dynamic scaling enabled (.useDynamicScale = true) are managed by ScalableBufferManager. Unity issues a global update call to update the resolution (probably keeping which slice of the render target to use, managed in the C++ side of the engine).

Which one to choose?

The reason Unity used Hardware and Software to label the two implementations is probably, it is intuitive that if dynamic scaling is supported on a "hardware" level, it is better. And that is actually true. But here are the details why memory aliasing is better than viewport when both options are available:

When the GPU loads a texture (via OP_LOAD), only the mapped memory of VkImage is pulled. The viewport implementation always loads the full resolution texture, but only draws on part of it, resulting in a waste of GPU bandwidth. In contrast, memory aliasing only loads what is necessary.
The viewport implementation only assures the specific area is written to, but does not prevent reading from areas that are "out of bounds". This means if you were to sample a texture subject to viewport implementation dynamic resolution before the upscaling pass, not only you don't save any sampling calls, you must also take precautions that there might be invalid data. Unity hides this fact as usually, you would sample those textures in a post processing pass, while the upscaling pass is placed conveniently before post processing by default.

The conclusion is: prefer Hardware whenever possible, and take precautions when sampling as not all devices support Vulkan/DX12 and will cause a fallback to Software.

The Tag in Text Mesh Pro

Mon, 16 Sep 2024 00:00:00 GMT

Changing font regularly might not be a common use case, let alone applying only to some text in a line of text. But sometimes this is necessary, for instance, encrypting a word as part of a linguistic puzzle.

When first using the font tag, the immediate result would be it does not work.

First Intuition: Erroneous Settings

When this happens, whether you have googled or thanks to a detective instinct, you would realize something hasn't been setup properly. And peeking at the Project Settings > TextMesh Pro > Settings, there is indeed a path that points to a resources directory. (Note the screenshot isn't showing the default value, expect something like Fonts & Materials if this is the first time you see this).

Moving your font assets under the indicated path would resolve the issue.

Though, if you try to change the path to a custom path as I did, there's a chance that it doesn't work. This is because TMP internally stitches the path with a simple string concat rather than Path.Combine, and if you forgot to add a slash ("/") at the end of your custom path, the stitched path would be incorrect. For example, if you are trying to use <font="DeliriumHW">:

Filling "Fonts" in the settings will cause TMP to call Resources.Load("Resources/**Fonts**DeliriumHW")
Filling "Fonts/" in the settings will cause TMP to call Resources.Load("Resources/**Fonts/**DeliriumHW")

Second Intuition: Custom Loader

Resources.Load is definitely not the best practice to load things. What's more, in Addressables, assets in Resources will be duplicated.

With some research, however, I was able to find an event that gets called before resorting to Resources.Load, and it is indeed intended to give us a chance to inject a custom loader. The event is:

 TMP_Text.OnFontAssetRequest

This would be invoked with an integer and a string, with the string matching the requested font asset name.

Strangely, this event isn't mentioned in TMP's font tag documentation.

Example of a custom loader:

 private TMP_FontAsset LoadFont(int arg1, string arg2)
 {
     TMP_FontAsset fontAsset;
     if (Application.isEditor && !Application.isPlaying)
     {
         // when in editor and not in play mode, we load with AssetDatabase
         fontAsset = AssetDbShorthand.FindAndLoadFirst<TMP_FontAsset>(arg2);
     }
     else
     {
        // load with Addressables
         var handle = Addressables.LoadAssetAsync<TMP_FontAsset>($"Fonts/{arg2}.asset");
         fontAsset = handle.WaitForCompletion();
     }
     return fontAsset;
 }

Note that using this example directly is not advised, as it doesn't keep the Addressables handle and can cause resource management problems

With the addition of this event, we can eventually manage font resources reasonably. It is also possible to preload the fonts in the loading phase to eliminate the impact of a synchronous load.

Some other events can be found around the OnFontAssetRequest event, e.g. sprite request. This would solve other related resource management issues that are strangely not covered by the official documentation.

Bypassing Unity Undo System's Memory Limitation

Mon, 03 Jun 2024 00:00:00 GMT

It is said that in the crumbled city of Uniograad, there is a catacomb buried deep under the debris. Whenever wind travels through the ruins, the empty chambers of the catacomb would produce a wailing sound. And if you dare to dig into the catacomb, you would find... the unspeakable abomination lurking in the walls...

Which is Unity's Undo system.

Usually this has not much of impact if you don't do a very specific thing, but if you dare to crank Terrain resolution to 4096, the undo system simply can't handle it. When you paint texture, chances are you are unable to undo.

The reason is simple, undo system has a built-in buffer size limitation. If your object is too large and surpasses this size, the undo system will not record it. A terrain with 4096 resolution and 16 layers will have 4 Alpha Map each has 4096x4096 in size, 4 channels, and that would give you something like 1 GB per undo step. It is still reasonable that Unity limited the size because this would quickly blow up your RAM or VRAM, but being unable to do even one undo is kinda... extreme.

So how to solve this problem?

Well, this complaint has been on Unity Forum for... more than 8 years! And Unity still hasn't exposed this buffer size limitation. Also, the Undo system is written in C++, so you cannot just use reflection to simply adjust a value.

A dead end, and another day to curse Unity for being unable to solve a simple problem.

But really, we can do nothing about it?

The Detour

Well, there is a way to hack our way out of this... It is still more than necessary, but if you are so desperate, this might be the only way out.

First, when you paint, the Terrain Tools system will invoke a method in Undo system, requesting the Undo system to save the current state of the objects. And these objects, for painting terrain texture, are simply 4 texture 2D. The instance id of these textures does not change, which means no matter how many strokes you make on a terrain, the 4 textures all keep their instance id. This information is useful for restoring the state of the alpha maps.

Now... if only we can hijack this invocation...

Oh, we do have something. There is a plugin called Harmony that can intercept a method call and run some code before actually calling the method. And you can also use the injected code to decide whether to run the original method. This "some code" is called a "prefix" in Harmony terminology.

But! There is a caveat, Harmony can't take extern methods. And the bad news is that most of the methods in the Undo system are extern. The good news is, the call made by Terrain Tools system, lands on a non extern method.

So, heh, after intercepting the objects, we need to create copies for them. It is essentially copying data from one texture to another. And according to this blogpost, the best way to do this is to use Graphics.CopyTexture, which takes no noticeable time. However, this means the data lives in GPU, so you probably need to be careful about it, as the textures are fairly large.

To restore the textures, first we will need to find alpha maps from TerrainData that have matching instance id as your recorded textures (you did record the instance id or a reference to the original texture as well, right? Because creating a new texture will have a different instance id). Then... you will need to use SetAlphaMaps to update the original alpha maps. This method requires a float[,,] to work, but we don't have it yet...

The best approach I found is to use GetPixels on the copied texture to download all pixel data from GPU to your RAM, and use a Parallel.For to create the 3 dimensional float array. This should reduce the undo operation time to about 1~2 seconds.

Oh, and finally... It is sad that we can't intercept the PerformUndo call since it is extern, so I enforced Alt+Z to undo when painting texture. This can be done with a simple [EditorMenu] attribute.

Anyway, this is not perfect, but still this is the best I can do if Unity keeps hiding that precious buffer size limitation.

Example Code

using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Threading.Tasks;
using HarmonyLib;
using UnityEditor;
using UnityEditor.SceneManagement;
using UnityEngine;
using Object = UnityEngine.Object;

namespace UndoHijack
{
    public static class UndoHijackSys
    {
        private static Harmony _harmony;

        [MenuItem("Help/Hacks/Hijack Undo")]
        [InitializeOnLoadMethod]
        public static void Init()
        {
            _harmony = new Harmony("UndoHijack");

            var original = typeof(Undo).GetMethod("RegisterCompleteObjectUndo",
                new[] { typeof(UnityEngine.Object[]), typeof(string) });
            var prefix = typeof(UndoHijackSys).GetMethod("HijackTexturePaintingUndo",
                BindingFlags.NonPublic | BindingFlags.Static);
            _harmony.Patch(original, prefix: new HarmonyMethod(prefix));
        }

        internal class UndoElement: IDisposable
        {
            public int instanceId;
            public Texture2D values;

            public void Dispose()
            {
                Object.DestroyImmediate(values);
            }
        }

        public const int maxUndo = 6;
        private static List<UndoElement[]> _undo = new();

        private static bool HijackTexturePaintingUndo(UnityEngine.Object[] objectsToUndo, string name)
        {
            if (name != "PaintTextureTool - Texture") return true;
            if (_undo.Count >= maxUndo)
            {
                var excessiveStep = _undo[0];
                _undo.RemoveAt(0);
                foreach (var element in excessiveStep)
                {
                    element.Dispose();
                }
            }

            _undo.Add(objectsToUndo.Select(x => CreateUndoStep(x as Texture2D)).ToArray());
            Debug.Log($"Adding undo ({objectsToUndo.Length})");
            return false;
        }

        private static UndoElement CreateUndoStep(Texture2D t2d)
        {
            var result = new UndoElement();
            result.instanceId = t2d.GetInstanceID();
            var width = t2d.width;
            var height = t2d.height;
            result.values = new Texture2D(width,height);
            Graphics.CopyTexture(t2d, result.values);
            return result;
        }

        [MenuItem("Help/Hacks/Alt Undo &z")]
        private static void Undo()
        {
            if (_undo.Count < 1)
            {
                Debug.LogWarning("Texture paint undo threshold reached.");
                return;
            }
            var stepToUndo = _undo[^1];
            var step = stepToUndo.ToDictionary(x => x.instanceId, x => x.values);
            var scene = EditorSceneManager.GetActiveScene();
            var terrains = scene.GetRootGameObjects()
                .SelectMany(x => x.GetComponentsInChildren<Terrain>()).ToList();
            foreach (var t in terrains)
            {
                /*
                 *  Conclusion after conducting experiments:
                 *  A terrain-data's Alphamap Texture's instance id does not change.
                 *  BUT! These Alphamaps are created on the fly in the get method.
                 */
                // Only compare the first. If the first doesn't match, none will match.
                var allAlphamapTexures = t.terrainData.alphamapTextures;
                var alphamapTexture = t.terrainData.alphamapTextures[0];
                var id = alphamapTexture.GetInstanceID();
                if (!step.ContainsKey(id)) continue;
                // write back
                var width = t.terrainData.alphamapWidth;
                var height = t.terrainData.alphamapHeight;
                var map = new float[width, height, t.terrainData.terrainLayers.Length];
                var counter = 0;
                foreach (var texture in allAlphamapTexures)
                {
                    var instanceID = texture.GetInstanceID();
                    var undoTexture = step[instanceID];
                    WriteToArray(undoTexture, map, width, height, counter);
                    counter++;
                }

                t.terrainData.SetAlphamaps(0, 0, map);
            }

            foreach (var element in stepToUndo)
            {
                element.Dispose();
            }
            _undo.RemoveAt(_undo.Count - 1);
        }

        private static void WriteToArray(Texture2D t2d, float[,,] map, int width, int height, int layer)
        {
            var res = t2d.GetPixels();
            var result = Parallel.For(0, width, i =>
            {
                for (int j = 0; j < height; j++)
                {
                    map[i, j, layer * 4 + 0] = res[i * width + j].r;
                    map[i, j, layer * 4 + 1] = res[i * width + j].g;
                    map[i, j, layer * 4 + 2] = res[i * width + j].b;
                    map[i, j, layer * 4 + 3] = res[i * width + j].a;
                }
            });
        }

        
    }
}

Loading Raw 3D Models in Runtime

Sat, 18 May 2024 00:00:00 GMT

This was a strange thing to tackle...

TL;DR: Attempt to import FBX is going to be a pure fantasy. OBJ, okay, but lacking advanced features. Your best pal would be GLB.

Now, explanations.

.glb/.gltf is a rather new format (compared to old fossil like obj), that is designed to be "the jpeg for 3D models". And indeed it is. It is lightweight, can have material/texture/animation info, and even lights and cameras (if... that matter). I would recommend conduct some further reading on this format.

But why are there 2 extensions? Well, .gltf isn't a standalone file, it consists of a gltf file in json format, and a data file in binary format. .glb however, packs everything in one file (b for binary). And of course we would be interested in .glb due to its portability. gltf can come in handy if you really want to debug your model for some reason.

Now, how does Unity work with gltf/glb?

It is officially supported by Unity via the glTFast package (link). It isn't well-known since 1. it isn't in package manager and 2. gltf is a format for presentation, not really for editing.

I tested runtime import of glb files and to my surprise, not only it works, it also creates correct materials and animations... which isn't possible with obj.

And most software support gltf/glb. Blender, for example, does it perfectly. And I tested Houdini by snatching a friend's computer. It can handle it partially, but if you try to animate an attribute that you created yourself... it will not work, Unity complains about blend shape weights. But if you simply import the glb in Blender and do nothing, export it, it will actually make Unity happy. The downside is that the glb would grow about 3 times in size; I suppose something hacky done by Houdini is reverted to a simpler state by Blender upon import.

Something about Animations

When you have animations in glb files, import it in the editor creates animations, and loading the asset creates a model instance with Animator. However, this is not the case when you import a model in runtime. There are two reasons:

the glTFast package defaults legacy animation system. and
You can't create animator controller in runtime.

I actually tried to force glTFast to use Mechanim import instead of legacy, but it doesn't solve the animator controller problem, as there's no information in glb files to author the creation of animator controller.

My final solution is to overthrow the animator controller and rewrite it using the Playables API. And I had to use a manual update to control the playable graph because imported animations are one-shot. I had to manually control the "elapsed time". Hopefully I will explain how this system is built in a future post.

Pixel Perfect Outline in Lo-Res

Mon, 04 Mar 2024 00:00:00 GMT

Pixel Perfect outlines for sprites aren't hard, but for 3D, I've been through the darkest corners.

BUT WAIT, you may say, why would it be hard? After all, there are so many articles out there who demonstrate how outline could be done on 3D meshes. Whether it is extruding the normals, screen space edge detection, there got to be an easy way to do this.

Well, I am a little peculiar when it comes to trace outlines. You see, I am not satisfied with the contour; I need hard edges that are inside the contour. Usually, this can be done by sampling the normal map.

But.... I have transparent objects. And they don't write to screen normals.

This causes 2 problems:

Any line drawn based on screen normals but masked by transparent objects is still drawn. Which isn't visually appealing.
Without normals we lose the clarity of outlines on transparent objects. In fact, with how renderers are normally set up, we don't even have depth. This would result in transparent objects rely solely on color.

Just before we delve deeper, let me explain why I can't use the normal extrusion method. If you have no idea what that is, we draw an object with vertices extruded in the direction of vertex normals, and give it the color of the outlines. This is basically doing outlines for 2d sprites - sample the original sprite with 1px offset in each direction, effectively extruding the sprite. The method requires smooth and watertight meshes, otherwise it would cause broken looking outlines, which is something I can't impose based on the game. Also, normal extrusion can't (afaik) achieve screen space pixel perfectness, which is a significant drawback.

Now, back to our subject.

Edge Detect

When I first started, I used the method that I am most familiar with: Edge detection. I stumbled across an article that uses Roberts kernel on depth and normal map, which has a problem at acute viewing angles since that causes depth value to change dramatically. The author compensated that by using the screen normals - the more perpendicular the screen normal is to the view vector, the less weight should be put on the normal value.

I implemented the Roberts kernel, but also a Laplacien kernel, which I use more often in computer vision projects. The result is interesting, as the Laplacien kernel does not have the acute viewing angle artifacts that are present when using Roberts kernel. Aside from that, there's no significant difference in outline quality. The Laplacien kernel is heavier though, because it is a 3x3 kernel while Roberts is 2x2. However, considering we don't need to apply view angle adjustments and we are in low resolution, I'd say it is a gain.

Here I am only showing Laplacien, as I have deleted everything about Roberts from my shader.

Laplacien Edge Detect:

This post explains the building blocks of this convolution node.

Edge detection on normals texture comes with a natural problem when we are in low resolution though, when we detect a sudden change in value, two color channels are affected, with a displacement of 1px. This is not prominent in high resolution, but in low resolution, 2px is easily told apart from 1px. I haven't come up with a way to solve this but perhaps later.

Anyway I did this for color, depth and screen normals.

Retrieve Depth and Screen Normals (Shader Graph)

Depth is straightforward, we need the SceneDepth node. But since it takes UV and outputs depth value directly, we can't use the same graphs as with the color texture. Feasible but tedious.

For normals, we need the name of a global variable, which is _CameraNormalsTexture. Create a Texture2D variable and name it CameraNormalsTexture, set the scope to Global, and ticking off Show In Inspector.

It can now be sampled as any other texture, but here's a trap: Unity stores normal vectors as Vector2 (r,g) instead of Vector3 (r,g,b). This is because normal vector is a unit vector ( magnitude = 1 ), thus it has only 2 degrees of freedom. The third component of a normal vector can be calculated with the 2 known by using Pythagorean Theorem.

Note on Depth and Screen Normals Texture

The shader I am developing was previously a full-screen effect. Later I discovered that I only need to render 3D objects in a more constrained area of the screen, thus it is turned in to an effect that applies to 2D textures. I still conserved the ability to apply it full-screen though, as a matter of fact, it is just the difference between using _MainTex or _CameraSortingLayerTexture.

But what about depth and screen normals? Turned out that these two, if I render the result to a render texture, will have size equal to that render texture. And if we don't have other things that write to depth and screen normals in different size, they will always remain of that size.

This is what happens if we render the 3D objects to a 200x200 render texture while enabling depth and normals based outline, using the "full screen" mode of the shader:

We can see that only the color based contours are correct, because the screen color texture is of the correct size, but depth and normals based contours are like stretching a 200x200 texture to full screen.

Intermezzo: Camera Sorting Layer Texture

I write this because a funny problem has occurred to me twice, each time I have no idea what is happening in the first 10 minutes.

This is simply due to the fact that the shader using the camera sorting layer texture renders to it as well.

For example, this is the canvas containing a shader that uses the camera sorting layer texture.

And in the URP settings, the foremost camera sorting layer texture is UI.

The solution is therefore to make sure everything using the camera sorting layer texture should be put in the layer above the foremost sorting layer texture. In this case, OverlayA.

Transparent Objects

Now here is a sort of like untraveled hinterland, because not much attempt has been made on drawing contours/structure lines on transparent objects.

The difficulty, as I have explained earlier, is that transparent objects don't write to depth and screen normals by default. It is, however, possible to force write to depth buffer.

Default (transparent objects don't write to depth buffer)

Write Depth with Less comparison

It looks... fine. But there's reason why transparent object don't write to scene depth by default.

Here is a scene where a transparent object is not in front of the cube which looks correct - we are only seeing the cube

Now if we take another transparent object in front of the cube, miracle happens, the transparent object behind the cube is showing up...

As I tested many other options, there is no depth write option that ensures a correct perspective under all circumstances.

My current approach, unfortunately, is to rely only on color for transparent objects. It looks OK, with the only problem that contours that should be obscured by transparent objects are showing through.

What if... we are able to mark all pixels that are affected by transparent objects?

My first attempt was to add another camera that renders transparent objects only, to a render texture. But that doesn't work, as we are not able to detect any opaque objects in front of transparent objects. And the ordering of the renderer is strange, if below normal 3D renderer it would clear depth and normals texture, and if above, it clears color.

After days struggling with URP's internal access modifiers, I finally did it:

I've created a global texture called _CustomColorTexture that can be accessed from shaders, and the Custom Render Objects Pass renders objects to that texture (only color, we still use depth texture form main camera, but since transparent objects don't write depth, it is effectively read-only).

It isn't magic though, I simply copied the built-in RenderObjectsPass, fix access modifier problems with cached reflections, and replaced render attachment with the self-allocated render texture (allocating with RTHandles.Alloc(...) and import with renderGraph.ImportTexture(...) ).

Bind to global shader variable is done by using builder.SetGlobalTextureAfterPass(...).

Unity is shifting towards Render Graph, a technology with currently very poor documentation. I just hope my code still works after the release of Unity 6... I might try to write something about it later.

Please Unity, be open (public)

NOTE: And indeed, one of them became publicly accessible in Unity 6.

With this obstacle cleared the rest should be relatively simple.

The contour lines obscured by transparent objects are no longer showing through.

Finale

I end this article here because I think all the necessary puzzle pieces are in place. With these we can easily enhance the quality of transparent object's outline using the custom color texture, dither the contour line based on how many layers of transparent objects are in front of that object, and other refinements if required.

IMPORTANT UPDATE

Past Unity 6000.0.5f1, _CameraNormalsTexture no longer works as intended. This is because GBuffers are no longer global.

I temporarily solved it by adding a pass after deferred lighting that copies GBuffer2 to a persistent texture. Now think of it, I should just declare using the GBuffer2 and then the Render Graph will solve this issue for me.

Multi-pass Convolution with Shader Graph

Sat, 23 Sep 2023 00:00:00 GMT

NOTE: This article requires you to already know what convolution in image processing is.

NOTE: Check out this repo which features the assets mentioned in this blog post.

Convolution, being the most essential and useful technique in image processing, can create a range of effects, such as Gaussian blur, edge detection, sharpening, and so on.

The most standard implementation of a multi-pass convolution effect is by using custom render features. (well, essentially, the convolution is still implemented inside a shader). But here I would like to take a step further, to see if it is possible to do something similar as what I did with the Kawase blur effect - limiting the effect to a certain zone instead of a global post processing that covers the whole screen, and can be resized with an animation.

And with the help of Custom Render Texture, it has been done and is very modular.

Implementation Detail

First, a custom render texture, just like render texture, can hold the result of a convolution operation and be fed to another as input. The documentation states that by setting a custom render texture as the initial texture of another, Unity will handle the update order. With this in mind, what left to do will be to create a custom render texture for each pass and assign a convolution operation (material) to each of them.

Implementing convolution is simply sampling the input texture with an offset and then multiply it by a kernel value. For a 3x3 kernel, that means to sample in a tic-tac-toe pattern, and then sum the cells up.

the following image shows the "Convolve Cell" sub-graph, noting that it also expects UV so that you can customize whether to use texture UV0 or screen position.

The channel mask is to select which channel or perceptual grey scale to use when running the convolution.

Therefore, it is straightforward to construct a 3x3 convolution node.

And with perhaps a little bit of work, a 5x5 or even 7x7 convolution node. But unfortunately Unity only supports up to Vector4 thus making a proper editor can be potentially demanding.

(Oh, by the way, Matrix3x3 is not available for the inspector, thus I used 3 Vector3 instead)

Support for Custom Render Texture in Shader Graph seems to be a very new thing, because the documentation is still stated that you can only write shader for it with code. But that needs to be updated as you can already find "Custom Render Texture" in the shader graph creation menu.

Well anyway, since we already have the convolution node, it is faster to create a shader for custom render textures. Strangely, I hoped that there is a magic property like _MainTex that automatically binds the custom render texture to that property, but I've tried and nothing worked.

Finally, create materials for common kernels, like the one showing on the right.

Extra

I've created a little script that helps to ease the pain of configuring the pipeline. By clicking Configure after setting everything else up, it will automatically create custom render textures, copy the materials, and link them in top-to-bottom order. This is currently written with Odin. The created assets will become sub-assets of the Scriptable Object.

An example of what happens if you click the Configure button. Each Pass N will result from each pass. Fetch the last one to get the final result.

NOTE: For a more code-oriented approach, check out this repo which does the same thing but without the hassle of setting up custom render textures. And honestly, this is the lib I have been using practically.

Irregular Puzzles

Thu, 01 Sep 2022 00:00:00 GMT

I have seen many puzzle games that use a mechanism that simulates putting together pieces of shredded paper... It is a simple mechanism, but when actually implementing it, it can give you a slight headache...

First, what are the use cases such a system needs to achieve?

Drag around pieces in a limited area.
When two pieces of matching edges connect, they are "merged", and further dragging should treat those pieces as one single piece.
Merged pieces should still be inside the limited area
The game should know when the puzzle is completed.

The difficulty is mostly in the 2nd use case, since "detecting edges" like in computer vision is not feasible, and you also need to keep track of different groups of merged pieces... also, imagine you have a group of 2 pieces and another group of 2 pieces, you will need to merge the two merged pieces as a whole...

So, here is my non-perfect approach...

I. Drag and Drop

I am a faithful Input System user, so when I started the project, I went for Input System without second thoughts. However, when I started implementing it, Input System fell short... at least not intuitive enough as the old system. Going though the setup of Actions as button based games can be painful, so I searched elsewhere...

The final answer is to make whatever you want to drag and drop, to implement the following interfaces:

IPointerDownHandler (When you press the left mouse button)

IBeginDragHandler (When mouse button is held and you start to move the mouse)

IEndDragHandler (When you stop dragging)

IDragHandler (When you are actually dragging)

Like the old system, these handlers are triggered by raycasting on a collider, so you will need to attach a Collider2D to the object.

There is still one thing you need to add: a Physics 2D Raycaster to your camera.

Ok, now about those handlers. They are actually from the UI system, so the PointerEventData that is passed as a parameter can be confusing... I have tried to use the mouse position in that data, but it gives ridiculously huge values even after applying ScreenToWorldPoint...

Well, use mouse position of something else, and that is:

Mouse.current.position.ReadValue();

Now you can use Camera.main.ScreenToWorldPoint() to convert this value to world position and use it to update the position in your OnDrag.

II. Limiting Movement

The most common practice is to use Mathf.Clamp() to restrict the position. However, it needs information about the delimiting area. It can be done by directly supply values of borders, but for me I usually want something more visual...

My method of choice is to create a dedicated class to store this information, and in most cases, the delimiting area is a rectangle. This class will expose the 4 values of the borders (up, down, left, right), and retrieve this information in its Awake(). To define this information, it needs to keep reference of two Transforms, one representing the upper left corner, and the other the lower right corner. After reading the corresponding values, these two objects will be set inactive.

The purpose of doing this is that you can reposition the two corners with drag and drop in editor. You can add a OnDrawGizmos() like I did to visualize the borders.

III. Determine the Correct Position

As stated before, when determine if two pieces would fit together, it is not feasible to judge based on the shape of the edges. Instead, the viable method that I found is to record the relative position of each of the two adjacent pieces when they are in a completed state. This method is certainly not perfect, because the more pieces you have, the more information you need to record. In my case, 6 pieces require 10 adjacency information to well define the puzzle.

Each adjacency information will contain 3 information: 2 references to the involved pieces, and a Vector2 that represents the relative position (transformB.position - transformA.position when in connected state). Now, with a context menu function or custom editor call, you should be able to generate the relative position fairly easily. The hard work is how to reduce the pain of assigning references...

The trick that I found is to use Raycasting. It does NOT guarantee the detection of all adjacencies but will significantly reduce the need for manual operation.

The idea is, for each pair of two pieces A and B, draw a line (RaycastAll()) from A to B and see if the second hit collider is on B. Why second? because the first hit will always be A.

This will work most times, EXCEPT if the contacting point is not on the drawn ray (like A and B in the graph above). To reduce these exceptional cases, you can try to sample multiple points on the same pair and see if at least one raycast is successful. Again, even if you do so, it will still not guarantee that you detect all the adjacencies, as a well-designed counter-example can easily make your effort useless, though such a case is very rare.

Besides that, I suggest adding OnDrawGizmos to the class by storing the adjacency information to visually show the adjacency so that it is easier to detect missing adjacencies.

To sum up, when you have properly created your adjacency recording system, first you need to finish the puzzle in the editor manually (so, if you have a large amount of pieces that can be problematic), and use the system to generate potential adjacency information, spot any missing adjacencies and add them manually, and finally use the system again to calculate the relative position in each adjacency.

To determine if two pieces should merge during runtime, first you need to know which piece is being dropped, and then verify for each adjacency information if the piece is involved. If it is involved (say, it is pieceB), calculate the absolute position based on the position of the other piece (pieceA), and if the dropped piece is in a certain range (Tolerance) of the absolute position, they are merged. Modify the position of the dropped piece directly to the absolute position, and delete the adjacency information (it is not useful anymore).

To know which piece is being dropped in the adjacency storage class, my approach is to create an Action<DragAndDrop> Drag in the DragAndDrop class and invoke the action in Drag(). Since in the adjacency class you have reference to all pieces, simply subscribe to your merge verification of that action.

Deleting information after successful merge has two effects: (1) reduce the search time for further merge attempts, and (2) when the list of adjacency information is empty, you know all pieces are connected and the puzzle is complete.

IV. Move Pieces as One

Now here is another tricky part. I have actually thought about two methods; I implemented the first, and the second is purely theoretical and I will not really discuss it. I will leave the judgement of which one is better for you.

The First Method

This method is straightforward: you store all pieces linked to the piece you are moving in a list, and move them too. So each drag-and-drop object will also keep a list of references to the linked pieces. There are two critical problems that need to be solved though: (1) how to propagate the information when a new piece or group of pieces is being merged into a group? (2) how to calculate the correct displacement to ensure all the pieces do not exceed the delimiting area and keep the global shape of the group unchanged?

For the first (1) point: The process is actually like mathematical induction. We first assume the piece that should be linked contains all pieces currently linked to that piece (excluding itself). Then, follow the following steps illustrated below:

Basically, you construct two lists of linked objects in each group, and for each object in one list, you link it with each object in the other list.

The presence of a group of pieces means you will also need to change something when you detect merges (Step III.). You will also need to check if there is a merge for each linked piece of the dropped piece, otherwise you will have undeleted adjacency in your list even if visually the puzzle is complete.

For the second (2) point: You cannot apply movement directly to each piece. Instead, you need to separate the Attempt to move the piece and actually Moving the piece. The Attempt returns a displacement of the piece, which can be less than the actual mouse delta since it can be truncated by the delimiting area. When you drag the piece under your mouse cursor, you gather all the displacement returned by the move attempts of all the linked pieces, as well as the piece itself. You can then take the shortest displacement to ensure that no piece will exceed the borders and call the actual Moving function with this shortest displacement provided as a parameter.

The Second Method

So this method delegates the moving part to Unity. My thought is that you can re-parent the linked pieces to an object created at runtime, and whenever you click a piece in a group, the movement is redirected to the parent object (you can do this by using delegates).

However, when rethinking this potential idea, it can get a little messy, and since my first method is functioning, I quickly abandoned the second method...

Weapon Trail

Mon, 20 Jun 2022 00:00:00 GMT

This is an effect I have been doing a lot in 2D, but in 3D it adds many complications, notably the plane of slash is no longer a flat plane and it varies a lot depending on the animation, which caused my usual method of using a pre-made mesh + shader + vfx to render the trail unusable.

With a little digging, I found a method to create the effect in 3D. But I have to say this method is not at all simple and took me 3 days of research to achieve the result.

In fact, in this method, (which I believe should be the same method used in the classic asset X Weapon Trail by watching the presentation video), you will create the mesh in runtime, and apply a material to render it). Most of the hard work really is in the part of creating a mesh, and I will explain why it can cause a headache.

Mesh Generation

Introducing the two most important components: Mesh Filter and Mesh Renderer. Mesh Filter is actually very simple, it keeps a reference to a mesh and allows the mesh renderer to render the mesh. The Mesh Renderer keeps information about rendering details (notably the material). With these two components set up, it is possible to assign a mesh created at runtime to the Mesh Filter (MeshFilter.mesh) and get it rendered.

However, there is a little twist.

When I first try this method, I created the two components in edit mode and reference them in my script. The object (I call it Trail Object), is a child of the weapon. This will create strange results, since render result will be originated from the Mesh Filter's position, plus any rotation applied to it. To avoid such complications, it is better to create the object in Start and leave it in the root hierarchy (parent to nothing). Since it is in Start, the overhead can be ignored.

Now, to construct the mesh, we start with something easy. A mesh should at least contain 2 basic informations: Vertices, an array of Vector3, and Triangles (sometimes called Indices), an array of int. If you don't already know, the computer only draws triangles (so if you have a rectangle, the computer draws 2 triangles). Vertex (singular for Vertices), is the point on the corners of a triangle, and an Index (singular for Indices), is a way to tell the renderer how to draw the vertices (how should the triangle be constructed from the vertices).

Observe the example here above. If your Vertices contain {V1, V2, V3}, to draw a triangle following the order V1=>V2=>V3, you would put {0,1,2} in your Indices array. The number in indices array corresponds to the vertex's index number in Vertices array.

For each triangle, you need 3 vertices to signify their corners, and actually 6 indices to correctly draw the triangle. This is because if you only draw your triangle in one order, your triangle will be single-sided. That means when the camera is not facing the triangle's normal (vector perpendicular to the triangle's plane), the triangle will not be rendered. The direction of the triangle is dependent according to which order you draw your triangle. So, in our case, to make the triangle visible in any angle, the triangle should be drawn in both clockwise and counter-clockwise.

With that in mind, we can start working on our mesh. We will assume the weapon is a straight sword, and the blade can be described with two points: base and tip. I call the segment formed by base and tip stem. For each frame, base and tip will be updated by animation, and if we keep track of the history positions, we should have a list of stems. By connecting adjacent tips and bases, we will have a zig zag plane that describes the trail of the weapon. Now, since each pair of adjacent stems contains 4 vertices, we should draw 4 triangles (2 unique shapes * 2 directions to ensure that the mesh is double sided), that means 12 indices for each pair.

For now, it is relatively easy to implement. You will also need to define a maximum frame length so that when a stem's life exceeds this limit, it is discarded and replaced by a fresh stem (this looks like a Queue, but I did it manually with a List since it is easier to iterate). By defining this maximum frame length, you can also more easily allocate the space needed for arrays.

Interpolation

You may quickly find a problem: If an action is so quick that the two adjacent stems are far away from each other, your trail will not look good (sharp edges). To solve this, and this is also the hardest part, you will need to apply for an Interpolation Algorithm. What Interpolation means is that the algorithm generates stems to fill in the gaps between the two adjacent stems. And the generated stems will be positioned so that the edge is a curve instead of a straight line. I found an article that discuss some of the mechanism behind X Weapon Trail, and in the article, the Interpolation used is Catmull-Rom.

A Catmull-Rom spline requires 4 points (P) to produce the interpolated point (Q) - 2 points prior to the interpolated point and 2 points after the interpolated point. It also requires a parameter u (in some reference, t ), which represents how far in the curve the interpolated point is. This can be defined by the proportion between the distance P1Q and P1P2 as shown above. The calculation is defined by a matrix multiplication:

The tau value is called the tension of the curve, which directly affects how the resulting curve looks like. The value is normally 0.5.

You may notice that P in the above function is a vector (it is bold). And since matrix multiplication is linear, you can apply this function to each component of the vector (x, y and z) to produce the resulting component.

So you should be good to translate the function to code. The only possible issue is how would you produce the u value ?

A possible solution is that you don't care about the actual position of the stems. You only use them to interpolate points. Your actual mesh will be separated by evenly spaced interpolated points:

You define u before finding which 4 points should be used for interpolation. If you know the total length of your mesh (preferably the length of the zig-zag curve created by the middle points of the stems), you can divide this length by a user defined integer (resolution of the mesh) to have the spacing between each interpolated stem. Note that the step is not yet the parameter u. To demonstrate how the parameter u can be found, consider the following example:

For instance, the fourth interpolated stem (red) has a distance of 3x step to the beginning of the mesh (stem0). Visually, this interpolated stem should be interpolated by using stem0, stem1, stem2 and stem3, and the value u is given by:

$$ ( 3*step - dist( stem1 , stem0 ) ) / dist( stem2 , stem1 ) $$

Where dist() gives the distance between the two stems.

So what you end up with is to find the stem P1, which should be the closest stem on the left (closer to the start of the mesh). By closest, you can define as the distance between the mid-points of the stems. If you keep a record of the stems' distance to the first stem (the final one will contain the total length of your mesh), you can just iterate through the list and find the stem P1. And there you go you should have all the parameters needed for Catmull Spline.

Applying Material

If you did all the above correctly, you should have something that works. But what about material? Well, you will need to specify the UV coordinates of the vertices.

UV tells the renderer how to map the texture on your mesh, and it is an array of Vector2. Each vertex will have its own UV value. UV coordinates should be clamped between 0 and 1, with (0,0) the top left corner and (1,1) the bottom right corner. In our case, you need to know the distance between two adjacent tip or base vertices and the total length of your mesh, and you can calculate the relative proportion and use that as UV's horizontal value. For vertical, it is just 0 and 1 since the stem should cover the total vertical length.

With all these settled, it is possible to create a powerful weapon trail plugin, but surely it demands dedication... I am still thinking about whether it can be fun to support extremely curved swords (which will need extra points to define the blade), or even whips (that can be diabolical).