I've spent the better part of two weeks trying to figure out what changed in the REX renderer that breaks our custom shadow projection shader. We use some built-in GLSL global variables to transform the model space vertex to our own shadow view volume using the following:
In the fragment shader, we are just testing this texture coordinate to see if the current fragment is within the volume:
if (gl_TexCoord.s > 0.0 && gl_TexCoord.p
// do something
I've played with the function ordering declaration in your vertex shader so that it is called last. I've also put it through GPUPerfStudio to see if I could trace the built-up shader code looking for anything that would change texture coordinate values. The only thing I can think of is the texture unit 2 may be getting used, but I tried several higher indexes to no avail. I even changed to a custom global variable to avoid the gl_TexCoord[n] writing to see if that was being overridden - no luck. I even turned off most of the REX functionality to avoid morphing in the .earth file:
I forced the texture unit to 3 all through the pipeline and it's still not showing in the render.
Does the new REX engine do anything to the gl_EyePlaneS array? I'm only taking a viewspace position, transforming that into the projection frustum using the gl_EyePlaneS, gl_EyePlaneT, gl_EyePlaneR, gl_EyePlaneQ vectors. This transformed point is then written to the gl_TexCoord array. At the fragment pass, I'm simply testing if the gl_TexCoord.s > 0 and less than gl_TexCoord.q, likewise for t and p < a user defined depth value. If it passes this test, I write a green pixel.
The only thing I can think of would be the transform isn't correct. But that would mean it's not writing to the gl_EyePlane matrix properly?
Down the road you might want to consider rewriting your code without TexGen. It's been deprecated since GL 3.1. The modern approach is to calculate your texture projection matrix on the CPU and send it to the shader via uniform.
I do actually compute it in code and pass it down as a uniform. I write to the gl_EyePlane matrix in the code.
I see in RexTerrainEngineNode.cpp that you reserve all texture units from 0-5 from the SamplerBinding class. I therefore changed my texture unit to 6 and it's still not passing my frustum texCoord test.
I'm going to look into some more code and see what I can find.