Tangent space normal maps

Alright, this is some useful stuff. I didn't find very many useful articles online about this, except for Blender.

The basic idea is, if you want to use bumpmapping/shiny/normal mapping, you can do this either for static objects (objects that NEVER have their mesh change, like trees, terrain, walls...) or dynamic objects (animated characters). For static objects, you can use what is called 'object space normals', aka, a texture with normal data relative to the object baked in. For dynamic objects, you are forced to use 'tangent space normals' where tangent vectors are calculated for each vertex each frame in the shader.

This works, because just like matrix palette deformations, you multiply a vertex by a matrix to transform it, either locally or globally. In this case, we have each vertex with a vertex normal, and a tangent, which means we know each verticies local space. This allows us to initially calculate better lighting parameters per-vertex using a vertex shader. As you know, Normal cross tangent or binormal will ive you the other missing vector in the 3x3 rotation matrix.

Now that you have a vertex shader that generates the parameters needed (has to send them via texture coords or other fragment parameters in the shader...) you can write a fragment shader that does your lighting computations, and calculates the per-pixel normal, tangent, and binormal, which then samples the normal map texture for the normal value at that pixel, thus resulting in normal mapped surface.

although this is trivial to do with shader parameters (add an extra array with tangent vectors) the fragment shader is annoying, because per fragment, you must renormalize the interpolated normal and tangent per vertex before applying the transformation to determine the normal to use for lighting equations.

All this blah and hooplah I did using fragment and vertex program. It's pretty kickass, and very simple.

Here is a example showing a generic 'snake' normal map on a realtime deformable tube using hemispherical lighting:



And here is no normal map applied using open GL lighting



This is the normal map, generated from Blender via 'Bake Selected to Active' (select all your scales and bumps, then the cylinder object which has the UV map you want to bake to, and done!)



And here's some nitty gritty low level shader code:

...vertex shader...
# ...pass normal & binormal to fragment as texture coords in 0 and 1
MOV oTex0.zw, vNorm.xyxy; #tex0.zy = normal.xy
MOV oTex1.x, vNorm.zzzz; #tex1.x = normal.z, tex1.yzw = binormal.xyz,
MOV oTex1.yzw, vBin.xxyz; #
MOV oTex0.xy, vTex0;
#...

...fragment shader...

!!ARBfp1.0
# Setup
ATTRIB fTex0 = fragment.texcoord[0]; #tex and normal
ATTRIB fTex1 = fragment.texcoord[1]; #Normal + Binormal

PARAM EyeDir = state.matrix.mvp.row[2]; #Get mvp/eye direction
PARAM Light0Dir = state.light[0].position; #state.light[0].spot.direction
PARAM HemiParam = program.local[4]; #{ 0.5, 0.5, 8.0, 0.125 }#Hemi Light Constants (darkness, 1-darkness, hemi_power, hemi_scale)
PARAM FlipValues = program.local[3]; #Direction flipping values

PARAM ColorAmb = state.material.front.ambient;
PARAM ColorDif = state.material.front.diffuse;
PARAM ColorSpec = state.material.front.specular;
PARAM ColorEmit = state.material.front.emission;
PARAM ColorShininess = state.material.front.shininess;

PARAM K = {1, 0.5, 0, 2.0};

TEMP vNorm, vBin, vTang, temp1, temp2, colTex;

#---Get texture normal (tangent space normal)---
TEX temp1, fTex0, texture[0], 2D; #Sample texture (ext normals)
TEX colTex, fTex0, texture[1], 2D; #Sample texture as color... rgb = xyz, a = ?ColorShininess? would be cool.

MAD temp1, temp1, K.w, -K.x; #Convert texnormal to normal [-1,1] (works)

DP3 temp1.w, temp1, temp1; #Normalize texnormal
RSQ temp1.w, temp1.w;
MUL temp1, temp1, temp1.w;

#---Calculate basis for tangent stuff (realtime, for deforming models. Static models can save the vectors and avoid this)---
MOV vNorm.xy, fTex0.zwxy; #Load in normal
MOV vNorm.z, fTex1.zwxy;
DP3 vNorm.w, vNorm, vNorm; #Normalize normal (correct it)
RSQ vNorm.w, vNorm.w;
MUL vNorm, vNorm, vNorm.w;

MOV vBin.xyz, fTex1.yzwx; #Load in binormal
DP3 vBin.w, vBin, vBin; #Normalize binormal (correct it)
RSQ vBin.w, vBin.w;
MUL vBin, vBin, vBin.w;

MUL vNorm, vNorm, FlipValues;
MUL vBin, vBin, FlipValues;

XPD vTang, vBin, vNorm; #Calculate tangent (pseudo-normalized)

#Make texture normal Global: ??? z/blue = out, y/up = down, x/left = right
MUL vNorm, temp1.z, vNorm;
MAD vNorm, temp1.x, vBin, vNorm;
MAD vNorm, temp1.y, vTang, vNorm;

#---Start of full hemi shading:---
#Req:
DP3 temp2, EyeDir, vNorm; #Eye dot normal = specref vector
MUL temp2, vNorm, temp2.w; #Scale normal by dp and 2.0
MUL temp2, temp2, K.w;
SUB temp2, EyeDir, temp2; #Sub to get specular normal

DP3 temp2.y, temp2, Light0Dir; #Specular dot produxt (specular normal dot light)
DP3 temp2.x, vNorm, Light0Dir; #Diffuse dot product (normal dot light)

#---Hemispherical lighting---
MAD temp2.x, temp2.x, HemiParam.x, HemiParam.y; #t = t*0.5 + 0.5 (hemispherical lighting = cool)

#---Scale the light to be the hemisphere---
POW temp2.x, HemiParam.z, temp2.x; #t = pow(t, hard), flip this to halo (A^B)
MUL temp2.x, temp2.x, HemiParam.w; #t *= spec

#---Calculate final hemi diffuse color---
LRP temp1, temp2.x, colTex, ColorAmb; #Interpolate color to ambient color (mult op)

#---Specular highlight---
MOV temp2.w, ColorShininess.x; #Load specular value into LIT vector... (0..127)
LIT temp2, temp2; #compute diffuse & specular coefficients

#---Calculate final diffuse + specular color---
LRP temp1, temp2.z, ColorSpec, temp1; #Interpolate diffuse to specular color (white = K.xxxx) based on specular component

#---Emissive color addition---
ADD temp1, temp1, ColorEmit; #Add in emission color

MOV result.color.xyz, temp1;
MOV result.color.w, fragment.color.w; #Load final color with initial color alpha
#---End of full hemi shading:---

END



Well, that's all. AS you can see, there is yet another 'art' component to matching good normal maps to good models, but hell, if the quality jumps this much using such a simple configuration, it'll be cakewalk to make next-gen looking characters, well, at least unrealistic ones.

Peace ya'll.

-Z