An XMSL shader baking hack for Max 2011

[Zoltan]

Unfortunately the usual way of rendering to texture (still) doesn't work in 3DS Max when
using XMSL materials. However it is possible to hack around the bug.

So you have a high definition model with a very complex XMSL shader, the look of which you
want to transfer to a low def mesh.
Instead of the complex shader, load a simple one with only a diffuse and a normal map, and
set it's glossiness to 0. (With this it will be easier to see how the hack works.)
Make sure that the XMSL shader is properly set up for Max. (See the rules in the help file
and here in the forums.)

Open the slate material editor. The HD object has a DirecX shader, which is fine while
editing the shader in the HW shaded viewport. However before baking, you need to add two
more nodes: a "Material to Shader" and an "Arch&Design". Link them like this:

[DirectX Shader] - [Material to Shader] - [Arch&Design(Self Illumination Map)]

In the last node set the following parameters:

- Diffuse level: 0.
- Reflection: 0.
- Self illumination: on.
- Luminance: unitless.
- Visible in reflections: off.

Assign this material to the HD mesh and set up the LD mesh for baking. For output maps
add "CompleteMap" and "NormalMap" (others will be black anyways).

If you do a test baking at this point, you'll see that the "CompleteMap" captures the
shaded look of the HD mesh (make sure you have some lights in the scene). Bump or
normal maps will have no effect due to a bug, but let's not worry about that.

A more evenly distributed lighting is needed so turn off the lights for now and make a box
around the HD mesh. Flip its polygons, in the object properties turn on "Backface cull"
and turn off "Visible to camera" and "Receive/Cast shadows".

Create an "Arch&Design" material and assign it to the box. In the material set these
properties:

- Diffuse level: 0.
- Reflectivity: 0.
- Self illumination: on.
- Luminance: Physical Units: 700 cd/m2.
- Visible in reflections: on.
- Illuminates the scene: on.

Bake stuff and if everything went well you'll see the diffuse texture (with some applied AO)
saved as the "Complete Map".

To bake the rest of the channels (specular, normal, etc) you need to add a few "Math
conditional" nodes to the XMSL shader. Make sure the condition parameters are used as
inputs for the material node in MentalMill.
Set them up so the shader behaves like this:

- If all newly added conditions are false then the shader works normally in the HW shaded
viewports. This mode is used for designing the shader and tweaking the visuals.

- The first condition, let's call it "BakeDiffuse" is turned on: The surface has only the
diffuse map applied, otherwise it has 0 glossiness, 0 reflections, 0 normal maps, 0
bumpmaps, 0 self illumination. This way the diffuse map will be baked.

- If the second condition, "BakeSpecular" is turned on as well then the specular map is
used as diffuse texture, everything else is zeroed out. Now the specular map is baked.

And so on, the rest of the channels follow the same logic.

It should be noted that the "normalmap channel" baked this way (so when the normal
texture is used as diffuse color) will not contain the geometry difference between the HD
and LD meshes. That data is stored in the "NormalsMap" output of the render to texture
function. The two image should be mixed later in an image editor.

Well, this is the basic idea. Now let's recap the workflow:

- Model HD and LD meshes.
- Create the HD XMSL shader using a hardware accelerated viewport (MentalMill or Max).
- Add the neccessary setup to the shader so it can relay different aspects of the material
as diffuse color.
- Make sure you link the conditions as material inputs.
- Set up baking in Max: Render to texture setup, lights off, illumating box around the HD
model.
- Instead of the original DX shader, use the self illuminating "Arch&Design" shader on the
HD mesh.
- Set conditions to what you want to bake.
- Render to texture.
- Rename the generated CompleteMap according to its contents.
- Change shader conditions, render, rename, rinse, repeat.
(You can also bake arbitrary surface data this way: physics material IDs, guide gradients
for special effects, masks for color customizations, etc.)

There is a problem however, a big one: there are rendering artifacts on the baked image:



Their location depends on the orientation of the camera/perspective view. I have no idea
what's going on or how to fix it. :\

So the workflow is convoluted, the XMSL shader is made outside of max and needs extra
care in order to work with the hacky baking process. Why bother with all this when
everything can be done using the standard Max/MR materials?

My main reason: instant feedback. I don't want to wait for the software renderer to see
what I'm working on. I'm making assets for games so I don't need many of the features of
MentalRay: raytraced reflection, refraction, dispersion, physically correct lighting are all
unneccessary to me.
What I want is realtime feedback instead.

Today's DX10 cards are so ridiculously powerful that I must work really hard to make
them sweat. They can easily handle the high def model with a few million polygons and a
dozen 4K texture maps. And even if the viewport slows down to 10 fps, its still way better
than waiting even just 5 seconds for a render.

So that's it. Any and all feedback is appreciated, especially suggestions regarding the
deal breaker render artifacts.

[dutch]

have you found a way to do this in earlier versions of max, without the slate editor?

Thanks

[Zoltan]

I didn't try this in earlier versions. (I really hate the default material editor so I refrained from
using it.)

[dutch]

everytime i try to load an xsml file into a direct x node, mental ray causes max to crash.

How are you getting this to work?

[Zoltan]

I've never had a crash with XMSL. They showed warnings if something was wrong, but no
crash while loading, assigning, rendering or baking the shaders.

Try searching the forums to see if others had this problem.

EDIT: Suggestion removed because an offline renderer won't care about video drivers.

[dutch]

Just fyi, this is what is popping up in mental rays log under the material editor tab.

SYSTEM: Material Editor renderer is changed to mental ray Renderer. Previous messages are cleared.
MSG 0.0 error: Custom translation interface for mtl/tex 'Material #31 ( DirectX Shader )': Shader declaration "HelloWorld" not found.
MCMP 0.0 error 771011: Can not find input parameter 'color1'