max 2010: custom shader works in viewport, but not software

[focomoso]

Hi all,

I'm new to mental mill and mental ray in general (long time max user) and I have to say, I'm pretty excited about how well designed this workflow is. I do have an issue though. I browsed the other max 2010 software render issues and I think this is different.

I wrote a simple custom shader just to test metaSL. It works fine in mental mill and fine in the max 2010 viewports, but when I render in software I get a shader not found message. I checked the .xmsl file and it has an absolute path to the shader in question. Any thoughts as to why it would work in the viewports, but not software?

Thanks,

James

[Olaf Finkbeiner]

Hi there,
there is a bug that if you have "," as windows System Units it might not work. Change your Windows System Units to "." and reboot windows, now it should work. But this is just a workaround, i think a hotfix for this is been worked on at autodesk...

kind regards

Olaf

[PiXeL_MoNKeY]

Olaf, it is a different issue I tested the .msl and .xmsl files on 2 US based machines and same results. Viewport Version works, but medit and actual render return black with error of the missing shader. focomoso, you may want to post the actual files for others to test with.

-Eric

[focomoso]

Thanks Olaf, but as Eric said, this is a US based machine already using ".".

I'm uploading a zip that has 3 files. One is a .xmsl with a relative path, one uses an absolute path and the .msl is the simple shader I'm trying to get to work.

In both cases, using gridTestAbs.xmsl or gridTestRel.xmsl, I can see the shader in mental mill and in the max viewports, but I get the shader not found error when I try to render.

Thanks for any help...

[ruediger]

You are using a custom msl shader that you wrote which isn't prepared for mental ray yet. You will need to precompile your shader and out the .dll into the right directory in 3ds Max. Another way to do this is to get cil compilation to work with mental ray in 3ds Max. We're working on a forum thread that will explain the steps necessary to get this activated. So for now you can try building your effect using base (math) shaders that come with the mental mill Artist Edition. This is a little challenge, but you can get the very same result.

Best Regards,
Ruediger

[focomoso]

Cool. I'll give it a shot with the math shaders and wait for the further info.

Thanks.

[Jerome]

You are using a custom msl shader that you wrote which isn't prepared for mental ray yet....Another way to do this is to get cil compilation to work with mental ray in 3ds Max. We're working on a forum thread that will explain the steps necessary to get this activated...

Hello Ruediger,

Do you know when Mental Ray will be ready to accept custom msl shaders?
Is the thread you were talking about available?

Thank you,

[ruediger]

Automatic MetaSL shader compilation has always been working with the standalone version of mental ray starting from version 3.7+
You just simply include a MetaSL shader in your .mi file like this:



$include "my_shader.msl"
$include "my_workspace.xmsl"


You then have access to the shader inside these files. So if your shaderfile contains a shader called "custom_illumination" you need not put a shader declaration in the mi file, but you can create a shader instance directly:



color texture "tex_01" "C:\textures\image.jpg"

shader "shader_instance1" "custom_illumination"
(
"diffuse" 0.9 0.9 0.9 1,
"specular" 1 1 1 1,
"diffuse_tex" "tex_01"
)

At a later point you can simply assign your shader in a material:


material "material_instance"
= "shader_instance1"
end material

That's the way how it works from the command line:
Start the Visual Studio Command promt - this ensures that all environment variables that are required for compilation are set. Now you can change to your mental ray directory and kick off the render:
When mental ray starts rendering, it will compile the .msl shaders on the fly and use them. Make sure to set the verbosity to 4 to see all the progress messages, e.g.

ray my_scene.mi -v 4

Using this verbosity level you get all the information about what is going on. If there are compiler errors, you can find this out here.

I wasn't successful with the Max cil compilation however. I hope to follow up on that one. I hope that helps. Let me know if you have any open questions.

Cheers,
Ruediger

[Olaf Finkbeiner]

I wasn't successful with the Max cil compilation however. I hope to follow up on that one. I hope that helps. Let me know if you have any open questions.

I would also love to know how to get this done without standalone mr!
But with 3dsmax and the standard edition... According to your standard edition feature list it should actually work... :rolleyes:

thx

Olaf

[Moosley]

Hi!

I was wondering if there were any further updates to this. I have some custom written msl nodes for our shader pipeline, but am still unable to render them using mental ray in 3ds max 2010:

MSG 0.0 error: Custom translation interface for mtl/tex not found.
LINK 0.3 error 191002 modult compilation ... failed.
PHEN 0.3 error 051011: shader '' not found

and several more like it for each of my custom coded nodes.

These shaders are to be deployed to several artists who run just 3ds max 2010, so need to know what I have to deliver to them so they can render the results directly.

Cheers,

SImon.

[ruediger]

Moosley,

can you tell us the steps that you took in order that we can reproduce this?

This includes:
* Which mental mill version you are using
* where you put the files that you created
* which files from 3ds Max you modified (if any)
* etc...

Also, check out this thread here:
It shows how to activate CIL compilation in 3ds Max 2010. Maybe this will do the trick for you.
http://forum.mentalimages.com/showthread.php?t=5353

Thank you,
Best Regards,

- Ruediger

[Moosley]

Hi!

I am using Mental Mill Standard Edition (1.0, build 45261.50496)

I have dropped the files both in our game assets folder on alienbrain (referened in the max material through a Z:\ drive mapping) and also I have put the .msl files (4 of them) in the '3ds Max 2010\hardwareshaders\MetaSL' folder. I have also put a copy of the .xmsl file in there too for good measure. The .msl files are referenced in the .xmsl from the AB location:

<metasl_code file_name="Z:\game_data\shaders\include\cm_hueShift.msl"/>

I'm running 3dsmax 2010 SP1 with the recent hotfix applied.

Would it help if I sent you the files directly? In the meantime please could you let me know where the CIL keys should go so I could give that a go too?

Thanks

Simon.

[ruediger]

You need to enter these values into the 'rayrc' file of mental ray. You should find it under the mentalray directory of your 3ds max installation.

- Ruediger

[Moosley]

Thanks, I added the lines to the rayrc file. It seems to go through a compile step, certainly there is a slight pause the first time you render, however I'm getting the followng:

MSG 0.0 error: Custom translation interface for mtl.tex ' ( DirectX Shader )':Shader declaration "shader_name" not found.
LINK 0.3 error 191034: Cannot load C:\...path_to_localSettings...\Temp\raylink_04116_ aaaa_.dll, The specified module couldnot be found.
PHEN 0.3 error 051011: shader "different_shader" not found
PHEN 0.3 error 051011: shader "different_shader" not found

Where "shader_name" in the first shader is my phenomonen and "different_shader" in the other errors is reffering to one of the 5 or 6 custom nodes I have coded, perhaps it's the first one it tries to get to or something.

Are there any other things I need to setup to declare the shader or should the xmsl file take care of this?

Thanks for the help!

Simon.