This post was updated the 15 August 2013.
When UDK imports any FBX it tries to find any smoothing group. With Maya there is no smoothing groups but the FBX exporter is in charge to make them. Smoothing group while defined by faces in any softwares supporting this system are stored per vertex from what I know. In Maya there is multiple ways to manipulate normals, the most commons are :
-Set to Face/Average normals
-Harden/Soften edge normals
The visual result in Maya is the same, but behind the scene it’s not the case. While the Soften/Harden actually work correctly and merge similar vertex normals, the Average/Set to Face orient the normals BUT don’t merge them if there are similar. So if you make a faceted mesh (every face has its own shading) and then do an “Average normal” on top, you get visually a smoothed and uniform shading, but every vertex normals are still separated, even after the export.
Here is a little test with two cubes :
What is the problem ?
Since the release of Maya 2013 and 3DS Max 2013 the developers at Autodesk did some updates to the FBX file format. Mostly a new way to compute bi-normals and tangent which are stored with the vertex normals of your meshes. Unfortunately, since this update, the UDK (tested with UDK November 2012 and UDK February 2013) is unable to import correctly your meshes, at least the FBX exported from Maya.
Have you an example ?
The result of a mesh imported with this error is really bad. The UDK is unable to read the bi-normals and tangents correctly and therefor splits them on every vertices. So the meshes imported get a lot of wrong vertices.
For example, a skeleton mesh made with 20 000 triangles is imported with 73 000 vertices (which is over the 65 535 limit of the 16 bit index used to store the vertices with static meshes). The result is an heavy mesh than is full or bad geometry, errors and so on. The same mesh, final version with 30 000 triangles, imported as a skeletal mesh is only 17 000 vertices.
Other example below, some 3D letters that I made with no vertex normal splits (aka one smoothing group). you can see the incorrect vertex count on the left with the FBX from maya 2013, while on the left with FBX made in blender everything is correct :
Hey, I just have to install an older version !
Unfortunately, using an older preset for exporting doesn’t solve the problem, neither installing an older version of the FBX plugin. Maya 2011 with the FBX 2011 plugin doesn’t work because the UDK is expecting FBX 2012 files. And under Maya 2013 you can’t go back to Maya 2011 or even before (because good version of the plugin doesn’t exist). The FBX converter tool doesn’t work too.
In the end, there is no way to get a “good” FBX file ready to be imported correctly.
What is the solution ?
From my experiences (based on a lot of tests and tries) you can only get a good FBX with the following situations :
-exporting from Blender with the default FBX exporter
-exporting from Modo 701
-exporting from 3DS Max 2010 with FBX 2012.2
As Modo and 3DS Max are expensive software, I decided to create a batch converted based on Blender (which is free).
The following zip archive contains a standalone Blender (plus an OpenCollada fix for the Maya 2013 crash) and the Maya script to export and convert your meshes. You will to install the OpenCollada plugin to be able to export.
How to install ?
Put froFBX.mel inside your Maya script folder
(like : C:\Users\Froyok\Documents\maya\2013-x64\scripts )
Install the OpenCollada plugin. If you are under Maya 2013 x64, you will have to apply the fix given in the zip archive, otherwise the exporter will crash every time you will try to export.
To launch the maya script use the following command :
source froFBX; froFBX();
How to use ?
Specify your export folder and the path to Blender.exe The select the meshes that you want to export and use the export button. You will see a command line window for every meshes that you will export. Each window call Blender to convert your Collada file as an FBX file.
Once done, simply import your mesh in UDK with :
-“Import tangents” disabled
-“Explicit normals” enabled
If you want to use your own Blender installation, you will have to put the “run.py” script inside the “Blender266_x64″ folder in you own blender folder (where Blender.exe is).
If you want to test by comparison, I made one of my own mesh available : Download test mesh
This mesh when imported correctly in UDK is made only of 1607 vertices. If you get a bigger number it means the import is incorrect.
–Why not using the Collada Exporter from the FBX plugin ?
Simply because under Maya 2013 x64, after 3 export as Collada, Maya will crash. Also, the FBX Collad exporter as less export options than the OpenCollada exporter.
–Vertex normals impossible
Since we are using Blender to export, you will lose the control of your vertex normals, because blender compute the vertex normals based on a smoothing group system. Blender also doesn’t understand vertex normals split unless you break the geometry. That’s why I have added an option to “split” the geometry during the export. So in the end you have the choice between : smoothed or broken, nothing more unfortunately.
xNormal and handplane should not be affected by this problem, you can export and import like before but be sure that your normals used to bake will be the same as the ones generated by Blender, otherwise your normal will not match your geometry.
–With which version of Maya this script is compatible ?
It has been tested with Maya 2011, Maya 2012 and Maya 2013. Also, only blender 2.63 and Blender 2.66 have been tested.
–Is that compatible with Maya 2014 ?
Maya 2014 use the FBX 2014 version which is also incompatible with the UDK importer, unfortunately the OpenCOLLADA plugin is not available yet with Maya 2014. The script should be able to export with the FBX Collada.
–I don’t get an FBX file after the export (I get a 0ko file without a file extension)
Be sure to use a clean name for your mesh under Maya, without a namespace (like “imported_obj:mesh”), before exporting in Collada.