Here is a simple way to set up render passes to render out different channels of your 3D image for use in compositing and post processing. Maya 2011 has this process more or less built in and it's able to render out a multi layered Photoshop file or a set of tiff or iff files that can be imported into Photoshop, After Effects or another compositing program.
If you're looking to just render an Ambient Occlusion image by itself, check out this post.
To get started, launch Maya and open up the scene you would like to render. For this example I am just going to use a simple arrangement of primitives to illustrate the principles of multi-pass rendering. In this example we will set up render passes for specular, indirect, reflection, ambient occlusion and diffuse.
This scene consists of a plane, sphere, cube and pyramid primitive, and an infinite light as the primary light source. I used mia_material_x_passes as the primary material type for the materials in this scene. It is not absolutely necessary to use _passes materials in your renders, although it does give you greater flexibility.
I loaded an HDR image into the image based lighting option under the indirect lighting tab (Click create under the Image Based Lighting and load an HDR image into the file field in the Attribute Manager). This will add a touch of indirect and warm lighting to our scene, and it will also be on its own adjustable layer.
I then unchecked the box for primary visibility for the image based lighting mental ray node, so the HDR image would show up in the reflections and global illumination but the actual image would not show up in the background of the render.
Once all of the elements of the scene are set up, you can begin to tweak the render settings.
Under the Render Settings dialogue, make sure Render Using is set to mental ray.
If mental ray is not an option you may have to load the plug-in by going to Window -> Settings/Preferences -> Plug-in Manager.
Check the box to load the Mayatomr.bundle plugin. It is worth it to check the box to auto-load this plug-in every time Maya starts, if it is not already checked.
In the File Output section you can right click in the File name prefix text field and choose some predefined naming conventions to better organize the name of your rendered files. I like to use scene name. If the final output of your project is a still image, I recommend choosing PSD Layered (psd) as the Image format. Otherwise, if you are rendering to a movie I recommend using tiffs or iffs and importing the image sequences as a separate layers/tracks in your compositing program (if using separate image sequences it helps to use the File name prefix:
Next up is the Passes tab. The Passes tab is split into two parts, Scene Passes and Associated Passes. Scene Passes are the global passes that can be applied to the entire scene. The Associated Passes are the passes that are applied to the current Render Layer. You can go through and set up all the different passes you think you need in the Scene Passes panel and then you can selectively apply these passes to each Render Layer as you need them.
For example, if you have a bunch of low-res, hard to see background objects, it will decrease your render time if you put them on a separate Render Layer and don't include the Ambient Occlusion Pass on that layer. That way Ambient Occlusion will not be calculated on the objects that are in the background and it most likely won't make any noticeable difference.
Click on the add new passes button in the Scene Passes to add passes for Ambient Occlusion, Diffuse, Indirect, Reflection and Specular.
Now that you've created these passes you are now ready to add them to the Render Layers you want. Since this is a simple scene, there is only the Master Render Layer and that is the layer that is currently selected. You can highlight all the Scene Render Passes and click the Associate selected passes with current render layer button.
All the Render Passes should switch from the Scene Passes to the Associated Passes panel.
Under the Quality tab choose the Production Quality quality preset. If rendering is becoming too expensive you might want to start at the Production Quality preset and gradually lower the settings/quality until the render time becomes more acceptable.
Now that the Render Passes are associated with the Render Layer you just need to set up some additional parameters to get everything to render correctly. Under the Features tab, make sure Raytracing, Global Illumination, Final Gathering and Ambient Occlusion are checked.
Also, under the indirect lighting tab I made sure Global Illumination and Final Gathering were checked to add a bit of indirect lighting to the scene. You may need to play with the Accuracy and Secondary Diffuse Bounces to achieve the level and quality of indirect lighting for your scene.
You can now switch to the Rendering menu set and choose Batch Render from the Render drop down menu.
You can open up the Script Editor to view the progress of your render.
Once the render is finished you can open up the multi layered Photoshop file in Photoshop to begin compositing your image. By default all layers are set to the blend mode of Normal and the overall Beauty Pass will be visible as well.
Hide the Beauty Pass and move the Diffuse pass to the bottom of the layer stack, just above the background layer.
Move the Ambient Occlusion layer to the top of the layer stack and change the blend mode to Multiply.
Change the blend mode of the Specular layer, Reflection Layer and Indirect layer to Linear Dodge (Add).
Your image should now look identical to the original Beauty Pass layer, however now each element of the render is separated out to its own layer. You can change the opacity of the Ambient Occlusion layer to adjust the intensity of the effect. Likewise, you can adjust the opacity of the reflections/specular to change the material properties of your objects. You can also add adjustment layers to the diffuse pass to change the color of your objects without affecting the reflections or Ambient Occlusion.