How to Achieve a Cell Shader Effect
Stanford Dragon, a Vicki, a Bryce sphere and an infinite plane – all selected.
Modify the materials setting the Specular Halo to fully white.
Ambience (global is set to grey as default) controls the appearance of the shadow regions and specular the lit area. No diffusion is needed.
Further modify the ground plane only as shown. Again no diffusion is needed.
This is the basic effect but with a bit of work it is possible to create more complex variants.
Select the perspective camera.
Click "Edit" > "Copy Matrix"
Create a radial light and edit it.
In the Light Lab set Specular to maximum and Diffuse to 0.
Click "Edit" > "Paste Matrix" to place the light on top of the camera origin.
Create another radial light, this time use fully negative Specularity and apply paste matrix to place as before to place the light on top of the camera origin.
Now set movement to object space. This will mean the negative light – which should be still selected, can now move back and forth in line with the camera Z axis.
Move the negative light back a bit as shown.
The effect should now be similar to this.
Now select the positive light so we can fine tune it.
Using the preview window reduce the Specular effect to get the effect you want.
The final result as seen in Bryce 7.1 Pro. Note that you can still achieve some outlining in Bryce 5 onwards using negative lighting.
Shadow regions can cause issues in complex scenes so you might need to turn off the "Cast Shadows" toggle. You will need to do this for both the positive and negative lights.
Feel free to experiment with different colours and sun positions.