Concepts

Full Scene AntiAliasing is a relatively new hardware method to smooth jaggy edges. The exact implementation differs depending on the GPU but it generally works by allocating multiple (currently 2, 4, or 6) samples per pixel, rendering to each of them with small offsets, and then averaging the results. This improves the quality of the image compared to regular aliased rendering, but at a significant cost in terms of video memory and fill rate. Regular (GL_SMOOTH) antialiasing and FSAA are mutually incompatible. Also, some GPUs have sizing bugs with FSAA point and line rendering.

Results

Here is the test pattern in the supported FSAA modes for each OpenGL renderer under Mac OS X 10.3.2 (GL_MULTISAMPLE_ARB enabled and GL_MULTISAMPLE_FILTER_HINT_NV set to GL_NICEST.)

Software renderer, Rage 128 and Rage 128 Pro Does not support FSAA. Everything appears aliased.
Radeon 7000 Supports 2x and 4x. 2x doubles the samples vertically, 4x then doubles the samples horizontally. Points larger than 1 px are unaffected. Line width and stipple length is halved. Textures are smudged (as visible in the Renderer name at the bottom.)
Radeon 7500 Supports 2x and 4x. Identical to Radeon 7000, except for slightly different blending near the center of test N.
Radeon 8500, 9000 and 9200 Supports 2x and 4x. Identical to Radeon 7500, except for slightly different rounding in test B.
Radeon 9600, 9700 and 9800 Supports 2x, 4x and 6x. 2x and 6x add a sample diagonally right-up/left-down. 4x adds a sample diagonally left-up/right-down. Point and line scale are unaffected and textures are not smudged, but even at 6x there is obvious banding since only 5 intermediate shades are created.
GeForce2 MX Unofficially supports 2x and 4x. 2x doubles the samples horizontally, 4x then doubles the samples vertically. Point size and line width are both halved, stipple length is unaffected. Textures are smudged.
GeForce4 MX Unofficially supports 2x and 4x. 2x smudges everything diagonally. 4x is identical to GeForce2 MX 4x.
GeForce4 Ti Supports 2x and 4x. 2x smudges everything diagonally. 4x smudges everything some more.
GeForceFX 5200 Supports 2x and 4x. 2x smudges everything diagonally. 4x smudges everything some more, with slightly different results than the GeForce4 Ti.

Summary

As you can see, in addition to all of the Hardware AA problems, FSAA adds a few more; the sizes are wrong on some GPUs, and because everything is rendered aliased before averaging, points always show up as squares. In addition, the antialiasing quality of FSAA, even 4x or 6x, is worse than the regular antialiasing of some GPUs (Radeon 7000-9200 for example) since the multisampling method of FSAA results in at most 5 intermediate shades, while 3 subpixel bits gives us 7 intermediate shades. (Quartz uses 255 intermediate shades.) Finally, although the specification says that you can disable multisampling in a FSAA buffer, in order to mix aliased and antialiased rendering, this in fact does not work on some GPUs. So once you create a FSAA buffer, all primitives are antialiased or blurred.

FSAA might be an OK solution for 3D scenes using only triangles, where the geometry complexity makes sorting back-to-front prohibitive, if you have the VRAM and fillrate to spare, but the quality is too poor for my applications, and the size halving problems with points and lines would require a lot of GPU-specific workarounds.

Another solution is to use the texturing hardware to perform antialiasing. It turns out that textured triangles are pretty reliable across all of the supported Mac OS X GPUs. This approach requires more effort on the part of the developer, but produces much better results.