Yes, you can get the pixels back from the screen by using FrameBuffer.getPixels(); As long as you are using the normal GL renderer, this should work fine. When using the AWTGLRenderer, make sure that you don't call it from inside the awt event dispatch thread or else you'll wait forever for the pixels.
However, i'm not sure if this is the best method. Wouldn't it be better to not allow these modes based on the hardware beforehand?