You can use get() to find the color of a specific pixel... cycling through all of the pixels will allow you to find areas of the screen that aren't drawn on. Using loadPixels() / updatePixels() and the pixel array would be faster...

A different approach: If you are displaying the rectangles at random locations, could you store the random values for the check? That might overcomplicated things...

Also, you don't need to use beginShape() / endShape() / vertex() to display rectangles... you can use rect()...

I think I understand... basically, you can't draw anything until you have already calculated all of the locations of the polygons. What about using a PGraphics?

Well I have already done the part of drawing polygons using beginShpae already. I just draw one with knowing the position (since it is only one corner of each polygons is random, others corners are same position all the time), then use translate to draw ..let's say 30 set of polygons(three polygons as one set). 

Now my problem only need to check those polygons are covered the restricted area or not.

Should I change my program and use PGraphics to draw those polygons at the beginning?Is it more easy to detect which area is not covered? If I use PGraphics to do it, how can I detect those area that not covered? 

PGraphics is just like the regular canvas, except it's offscreen (it won't draw to the screen) and you have to prefix all of the drawing calls everything with the name of the object... I don't have much time right now, I wish I could give a better explanation...