This is not a library, but one method would be to compare the square / circle that appears on the screen to a perfect square / circle. You can do this by:
1. Set the pixels making up the square / circle in question to either exactly black or white. For example, you could say all pixels with value less than 128 are black, otherwise white.
2. Get the topmost, rightmost, bottommost, and leftmost pixels of the square / circle in question. Subtracting leftmost from rightmost gives you the width of the shape in question and the same method can be used with topmost from rightmost to get the height.
3. Compare it to a perfect square and then to a perfect circle. This can be done by checking each black pixel's nearest neighbor in the perfect square / circle. A poorly drawn square will still be more like a perfect square than it will be like a perfect circle. This step actually involves a few things:
i. Resize the perfect square / circle being tested to match the shape in question. The left and top position of the shape in question was found in step 2. Set the left and top of the perfect square / circle to this. Then set the width / height to be the same as the ones from step 2 as well.
ii. Now that the testing shape and the perfect shapes are aligned, perform the nearest neighbor check. Unfortunately, this is slow. What you have to do is: for every testing shape's black pixels... compare it to all of the perfect shape's black pixels... find the closest one... add it to a tally... do this for all testing shape's black pixels.
iii. Once the tally is found for the distance to a perfect circle and the distance to a perfect square, then the one that is less distant is your answer. This can be wrong occasionally if the shape in question is ambiguous to the computer.