for a circle, you just have to check the distance.
Btw you dont have to call the first rec/ellipse in the setup.


int px,py;

void setup() {
size(400,400);
background(150);
stroke(0,0,50);
noFill();
ellipseMode(CENTER);

px = 200;
py = 200;
//ellipse(px,py,50,50);
}

void draw() {

if(dist(mouseX,mouseY,px,py)<=25) {
 
    stroke(0,200,250);
    ellipse(px,py,50,50);
}
else {
  stroke(0,0,50);
  ellipse(px,py,50,50);
}
}

It is indeed non trivial for triangles (surprisingly!), you will find lot of articles on the topic on the Net (eg. on Paul Bourke's site, a good source of geometric information). And it becomes increasingly hard for other shapes.
The solution generally used is to render the shape with a monochrome color on a different background in a test PImage. Then you check the color at mouse position: if it is of background color, it is outside of the shape.

I believe that the generally accepted math solution to this problem is to take each side of a polygon, figure out the formulas for those lines, and use that particular formulation of the distance-from-point-to-line formula that yields positive numbers on the "left" side of the line and negative numbers on the "right" side.

If you arrange all the sides of the polygon in clockwise order (or counterclockwise, as long as you're consistent), then points that are inside will yield the same sign of the answer from that distance formula.  Points that are on the outside will yield mixed positive/negative results.