What defines the boundary of a square?  not so difficult to define and test for in a set of conditions...  So when you place your dots check that they're not within the bounds of the square.  If they are place them somewhere else...

Well it's pretty easy which is why I didn't bother posting any code.

First things first: check how you're drawing the square (see  rectMode)

Now check if the point is inside the square:

Code:

// assuming rectMode(CORNER)
if (dot.x > square.x && dot.x < square.x + square.width && dot.y > square.y && dot.y < square.y + square.height) {
  // dot is inside the square
} 

Note the && (AND) means you combine all the conditions in one test:  all have to be true for the condition to return true.

So, calculate new x,y coords for your point.  If they're inside the square ditch them and try again.  You could use a do while loop to achieve this, though that might be dangerous if the square is particularly large in relation to the size of your sketch (i.e. it will have to continue until it finds a safe point which could take some time).