Indeed, it can be simpler if the polygon is convex!

But overall, it is just matter of computing the normal of the polygon edge being hit, then compute the angle symmetrically with regard to this normal.
In most simple simulations, there is no friction and velocity is kept as is, although you can multiply it by an arbitrary amount (below 1).
If you need a realistic billiard simulation, I fear you have to search a bit the Internet, as the involved physics are quite complex.

Do you mean "convex" or "complex"?

I'm bouncing circles around inside a hexagon's walls and against one another.