Well, all you have to do is to make an algorithm to layout your icons in a given rectangular space.
When your rectangle enters, it defines a reduced rectangular space for the icons, so you apply the layout again on the remaining space.

Yes, exactly.

A way to do "squiggle" could be to use a library like Traer or Toxiclib, with "physics" based on attractors and repulsors. This allows a relative stable state after a while. I have little experience with these libraries, so I don't know how to constraint the icons in an area.

One solution would be to determine the layout the icons after the rectangle has already expanded (before it happens assuming it expands only to a certain point). For each icon, store each's place in this layout as a targetX and targetY coordinate for later. Now place your icons wherever you want at the beginning of the program. When the new rectangle comes in / expands in, do a collision test to see if it is touching one of your icons. If it is, then move your icon in the direction of the target coordinate (and then display so it seems like the rectangle never touched it). The only problem with this is that your icons may overlap each other while they are traveling towards their target so either you will have to do further collision testing / path finding to avoid this or not actually place them randomly but in a way that they won't collide.

Yes, I have looked at the examples of Toxiclib, under the verletphysics folder. There is the Shiffmann_ForceDirectedGraph example which seems appropriate. It uses clusters of nodes, but you can simplify it to be simple nodes, not to show connections, and in Node, to display the icon instead of the circle. The code uses setWorldBounds so they can be restrained in a rectangle. Looks like a way to get quickly what you want.