No experience to share, but both solutions seem easy to try, so you should implement both and see what fits more your needs. Basically, it is slow start up time vs. a little slow down on each load (unless you use threaded loading).

Hi PhiLho,

Threaded loading is more used with images, I don't know if it works well with Minim. The idea is to create a thread that would load the sound. The interest is that it can be a real system thread, so it can be run in parallel with the visual animation without slowing it down.
That said, loading all your files in memory seems doable, and since the start up time isn't important, it seems like a good strategy: it can be more stable that loading and unloading files permanently.