no solution?  

you get better results when using the P2D renderer

 size(600,400,P2D);

?!

Is this perhaps a platform specific issue? On my machine, (MacOSX 10.6 running Processing 1.2.1), smooth() seems to work as expected. Below is a rotated image without smooth() (left) and with smooth() (right):

PImage img;
float rotAngle;
boolean paused;

void setup()
{
  size(400,400);
  smooth();    // Comment this line out to see unsmoothed image.
  img = loadImage("apple.jpg");
  rotAngle = 0;
  paused = false;
}

void draw()
{
  background(255);
  if (!paused)
  {
    rotAngle+= 0.01;
  }
   
  // Rotate and draw the image
  translate(width/2,height/2);
  rotate(rotAngle);
  translate(-img.width/2,-img.height/2);
  image(img,0,0);
}
  
// Allow the rotation to pause to see the anti-aliasing.
void keyPressed()
{
  if (key == ' ')
  {
    paused = !paused;
  }
}

Indeed, on Windodws XP, with smooth(), it is much slower (doing a better job inside, seen on text for example) but still has jagged borders.

yes it's exactly my problem (I have win 7) :

on Windodws XP, with smooth(), good  job inside but still has jagged borders.

But I have to get correct issue with Windows. I'm sure there is someone who managed to fix that, it's too problematic to be ignored!

I have an idea : add a transparent border to the image, so that the border will be inside, where smooth is ok. I'll try this!