Spyro,

Basically, you need to convert the image (frame) from Cartesian coordinates to polar, change what you need and convert back to Cartesian coordinates (you would need to interpolate the values in order to avoid artifacts)..

Here is a link  that you might find useful:
http://dem.ocracy.org/libero/photobooth/

It implements several several photobooth-like "filters" and gives away the code as well. It is in cpp and uses GLSL shaders

I have been trying to implement this in openframeworks, but I don't seem to be able to make the shaders work...

It should be a matter of a couple of hours work for someone that is comfortable with opengl shaders.

In principle you don't need to implement it using shaders, but it can run on the cpu. I wouldn't know though how it would do in terms of performance (framerate).

By the way, someone has already implemented it in vvvv

Afta gia twra,

Nikolas