Not sure if there are guidelines for providing callback functionnality. It's more like a design-pattern :
http://en.wikipedia.org/wiki/Callback_(computer_science)

When you are coding a library and you want to provide the ability to execute custom codes on some events, you use callbacks. In Processing, it's just a function that will be called by the library when some events occur.

For example, proMidi will call noteOn() when a note is played. SoundCypher will call your handleCallbacks() method when it reaches a previously registered callback. You could think of the core UI methods - mouseClicked(), keyPressed(), etc. - as callbacks too (methods call by Processing's core on mouse or keyboard events).

To provide a callback, you just need to make sure that the method you will call does exist. There is a how-to on the wiki :
Library Basics > Adding Your Own Library Events

A more used pattern in Java (and lot of OOP languages) is to pass an object defining a function of a given name. In general, the name is defined by an interface that these objects must implement. Thus, you have a guarantee that the object will have this function. And the method getting this object knows it can call this function.
That's how event listeners function in Swing, for example.

can you give a small example? I got lost :)

Actually, I was answering the original post, without noticing it was a year old...
Anyway, let's illustrate the method with interfaces with a lame but simple example:

1. C c;
2. R r;
3. // Define the elements of the list by their least common denominator...
4. List<Growable> items = new ArrayList<Growable>();
5.  
6. void setup()
7. {
8.   size(500, 500);
9.   smooth();
10.   c = new C(20);
11.   r = new R(30);
12.   items.add(c);
13.   items.add(r);
14. }
15.  
16. void draw()
17. {
18.   background(255);
19.  
20.   for (Growable g : items)
21.   {
22.     g.draw();
23.   }
24. }
25.  
26. void keyPressed()
27. {
28.   int gr = 0;
29.   if (key == 'g')
30.   {
31.     gr = 5;
32.   }
33.   else if (key == 's')
34.   {
35.     gr = -7;
36.   }
37.   for (Growable g : items)
38.   {
39.     g.grow(gr);
40.   }
41. }
42.  
43. interface Growable
44. {
45.   // Objects can grow
46.   void grow(int n);
47.   // And can be drawn (could be put in an interface named Drawable, for example)
48.   void draw();
49.   // You can define other methods
50. }
51.  
52. class C implements Growable
53. {
54.   int diam;
55.  
56.   C(int r)
57.   {
58.     diam = r;
59.   }
60.  
61.   //@Override
62.   void draw()
63.   {
64.     fill(#55AA88);
65.     ellipse(100, 100, diam, diam);
66.   }
67.  
68.   //@Override -- Not usable in 1.5, and 2.0 is still at Java 5 syntax, not usable on interface implementations!
69.   void grow(int n)
70.   {
71.     diam += n;
72.   }
73. }
74.  
75. class R implements Growable
76. {
77.   int size;
78.  
79.   R(int s)
80.   {
81.     size = s;
82.   }
83.  
84.   //@Override
85.   void draw()
86.   {
87.     fill(#3388AA);
88.     rect(300, 300, size, size);
89.   }
90.  
91.   //@Override
92.   void grow(int n)
93.   {
94.     size += n;
95.   }
96. }

You can use that in a library, but indeed the callback by reflection is more in the style of Processing.

thanks.

Note on your code using reflection: you search a method with a FancyLibrary parameter, but you wrote it without parameters.
Here is a working version:

1. import java.lang.reflect.Method;
2.  
3. FancyLibrary fancyLibrary;
4.  
5. void setup() {
6.   fancyLibrary = new FancyLibrary(this);
7. }
8.  
9. void draw() {}
10. void mousePressed()
11. {
12.   fancyLibrary.makeEvent();
13. }
14.  
15. void fancyEvent() {
16.   println("works!!!"); 
17. }
18.  
19. class FancyLibrary {
20.   Method fancyEventMethod;
21.   PApplet parent;
22.  
23.   public FancyLibrary(PApplet parent) {
24.     // your library init code here...
25.     this.parent = parent;
26.     // check to see if the host applet implements
27.     // public void fancyEvent(FancyLibrary f)
28.     try {
29.       fancyEventMethod =
30.         parent.getClass().getMethod("fancyEvent");
31.     }
32.     catch (Exception e) {
33.       // no such method, or an error.. which is fine, just ignore
34.       println("noooo");
35.     }
36.   }
37.  
38.   // then later, to fire that event
39.   public void makeEvent() {
40.     println("Calling " + fancyEventMethod);
41.     if (fancyEventMethod != null) {
42.       try {
43.         fancyEventMethod.invoke(parent);
44.       }
45.       catch (Exception e) {
46.         System.err.println("Disabling fancyEvent() for because of an error.");
47.         e.printStackTrace();
48.         fancyEventMethod = null;
49.       }
50.     }
51.   }
52. }

I changed it a litle.