This is indeed a problem I've encountered.

So far every library I've seen has used its own form of Vector. For example, in my own personal "library" I use Vec2 for 2D, although I know that similar ones exist in other libraries.

The problem with making a "generic" vector library is that everyone has their own style of doing things. If Traer used my Vec3 objects I think it would drive him nuts. I've added a ton of functionality to it, including VecX support for all shapes vertex(Vec3 v) for example. Also, for Vec2 I've included polar coordinates support so you can easily get x,y from an angle and distance.

Personally I feel this is better left untouched, however I wouldn't cross my fingers on that. Rumor has it there's a PShape class coming in the future...

I would tend to agree with all of those pros and cons; however, I've got to say that I am pretty strongly in the pro-Vector3d camp, with some caveats.  The fact is that just about everybody uses a version of this class, and this is going to become quite a problem when the number of available libraries grows further; there are only so many names you can come up with for an N-dimensional vector, after all, so when Processing leaves beta and the libraries start really pouring in, we're going to be in for a lot of nasty namespace conflicts, not to mention the need to build vector-vector adapters for each pair of libraries that don't happen to conflict with each other if you ever want to let them interconnect.

I understand your speed concerns completely.  When you have vectors as a class you tend to throw them around just about everywhere, creating and abandoning them recklessly just because the math makes better sense that way.  So they certainly should not be used in the core internals, especially the crucial areas.  However, in most sketches that I've seen and written, the bottleneck is almost never caused by the difference between a few floats and a Vector3d object unless you're doing something fairly naughty anyways.  Generally speaking, I find that the increase in productivity from using vector objects heavily outweighs the minor decrease in speed, though that's not to say that careful inlining of frequently used expressions that would otherwise be fully vector-based doesn't speed things up immensely in a pinch!

Given my druthers, I would probably prefer to see a well chosen vector library included in the default list of libraries rather than as a change to the core, mainly because you're likely to break a lot of code by adding a new internal class (though I suppose few people are likely using PVector3d, so I guess that trouble could be minimized).  This would have the effect of establishing an effective standard without forcing it on anybody, and I think it would be a fair and useful compromise; while I am perfectly aware that standard vector libraries exist, the fact is, I have never used them because they are a few clicks further away than the "Sketch" menu.  If you did decide to integrate it into the core, though, one option to consider on the speed front would be adding some automatic unrolling of simple vector operations as a preprocessing step - I remember from programming in Flash (where the object penalty is significantly higher than in Java) that some vector operations could be fairly easily recognized and inlined into equivalent float manipulations, especially when local vectors were involved.  This might get a bit messy, and I suspect adding another layer of preprocessing is not something you would probably like to do, but in a lot of cases it can really speed things up, especially if you are aware of the types of things that can and cannot be automatically optimized in this way.

Dimensionality: Clearly 3, maybe 2, though I don't know whether it's even worth it; I usually use 3 anyways so I can do stuff with cross products.  4 can be occasionally useful but it seems like overkill since we already have the graphics renderer underneath - are any people using 'em in Processing?

API: IMO, if the operation has a symbol that an undergrad math major would recognize, it should probably be in there - arithmetic, dot, cross, normalize, norm, normSquared (save a square root if you can) and projection are the big ones for me.  Apart from that, maybe some simple rotation functions and a way to be acted on by a matrix or quaternion.  On the other hand, if this stuff was added merely as a standard library, it might make sense to include a whole crapload of utility functions, I'm not sure.  The only other thing I would suggest is to have both mutator and vector methods - i.e. vecA.plus(vecB) returns a new vector, whereas vecA.add(vecB) merely alters vecA.