You need an optimization algorithm such as Simplex, since this is a minimization problem.
In vector geometry, a point
p of a plane is described as
p =
p
0 + s *
v
1 + t *
v
2 (underlines taken to design vectors)
p
0 = some start point
v
1 ,
v
2 = direction vectors,
v
1 not equal
v
2
s, t are scalars. For simplicity we can assume them being = 1.
Your four facet vertices
f
1,
f
2,
f
3,
f
4 need to have minimum distance to
p (being unknown), which means that the sum of the squared distances must be minimal:
R = sum{i = 1...4} (
f
i -
p)
^2 = minimal
This is the equation to be solved. So, for p you have
p
0x v
1x v
2x
p = (p
0y) + (v
1y) + (v
2y)
p
0z v
1z v
2z
with the individual components p
0x,p
0y,....v
2y,v
2z as unknowns (i.e. you have 12 knowns with your 4 vertices and 9 unknowns for your plane).
For the optimization algorithm, you specify a good initial guess for
p, e.g. something like
0 1
0
p = (0) + (0) + (1) (xy plane) or the guess phi.lho has suggested,
0 0
0
and calculate R. The optimization algorithm then varies your unknowns so that R is minimized.
A simple algorithm (which however, depending on your start value and the form of the R landscape might use a lot of optimization steps) is Simplex (do a search for Simplex + Java + algorithm). More complicated algorithms use gradients or the Hessian matrix and may converge faster, but for the low dimensionality of your problem (only 4 points in 3 dimensions) might be overkill.
When you have a optimized parameters for the equation for
p, you then can calculate the projection of the facet vertices on the plane.
(Comment: In Edit mode, the message shows sub- and superscripts, but when published, unfortunately it does not!).