Do you have wheels with more offset fitted?
One of the reasons for the ball joints is to get neutral or negative scrub radius to minimise wobble problems.
I also would look at the bushes in the panhard rod. I once had a set of poly bushes last one trip to the centre.
On balance the first thing I would do is change the panhard bushes.
Regards Philip A

