Damn, damn, dammit! Just finished time synching some servers in seperate networks and if I had these tools before it would have taken half my time.
First the tool I used to diagnose the problem and check which servers were in sync: NTPmonitor.
Then I used this document to setup the server to use NTP sources and synced everything to a reliable NTP source.
Of course I used a stratum 2 source as these servers will not be available to the public. Use your preferred search engine to find public NTP servers.