Which is the best ticket for a day trip to Versailles from Gare du Nord?

If you're only going from Paris Gare du Nord to Versailles and back without any other side trips during the day, then the cheapest option would be single-use Origin-Destination RER tickets which are customized for a starting and ending station.  These are known as "billets Ile-de-France" and a ticket agent at any RER station ticket window can create these for you.  Or you can try your hand at creating them at automated ticket machines.
 
Each of these one-way tickets (you'll buy 2 per person if you want round trip fares) will cost €4.05 (so €8.10 in total) and will cover you for both Metro and RER portions of the trip.
 
From Gare du Nord you would take Metro line 5 in direction of Place d'Italie until Gare d'Austerlitz, then transfer to RER C in direction Versailles Chateau Rive Gauche (using the same ticket).
 
Be careful when transfering to the Paris RER C train line at Gare d'Austerlitz as there are several terminus stations (directions) and you'll specifically want "Versailles Chateau Rive Gauche".  That name will likely be abbreviated on signs as it's very long, but you'll need this specific direction, not any others.