Hi Xian,
See the article:
Paris to Chateau Fontainebleau for detailed instructions on that trip. From the Louvre area you would take Metro Line 1 in direction Ch. de Vincennes until
Gare de Lyon and then follow the instructions in the above article.
Chateau de Vincennes is also very easy to get to. Metro Line 1 from Louvre-Rivoli (or whichever Line 1 station is closest to you) in direction of Chateau de Vincennes will take you directly to the doorstep of the Chateau. I quite like this chateau. It's very "classic" with its moat and well thought out design.
To visit Chateau Malmaison take the
RER A train from
Chatelet Les Halles station (north of the Louvre) in direction St-Germain-en-Laye or Poissy or Cergy-le-Haut until La Défense (Grande Arche). From this station you'll need to take Bus 258 until stop "Le Chateau". You can also start the trip on Metro Line 1, direction La Défense, taking it right to the end, station La Défense, and continuing on the bus as before.
You can purchase either single tickets (
billets Ile de France) for these trips, or simply purchase a
Paris Visite transit pass covering you for
all 6 zones.