If you used grease (like I'm sure you did), it probably needs time to work its way into the hinge. Depending on the temperature this could take a while.

What you might try is warming the hinge with a hair dryer or low setting on a heat gun and work the door back and forth. That would cause the grease to melt its way into the joint, which is probably still pretty tight as you say the passenger side gets used way less than the driver's side.