"..." He can't really say anything to defend himself here-

That's exactly what he'd planned on doing. His family knows. The crew would learn eventually. Jim wasn't supposed to find out until after he got back- above all else he needed to know they'd be able to focus on the mission. Keep their minds out in the stars where they belonged.

"I came here to be comfortable." Which is as much of an admission as it is a truth.