I just watched The Enemy Within, and yes,
this is what I thought of Spock's comment, too.
(In context: imposter!Jim ("the imposter") was Jim's "evil" side, which tried at one point to make sweet, sweet rape to the Yeoman that Spock's speaking to here, 'cause, y'know, Jim has feelings for her for like the entire first half of the first season, which Spock knows due to Nakie Time. Yeoman was traumatized, but not actually against it being Jim, I think, and/or forgave him, and sort of said something that vaguely implied to that effect to Jim just before Spock makes this comment.)