Sexual harassment goes deeper than that in the work place. The other two women could have complained of sexual harassment just by hearing that comment.
As a man in the workplace, you can't say shit sexually towards a woman directly or within an earshot without the potential of it being sexual harassment.
Hell, you can't say that around a man either
Keep that shit 100% professional, this goes for your neighbours too, if you have a side piece never let any of your neighbours know no matter how cool they might seem.