The phrase “not all men” is a false argument. It’s a lot like “no true Scotsman”. Its a false equivalent, or an informal fallacy. From wikipedia:
No true Scotsman is an informal fallacy, an ad hoc attempt to retain an unreasoned assertion. When faced with a counterexample to a universal claim (“no Scotsman would do such a thing”), rather than denying the counterexample or rejecting the original universal claim, this fallacy modifies the subject of the assertion to exclude the specific case or others like it by rhetoric, without reference to any specific objective rule (“no true Scotsman would do such a thing”). It can also be used to create unnecessary requirements.
Not all men is the same thing. Its an informal fallacy. But it goes a lot further than that. The phrase is used in another way, not to show that a specific gender is completely without fault, but a much more sinister use. Its used to downplay the actual arguments for which men are often guilty of.
We don’t have to look very far to see its use. And its use has been in popular terminology for a few years now. When a woman speaks up about having to deal with sexist comments, threat of violence, rape, being treated as inferior at a job, the common outcry from MRAs (and I use the term “MRA” much like I’d use the term douchebag) happens to be “well, not all men act that way”. This doesn’t address the concerns the woman (or trans woman, or even trans man) is attempting to define. The phrase “not all men” has no suggestions or solutions as to how the problem can be changed. It doesn’t address anything about the problem at all.
In the same light, when a woman brings up these issues, its often suggested that there could have been things that the woman could have done differently. Sadly, when faced with the threat of violence, there is literally nothing the woman could have done differently. So trying to focus an attack on women as the woman’s fault is also attempting to avoid the real issue.
While the phrase “not all men” does have some truth to it, it still does not do anything to address the very real problems that, yes, all women do face threats of violence, rape, inequality, body shaming, and body autonomy on a daily basis. The problem isn’t solved by saying “not all men”, but we make a step when we say not enough men are addressing the real problem.