Can you be against feminism without being a misogynist?

I think the answer to this question lies in distinguishing feminism as a set of movements and feminism as an ideal.

There are valid criticisms of feminist movements, and those that genuinely and honestly dislike feminism as a movement because of those things are not necessarily woman haters, and in some cases are anything but.  A non-feminist identified womanist who feels alienated by the history of erasure of woman of color and racism by some sections of the feminist movement is not a misogynist.  A trans woman who criticizes histories of transphobia within certain sections of the feminist movement is not engaging in woman hating solely by doing so.

On the other hand, attacks on feminism qua feminism, as a movement for rights of women, as women's non-inferiority and personhood, etc. are inherently misogynist.  Opposition to the notion that women are full people is misogynist to its very core, assertions that women should be denied bodily autonomy or access to social participation is intrinsically misogynist.  There is no real way to attack the idea of opposition to oppression of women as such and not being a misogynist.

Misogynist anti-feminist love to conflate their intrinsically misogynist attacks on feminism with the not intrinsically misogynist genuine criticisms of various feminist movements, but they are not the same thing at all.  The form, content, and intent of criticisms of feminism absolutely do matter.

If someone says that they do not like feminism because they have experienced a great deal of classism in feminist spaces, that alone does not make them a misogynist or indicate they are a misogynist.  However, those that take issue with feminism because it aims to advocate against sexist oppression, they are simply misogynists.