A gentle but always amused tip that the expression you mean to say is "Rubs me the wrong way," not "Rubs me off wrong." The expression is meant to evoke the feeling of stroking an animal's fur backwards, which they don't like.
To "rub off" is a different kind of stroking entirely.