If your scroll wheel is jittering in the wrong direction, or inconsistent, it is likely that something is wrong with the encoder.
Most likely, this is caused by dust, dirt, or fluid that gets trapped inside the encoder, interfering with its signal. In most cases, this can easily be fixed by cleaning your mouse.
One way to get the dust out of your encoder is to roll the scroll wheel on a flat surface very fast until the dust is cleaned out. If this does not work, you can also try to clean your encoder with compressed air.
If this does not work, it is likely that your encoder is defective, and needs to be replaced with a new one.