It can be hate or phobia of numbers or by extension hatred of mathematics or arithmetic. It can also be called arhythmophobia.
