Can writing out calculations slow you down? Well, yes - of course! What will happen if you don't write it down however? Can you do the math in your head (or estimate it)? If so, don't write things down. If you can't, then you really only have two options: (1) your write the math down, "waste" a little bit of time, and have a chance to get to the right answer, or (2) you don't write it down, get the wrong answer, and never even get a chance to get an interview.

What you want to do it practice quick math enough that you have time to finish the whole test within the allocated time and with sufficient accuracy that your score helps you rather than hurts you. PrepLounge has good material, caseinterviewmath.com is a fantastic resource, and you will also find dozens of books on the suject at your local library, Barnes & Noble or Amazon. There's no shortcut, you will have to put in the work sorry.