Python (59 57 56 bytes)
lambda n:0**n+sum((-(n%(x-~x)<1))**x*4for x in range(n))
As with my CJam answer, this uses Möbius inversion and runs in pseudoquasilinear time.
Thanks to Sp3000Sp3000 for 2 bytes' savings, and feersumfeersum for 1.