C, C++, 47 bytes ( #define ), 51 bytes ( function )
-1 byte thanks to Zacharý
Macro version :
#define F(n)n*n*(n+1)*(n+1)/4-n*(n+1)*(2*n+1)/6
Function version :
int f(int n){int p=n*n+n;return p*p/4-p*(2*n+1)/6;}
Thoses lines are based on thoses 2 formulas :
Sum of numbers between 1 and n = n*(n+1)/2
Sum of squares between 1 and n = n*(n+1)*(2n+1)/6
So the formula to get the answer is simply (n*(n+1)/2) * (n*(n+1)/2) - n*(n+1)*(2n+1)/6
And now to "optimize" the byte count, we break parenthesis and move stuff around, while testing it always gives the same result
(n*(n+1)/2) * (n*(n+1)/2) - n*(n+1)*(2n+1)/6 =>
n*(n+1)/2*n*(n+1)/2 - n*(n+1)*(2n+1)/6 =>
n*(n+1)*n*(n+1)/4 - n*(n+1)*(2n+1)/6
Notice the pattern p = n*n+1 = n*n+n, so in the function, we declare another variable int p = n*n+n and it gives :
p*p/4 - p*(2n+1)/6
For p*(p/4-(2*n+1)/6) and so n*(n+1)*(n*(n+1)/4 - (2n+1)/6), it works half the time only, and I suspect integer division to be the cause ( f(3) giving 24 instead of 22, f(24) giving 85200 instead of 85100, so we can't factorize the macro's formula that way, even if mathematically it is the same.
Both the macro and function version are here because of macro substitution :
F(3) gives 3*3*(3+1)*(3+1)/4-3*(3+1)*(2*3+1)/6 = 22
F(5-2) gives 5-2*5-2*(5-2+1)*(5-2+1)/4-5-2*(5-2+1)*(2*5-2+1)/6 = -30
and mess up with the operator precedence. the function version does not have this problem