Python 2, 43 bytes
f=lambda a,b,s=0:b/a and 2*a*s+f(a+1,b,s+a)
Test it on Ideone.
How it works
Call the function defined in the specification \$g(a, b)\$. We have that
$$
\newcommand{\sumab}[2]{\sum_{a \le #1 \le b} #2 \:}
\begin{align}
g(a,b) & = \left( \sumab n n \right)^2 - \sumab n {n^2} \\
& = \sumab {i,j} {ij} - \sumab n {n^2} \\
& = \sumab {i<j} {ij} + \sumab {i=j} {ij} + \sumab {j<i} {ij} - \sumab n {n^2} \\
& = 2 \sumab {j<i} {ij} \\
& = 2 \sumab {i} {\sum_{a \le i < j} ij} \\
& = \sumab {i} {\left( 2i \sum_{a \le j < i} j \right)}
\end{align}
$$
Define the function \$f(x, y, s)\$ recursively as follows.
$$ f(x, y, s) =
\begin{cases}
2xs + f(x+1, y, s+x) & \text{ if } x \le y \\
0 & \text{ if} x > y
\end{cases}
$$
By applying the recurrence relation of \$f(a, b, 0)\$ a total of \$b - a\$ times, we can show that:
$$
\begin{align}
f(a, b, 0) & = 2\cdot0 + f(a+1, b, a) \\
& = 2\cdot0 + 2\cdot(a+1)\cdot a + f(a+2, b, a+(a+1)) \\
& = 2\cdot0 + 2\cdot(a+1)\cdot a + 2\cdot(a+2)\cdot(a+(a+1)) + f(a+3, b, a+(a+1)+(a+2)) \\
& \vdots \\
& = \sumab {i} {\left( 2i \sum_{a \le j < i} j \right)} + f\left(b+1, b, \sumab j j\right) \\
& = \sumab {i} {\left( 2i \sum_{a \le j < i} j \right)} \\
& = g(a, b)
\end{align}
$$
This is the function f of the implementation. While b/a
returns a non-zero integer, the code following and
is executed, thus implementing the recursive definition of f.
Once b/a
reaches 0, we have that b > a and the lambda returns False = 0, thus implementing the base case of the definition of f.