Racket 86 bytes
-

Following simple function works for smaller matrices but hangs on my machine for larger matrices: 

    (for/sum((p(permutations(range(length ll)))))(for/product((l ll)(c p))(list-ref l c)))

Ungolfed: 

    (define (f ll) 
      (for/sum ((p (permutations (range (length ll))))) 
        (for/product ((l ll)(c p)) 
          (list-ref l c))))

The code can easily be modified for unequal number of rows and columns.
    
Testing: 
    
    (f '[[ 1 -1 -1  1]
         [-1 -1 -1  1]
         [-1  1 -1  1]
         [ 1 -1 -1  1]])
    
    (f '[[ 1 -1  1 -1 -1 -1 -1 -1]
     [-1 -1  1  1 -1  1  1 -1]
     [ 1 -1 -1 -1 -1  1  1  1]
     [-1 -1 -1  1 -1  1  1  1]
     [ 1 -1 -1  1  1  1  1 -1]
     [-1  1 -1  1 -1  1  1 -1]
     [ 1 -1  1 -1  1 -1  1 -1]
     [-1 -1  1 -1  1  1  1  1]])
    
Output: 

    -4
    192

As I mentioned above, it hangs on testing following: 

    (f '[[1 -1 1 -1 -1 1 1 1 -1 -1 -1 -1 1 1 1 1 -1 1 1 -1]
     [1 -1 1 1 1 1 1 -1 1 -1 -1 1 1 1 -1 -1 1 1 1 -1]
     [-1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1 -1 -1 -1 1 -1 -1]
     [-1 -1 -1 1 1 -1 1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 1 -1]
     [-1 1 1 1 -1 1 1 1 -1 -1 -1 1 -1 1 -1 1 1 1 1 1]
     [1 -1 1 1 -1 -1 1 -1 1 1 1 1 -1 1 1 -1 1 -1 -1 -1]
     [1 -1 -1 1 -1 -1 -1 1 -1 1 1 1 1 -1 -1 -1 1 1 1 -1]
     [1 -1 -1 1 -1 1 1 -1 1 1 1 -1 1 -1 1 1 1 -1 1 1]
     [1 -1 -1 -1 -1 -1 1 1 1 -1 -1 -1 -1 -1 1 1 -1 1 1 -1]
     [-1 -1 1 -1 1 -1 1 1 -1 1 -1 1 1 1 1 1 1 -1 1 1]
     [-1 -1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 1 1 1 -1 -1 -1 -1]
     [1 1 -1 -1 -1 1 1 -1 -1 1 -1 1 1 -1 1 1 1 1 1 1]
     [-1 1 1 -1 -1 -1 -1 -1 1 1 1 1 -1 -1 -1 -1 -1 1 -1 1]
     [1 1 -1 -1 -1 1 -1 1 -1 -1 -1 -1 1 -1 1 1 -1 1 -1 1]
     [1 1 1 1 1 -1 -1 -1 1 1 1 -1 1 -1 1 1 1 -1 1 1]
     [1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 1 -1 1 -1 -1 -1 -1 -1]
     [-1 1 1 1 -1 1 1 -1 -1 1 1 1 -1 -1 1 1 -1 -1 1 1]
     [1 1 -1 -1 1 1 -1 1 1 -1 1 1 1 -1 1 1 -1 1 -1 1]
     [1 1 1 -1 -1 -1 1 -1 -1 1 1 -1 -1 -1 1 -1 -1 -1 -1 1]
     [-1 1 1 1 -1 -1 -1 -1 -1 -1 -1 1 1 -1 1 1 -1 1 -1 -1]])