Japt -h, 14 bytes

Outputs undefined instead of 0 (pending confirmation).

ã ke"%(%)" mÊÍ

Try it

9 bytes

This version takes input as a string of 1s & 0s, representing ( & ) respectively, which the comments on the challenge would seem to allow.

ã keA mÊÍ

Try it

Japt -h, 14 bytes

Outputs undefined instead of 0 (pending confirmation).

ã ke"%(%)" mÊÍ

Try it

9 bytes

This version takes input as a string of 1s & 0s, representing ( & ) respectively, which the comments on the challenge would seem to allow.

ã keA mÊÍ

Try it

ã ke"%(%)" mÊÍ     :Implcit input of string
ã                  :Substrings
  k                :Remove items that return truthy (non-empty string)
   e               :  Recursively replace
    "%(%)"         :  RegEx /\(\)/ (or A=10, in the second version)
           m       :Map
            Ê      :  Length
             Í     :Sort
                   :Implicit out output of last element