JavaScript (ES7), 121 117 bytes
x=>(a=b=0,[for(c of x)for(d of'1234')(e=c.charCodeAt()/26|0)==d?a^=1<<d:b^=(a>>d&1)<<d*4+e],f=y=>y&&y%2+f(y>>1))(b)/2
Wow. That was fun. I sketched out an answer idea when this challenge first came out, but it was over 150 bytes long and I didn't want to put in the effort to golf it. I ran across this idea in my notebook yesterday and decided I wouldn't stop thinking about it until I had fully golfed it. I ended up writing out two entirely new algorithms, the first of which ended up several bytes shorter after golfing off around 25 bytes with tons of bit-hacking.
How it works
First we set variables a
and b
to 0
. a
is a 4-bit binary array of which bracket pairs we are currently inside, and b
is a 16-bit binary array of which bracket pairs are linked together.
Next, we loop through each character c
in x
, and each char d
in '0123'
. First we determine what type of bracket c
is with e=c.charCodeAt()/26-1|0
. The decimal char codes of each bracket type are as follows:
() => 40,41
<> => 60,62
[] => 91,93
{} => 123,125
By dividing by 26, subtracting 1, and flooring, we map these to 0, 1, 2, and 3, respectively.
Next we check if this number is equal to the current value of d
. If it is, we are either entering or exiting the d
th bracket type, so we flip the d
th bit in a
with a^=1<<d
. If it's not, but we are inside the d
th bracket type, we need to flip the e
th bit in the d
th 4-bit section of b
. This is done like so:
b^=(a>>d&1)<<d*4+e
(a>>d&1)
Returns the d
th bit in a
. If we are inside the d
th bracket type, this returns 1; otherwise, it returns 0. Next, we shift this left by d*4+e
bits, and XOR b
by the result. If we are inside the d
th bracket type, this XORs the d*4+e
th bit of b
; otherwise, it does nothing.
At the end of all the looping, b
will contain a number of 1-bits equal to twice the desired return value. But we still need to figure out how many bits this is. That's where the sub-function f
comes in:
f=y=>y&&y%2+f(y>>1)
If y
is 0, this simply returns 0. Otherwise, it takes the last bit of y
with y%2
, then adds the result of running all but the last bit y
through the function again. For example:
f(y) => y && y%2 + f(y>>1)
f(0b1001101) => 1 + f(0b100110) = 4
f(0b100110) => 0 + f(0b10011) = 3
f(0b10011) => 1 + f(0b1001) = 3
f(0b1001) => 1 + f(0b100) = 2
f(0b100) => 0 + f(0b10) = 1
f(0b10) => 0 + f(0b1) = 1
f(0b1) => 1 + f(0b0) = 1
f(0b0) => 0 = 0
We run b
through this function and divide the result by 2, and there is our answer.