J, ³² 25 bytes

[:-:/[:(|/.~/:~"1)2-/\,&0

Try it online!

Based on Leo's excellent idea

• 2-/\,&0 Adjacent elementwise row deltas, with an extra row of zeros appended first.
• (|/.~/:~"1) Group by sorted rows and take absolute value. This puts all rows with a single 1 into one bucket, and all rows with a single -1 into another.
• [:-:/ Are those buckets equal?