Regex (ECMAScript), 37 bytes
Warning: Despite this regex's small size, it contains a major spoiler. I highly recommend learning how to solve unary mathematical problems in ECMAScript regex by figuring out the initial mathematical insights independently. It's been a fascinating journey for me, and I don't want to spoil it for anybody who might potentially want to try it themselves, especially those with an interest in number theory. See this earlier post for a list of consecutively spoiler-tagged recommended problems to solve one by one.
So do not read any further if you don't want some advanced unary regex magic spoiled for you. If you do want to take a shot at figuring out this magic yourself, I highly recommend starting by solving some problems in ECMAScript regex as outlined in that post linked above.
This is the simplest natural number unary decision-function that, in order to be implemented in ECMAScript regex, absolutely requires an algorithm for generalized multiplication. This is in contrast to matching perfect squares and other powers, which can be done by dividing out prime factors individually (although the best golf is not achieved that way). Generalized multiplication can be implemented by prime factorization, and I did so on my initial try, but it takes 600+ bytes.
Remarkably, generalized multiplication can be implemented quite concisely in ECMAScript regex using the Chinese remainder theorem. The algorithm is discussed in my abundant numbers answer.
teukon posed the Triangular Numbers puzzle in 2014, after we had both figured out the best algorithm for generalized multiplication. I was not able to beat his golf on this problem; he managed to get it down to 37 bytes, but I was only able to get it down to 48 bytes. He's unlikely to ever create an account on CGCC, and I haven't been able to even get in contact with him since 2014. I feel this very elegant regex belongs on CGCC, though, and the ideas behind it were a joint effort between us, so I'm posting on his behalf:
^((((x*)x?)\3)x)(?=\4(\1*)\2*$)\1*$\5
Try it online!
^ # N = input number
( # \1 = n*2-1 or n*2+1
( # \2 = \1-1
( # \3 = n-1 or n
(x*) # \4 = n-1
x?
)
\3
)
x
)
(?=
\4(\1*)\2*$ # iff \1+\4*\1 == N, then the first match here must result in \5 == 0
)
\1*$\5 # test for divisibility by \1 and for \5 == 0 simultaneously
Including zero in the "truthy" category only costs 1 byte (bringing it to 38 bytes), using ECMAScript NPCG (non-participating/unset capture group) behavior:
^((((x*)x?)\3)x)?(?=\4(\1*)\2*$)\1*$\5
Try it online!
Doing so with NPCG-independent behavior costs 3 bytes, bringing it to 40 bytes:
^((((x*)x?)\3)x)(?=\4(\1*)\2*$)\1*$\5|^$
Try it online!
Returning the triangular root
48 bytes with zero-handling, based on the 38 byte true/false version:
^(?=((((x*)x?)\3)x)?(?=\4(\1*)\2*$)\1*$\5)x?\3\4
Try it online!
46 bytes without zero-handling, based on the 37 byte true/false version:
^(?=((((x*)x?)\3)x)(?=\4(\1*)\2*$)\1*$\5)x\3\4
Try it online!
I haven't tried golfing this down yet. There may be specific golf optimizations for returning the triangular root.
