Jelly, 23 22 bytes
ḟ0ṢṚ0_1x$}1ĿɗL<¥ß`ɗL<¥?Ḣ$1L?
A full program that takes the input list of answers as its argument and returns 1 for valid and 0 for impossible.
Thanks to @JonathanAllan for noting an issue with the second version.