Ruby 2.7, ^{695 693} 687 bytes

R=*0..7
L,*b='+-'*8+?+
a,o,*e=?X,?O
l={}
[*0..3,8,9,10,16,17,24].map{b[_1],b[63-_1]=o,a}
loop{puts"
 "+[*?a..?h]*' ',R.map{|r|L+"
|#{b[56-8*r,8].map{_1||?.}*?|}| #{8-r}"},L
M,*m=->c,r,w=0,*s{9.times{[x=c+u=_1/3-1,y=r+v=_1%3-1]-R==e&&(k=b[x+8*y]
w<1?k!=o&&s<<[x,y]:b[z=c+8*r]==a&&(w<2?l[a]!=z&&(eval(f="[x+=u,y+=v]-R==e&&!b[x+8*y]")&&(k==a||k&&N[c,r]-N[x,y]!=e)&&s<<[c,r,x,y]
x,y=c,r
s<<[c,r,x,y]while eval f):s<<k))}
w>1?(a,o=o,a;([p,a]&s)[0]):s}
N,*q=->c,r,n=M[c,r]{n==n|=n.flat_map{M[*_1]}or redo;n}
R.product(R){M[*_1,2]&&(p a;Q);m+=s=M[*_1,1];M[*_1,2]&&q+=s}
a,o=o,a
q!=e&&m=q
m==e&&(p o;Q)
c,r,x,y=i=($><<a+?>;gets.bytes-[10,32]).map{~-_1%8}until[i]-m==e
b[c+8*r],b[l[a]=x+8*y]=p,a}

Try it online! (695 bytes because the older Ruby on TIO doesn't support numbered block parameters.) A simple example game is included that illustrates most of Quagmire's rules. Just for clarity, the input is commented and invalid moves are indented.

Input is in the format x# y#, terminated either by newline or EOF. The input prompt is O> for player 1 and X> for player 2. (In an interactive terminal the input appears as it is typed, after the prompt.) The prompt is repeated until a valid move is entered, then the updated board is printed. When the game ends, the winning player's symbol is printed in double quotes; I like to imagine it is raising its arms in victory.

Explanation

While perhaps not a brilliant board game, it ought to be a fun code golf!

Steel your Brain

The first 5 lines perform initialisation. The x,*y=z construct is equivalent to x=z;y=[] but 2 bytes shorter. R is a list of row numbers and L is a fixed string representing the edges of the board. a and o are the 'active' and 'opponent' (enemy) piece symbols and are swapped each turn. l is a hash that stores the board index of the piece moved on each player's previous turn (which cannot be moved on their next turn). b stores the contents of each square on the board (O, X, or nil). Empty squares, stored as nil, are replaced by . on output. e is just an empty array, which is used often enough that defining this single-letter variable is shorter than using [].

Then the main loop begins, running until one player wins. Following the 3 lines that display the board, we get to the real meat of the program: the lambdas M and N.

Mary had a little lambda

Let's start with N. N finds the set of all squares that are connected to the square at column c and row r (0-indexed). Here, connected means that there is a path between all the squares that is not blocked by enemy pieces. N fans out from the current square, adding its neighbours and their neighbours (by repeatedly calling M in w=0 mode; see below) to the set n until n no longer changes (n==n|=n.flat_map{M[*_1]}or redo). Source and destination squares must be unconnected as a prerequisite for jumping an enemy piece.

M probes the immediate neighbours (8-neighbourhood) of the square at column c and row r. Its exact function is determined by the switch w, which may take a value of 0, 1, or 2:

• When w=0, M returns an array s of the co-ordinates of all neighbours of the current square that do not contain enemy pieces (k!=o). Lambda N uses this information to find the set of all squares that are connected to the current square, as described above.

• When w=1, M returns an array s containing all possible moves for the active player originating from the current square. If the current square contains an active piece (b[z=c+8*r]==a) and that piece wasn't moved on the previous turn (l[a]!=z), then

  • a jump is possible if the destination square exists ([x+=u,y+=v]-R==e, i.e. x and y are both in R), is empty (!b[x+8*y]), and either another active piece is in between (k==a) or an enemy piece is in between and the source and destination squares are not connected (k&&N[c,r]-N[x,y]!=e);
  • any move in a straight line is possible if the destination square and all previous squares on the same path are empty. The code to test such moves is identical to part of the jump code and is hence recycled (while eval f).

  Moves are stored in the format [source column, source row, destination column, destination row] to match the (processed) input format.

• When w=2, M swaps the active/enemy pieces (a,o=o,a) and then tests whether an enemy piece in the current square is in quagmire, returning a truthy/falsey value accordingly. The piece is in quagmire if there are no empty neighbours and at least one neighbour is an active piece (([p,a]&s)[0]). The reason for swapping active/enemy pieces here is that quagmire checks are performed twice per square, once to find whether the opponent has left a piece in quagmire (and hence lost the game) and again to find whether the active player has been quagmired by the opponent's previous move.

With their uppercase names, M and N are constants by Ruby convention. Here they are defined anew on each iteration of the loop, causing warnings to be emitted to STDERR. (Ruby may be called with the -W0 option to suppress these warnings.) This choice is necessary because the lambdas call each other and so each must be in scope when the other is called. They are placed inside the loop, rather than outside, so that the arrays m and q, which need to be reset at each iteration, can be initialised cheaply.

Into the quagmire

The remainder of the program is relatively straightforward. On each turn, loop over all squares of the board by forming pairs of column and row indices (R.product(R)). If any square contains a quagmired enemy piece (M[*_1,2]) the game is won; print the active player's symbol (p a) and terminate by throwing a tantrum (Q is deliberately undefined). Otherwise, add all possible moves for the active player to m and keep a copy of possible moves for the current square in s (m+=s=M[*_1,1]). Then check whether the current square contains a quagmired active piece (M[*_1,2]); if so, add the previously saved moves in s to q (q+=s). Once the loop has finished, swap the active/opponent players again (a,o=o,a).

We now have two arrays of moves for the active player: m contains all possible moves (not accounting for quagmire) whereas q contains possible moves by any quagmired pieces. If q is nonempty (q!=e) it takes precedence over (replaces) m (m=q). If the active player has no possible moves (m==e) the game is lost; print the opponent's symbol and poQ (p o;Q).

Finally it's time for some input! Until a valid move is entered (until[i]-m==e), display the prompt ($><<a+?>), convert the input to ASCII codes with newlines and spaces removed (gets.bytes-[10,32]), then convert each (valid) byte to a 0-indexed column/row number (map{~-_1%8}). The final line updates the source and destination squares, saving the destination index in l.