ŒgL⁼¥Ðf
,UŒD€;Z,$瀀4FṀ

ŒgL⁼¥Ðf
;UŒD€;Z;$ç€4FṀ

How?

ŒgL⁼¥Ðf - Link 1, runs of given length: list A, length B  e.g. "XYYYXXO", 4
Œg      - group runs of equal elements of A                     ["X","YYY","XX","O"]
     Ðf - filter keep:
    ¥   -     last two links as a dyad:
  L     -         length                                         1   3     2    1
   ⁼    -         equal to B?         (none kept in this case->) 0   0     0    0

;UŒD€;Z;$ç€4FṀ - Main link: list of list of chars (or list of stings) I
 U             - reverse each row of I
;              - I concatenated with that
  ŒD€          - positive diagonals of €ach (positive and negative diagonals)
        $      - last two links as a monad:
      Z        -     transpose of I (i.e. the columns)
       ;       -     concatenated with I (columns + rows)
     ;         - concatenate (all the required directional slices)
         ç€4   - call the last link (1) as a dyad for €ach with right argument = 4
            F  - flatten the result
             Ṁ - take the maximum ('Y'>'X'>'O') - this has the bonus effect of returning:
                               'Y' or 'X' for a winning board; and
                               'O' or '' for a (valid) game in progress.
       

Jelly, 25 bytes

ŒgL⁼¥Ðf
,UŒD€$$;Z,$瀀4FṀ

Takes a list of strings (or list of list of characters) formed of X, Y, and O (would also work with replacements such that the space has a lower ordinal than both counters).

Try it online! or run an augmented version that takes a multiline string

Jelly, 25 22 bytes

ŒgL⁼¥Ðf
,UŒD€;Z,$瀀4FṀ

Takes a list of strings (or list of list of characters) formed of X, Y, and O (would also work with replacements such that the space has a lower ordinal than both counters).

Try it online! or run an augmented version that takes a multiline string.