2 added 119 characters in body

# Pyth, 6565 63 bytes

eSm+FM}3-BGHeSm+F.n<VVM.:.umm}3-BFbCkC,uCmsMugVVuCmsM.:++0k03G2NNd)2muCm.[Hk0GQd^^U2EE


Test suite

Input is taken as:

n,m
x
y


On the input 3,3,12,12, I got 594, not 596, with the same grid as the OP, so I'm not sure what's up.

How it works:

^^U2EE: Generate all possible patterns

muCm.[Hk0GQd: Pad them to the appropriate size. Written as pad, rotate, repeat.

uCmsM.:++0k03G2N: Generate the number of live neighbors for each cells. Written as pad, take subsequences of size 3, sum them, rotate, repeat.

mmM}3-BFbCkC,BGH ... gVV ... N: Pair that with the original grid, use both to calculate the new cells. We do this by checking whether live neighbors including the center, optionally minus whether it was alive, contains a 3. I really like this VV trick, so I wrote a helper function (M}3-BGH) so I could do it twice. (see below)

.u ... d): Do that until it stops changing, return all intermediate steps.

.: ... 2: Take all consecutive pairs.

<VVM: Map each pair to the number of new cells born. I'm particularly proud of <VVM, I thought that was very clever.

m+F.n: Map each grid to its total number of cells born.

eS: Max

# Pyth, 65 bytes

eSm+F.n<VVM.:.umm}3-BFbCkC,uCmsM.:++0k03G2NNd)2muCm.[Hk0GQd^^U2EE


Test suite

Input is taken as:

n,m
x
y


On the input 3,3,12,12, I got 594, not 596, with the same grid as the OP, so I'm not sure what's up.

How it works:

^^U2EE: Generate all possible patterns

muCm.[Hk0GQd: Pad them to the appropriate size. Written as pad, rotate, repeat.

uCmsM.:++0k03G2N: Generate the number of live neighbors for each cells. Written as pad, take subsequences of size 3, sum them, rotate, repeat.

mm}3-BFbCkC, ... N: Pair that with the original grid, use both to calculate the new cells. We do this by checking whether live neighbors including the center, optionally minus whether it was alive, contains a 3.

.u ... d): Do that until it stops changing, return all intermediate steps.

.: ... 2: Take all consecutive pairs.

<VVM: Map each pair to the number of new cells born. I'm particularly proud of <VVM, I thought that was very clever.

m+F.n: Map each grid to its total number of cells born.

eS: Max

# Pyth, 65 63 bytes

M}3-BGHeSm+F.n<VVM.:.ugVVuCmsM.:++0k03G2NNd)2muCm.[Hk0GQd^^U2EE


Test suite

Input is taken as:

n,m
x
y


On the input 3,3,12,12, I got 594, not 596, with the same grid as the OP, so I'm not sure what's up.

How it works:

^^U2EE: Generate all possible patterns

muCm.[Hk0GQd: Pad them to the appropriate size. Written as pad, rotate, repeat.

uCmsM.:++0k03G2N: Generate the number of live neighbors for each cells. Written as pad, take subsequences of size 3, sum them, rotate, repeat.

M}3-BGH ... gVV ... N: Pair that with the original grid, use both to calculate the new cells. We do this by checking whether live neighbors including the center, optionally minus whether it was alive, contains a 3. I really like this VV trick, so I wrote a helper function (M}3-BGH) so I could do it twice. (see below)

.u ... d): Do that until it stops changing, return all intermediate steps.

.: ... 2: Take all consecutive pairs.

<VVM: Map each pair to the number of new cells born. I'm particularly proud of <VVM, I thought that was very clever.

m+F.n: Map each grid to its total number of cells born.

eS: Max

1

# Pyth, 65 bytes

eSm+F.n<VVM.:.umm}3-BFbCkC,uCmsM.:++0k03G2NNd)2muCm.[Hk0GQd^^U2EE


Test suite

Input is taken as:

n,m
x
y


On the input 3,3,12,12, I got 594, not 596, with the same grid as the OP, so I'm not sure what's up.

How it works:

^^U2EE: Generate all possible patterns

muCm.[Hk0GQd: Pad them to the appropriate size. Written as pad, rotate, repeat.

uCmsM.:++0k03G2N: Generate the number of live neighbors for each cells. Written as pad, take subsequences of size 3, sum them, rotate, repeat.

mm}3-BFbCkC, ... N: Pair that with the original grid, use both to calculate the new cells. We do this by checking whether live neighbors including the center, optionally minus whether it was alive, contains a 3.

.u ... d): Do that until it stops changing, return all intermediate steps.

.: ... 2: Take all consecutive pairs.

<VVM: Map each pair to the number of new cells born. I'm particularly proud of <VVM, I thought that was very clever.

m+F.n: Map each grid to its total number of cells born.

eS: Max