Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
4 Some clarification.
source | link

Non-recursive stack overflow in C

Calling convention mismatch.

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) 
    goodbye(0);
}

Compile with gcc -O0.

__cdecl functions expect the caller to clean up the stack, and __stdcall expects the callee to do it, so by calling through the typecast function pointer, the cleanup is never done -- main pushes the parameter onto the stack for each call but nothing pops it and ultimately the stack fills.

Non-recursive stack overflow in C

Calling convention mismatch.

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) 
    goodbye(0);
}

Compile with gcc -O0.

__cdecl functions expect the caller to clean up the stack, and __stdcall expects the callee to do it, so by calling through the typecast function pointer, the cleanup is never done.

Non-recursive stack overflow in C

Calling convention mismatch.

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) 
    goodbye(0);
}

Compile with gcc -O0.

__cdecl functions expect the caller to clean up the stack, and __stdcall expects the callee to do it, so by calling through the typecast function pointer, the cleanup is never done -- main pushes the parameter onto the stack for each call but nothing pops it and ultimately the stack fills.

3 deleted 425 characters in body
source | link

Non-recursive stack overflow in C

Calling convention mismatch. No assembler or alloca().

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) 
    goodbye(0);
}

Compile with gcc -O0.


Explanation

It works because __cdecl__cdecl functions expect the caller to clean up the stack, and __stdcall__stdcall expects the callee to do it, so by castingcalling through the typecast function pointer, the cleanup is never done.

Try this and watch the stack pointer decrease until it overflows:

#include <stdio.h>

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) {
    goodbye(0);
    unsigned long v;
    asm("movl %%esp, %0;":"=r"(v));
    printf("%lu\n", v);
  }
}

Non-recursive stack overflow in C

Calling convention mismatch. No assembler or alloca().

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) 
    goodbye(0);
}

Compile with gcc -O0.


Explanation

It works because __cdecl functions expect the caller to clean up the stack, and __stdcall expects the callee to do it, so by casting the function pointer, the cleanup is never done.

Try this and watch the stack pointer decrease until it overflows:

#include <stdio.h>

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) {
    goodbye(0);
    unsigned long v;
    asm("movl %%esp, %0;":"=r"(v));
    printf("%lu\n", v);
  }
}

Non-recursive stack overflow in C

Calling convention mismatch.

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) 
    goodbye(0);
}

Compile with gcc -O0.

__cdecl functions expect the caller to clean up the stack, and __stdcall expects the callee to do it, so by calling through the typecast function pointer, the cleanup is never done.

2 added 43 characters in body
source | link

Non-recursive stack overflow in C, no assembler

Calling convention mismatch. No assembler or alloca().

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) {
    goodbye(0);
  }
}

Compile with gcc -O0.


Explanation

It works because __cdecl functions expect the caller to clean up the stack, and __stdcall expects the callee to do it, so by casting the function pointer, the cleanup is never done.

Try this and watch the stack pointer decrease until it overflows:

#include <stdio.h>

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) {
    goodbye(0);
    unsigned long v;
    asm("movl %%esp, %0;":"=r"(v));
    printf("%lu\n", v);
  }
}

Non-recursive stack overflow in C, no assembler

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) {
    goodbye(0);
  }
}

Compile with gcc -O0.

It works because __cdecl functions expect the caller to clean up the stack, and __stdcall expects the callee to do it, so by casting the function pointer, the cleanup is never done.

Try this and watch the stack pointer decrease until it overflows:

#include <stdio.h>

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) {
    goodbye(0);
    unsigned long v;
    asm("movl %%esp, %0;":"=r"(v));
    printf("%lu\n", v);
  }
}

Non-recursive stack overflow in C

Calling convention mismatch. No assembler or alloca().

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) 
    goodbye(0);
}

Compile with gcc -O0.


Explanation

It works because __cdecl functions expect the caller to clean up the stack, and __stdcall expects the callee to do it, so by casting the function pointer, the cleanup is never done.

Try this and watch the stack pointer decrease until it overflows:

#include <stdio.h>

typedef void __stdcall (* ptr) (int);

void __cdecl hello (int x) { }

void main () {
  ptr goodbye = (ptr)&hello;
  while (1) {
    goodbye(0);
    unsigned long v;
    asm("movl %%esp, %0;":"=r"(v));
    printf("%lu\n", v);
  }
}
1
source | link