Classic ret2win: the return address of vuln() is overwritten with the address of win(), which calls system("/bin/sh"). The binary is compiled with Intel CET (SHSTK + IBT), which adds an extra complication the standard TryHackMe hint doesn't cover.
buf is 32 bytes, so the return address is probably at offset 40 bytes (32 bytes buffer + 8 bytes saved frame pointer).
This is not correct for this specific binary, and it's a classic pitfall:
c
char *buf[0x20];
This is not an array of 32 chars (char buf[0x20]), but an array of 32 pointers — on x86-64 each pointer is 8 bytes, so the actual buffer size is 0x20 * 8 = 256 bytes, not 32.
The disassembly of vuln() confirms this:
asm
sub $0x100,%rsp ; 0x100 = 256 bytes reserved, not 32lea -0x100(%rbp),%rax ; buf starts at rbp-0x100mov $0x200,%edx ; read() reads up to 512 bytes
So 264 bytes of padding, not 40. The cyclic-pattern method from the hint (cyclic(200) + crash + look up the offset) is fine as a technique and generally applicable, but the hint misjudges the buffer size itself. Correct is:
python
from pwn import *print(cyclic(300)) # larger than 264, so you're sure to hit the return address
followed by actually working it out via disassembly (or cyclic_find() on the crashed value), instead of blindly trusting the C declaration.
The hint stops at "send padding + address of win()". For this binary that's not enough — a bare ret2win produces a crash. Important to distinguish:
SHSTK (Shadow Stack) would produce a control protection fault if the return address itself were blocked. That didn't happen here.
The actual fault in the kernel log was a general protection fault in libc.so.6, consistently at the same offset:text
traps: tryretme[...] general protection fault ip:...843b sp:... in libc.so.6[...]
This points to a stack-alignment problem: by jumping straight into win() via ret (instead of via a normal call), the stack is 8 bytes off from what glibc's internal SSE instructions (e.g. in system()) expect.
0000000000401180 <__do_global_dtors_aux+0x20>: 401180: c3 ret
By jumping to this address first (which does nothing but "eat" 8 bytes of the stack and jump on), the alignment is corrected by 8 bytes before win() and system() run.
264 bytes (256 buf + 8 rbp), because char *buf[0x20] = pointer array, not char array
Payload
padding + win() address
padding + standalone ret gadget + win() address
Reason for extra step
not mentioned
CET-compiled binary requires correct stack alignment before the system() call in win(), otherwise a general protection fault in libc
The generic hint describes the base technique correctly, but assumes every char buf[N]-like declaration is literally N bytes and that a bare ret2win always works. Both assumptions were wrong here, and required disassembly verification and an extra alignment gadget respectively to get the exploit working.