s1protocol – Pwn2Win 2020 CTF

Points: 303 (dynamic) Solves: 18


  1. Reverse the binary and figure out the cryptosystem
  2. Given N and enc_flag, find the flag such that enc_flag = pow(N+1, flag, N**3*(N+1))
  3. Just do ((enc_flag % (N**2)) - 1)//N to trivially get the flag


We are given a statically linked, stripped binary. Library signatures can often help in figuring out most of the functions used in cases like this. We tried to apply both Ghidra Function ID and FLIRT signatures with no success. We simply could not find the correct versions of GMP/libc used.

We tried this database for Ghidra Function IDs, and this and this for FLIRT signatures. We used this script to make FLIRT signatures work on Ghidra

And a few others. Here is a video from LiveOverflow explaining how to use function signatures.

Fortunately, the code was simple and the used library functions were gueassable:

int main(int argc, const char **argv, const char **envp)
    get_random_bytes(y, 21); /* Random 168 bit pattern in p */
    for ( i = 0; i < 128 / 21; ++i )
      copy_pattern(rand + i * 21, y, 21);
    get_random_bytes(rand + i * 21, 128 % 21); /* Add r (16 random bits) to p */
    long2bytes(&prime1, rand, 128);
    set_bit_at(&prime1, 8 * 128 - 2);
    set_bit_at(&prime1, 8 * 128 - 1);
    set_bit_at(&prime1, 0LL);
  while ( !mpz_probab_prime_p(&prime1, 0x1Eu) );

    get_random_bytes(rand, 128);
    long2bytes(&prime2, rand, 128);
    set_bit_at(&prime2, 8 * 128 - 2);
    set_bit_at(&prime2, 8 * 128 - 1);
    set_bit_at(&prime2, 0LL);
  while ( !mpz_probab_prime_p(&prime2, 30u) );

  fp = fopen("flag.txt", "r");
  if ( !fp )
    puts("Error opening file!", "r", v6);
  num_read = fread(buf, 1uLL, 128, fp);
  long2bytes(&flag_bytes, buf, num_read);

  get_random_bytes(rand, 128 - 1);
  long2bytes(&h, rand, (128 - 1));
  mpz_mult(modulus, &prime1, &prime2);
  mpz_mult(mega, modulus, modulus);
  mpz_mult(mega, modulus, mega);
  mpz_powm(&h, &h, modulus, mega);
  mpz_add(modulus, modulus, 1LL);
  mpz_mult(mega, modulus, mega);

  mpz_powm(&flag_enc, modulus, &flag_bytes, mega);
  printf("%Zd\n\n", modulus);
  printf("%Zd\n\n", &h);
  printf("%Zd\n\n", &flag_enc);
  return 0;

First, the primes are generated. q is a prime generated in the usual way. p has a pattern of 168 bits that repeats 6 times (the first loop in the program is copying that pattern to a buffer that will contain the whole number p). The remaining 16 bits of p are random (well, except the lsb of course, which must be 1 because p is prime).

Next, it reads the flag from flag.txt and computes mega = (N)^3 * (N+1) and finally flag_enc = (N+1)^flag mod mega . The program ends after printing 3 numbers. N+1 (first), flag_enc (third) and the second number you can safely ignore. We later found that the number was there because this challenge was an attempt to implement a cryptosystem similar to Paillier



Factoring N

We spent a few hours trying to factor N. We managed to do it but it revealed to be useless, or at least there is a much simpler way to get the flag. For the curious ones, here is how we factored N:

Let y be the random 168 bit pattern in p that repeats 6 times. r the 16 random bits (p lsbs) where r & 1 == 1 because p is prime,

Then, we can say p = y*K + r where K = 2^16 + 2^(16+168) + 2^(16+168*2) + ... + 2^(16+168*5) This means that N = (y*K + r)*q = y*K*q + r*q and therefore N % K = r*q % K so if we bruteforce r we know that q % K = (N % K)*r^(-1) % K Let x be number of times that q wraps around K so that q = (q % K) + x*K. x is a 168 bit number because K is a 856 bit number and q is a 1024 bit number.

We know now that N = p*q = (y*K + r)*((N % K)*inverse(r, K) + x*K) where the only unknowns are x and y and they are both small (168 bits each). Factoring N can then be reduced to finding small roots of the following polynomial: P(x,y) = (y*K + r)*((N % K)*inverse(r, K) + x*K) - N. Using Coron’s reformulation, we can find these roots and therefore our factorization.

We used the following script to factor N in 10 minutes using 8 cores: solve.sage

Get the flag

The only thing you had to do to get the flag after reversing the binary was this:

Expanding flag_enc yields N^flag + flag*N^(flag-1) + ... + flag*N + 1 mod mega Given that N^2 is a factor of mega, we can compute flag_enc mod N^2 = flag*N + 1 mod N^2 which is simply flag*N + 1 because that number is smaller than N^2.

To get the flag, just subtract 1 and divide by N…

flag: CTF-BR{the_S1_Protocol_1s_pa1lli3r_but_w3ird_and_puzzle}