Vulnerability Report - Integer Overflow in libgsl

[Silvia De Francisci via Bug reports for the GNU Scientific Library]

Dear libgsl development team,

I hope this message finds you well.

We are writing to inform you about a potential security vulnerability that
Professor Gabriele Costa (in cc) and I have discovered in your library,
libgsl. Specifically, the issue is an integer overflow (CWE-190
<https://cwe.mitre.org/data/definitions/190.html>), which can lead to
illegal memory access. Please find a detailed description below.
Vulnerability Details:

   -

   Type of vulnerability: Integer Overflow
   -

   Location: function gsl_siman_solve_many in file  gsl/siman/siman.c
   -

   Description: The vulnerability occurs when params.n_tries is set to a
   negative integer, which leads to incorrect memory allocation in the
   following lines of code:

 x = (void *) malloc (params.n_tries * element_size);
new_x = (void *) malloc (params.n_tries * element_size);
energies = (double *) malloc (params.n_tries * sizeof (double));
probs = (double *) malloc (params.n_tries * sizeof (double));
sum_probs = (double *) malloc (params.n_tries * sizeof (double));
…
memcpy (x, x0_p, element_size);

Steps to Reproduce:

Here is a minimal code example that demonstrates the vulnerability:

   #include <gsl/gsl_siman.h>
  #include <gsl/gsl_rng.h>
  #include <gsl/gsl_vector.h>

   double objective_function(void *xp) {
      return 0.0;
  }

   void take_step(const gsl_rng *r, void *xp, double step_size) {
  }

   int main(void) {
      gsl_rng *r;
      gsl_rng_env_setup();
      r = gsl_rng_alloc(gsl_rng_default);

       // Initialize GSL Simulated Annealing parameters
      gsl_siman_params_t p;
      p.n_tries = -1;// Integer overflow vulnerability: invalid number of
trials

       gsl_vector *x0 = gsl_vector_alloc(1);

       // Call the simulated annealing solver (this is the key call for the
vulnerability)
      gsl_siman_solve_many(r, x0, objective_function, take_step, NULL, NULL,
sizeof(gsl_vector), p);

       return 0;
  }

To compile and run the above code:

   $ gcc test.c -o test -lgsl
  $ catchsegv ./test
  Segmentation fault (core dumped)
  *** signal 11

   Register dump:

   RAX: 0000000000000000   RBX: fffffffffffffff8  …

   Trap: 0000000e   Error: 00000006   OldMask: 00000000   CR2: 00000000

   …

   Backtrace:
  /lib/x86_64-linux-gnu/libc.so.6(+0x1a07e1)[0x752cfd3a07e1]
  /lib/x86_64-linux-gnu/libgsl.so.27(gsl_siman_solve_many+0x10d)[
0x752cfd749aed]
  ./test(+0x125b)[0x5d80a762525b]
  /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x752cfd229d90]
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x752cfd229e40]
  ./test(+0x10e5)[0x5d80a76250e5]



Suggested Fix:

We believe this issue could be mitigated by adding a sign check before
allocating memory. For example:

   if (0 <= params.n_tries) {

   x = (void *) malloc (params.n_tries * element_size);
}

Please feel free to reach out if you need further information,
clarification, or assistance in addressing this vulnerability. Also, if you
plan to release a CVE entry, we would be happy to co-author it.

Finally, please note that this vulnerability was found as part of a
research activity we are carrying out. Our plan is to include part of this
report in an upcoming research article. Following a responsible disclosure
approach, we are keeping this information confidential and preparing our
paper by anonymizing all the details given above. For this reason, we would
be grateful if we could agree on a grace period for the bug-fixing process,
after which we can safely publish my work.

Thank you for your attention to this matter, and we look forward to your
response.

Best regards,
Silvia De Francisci