[Nauty] Buffer overruns in nauty 2.9.0

[Brendan McKay]

Dear nauty people,

Jerry has identified a serious bug in dreadnaut for versions 2.9.0
and 2.9.1rc1.  If you installed either of those and rely on dreadnaut,
you should upgrade to 2.9.1rc3 which is now available at
https://users.cecs.anu.edu.au/~bdm/nauty/ .

2.9.1rc3 will be renamed to 2.9.1 in the near future if no further
problems are noted.

Regards, Brendan.

On 24/8/2025 7:29 am, Jerry James via Nauty wrote:
> Greetings!  Since updating the nauty package to version 2.9.0 for the
> upcoming Fedora 43 release, the GAP packages grape and fining have
> been failing their testsuites.  Both use dreadnaut, and both
> testsuites show that some invocations of dreadnaut fail like this:
>
> munmap_chunk(): invalid pointer
> Aborted                    (core dumped) dreadnaut < [input file]
>
> The attached file named "test" triggers this behavior.  Valgrind shows
> buffer overruns for the arrays allocated in dyn_all.  Those arrays are
> initially allocated with a call to dyn_all(1, WORDSIZE), but are not
> reallocated if n and m change because the 'n' parameter is set.  The
> following patch lets grape and fining pass their tests again:
>
> --- nauty2_9_0/dreadnaut.c.orig
> +++ nauty2_9_0/dreadnaut.c
> @@ -915,6 +915,10 @@ main(int argc, char *argv[])
>                   loops = 0;
>                   n = i;
>                   m = SETWORDSNEEDED(n);
> +#if !MAXN
> +                DYNALLOC2(graph,g,g_sz,n,m,"dreadnaut");
> +                dyn_all(m,n);
> +#endif
>                   freeschreier(NULL,&generators);
>               }
>               break;
>
> Please let me know if that looks correct.  Regards,
>
> _______________________________________________
> Nauty mailing list
> Nauty at anu.edu.au
> https://mailman.anu.edu.au/mailman/listinfo/nauty