[Nauty] Buffer overruns in nauty 2.9.0

[Jerry James]

Greetings!  Since updating the nauty package to version 2.9.0 for the
upcoming Fedora 43 release, the GAP packages grape and fining have
been failing their testsuites.  Both use dreadnaut, and both
testsuites show that some invocations of dreadnaut fail like this:

munmap_chunk(): invalid pointer
Aborted                    (core dumped) dreadnaut < [input file]

The attached file named "test" triggers this behavior.  Valgrind shows
buffer overruns for the arrays allocated in dyn_all.  Those arrays are
initially allocated with a call to dyn_all(1, WORDSIZE), but are not
reallocated if n and m change because the 'n' parameter is set.  The
following patch lets grape and fining pass their tests again:

--- nauty2_9_0/dreadnaut.c.orig
+++ nauty2_9_0/dreadnaut.c
@@ -915,6 +915,10 @@ main(int argc, char *argv[])
                 loops = 0;
                 n = i;
                 m = SETWORDSNEEDED(n);
+#if !MAXN
+                DYNALLOC2(graph,g,g_sz,n,m,"dreadnaut");
+                dyn_all(m,n);
+#endif
                 freeschreier(NULL,&generators);
             }
             break;

Please let me know if that looks correct.  Regards,
-- 
Jerry James
http://www.jamezone.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test
Type: application/octet-stream
Size: 60231 bytes
Desc: not available
URL: <https://mailman.anu.edu.au/pipermail/nauty/attachments/20250823/9967c946/attachment-0001.obj>