[Nauty] Undefined behavior in nauty (Going over bounds)

[Mathieu Dutour]

Dear all,

I found a memory problem in nauty. That is the array "int p[2]" in permnode
is accessed above the bound of 2.

The way to detect it is by compiling with
clang -fsanitize=thread -fsanitize=undefined -fsanitize=bool
-fsanitize=bounds -fsanitize=bounds -fsanitize=return -fsanitize=null

The compiling nautyex9 with "make nautyex9"

And then it goes:
(base) ➜  nauty27r1 ./nautyex9

enter n : 13
schreier.c:687:45: runtime error: index 12 out of bounds for type 'int [2]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior schreier.c:687:45 in
schreier.c:844:56: runtime error: index 3 out of bounds for type 'int [2]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior schreier.c:844:56 in
Automorphism group size = 78

Best wishes,

  Mathieu