From loganjerry at gmail.com  Thu Sep  3 08:10:07 2026
From: loganjerry at gmail.com (Jerry James)
Date: Wed, 2 Sep 2026 16:10:07 -0600
Subject: [Nauty] Patch for undefined behavior
Message-ID: <CAHCOHQm6ZX320d-wPw-zmQrsT8sv4hhRxfSSnSN+giw0CxNhBA@mail.gmail.com>

If nauty 2.9.3 is built with -fsanitize=undefined, two instances of
undefined behavior are identified while running the tests, both due to
signed integer overflow.  I would like to suggest the patch below so
that overflow only happens with unsigned integers.  In the nausha.c
case, data is an array of 1-byte quantities, which are automatically
promoted to int, a signed type, when used in an arithmetic expression.
The fix is to explicitly convert to an unsigned 32-bit type.  In the
watercluster2.c case, since qual can overflow, make it an unsigned
variable and cast to a signed type when returning the result.  All
tests pass after making these changes.

--- nauty2_9_3/nausha.c.orig
+++ nauty2_9_3/nausha.c
@@ -74,8 +74,8 @@ sha256_transform(SHA256_CTX *ctx, const
     nsword32 a, b, c, d, e, f, g, h, i, j, t1, t2, m[64];

     for (i = 0, j = 0; i < 16; ++i, j += 4)
-            m[i] = (data[j] << 24) | (data[j + 1] << 16)
- | (data[j + 2] << 8) | (data[j + 3]);
+            m[i] = ((nsword32)data[j] << 24) | ((nsword32)data[j + 1] << 16)
+ | ((nsword32)data[j + 2] << 8) | ((nsword32)data[j + 3]);
     for ( ; i < 64; ++i)
             m[i] = SIG1(m[i - 2]) + m[i - 7] + SIG0(m[i - 15]) + m[i - 16];

--- nauty2_9_3/watercluster2.c.orig
+++ nauty2_9_3/watercluster2.c
@@ -3130,13 +3130,13 @@ int i,start,end, problem, sum, maxsum;
 int getexpensivequality(graph x,graph y)
 {
 int j;
-int qual=0;
+unsigned int qual=0;

  FORALLELEMENTS(x,j) qual+=(saturated[j]<<4)-indeg_free[j];
  qual = qual<<6;
  FORALLELEMENTS(y,j) qual+=(saturated[j]<<4)-indeg_free[j];

- return qual;
+ return (int)qual;
 }

 int is_canonical_edge(BOOG list[],int last_positie)

Regards,
-- 
Jerry James
http://www.jamezone.org/

From Brendan.McKay at anu.edu.au  Thu Sep  3 16:37:17 2026
From: Brendan.McKay at anu.edu.au (Brendan McKay)
Date: Thu, 3 Sep 2026 16:37:17 +1000
Subject: [Nauty] Patch for undefined behavior
In-Reply-To: <CAHCOHQm6ZX320d-wPw-zmQrsT8sv4hhRxfSSnSN+giw0CxNhBA@mail.gmail.com>
References: <CAHCOHQm6ZX320d-wPw-zmQrsT8sv4hhRxfSSnSN+giw0CxNhBA@mail.gmail.com>
Message-ID: <72e96812-7a5c-4ebf-a2ac-f92c1c912ed3@anu.edu.au>

Hi James,

I had found the nausha.c issue myself by the same method.
Not the watercluster2.c issue though; now it is done.

As always, thanks for reporting.
Cheers, Brendan.

On 3/9/2026 8:10 am, Jerry James via Nauty wrote:
> If nauty 2.9.3 is built with -fsanitize=undefined, two instances of
> undefined behavior are identified while running the tests, both due to
> signed integer overflow.  I would like to suggest the patch below so
> that overflow only happens with unsigned integers.  In the nausha.c
> case, data is an array of 1-byte quantities, which are automatically
> promoted to int, a signed type, when used in an arithmetic expression.
> The fix is to explicitly convert to an unsigned 32-bit type.  In the
> watercluster2.c case, since qual can overflow, make it an unsigned
> variable and cast to a signed type when returning the result.  All
> tests pass after making these changes.
>
> --- nauty2_9_3/nausha.c.orig
> +++ nauty2_9_3/nausha.c
> @@ -74,8 +74,8 @@ sha256_transform(SHA256_CTX *ctx, const
>       nsword32 a, b, c, d, e, f, g, h, i, j, t1, t2, m[64];
>
>       for (i = 0, j = 0; i < 16; ++i, j += 4)
> -            m[i] = (data[j] << 24) | (data[j + 1] << 16)
> - | (data[j + 2] << 8) | (data[j + 3]);
> +            m[i] = ((nsword32)data[j] << 24) | ((nsword32)data[j + 1] << 16)
> + | ((nsword32)data[j + 2] << 8) | ((nsword32)data[j + 3]);
>       for ( ; i < 64; ++i)
>               m[i] = SIG1(m[i - 2]) + m[i - 7] + SIG0(m[i - 15]) + m[i - 16];
>
> --- nauty2_9_3/watercluster2.c.orig
> +++ nauty2_9_3/watercluster2.c
> @@ -3130,13 +3130,13 @@ int i,start,end, problem, sum, maxsum;
>   int getexpensivequality(graph x,graph y)
>   {
>   int j;
> -int qual=0;
> +unsigned int qual=0;
>
>    FORALLELEMENTS(x,j) qual+=(saturated[j]<<4)-indeg_free[j];
>    qual = qual<<6;
>    FORALLELEMENTS(y,j) qual+=(saturated[j]<<4)-indeg_free[j];
>
> - return qual;
> + return (int)qual;
>   }
>
>   int is_canonical_edge(BOOG list[],int last_positie)
>
> Regards,