[Nauty] Patch for undefined behavior

[Brendan McKay]

Hi James,

I had found the nausha.c issue myself by the same method.
Not the watercluster2.c issue though; now it is done.

As always, thanks for reporting.
Cheers, Brendan.

On 3/9/2026 8:10 am, Jerry James via Nauty wrote:
> If nauty 2.9.3 is built with -fsanitize=undefined, two instances of
> undefined behavior are identified while running the tests, both due to
> signed integer overflow.  I would like to suggest the patch below so
> that overflow only happens with unsigned integers.  In the nausha.c
> case, data is an array of 1-byte quantities, which are automatically
> promoted to int, a signed type, when used in an arithmetic expression.
> The fix is to explicitly convert to an unsigned 32-bit type.  In the
> watercluster2.c case, since qual can overflow, make it an unsigned
> variable and cast to a signed type when returning the result.  All
> tests pass after making these changes.
>
> --- nauty2_9_3/nausha.c.orig
> +++ nauty2_9_3/nausha.c
> @@ -74,8 +74,8 @@ sha256_transform(SHA256_CTX *ctx, const
>       nsword32 a, b, c, d, e, f, g, h, i, j, t1, t2, m[64];
>
>       for (i = 0, j = 0; i < 16; ++i, j += 4)
> -            m[i] = (data[j] << 24) | (data[j + 1] << 16)
> - | (data[j + 2] << 8) | (data[j + 3]);
> +            m[i] = ((nsword32)data[j] << 24) | ((nsword32)data[j + 1] << 16)
> + | ((nsword32)data[j + 2] << 8) | ((nsword32)data[j + 3]);
>       for ( ; i < 64; ++i)
>               m[i] = SIG1(m[i - 2]) + m[i - 7] + SIG0(m[i - 15]) + m[i - 16];
>
> --- nauty2_9_3/watercluster2.c.orig
> +++ nauty2_9_3/watercluster2.c
> @@ -3130,13 +3130,13 @@ int i,start,end, problem, sum, maxsum;
>   int getexpensivequality(graph x,graph y)
>   {
>   int j;
> -int qual=0;
> +unsigned int qual=0;
>
>    FORALLELEMENTS(x,j) qual+=(saturated[j]<<4)-indeg_free[j];
>    qual = qual<<6;
>    FORALLELEMENTS(y,j) qual+=(saturated[j]<<4)-indeg_free[j];
>
> - return qual;
> + return (int)qual;
>   }
>
>   int is_canonical_edge(BOOG list[],int last_positie)
>
> Regards,