[Nauty] Patch for undefined behavior

[Jerry James]

If nauty 2.9.3 is built with -fsanitize=undefined, two instances of
undefined behavior are identified while running the tests, both due to
signed integer overflow.  I would like to suggest the patch below so
that overflow only happens with unsigned integers.  In the nausha.c
case, data is an array of 1-byte quantities, which are automatically
promoted to int, a signed type, when used in an arithmetic expression.
The fix is to explicitly convert to an unsigned 32-bit type.  In the
watercluster2.c case, since qual can overflow, make it an unsigned
variable and cast to a signed type when returning the result.  All
tests pass after making these changes.

--- nauty2_9_3/nausha.c.orig
+++ nauty2_9_3/nausha.c
@@ -74,8 +74,8 @@ sha256_transform(SHA256_CTX *ctx, const
     nsword32 a, b, c, d, e, f, g, h, i, j, t1, t2, m[64];

     for (i = 0, j = 0; i < 16; ++i, j += 4)
-            m[i] = (data[j] << 24) | (data[j + 1] << 16)
- | (data[j + 2] << 8) | (data[j + 3]);
+            m[i] = ((nsword32)data[j] << 24) | ((nsword32)data[j + 1] << 16)
+ | ((nsword32)data[j + 2] << 8) | ((nsword32)data[j + 3]);
     for ( ; i < 64; ++i)
             m[i] = SIG1(m[i - 2]) + m[i - 7] + SIG0(m[i - 15]) + m[i - 16];

--- nauty2_9_3/watercluster2.c.orig
+++ nauty2_9_3/watercluster2.c
@@ -3130,13 +3130,13 @@ int i,start,end, problem, sum, maxsum;
 int getexpensivequality(graph x,graph y)
 {
 int j;
-int qual=0;
+unsigned int qual=0;

  FORALLELEMENTS(x,j) qual+=(saturated[j]<<4)-indeg_free[j];
  qual = qual<<6;
  FORALLELEMENTS(y,j) qual+=(saturated[j]<<4)-indeg_free[j];

- return qual;
+ return (int)qual;
 }

 int is_canonical_edge(BOOG list[],int last_positie)

Regards,
-- 
Jerry James
http://www.jamezone.org/