Integer Overflow

Not accounting for integer overflow can result in logic errors or buffer overflow.

Integer overflow errors occur when a program fails to account for the fact that an arithmetic operation can result in a quantity either greater than a data type's maximum value or less than its minimum value. These errors often cause problems in memory allocation functions, where user input intersects with an implicit conversion between signed and unsigned values. If an attacker can cause the program to under-allocate memory or interpret a signed value as an unsigned value in a memory operation, the program might be vulnerable to a buffer overflow.

Example 1: The following code excerpt from OpenSSH 3.3 demonstrates a classic case of integer overflow:

nresp = packet_get_int();
if (nresp > 0) {
 response = xmalloc(nresp*sizeof(char*));
 for (i = 0; i < nresp; i++)
  response[i] = packet_get_string(NULL);
}

If nresp has the value 1073741824 and sizeof(char*) has its typical value of 4, then the result of the operation nresp*sizeof(char*) overflows, and the argument to xmalloc() will be 0. Most malloc() implementations will allow for the allocation of a 0-byte buffer, causing the subsequent loop iterations to overflow the heap buffer response.

Example 2: This example processes user input comprised of a series of variable-length structures. The first 2 bytes of input dictate the size of the structure to be processed.

 char* processNext(char* strm) {
 char buf[512];
 short len = *(short*) strm;
 strm += sizeof(len);
 if (len <= 512) {
  memcpy(buf, strm, len);
  process(buf);
  return strm + len;
 } else {
  return -1;
 }
}

The programmer has set an upper bound on the structure size: if it is larger than 512, the input will not be processed. The problem is that len is a signed integer, so the check against the maximum structure length is done with signed integers, but len is converted to an unsigned integer for the call to memcpy(). If len is negative, then it will appear that the structure has an appropriate size (the if branch will be taken), but the amount of memory copied by memcpy() will be quite large, and the attacker will be able to overflow the stack with data in strm.