crypto: feed long messages to the AEAD ciphers in chunks

[ThomasWaldmann]

The AEAD ciphers (AES-OCB, ChaCha20-Poly1305) in crypto/low_level.pyx passed the whole message to EVP_EncryptUpdate() / EVP_DecryptUpdate() in one call and kept every length in a C int, although the OpenSSL update calls take an int input length:

• a message of 2^31-48 bytes or more failed with SystemError: Negative size passed to PyBytes_FromStringAndSize,
• above 2^32 bytes the payload was silently truncated to its length modulo 2^32.

Pack objects never get there (they are bounded by MAX_DATA_SIZE), but with #10406 the per-archive reference caches also go through the key's envelope, and an archive referencing ~60 million objects produces a cache of that size, which would crash borg compact and borg analyze on every run.

This feeds the input (data and AAD) to OpenSSL in chunks of at most 1 GiB and keeps every byte count in a Py_ssize_t. Both AEAD modes stream, so chunked calls give the same envelope as a single call. The chunk size is a module variable so a test can shrink it to 7 bytes and cover chunk boundaries that fall inside a cipher block (OCB buffers partial blocks across calls); a second test round-trips a 2 GiB message and is skipped unless BORG_TESTS_BIG_MEMORY is set (it needs ~4 GiB of RAM, ~3-4 s here).

The legacy AES-CTR classes used for reading borg 1.x repositories keep their int lengths: their messages are bounded by borg 1.x's object size limit.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.29%. Comparing base (d576bb4) to head (2c7f710).
⚠️ Report is 5 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10409      +/-   ##
==========================================
+ Coverage   88.28%   88.29%   +0.01%     
==========================================
  Files         103      103              
  Lines       18904    18904              
  Branches     2937     2937              
==========================================
+ Hits        16690    16692       +2     
+ Misses       1540     1538       -2     
  Partials      674      674              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[ThomasWaldmann]

Ran the test_aead_message_longer_than_int_max test body (from 2c7f710, worktree build of the extension) parametrized over more sizes, both ciphers, sequentially in one process on a macOS arm64 machine with 18 GiB RAM (Python 3.11).

Sizes: 0, 2**30-1, 2**30, 2**30+1, 2**31-1, 2**31, 2**31+1, 2**32-1, 2**32, 2**32+1.

Each run: encrypt, check envelope length == 16 + size, decrypt, check plaintext length and that it is all zero.

size	AES256_OCB	CHACHA20_POLY1305
0	pass	pass
2^30-1, 2^30, 2^30+1	pass, ~1.2 s each	pass, ~1.4 s each
2^31-1, 2^31, 2^31+1	pass, ~3 s each	pass, ~2.7 s each
2^32-1, 2^32, 2^32+1	pass, ~6 s each	pass, ~7 s each

20 passed in 65 s total, peak memory ~8 GiB for the 4 GiB cases. Too heavy for CI, which is why the in-tree test stays behind BORG_TESTS_BIG_MEMORY with a single size.

scratch test used

import pytest
from borg.crypto.low_level import AES256_OCB, CHACHA20_POLY1305

SIZES = [0, 2**31 - 1, 2**31, 2**31 + 1, 2**32 - 1, 2**32, 2**32 + 1, 2**30 - 1, 2**30, 2**30 + 1]


@pytest.mark.parametrize("size", SIZES, ids=[str(s) for s in SIZES])
@pytest.mark.parametrize("cipher_cls", [AES256_OCB, CHACHA20_POLY1305])
def test_aead_message_sizes(cipher_cls, size):
    data = bytes(size)
    cipher = cipher_cls(b"k" * 32, iv=0)
    envelope = cipher.encrypt(data)
    assert len(envelope) == 16 + size
    del data
    plaintext = cipher.decrypt(envelope)
    assert len(plaintext) == size and plaintext.count(b"\0") == size