iemejia commented on PR #3745: URL: https://github.com/apache/avro/pull/3745#issuecomment-4358768317
Ok my dear Claude says this (which I don't know if it should gives us confidence but it make somehow sense): > The DataFileWriter.buffer is used to accumulate raw uncompressed records before they're flushed as a block. The default sync interval is 64 KB, so normally this buffer only grows to ~64 KB. But users can set syncInterval up to 1 GB. If someone sets a sync interval > 200 MB, the write would fail. > However, there's a subtlety — this buffer goes through buffer.reset() at DataFileWriter.java:434 after each block flush, so it never accumulates more than one block's worth of data. The question is whether any real-world use case writes blocks > 200 MB. > Your concern is valid but limited in scope. Here's the precise answer: > > No, this will NOT be an issue for typical Avro MapReduce usage. Here's why: > 1. Default sync interval is 64 KB (4000 * 16). The DataFileWriter.buffer is flushed whenever accumulated data reaches this threshold, so it never grows beyond ~80 KB in the default case. > 2. MapReduce defaults are the same. AvroOutputFormat uses DEFAULT_SYNC_INTERVAL (64 KB) unless explicitly overridden via avro.mapred.sync.interval. > 3. Even with a large sync interval, you'd need > 200 MB per block. The setSyncInterval API allows up to 1 GB, but the limit only fires if a single uncompressed block exceeds 200 MB. Most production jobs use intervals of 64 KB to a few MB. > 4. The compress() path is safe. Compressed output is always <= uncompressed input, so the NonCopyingByteArrayOutputStream used inside DeflateCodec.compress() etc. won't hit the limit. > Where it COULD be an issue: > - Users who set syncInterval to a very large value (> 200 MB) to optimize for sequential read throughput — rare, but possible in batch analytics. > - The appendAllFrom() path calls decompressUsing() then compressUsing() — the decompression step would correctly be bounded (intended behavior), but this means files with blocks > 200 MB can't be recompressed between codecs. > - SortedKeyValueFile sets sync interval to 1 << 20 (1 MB) — safe. > The 200 MB default is generous enough that this won't affect MapReduce workloads in practice. The Avro default block size is 64 KB, Hadoop ecosystem tools typically use 1-16 MB, and even aggressive configurations rarely exceed 128 MB per block. The edge case would be someone who explicitly set syncInterval > 200 MB AND writes records that actually fill a single block to that size — an extremely uncommon configuration. > If you want to be safe, the system property org.apache.avro.limits.decompress.maxLength can be increased or set to -1 to disable the limit entirely for trusted environments. > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
