carlpayne commented on issue #25794: URL: https://github.com/apache/beam/issues/25794#issuecomment-1472215554
> f you want compression you should use the built-in compression support that the avro container file presents. If you're using beam that'd be [withCodec](https://beam.apache.org/releases/javadoc/2.45.0/org/apache/beam/sdk/io/AvroIO.TypedWrite.html#withCodec-org.apache.avro.file.CodecFactory-) on AvroIO. 100% agree and this is the action item I'm taking away for my team, gzipping files that have their own built-in compression support doesn't make much sense (plus reading these files presents all sorts of problems, in addition to what we see here with Beam, many other data-processing tools also fail to correctly read these types of files) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
