Hi, the SequenceFileWriter and the AvroKeyValueSinkWriter both support compressed outputs. Apart from that, I'm not aware of any other Writers which support compression. Maybe you could use these two Writers as a guiding example. Alternatively, you could try to extend the StreamWriterBase and wrapping the outStream into a GZIPOutputStream.
Cheers, Till On Wed, Mar 28, 2018 at 1:59 AM, [email protected] <[email protected]> wrote: > I want to upload a compressed file (gzip preferrably) using the Bucketing > Sink. What is the best way to do this? Would I have to implement my own > Writer that does the compression? Has anyone done something similar? >
