Nevermind, it seems snappy is supported by default for ParquetFormat. It was
not obvious with the fixed extension.
private final static String EXTENSION = ".parquet";
final Schema avroSchema = avroData.fromConnectSchema(record.valueSchema());
CompressionCodecName compressionCodecName = CompressionCodecName.SNAPPY;
final ParquetWriter<GenericRecord> writer =
new AvroParquetWriter<>(path, avroSchema, compressionCodecName, blockSize,
pageSize);
________________________________
From: Henry Kim
Sent: Wednesday, November 2, 2016 2:46:27 PM
To: [email protected]
Subject: HDFS Connector Compression?
Is it possible to add compression to the HDFS Connector out of the box? Or does
it require code change?
Thanks
Henry Kim
