Hi Bashir, I think it is just a case of somebody bothering to plumbing it in explicitly, e.g.
/** Specifies row group size. By default, DEFAULT_BLOCK_SIZE. */ public Sink withRowGroupSize(int rowGroupSize) { return toBuilder().setRowGroupSize(rowGroupSize).build(); } and this.writer = AvroParquetWriter.<GenericRecord>builder(beamParquetOutputFile) .withRowGroupSize(getRowGroupSize()) // Ze patch to set RowGroupSize .withSchema(schema) .withCompressionCodec(getCompressionCodec()) .withWriteMode(OVERWRITE) .build(); Etc. However, it might worth exploring if it can be set via the HadoopConfiguration “parquet.block.size” property, but I’m not sure that it actually can. We patched in something explicitly last year but didn’t contribute upstream as there was quite a bit of activity on the ParquetIO (e.g. conversion to SDF) at the time. The use case we had at the time was that some downstream consumers of the parquet (AWS S3 Select) couldn’t handle rowGroupSizes > 64MB uncompressed. I’m sure there are other use cases out there that need this fined grained control. Cheers, David David Hollands BBC Broadcast Centre, London, W12 Email: david.holla...@bbc.co.uk<mailto:david.holla...@bbc.co.uk> From: Bashir Sadjad <bas...@google.com> Reply to: "user@beam.apache.org" <user@beam.apache.org> Date: Friday, 12 March 2021 at 07:58 To: "user@beam.apache.org" <user@beam.apache.org> Subject: Setting rowGroupSize in ParquetIO Hi all, I wonder how I can set the row group size for files generated by ParquetIO.Sink<https://beam.apache.org/releases/javadoc/2.20.0/org/apache/beam/sdk/io/parquet/ParquetIO.Sink.html>. It doesn't seem to provide the option for setting that and IIUC from the code<https://github.com/apache/beam/blob/fffb85a35df6ae3bdb2934c077856f6b27559aa7/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java#L1117>, it uses the default value in ParquetWriter.Builder<https://github.com/apache/parquet-mr/blob/bdf935a43bd377c8052840a4328cf5b7603aa70a/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetWriter.java#L636> which is 128MB. Is there any reason not to expose this parameter in ParquetIO? Thanks -B