[
https://issues.apache.org/jira/browse/DRILL-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067554#comment-16067554
]
ASF GitHub Bot commented on DRILL-5379:
---------------------------------------
Github user kkhatua commented on the issue:
https://github.com/apache/drill/pull/826
@ppadma , Khurram [~khfaraaz] and I were looking at the details in the PR
and it's not very clear what new behavior does the PR allow. If you need to
specify the block-size as described in the [comment
](https://issues.apache.org/jira/browse/DRILL-5379?focusedCommentId=15981366&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15981366)by
@fmethot , doesn't Drill already do that? I thought Drill implicitly creates
files with a single row-group anyway.
My understanding of the JIRA's problem statement was that if the Parquet
block-size (i.e. the rowgroup size) is set to a large value that exceeds the
HDFS block size, using the flag would allow Drill to ignore the larger value in
the options and write with a parquet-blocksize that matches the target HDFS
location. So, I could have {{store.parquet.block-size=1073741824}} (i.e. 1GB),
but when writing an output worth 512MB, instead of 1 file... Drill would read
the HDFS block-size (say 128GB) and apply that as the parquet-block-size and
write 4 files.
@fmethot is that what you were looking for? An **automatic scaling down**
of the parquet file's size to match (and be contained within) the HDFS block
size?
> Set Hdfs Block Size based on Parquet Block Size
> -----------------------------------------------
>
> Key: DRILL-5379
> URL: https://issues.apache.org/jira/browse/DRILL-5379
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
> Affects Versions: 1.9.0
> Reporter: F Méthot
> Assignee: Padma Penumarthy
> Labels: ready-to-commit
> Fix For: Future, 1.11.0
>
>
> It seems there a way to force Drill to store CTAS generated parquet file as a
> single block when using HDFS. Java HDFS API allows to do that, files could be
> created with the Parquet block-size set in a session or system config.
> Since it is ideal to have single parquet file per hdfs block.
> Here is the HDFS API that allow to do that:
> http://archive.cloudera.com/cdh4/cdh/4/hadoop/api/org/apache/hadoop/fs/FileSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20short,%20long)
> http://archive.cloudera.com/cdh4/cdh/4/hadoop/api/org/apache/hadoop/fs/FileSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20short,%20long)
> Drill uses the hadoop ParquetFileWriter
> (https://github.com/Parquet/parquet-mr/blob/master/parquet-hadoop/src/main/java/parquet/hadoop/ParquetFileWriter.java).
> This is where the file creation occurs so it might be tricky.
> However, ParquetRecordWriter.java
> (https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java)
> in Drill creates the ParquetFileWriter with an hadoop configuration object.
> something to explore: Could the block size be set as a property within the
> Configuration object before passing it to ParquetFileWriter constructor?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
