[
https://issues.apache.org/jira/browse/IMPALA-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17236122#comment-17236122
]
Attila Doroszlai commented on IMPALA-9437:
------------------------------------------
bq. it is possible for HDFS to create a block size with a different one than
requested by Impala. So in order to track the correct block size for a file,
the file needs to be stat'ed after opening it. ... So we should investigate
the impact of not having this capability and consider adding it to the Ozone
client.
Ozone has a fixed block size, its {{FileSystem}} implementations ignore
{{blockSize}} argument passed to {{create()}} and {{createNonRecursive()}}.
Does that eliminate the need for any change in this task, or does Impala need
the configured block size from Ozone?
CC [~tarmstrong]
> Cannot get Ozone file block size after opening the file
> -------------------------------------------------------
>
> Key: IMPALA-9437
> URL: https://issues.apache.org/jira/browse/IMPALA-9437
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Sahil Takiar
> Priority: Major
>
> When creating a tmp file on HDFS, {{HdfsTableSink::CreateNewTmpFile}} first
> opens the file, and then stats the file ({{hdfsGetPathInfo}}) before actually
> writing any data to the file or closing the file. HDFS seems to allow this
> behavior. However, Ozone, S3A, and ABFS do not. Impala does this for HDFS in
> order to get the block size of the opened file. According to
> {{HdfsTableSink}} it is possible for HDFS to create a block size with a
> different one than requested by Impala. So in order to track the correct
> block size for a file, the file needs to be stat'ed after opening it. For S3A
> and ABFS this isn't a big deal, because they aren't block based filesystem,
> but Ozone is. So we should investigate the impact of not having this
> capability and consider adding it to the Ozone client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
