Joe McDonnell created IMPALA-15322:
--------------------------------------
Summary: Bypass libhdfs when reading local data files
Key: IMPALA-15322
URL: https://issues.apache.org/jira/browse/IMPALA-15322
Project: IMPALA
Issue Type: Improvement
Components: Backend
Affects Versions: Impala 5.0.0
Reporter: Joe McDonnell
When tables reference files local to Impala (e.g. on the local filesystem for a
single Impalad with a path like file:///path/to/file), the file IO is currently
going through libhdfs and the JVM like any other HDFS read. This is incurring
the libhdfs / JVM overhead without benefit, as these files are accessible
through standard UNIX file IO APIs. A prototype that switched file:/ URIs to
the LocalFileReader rather than HdfsFileReader showed performance improvements
on ClickBench queries for a variety of queries. This could also help the write
path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
