StephanEwen opened a new pull request #13394:
URL: https://github.com/apache/flink/pull/13394
## What is the purpose of the change
This PR helps to drastically reduce the number of RPC requests made to the
file system master (for example the HDFS Name Node) when enumerating the splits
in file sources. Previously, we enumerated first all files, and then made a
separate RPC call to the name node to get the block locations for each file.
Hadoop has introduced an optimization here that this PR exploits. When the
HDFS Client returns a `FileStatus` (description of a file) it frequently
returns a `LocatedFileStatus` which already contains all the `BlockLocation`
information.
In this PR, we mirror that mechanism in the Flink FileSystem APIs:
When the `FileStatus` obtained from Hadoop file systems during listing the
directory (or getting details for a file) already has the block locations, we
use that directly and expose in a Flink `LocatedFileStatus`. That saves the
extra RPC call per file to obtain that block location information.
## Brief change log
- 04e493f0c0d3ba80a2507c4550633a62af45a2e3 : [FLINK-19218] Remove
inconsistent/misleading locality information from Local File Splits.
This is a general cleanup/improvement and makes the implementation of
the main change easier.
- cdc711101c86a58196f0978272a315d03b87e303 : [FLINK-19221] Main change as
described above.
## Verifying this change
This change makes a "best effort" attempt and, on purpose, does not add hard
tests to enforce this behavior. That is because certain file system
implementations or setups may not eagerly expose the location information and
we don't want to break compatibility working with them.
The best way to verify this is running a file-heavy batch job against HDFS
with this change and observing the reduced number or Name Node RPC requests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **yes** *(a new class, no breaking changes)*
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
