Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/226#issuecomment-65705999
I'm really not an expert in streaming systems, but the behavior feels
unexpected coming from the batch world.
For Flink batch, we have an InputSplitAssigner running in the JobManager
that is assigning parts of the input to the workers in the cluster. Maybe we
need to run the "HDFS monitoring" centrally in the JobManager an assign the
reading to of the newly added files to individual workers (ideally respecting
the locality).
Maybe a good approach for now is to merge this change, document the
limitation and then fix the issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
