Hi, Regarding Shannon first point: I agree. We can improve the user experience a lot, and documenting the behavior is the first step we should do here. I see your points. I agree that we should use a separate thread for running the main method and report better to the front end what's happening.
Re 2.: The main() method is executed on the Application Master / JobManager / Web Interface JVM, when submitting the job through the web interface. When using the ./bin/flink tool, its executed there. As Gordon said correctly, Flink doesn't pick up newly added partitions. Leader changes are handled transparently since 1.0.0. I hope that clarify things. On Tue, Jun 14, 2016 at 5:57 AM, Tzu-Li (Gordon) Tai <tzuli...@gmail.com> wrote: > Hi Shannon, > > Thanks for your investigation on the issue and the JIRA. There's actually a > previous JIRA on this problem already: > https://issues.apache.org/jira/browse/FLINK-4023. Would you be ok with > tracking this issue on FLINK-4023, and close FLINK-4069 as a duplicate > issue? As you can see, I've also referenced a link to FLINK-4069 on > FLINK-4023 for your additional info on the problem. > > A little help with answering your last questions: > 1. We're doing the partition distribution across consumers ourselves: the > Kafka consumer connector creates a Kafka client on subtasks, and each > subtask independently determines which partitions it should be in charge > of. > There's also information on this blog here for more info: > http://data-artisans.com/kafka-flink-a-practical-how-to/, on the last FAQ > section. As Robert has mentioned, the consumer is currently depending on > the > fixed ordered list of partitions sent to all subtasks so that each of them > always determine the same set of partitions to fetch from across restarts. > 2. Following the above description, currently the consumer is only > subscribing to the fixed partition list queried in the constructor. So at > the moment the Flink Kafka consumer doesn't handle repartitioning of > topics, > but it's definitely on the todo list for the Kafka connector and won't be > too hard to implement once querying in the consumer is resolved (perhaps > Robert can clarify this a bit more). > > Best, > Gordon > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/API-request-to-submit-job-takes-over-1hr-tp7319p7558.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >
