Re: [DISCUSS] Reverting sink metric name changes made in 1.15
Thanks for raising the discussion, Qingsheng, +1 on reverting the breaking changes. In addition, we might want to release a 1.15.3 to fix this and update the previous release docs with this known issue, so that users can upgrade to 1.15.3 when they hit it. It would also be good to add some backwards compatibility tests on metrics to avoid unintended breaking changes like this in the future. Thanks, Jiangjie (Becket) Qin On Sun, Oct 9, 2022 at 10:35 AM Qingsheng Ren wrote: > Hi devs and users, > > I’d like to start a discussion about reverting a breaking change about sink > metrics made in 1.15 by FLINK-26126 > [1] and FLINK-26492 [2]. > > TL;DR > > > All sink metrics with name “numXXXOut” defined in FLIP-33 are replace by > “numXXXSend” in FLINK-26126 and FLINK-26492. Considering metric names are > public APIs, this is a breaking change to end users and not backward > compatible. Also unfortunately this breaking change was not discussed in the > mailing list before. > > Background > > > As defined previously in FLIP-33 (the FLIP page has been changed so please > refer to the old version [3] ), metric “numRecordsOut” is used for reporting > the total number of output records since the sink started (number of records > written to the external system), and similarly for “numRecordsOutPerSecond”, > “numBytesOut”, “numBytesOutPerSecond” and “numRecordsOutError”. Most sinks > are following this naming and definition. However, these metrics are > ambiguous in the new Sink API as “numXXXOut” could be used by the output of > SinkWriterOperator for reporting number of Committables delivered to > SinkCommitterOperator. In order to resolve the conflict, FLINK-26126 and > FLINK-26492 changed names of these metrics with “numXXXSend”. > > Necessity of reverting this change > > > - Metric names are actually public API, as end users need to configure metric > collecting and alerting system with metric names. Users have to reset all > configurations related to affected metrics. > > - This could also affect custom and external sinks not maintained by Flink, > which might have implemented with numXXXOut metrics. > > - The number of records sent to external system is way more important than > the number of Committables sent to SinkCommitterOperator, as the latter one > is just an internal implementation of sink. We could have a new metric name > for the latter one instead. > > - We could avoid splitting the project by version (like “plz use numXXXOut > before 1.15 and use numXXXSend after”) if we revert it ASAP, cosidering 1.16 > is still not released for now. > > > As a consequence, I’d like to hear from devs and users about your opinion on > changing these metrics back to “numXXXOut”. > > Looking forward to your reply! > > [1] https://issues.apache.org/jira/browse/FLINK-26126 > [2] https://issues.apache.org/jira/browse/FLINK-26492 > [1] FLIP-33, version 18: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883136 > > Best, > Qingsheng >
About the Current22 event
Hi my Flink fellas, The CFP for the Current22 [1] event is about to close. The Current event is the next generation of KafkaSummit. It expands the scope to cover **ALL** the technologies for real-time data, not limited to Kafka. Given Flink is a leading project in this area, the program committee is actively looking for speakers from the Flink community. Please don't hesitate to submit a talk [2] if you are interested! Thanks, Jiangjie (Becket) Qin [1] https://2022.currentevent.io/website/39543/ [2] https://sessionize.com/current-2022/
Re: [ANNOUNCE] Apache Flink Table Store 0.1.0 released
Really excited to see the very first release of the flink-table-store! Kudos to everyone who helped with this effort! Cheers, Jiangjie (Becket) Qin On Wed, May 11, 2022 at 1:55 PM Jingsong Lee wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink Table Store 0.1.0. > > Apache Flink Table Store provides storage for building dynamic tables for > both stream and batch processing in Flink, supporting high speed data > ingestion and timely data query. > > Please check out the release blog post for an overview of the release: > https://flink.apache.org/news/2022/05/11/release-table-store-0.1.0.html > > The release is available for download at: > https://flink.apache.org/downloads.html > > Maven artifacts for Flink Table Store can be found at: > https://search.maven.org/search?q=g:org.apache.flink%20table-store > > The full release notes are available in Jira: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351234 > > We would like to thank all contributors of the Apache Flink community who > made this release possible! > > Best, > Jingsong Lee >
Re: [Discuss] Creating an Apache Flink slack workspace
Thanks for the proposal, Xintong. While I share the same concerns as those mentioned in the previous discussion thread, admittedly there are benefits of having a slack channel as a supplementary way to discuss Flink. The fact that this topic is raised once a while indicates lasting interests. Personally I am open to having such a slack channel. Although it has drawbacks, it serves a different purpose. I'd imagine that for people who prefer instant messaging, in absence of the slack channel, a lot of discussions might just take place offline today, which leaves no public record at all. One step further, if the channel is maintained by the Flink PMC, some kind of code of conduct might be necessary. I think the suggestions of ad-hoc conversations, reflecting back to the emails are good starting points. I am +1 to give it a try and see how it goes. In the worst case, we can just stop doing this and come back to where we are right now. Thanks, Jiangjie (Becket) Qin On Fri, May 6, 2022 at 8:55 PM Martijn Visser wrote: > Hi everyone, > > While I see Slack having a major downside (the results are not indexed by > external search engines, you can't link directly to Slack content unless > you've signed up), I do think that the open source space has progressed and > that Slack is considered as something that's invaluable to users. There are > other Apache programs that also run it, like Apache Airflow [1]. I also see > it as a potential option to create a more active community. > > A concern I can see is that users will start DMing well-known > reviewers/committers to get a review or a PR merged. That can cause a lot > of noise. I can go +1 for Slack, but then we need to establish a set of > community rules. > > Best regards, > > Martijn > > [1] https://airflow.apache.org/community/ > > On Fri, 6 May 2022 at 13:59, Piotr Nowojski wrote: > >> Hi Xintong, >> >> I'm not sure if slack is the right tool for the job. IMO it works great as >> an adhoc tool for discussion between developers, but it's not searchable >> and it's not persistent. Between devs, it works fine, as long as the >> result >> of the ad hoc discussions is backported to JIRA/mailing list/design doc. >> For users, that simply would be extremely difficult to achieve. In the >> result, I would be afraid we are answering the same questions over, and >> over and over again, without even a way to provide a link to the previous >> thread, because nobody can search for it . >> >> I'm +1 for having an open and shared slack space/channel for the >> contributors, but I think I would be -1 for such channels for the users. >> >> For users, I would prefer to focus more on, for example, stackoverflow. >> With upvoting, clever sorting of the answers (not the oldest/newest at >> top) >> it's easily searchable - those features make it fit our use case much >> better IMO. >> >> Best, >> Piotrek >> >> >> >> pt., 6 maj 2022 o 11:08 Xintong Song napisał(a): >> >> > Thank you~ >> > >> > Xintong Song >> > >> > >> > >> > -- Forwarded message - >> > From: Xintong Song >> > Date: Fri, May 6, 2022 at 5:07 PM >> > Subject: Re: [Discuss] Creating an Apache Flink slack workspace >> > To: private >> > Cc: Chesnay Schepler >> > >> > >> > Hi Chesnay, >> > >> > Correct me if I'm wrong, I don't find this is *repeatedly* discussed on >> the >> > ML. The only discussions I find are [1] & [2], which are 4 years ago. On >> > the other hand, I do find many users are asking questions about whether >> > Slack should be supported [2][3][4]. Besides, I also find a recent >> > discussion thread from ComDev [5], where alternative communication >> channels >> > are being discussed. It seems to me ASF is quite open to having such >> > additional channels and they have been worked well for many projects >> > already. >> > >> > I see two reasons for brining this discussion again: >> > 1. There are indeed many things that have change during the past 4 >> years. >> > We have more contributors, including committers and PMC members, and >> even >> > more users from various organizations and timezones. That also means >> more >> > discussions and Q&As are happening. >> > 2. The proposal here is different from the previous discussion. Instead >> of >> > maintaining a channel for Flink in the ASF workspace, here we are >> proposing >> > to create
Re: flink1.12.2 "Failed to execute job"
Hi, Piotr is correct. The cause of this issue is likely because the instantiation of the SplitEnumerator is done in the JM main thread. FLINK-22282 has been created to address this issue. Thanks, Jiangjie (Becket) Qin On Wed, Apr 14, 2021 at 10:32 PM Piotr Nowojski wrote: > Hi, > > I haven't found anything strange in the logs (I've received logs in a > separate message). It looks like the problem is that split enumeration is > taking a long time, and currently this is being done in the Job Manager's > main thread, blocking other things from executing. For the time being I > think the only thing you can do is to either speed up the split enumeration > (probably difficult) or increase the timeouts that are failing. I don't > know if there is some other workaround at the moment (Becket?). > > Piotrek > > śr., 14 kwi 2021 o 15:57 Piotr Nowojski napisał(a): > >> Hey, >> >> could you provide full logs from both task managers and job managers? >> >> Piotrek >> >> śr., 14 kwi 2021 o 15:43 太平洋 <495635...@qq.com> napisał(a): >> >>> After submit job, I received 'Failed to execute job' error. And the >>> time between initialization and scheduling last 214s. What has happened >>> during this period? >>> >>> version: flink: 1.12.2 >>> deployment: k8s standalone >>> logs: >>> >>> 2021-04-14 12:47:58,547 WARN >>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - >>> Property [transaction.timeout.ms] not specified. Setting it to 360 >>> ms >>> 2021-04-14 12:48:04,175 INFO >>> org.apache.flink.client.deployment.application.executors. >>> EmbeddedExecutor [] - Job 1276000e99efdb77bdae0df88ab91da3 is submitted. >>> 2021-04-14 12:48:04,175 INFO >>> org.apache.flink.client.deployment.application.executors. >>> EmbeddedExecutor [] - Submitting Job with JobId=1276000e99 >>> efdb77bdae0df88ab91da3. >>> 2021-04-14 12:48:04,249 INFO org.apache.flink.runtime.dispatcher. >>> StandaloneDispatcher [] - Received JobGraph submission >>> 1276000e99efdb77bdae0df88ab91da3 >>> (Prediction Program). >>> 2021-04-14 12:48:04,249 INFO org.apache.flink.runtime.dispatcher. >>> StandaloneDispatcher [] - Submitting job 1276000e99efdb77bdae0df88ab91da3 >>> (Prediction Program). >>> 2021-04-14 12:48:04,250 INFO org.apache.flink.runtime.rpc.akka. >>> AkkaRpcService [] - Starting RPC endpoint for >>> org.apache.flink.runtime.jobmaster.JobMaster at >>> akka://flink/user/rpc/jobmanager_8 . >>> 2021-04-14 12:48:04,251 INFO org.apache.flink.runtime.jobmaster. >>> JobMaster [] - Initializing job Prediction Program (1276000e99 >>> efdb77bdae0df88ab91da3). >>> 2021-04-14 12:48:04,251 INFO org.apache.flink.runtime.jobmaster. >>> JobMaster [] - Using restart back off time strategy >>> NoRestartBackoffTimeStrategy for Prediction Program (1276000e99 >>> efdb77bdae0df88ab91da3). >>> 2021-04-14 12:48:04,251 INFO org.apache.flink.runtime.jobmaster. >>> JobMaster [] - Running initialization on master for job Prediction >>> Program (1276000e99efdb77bdae0df88ab91da3). >>> 2021-04-14 12:48:04,252 INFO org.apache.flink.runtime.jobmaster. >>> JobMaster [] - Successfully ran initialization on master in 0 ms. >>> 2021-04-14 12:48:04,254 INFO org.apache.flink.runtime.scheduler.adapter. >>> DefaultExecutionTopology [] - Built 10 pipelined regions in 0 ms >>> 2021-04-14 12:48:04,255 INFO org.apache.flink.runtime.jobmaster. >>> JobMaster [] - Using application-defined state backend: >>> org.apache.flink.streaming.api.operators.sorted.state. >>> BatchExecutionStateBackend@3ea8cd5a >>> 2021-04-14 12:48:04,255 INFO org.apache.flink.runtime.checkpoint. >>> CheckpointCoordinator [] - No checkpoint found during restore. >>> 2021-04-14 12:48:04,255 INFO org.apache.flink.runtime.jobmaster. >>> JobMaster [] - Using failover strategy >>> org.apache.flink.runtime.executiongraph.failover.flip1. >>> RestartPipelinedRegionFailoverStrategy@26845997 for Prediction Program ( >>> 1276000e99efdb77bdae0df88ab91da3). >>> 2021-04-14 12:48:04,255 INFO org.apache.flink.runtime.jobmaster. >>> JobManagerRunnerImpl [] - JobManager runner for job Prediction Program ( >>> 1276000e99efdb77bdae0df88ab91da3) was granted leadership with session >>> id ---- at akka.tcp://flink@flink >>> -jobmanager:6123/user/rpc/jobmanager_8. >>> 2021-04-14 12:48:04,255 INFO org.apache.flink.runtime
Re: FlinkKafkaProducer Fails with "Topic not present in metadata"
Hi Joseph, Thanks for the thorough information. Do you happen to have the trace level logging available? If so, do you mind puttng it somewhere so we can take a look? Thanks, Jiangjie (Becket) Qin On Thu, Dec 3, 2020 at 8:55 PM Joseph Lorenzini wrote: > Hi all, > > I have a flink job that uses FlinkKafkaConsumer to consume messages off a > kafka topic and a FlinkKafkaProducer to produce records on a Kafka topic. > The consumer works fine. However, the flink job eventually fails with the > following exception. > > Caused by: org.apache.kafka.common.errors.TimeoutException: Topic XXX not > present in metadata after 6 ms. > > I did find this issue but didn't have any details so I am not sure if its > related or not. > > https://issues.apache.org/jira/browse/FLINK-18757 > > Some details that might be important: > > - yes I verified the topic exists__ > - The kafka cluster that the flink job is integrating with is the > Confluent cloud platform at version 5.5.0. This means it should be > compatible with apache kafka 2.5.X. See here for details > https://docs.confluent.io/platform/current/installation/versions-interoperability.html > - ACLs and SASL SSL are turned on > - a springboot app that I wrote (which uses spring kafka) is able to write > to this same topic using the same credentials as what the flink job is using > - I am on flink 1.11.2 and using the flink-connector-kafka_2.11 at version > 1.11.2. > - I turned on trace logs and verified that metadata requests from the > flink job occur and metadata responses from the kafka broker are returned. > - I've set producer semantics to none and disabled checkpointing > > > > Privileged/Confidential Information may be contained in this message. If > you are not the addressee indicated in this message (or responsible for > delivery of the message to such person), you may not copy or deliver this > message to anyone. In such case, you should destroy this message and kindly > notify the sender by reply email. Please advise immediately if you or your > employer does not consent to Internet email for messages of this kind. > Opinions, conclusions and other information in this message that do not > relate to the official business of my firm shall be understood as neither > given nor endorsed by it. >
Re: State of Machine Learning with Flink and especially FLIP-39
Hi Niklas, We dropped the Flink ML lib in 1.9 and plan to replace it with a new machine learning library for traditional machine learning algorithms. And that library will be based on FLIP-39. The plan was pushed back a little bit because we plan to deprecate DataSet API and but haven't got the batch iteration support in DataStream API yet. So at this point we don't have an ML lib implementation in Flink. That being said, we are working with the community to add some ML related features to Flink. At this point, we have the following two projects available from Alibaba that will likely be contributed to Flink. You may also take a look at them. Alink - A machine learning library. https://github.com/alibaba/alink Flink-AI-Extended - A project helps running TF / PyTorch on top of Flink. https://github.com/alibaba/flink-ai-extended Thanks, Jiangjie (Becket) Qin On Fri, Nov 20, 2020 at 3:43 AM Arvid Heise wrote: > Hi Niklas, > > indeed some efforts on the machine learning libraries are pushed back in > favor of getting proper PyTorch and Tensorflow support through PyFlink. > > Native implementations in Flink have been done so far in the DataSet API, > which is going to deprecated in the next few releases in favor of the > unified DataStream API with bounded streams. I expect efforts for native > implementations to be picked up once DataSet is fully replaced to avoid > doubling the work. One of the most important features that is lacking is > proper iteration support in DataStream. > > On Thu, Nov 19, 2020 at 1:34 PM Niklas Wilcke > wrote: > >> Hi Flink-Community, >> >> I'm digging through the history of FlinkML and FLIP-39 [0]. What I >> understood so far is that FlinkML has been removed in 1.9, because it got >> unmaintained. >> I'm not really able to find out whether FLIP-39 and providing a >> replacement for FlinkML is currently worked on. The Umbrella Jira Ticket >> FLINK-12470 [1] looks stale to me. >> Was there maybe a change of strategy in the meantime? Is the focus >> currently on PyFlink to provide ML-Solutions (FLIP-96 [2])? >> It would be really interesting to get some insights about the future and >> roadmap of ML in the Flink ecosystem. Thank you very much! >> >> Kind Regards, >> Niklas >> >> [0] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs >> [1] https://issues.apache.org/jira/browse/FLINK-12470 >> [2] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-96%3A+Support+Python+ML+Pipeline+API > > > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng >
Re: What does Kafka Error sending fetch request mean for the Kafka source?
Hi John, The log message you saw from Kafka consumer simply means the consumer was disconnected from the broker that FetchRequest was supposed to be sent to. The disconnection can happen in many cases, such as broker down, network glitches, etc. The KafkaConsumer will just reconnect and retry sending that FetchRequest again. This won't cause duplicate messages in KafkaConsumer or Flink. Have you enabled exactly-once semantic for your Kafka sink? If not, the downstream might see duplicates in case of Flink failover or occasional retry in the KafkaProducer of the Kafka sink. Thanks, Jiangjie (Becket) Qin On Thu, Oct 22, 2020 at 11:38 PM John Smith wrote: > Any thoughts this doesn't seem to create duplicates all the time or maybe > it's unrelated as we are still seeing the message and there is no > duplicates... > > On Wed., Oct. 21, 2020, 12:09 p.m. John Smith, > wrote: > >> And yes my downstream is handling the duplicates in an idempotent way so >> we are good on that point. But just curious what the behaviour is on the >> source consumer when that error happens. >> >> On Wed, 21 Oct 2020 at 12:04, John Smith wrote: >> >>> Hi, running Flink 1.10.0 we see these logs once in a while... 2020-10-21 >>> 15:48:57,625 INFO org.apache.kafka.clients.FetchSessionHandler - [ >>> Consumer clientId=consumer-2, groupId=xx-import] Error sending >>> fetch request (sessionId=806089934, epoch=INITIAL) to node 0: >>> org.apache.kafka.common.errors.DisconnectException. >>> >>> Obviously it looks like the consumer is getting disconnected and from >>> what it seems it's either a Kafka bug on the way it handles the EPOCH or >>> possibly version mismatch between client and brokers. That's fine I can >>> look at upgrading the client and/or Kafka. But I'm trying to understand >>> what happens in terms of the source and the sink. It looks let we get >>> duplicates on the sink and I'm guessing it's because the consumer is >>> failing and at that point Flink stays on that checkpoint until it can >>> reconnect and process that offset and hence the duplicates downstream? >>> >>
Re: Status of a job when a kafka source dies
Hey Nick and Piotr, Sorry for the late reply. This email somehow failed to pass my mail filter. The KafkaConsumer in Apache Kafka itself does not throw any exception if the broker is down. There isn't any API in KafkaConsumer telling you that the brokers are not reachable. Instead, the consumer just keeps retrying to fetch the records. It is designed this way so that when there is a Kafka failure in an organization, people won't have to restart all the downstream applications after Kafka is up again. The Spring Kafka consumer is a community project which wraps the Java KafkaConsumer from Apache Kafka. The Spring Kafka consumer emits a special event if no message is received from the consumer.poll() call for some time. As Nick mentioned, it does not necessarily mean that the broker is down. It simply means that there isn't any message consumed from Kafka for some time. Nick, can you elaborate a little bit on why you would like to have an exception thrown in your Flink app when Kafka is down, rather than let it run until Kafka is up again? Thanks, JIangjie (Becket) Qin On Fri, Aug 14, 2020 at 4:28 PM Piotr Nowojski wrote: > Hey, > > But do you know what API is Kafka providing that Spring is using to > provide this feature? > > Piotrek > > czw., 13 sie 2020 o 17:15 Nick Bendtner napisał(a): > >> Hi Piotr, >> Sorry for the late reply. So the poll does not throw an exception when a >> broker goes down. In spring they solve it by generating an event [1] >> whenever this happens and you can intercept this event, >> consumer.timeout.ms helps to some extent does help but if the source >> topic does not receive any messages for the specified value then it still >> throws an exception. >> >> Best, >> Nick. >> >> >> [1] >> https://docs.spring.io/spring-kafka/api/org/springframework/kafka/event/NonResponsiveConsumerEvent.html >> >> On Wed, Aug 5, 2020 at 1:30 PM Piotr Nowojski >> wrote: >> >>> Hi Nick, >>> >>> Could you elaborate more, what event and how would you like Flink to >>> handle? Is there some kind of Kafka's API that can be used to listen to >>> such kind of events? Becket, do you maybe know something about this? >>> >>> As a side note Nick, can not you configure some timeouts [1] in the >>> KafkaConsumer? Like `request.timeout.ms` or `consumer.timeout.ms`? But >>> as I wrote before, that would be more a question to Kafka guys. >>> >>> Piotrek >>> >>> [1] http://kafka.apache.org/20/documentation/ >>> >>> śr., 5 sie 2020 o 19:58 Nick Bendtner napisał(a): >>> >>>> +user group. >>>> >>>> On Wed, Aug 5, 2020 at 12:57 PM Nick Bendtner >>>> wrote: >>>> >>>>> Thanks Piotr but shouldn't this event be handled by the >>>>> FlinkKafkaConsumer since the poll happens inside the FlinkKafkaConsumer. >>>>> How can I catch this event in my code since I don't have control over the >>>>> poll. >>>>> >>>>> Best, >>>>> Nick. >>>>> >>>>> On Wed, Aug 5, 2020 at 12:14 PM Piotr Nowojski >>>>> wrote: >>>>> >>>>>> Hi Nick, >>>>>> >>>>>> What Aljoscha was trying to say is that Flink is not trying to do any >>>>>> magic. If `KafkaConsumer` - which is being used under the hood of >>>>>> `FlinkKafkaConsumer` connector - throws an exception, this >>>>>> exception bubbles up causing the job to failover. If the failure is >>>>>> handled >>>>>> by the `KafkaConsumer` silently, that's what's happening. As we can in >>>>>> the >>>>>> TM log that you attached, the latter seems to be happening - note that >>>>>> the >>>>>> warning is logged by "org.apache.kafka.clients.NetworkClient" package, so >>>>>> that's not the code we (Flink developers) control. >>>>>> >>>>>> If you want to change this behaviour, unless someone here on this >>>>>> mailing list just happens to know the answer, the better place to ask >>>>>> such >>>>>> a question on the Kafka mailing list. Maybe there is some way to >>>>>> configure >>>>>> this. >>>>>> >>>>>> And sorry I don't know much about neither the KafkaConsumer nor the >>>>>> KafkaBrokers configurat
Re: ML/DL via Flink
Hi Max, Thanks for the question and sharing your findings. To be honest, I was not aware some of the projects until I see your list. First, to answer you questions: > (i) Has anyone used them? While I am not sure about the number of users of every listed project, Alink is definitely used by Alibaba. In fact, Alink team is trying to contribute the code to Flink repo to and become the new FlinkML library. Besides, I would like to add flink-ai-extended ( https://github.com/alibaba/flink-ai-extended) to the list. This project allows you to run TensorFlow / PyTorch on top of Flink. It is actively used and maintained by Alibaba as well. (ii) More specifically, has someone implemented *Stochastic Gradient > Descent, Skip-gram models, Autoencoders* with any of the above tools (or > other)? I think Alink has SGD there, but I did not find skip-gram / Autoencorder. Some more comments / replies below: > I assume it is more efficient to do all the training in Flink (somehow) > rather than (re)training a model in Tensorflow (or Pytorch) and porting it > to a flink Job. For instance, > > https://stackoverflow.com/questions/59563265/embedd-existing-ml-model-in-apache-flink > Especially, in streaming ML systems the training and the serving should > both > happen in an online fashion. I guess it depends on what exactly you want to do. If you are doing a training running for hours with a lot of rounds of iterations until it converges, having it trained separately and then porting it to Flink for inference might not lose too much efficiency. However, if you are doing online learning to incrementally update your model as the samples flow by, having such incremental training embedded into Flink would make a lot of sense. Flink-ai-extended was created to support both cases, but it is definitely more attractive in the incremental training case. 1) *FlinkML(DataSet API)* > > https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/libs/ml/index.html > This is not for streaming ML as it shits on top of DataSet API. In > addition, > recently the library is dropped > https://stackoverflow.com/questions/58752787/what-is-the-status-of-flinkml > but there is ongoing development (??) of a new library on top of TableAPI. > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs > https://issues.apache.org/jira/browse/FLINK-12470 > which is not in the 1.10 distribution. We removed the DataSet based FlinkML library because at that point it looks that there is no users of it. Removing it allows us to use cleaner package paths. That said, personally I agree that we should mark the library as deprecated and remove it from the code base in a later release. It looks you are looking for a ML algorithm library. Not sure if you are also interested in ML engineering part. We have an ongoing project called Flink AI Flow which allows you define an end-to-end online learning workflow, with datasets, models and metrics managed. I had a talk about it at the recent Flink Forward virtual event. The videos should be available soon. But feel free to reach out to me for more details. Thanks, Jiangjie (Becket) Qin On Wed, Apr 29, 2020 at 1:12 AM Timo Walther wrote: > Hi Max, > > as far as I know a better ML story for Flink is in the making. I will > loop in Becket in CC who may give you more information. > > Regards, > Timo > > On 28.04.20 07:20, m@xi wrote: > > Hello Flinkers, > > > > I am building a *streaming* prototype system on top of Flink and I want > > ideally to enable ML training (if possible DL) in Flink. It would be > nice to > > lay down all the existing libraries that provide primitives that enable > the > > training of ML models. > > > > I assume it is more efficient to do all the training in Flink (somehow) > > rather than (re)training a model in Tensorflow (or Pytorch) and porting > it > > to a flink Job. For instance, > > > https://stackoverflow.com/questions/59563265/embedd-existing-ml-model-in-apache-flink > > Especially, in streaming ML systems the training and the serving should > both > > happen in an online fashion. > > > > To initialize the pool, I have found the following options that run on > top > > of Flink i.e., leveraging the engine for distributed and scalable ML > > training. > > > > 1) *FlinkML(DataSet API)* > > > https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/libs/ml/index.html > > This is not for streaming ML as it shits on top of DataSet API. In > addition, > > recently the library is dropped > > > https://stackoverflow.com/questions/58752787/what-is-the-status-of-flinkml > > but there is ongoing development (??) of a new library on top of > TableAPI. > > > https://cwik
Re: New kafka producer on each checkpoint
A slightly more common case that may cause the producer to be not reusable is when there is no data for long time, the producer won't send any request to the broker and the tansactional.id may also expire on the broker side. On Tue, Apr 14, 2020 at 8:44 AM Becket Qin wrote: > Hi Maxim, > > That is a good question. I don't see an obvious reason that we cannot > reuse the producers. That said, there might be some corner cases where the > KafkaProducer is not reusable. For example, if the checkpoint interval is > long, the producer.id assigned by the TransactionCoordinator may have > expired on the broker side and the producer may not be reusable anymore. > But that should be a rare case. > > @Piotr Nowojski might know some more reasons that > the producers are not reused when it was initially implemented. > > Thanks, > > JIangjie (Becket) Qin > > On Mon, Apr 13, 2020 at 4:59 PM Maxim Parkachov > wrote: > >> Hi Yun, >> >> thanks for the answer. I did now increased checkpoint interval, but still >> I don't understand reason for creating producer and re-connecting to to >> kafka broker each time. According to documentation: >> >> Note: Semantic.EXACTLY_ONCE mode uses a fixed size pool of >> KafkaProducers per each FlinkKafkaProducer011 instance. One of each of >> those producers is used per one checkpoint. If the number of concurrent >> checkpoints exceeds the pool size, FlinkKafkaProducer011 will throw an >> exception and will fail the whole application. Please configure max pool >> size and max number of concurrent checkpoints accordingly. >> >> I assumed that this is also true for post 011 producers as well. I >> expected to have 5 (default) producers created and used without >> re-instantiating producer each time. In my case checkpoint is so fast that >> I will never have concurrent checkpoints. >> >> Regards, >> Maxim. >> >> >> On Wed, Apr 8, 2020 at 4:52 AM Yun Tang wrote: >> >>> Hi Maxim >>> >>> If you use the EXACTLY_ONCE semantic (instead of AT_LEAST_ONCE or NONE) >>> for flink kafka producer. It will create new producer when every new >>> checkpoint comes [1]. This is by design and from my point of view, the >>> checkpoint interval of 10 seconds might be a bit too often. In general I >>> think interval of 3 minutes should be enough. If you cannot offer the >>> source rewind time after failover, you could turn the interval more often. >>> >>> >>> [1] >>> https://github.com/apache/flink/blob/980e31dcc29ec6cc60ed59569f1f1cb7c47747b7/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java#L871 >>> >>> Best >>> Yun Tang >>> -- >>> *From:* Maxim Parkachov >>> *Sent:* Monday, April 6, 2020 23:16 >>> *To:* user@flink.apache.org >>> *Subject:* New kafka producer on each checkpoint >>> >>> Hi everyone, >>> >>> I'm trying to test exactly once functionality with my job under >>> production load. The job is reading from kafka, using kafka timestamp as >>> event time, aggregates every minute and outputs to other kafka topic. I use >>> checkpoint interval 10 seconds. >>> >>> Everything seems to be working fine, but when I look to the log on INFO >>> level, I see that with each checkpoint, new kafka producer is created and >>> then closed again. >>> >>> 1. Is this how it is supposed to work ? >>> 2. Is checkpoint interval 10 second too often ? >>> >>> Thanks, >>> Maxim. >>> >>
Re: New kafka producer on each checkpoint
Hi Maxim, That is a good question. I don't see an obvious reason that we cannot reuse the producers. That said, there might be some corner cases where the KafkaProducer is not reusable. For example, if the checkpoint interval is long, the producer.id assigned by the TransactionCoordinator may have expired on the broker side and the producer may not be reusable anymore. But that should be a rare case. @Piotr Nowojski might know some more reasons that the producers are not reused when it was initially implemented. Thanks, JIangjie (Becket) Qin On Mon, Apr 13, 2020 at 4:59 PM Maxim Parkachov wrote: > Hi Yun, > > thanks for the answer. I did now increased checkpoint interval, but still > I don't understand reason for creating producer and re-connecting to to > kafka broker each time. According to documentation: > > Note: Semantic.EXACTLY_ONCE mode uses a fixed size pool of KafkaProducers > per each FlinkKafkaProducer011 instance. One of each of those producers > is used per one checkpoint. If the number of concurrent checkpoints exceeds > the pool size, FlinkKafkaProducer011 will throw an exception and will > fail the whole application. Please configure max pool size and max number > of concurrent checkpoints accordingly. > > I assumed that this is also true for post 011 producers as well. I > expected to have 5 (default) producers created and used without > re-instantiating producer each time. In my case checkpoint is so fast that > I will never have concurrent checkpoints. > > Regards, > Maxim. > > > On Wed, Apr 8, 2020 at 4:52 AM Yun Tang wrote: > >> Hi Maxim >> >> If you use the EXACTLY_ONCE semantic (instead of AT_LEAST_ONCE or NONE) >> for flink kafka producer. It will create new producer when every new >> checkpoint comes [1]. This is by design and from my point of view, the >> checkpoint interval of 10 seconds might be a bit too often. In general I >> think interval of 3 minutes should be enough. If you cannot offer the >> source rewind time after failover, you could turn the interval more often. >> >> >> [1] >> https://github.com/apache/flink/blob/980e31dcc29ec6cc60ed59569f1f1cb7c47747b7/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java#L871 >> >> Best >> Yun Tang >> -- >> *From:* Maxim Parkachov >> *Sent:* Monday, April 6, 2020 23:16 >> *To:* user@flink.apache.org >> *Subject:* New kafka producer on each checkpoint >> >> Hi everyone, >> >> I'm trying to test exactly once functionality with my job under >> production load. The job is reading from kafka, using kafka timestamp as >> event time, aggregates every minute and outputs to other kafka topic. I use >> checkpoint interval 10 seconds. >> >> Everything seems to be working fine, but when I look to the log on INFO >> level, I see that with each checkpoint, new kafka producer is created and >> then closed again. >> >> 1. Is this how it is supposed to work ? >> 2. Is checkpoint interval 10 second too often ? >> >> Thanks, >> Maxim. >> >
Re: KafkaConsumer keeps getting InstanceAlreadyExistsException
Hi Rong, The issue here is that the PartitionDiscoverer has an internal KafkaConsumer which reuses the client.id set by the users for the actual fetching KafkaConsumer. Different KafkaConsumers distinguish their metrics by client.id, therefore if there are two KafkaConsumers in the same JVM with the same client.id, their metrics will collide with each other. This is why the exception was reported. Thanks, Jiangjie (Becket) Qin On Thu, Mar 19, 2020 at 12:04 AM Rong Rong wrote: > Hi Becket/Till, > > Thanks for the detail explanation. Just to confirm: > the issue in FLINK-8093 refers to multiple Kafka consumer within the same > TM - thus the fix should be to make consumer client.id unique for > different tasks ? > and the issue here is an issue internal to the Kafka consumer, where both > the polling consumer thread and the MBean JMX reporter thread share the > same client.id - thus we should fix this in the Kafka level? > > If this is the correct understanding, I think we should separate them > since they are in fact 2 different issues. > > -- > Rong > > On Tue, Mar 17, 2020 at 3:36 AM Becket Qin wrote: > >> Actually it might be better to create another ticket, FLINK-8093 was >> mainly complaining about the JMX bean collision when there are multiple >> tasks running in the same TM. >> >> Jiangjie (Becket) Qin >> >> On Tue, Mar 17, 2020 at 6:33 PM Becket Qin wrote: >> >>> Hi Till, >>> >>> It looks FLINK-8093 <https://issues.apache.org/jira/browse/FLINK-8093> >>> reports >>> the same issue, although the reported information is not exactly correct, >>> as this should not cause the producer to fail. I'll take care of the ticket. >>> >>> Thanks, >>> >>> Jiangjie (Becket) Qin >>> >>> On Tue, Mar 17, 2020 at 6:00 PM Till Rohrmann >>> wrote: >>> >>>> @Becket do we already have a JIRA ticket to track this effort? >>>> >>>> Cheers, >>>> Till >>>> >>>> On Mon, Mar 16, 2020 at 4:07 AM Becket Qin >>>> wrote: >>>> >>>>> Hi Sidney, >>>>> >>>>> The WARN logging you saw was from the AbstractPartitionDiscoverer >>>>> which is created by FlinkKafkaConsumer itself. It has an internal consumer >>>>> which shares the client.id of the actual consumer fetching data. This >>>>> is a bug that we should fix. >>>>> >>>>> As Rong said, this won't affect the normal operation of the consumer. >>>>> It is just an AppInfo MBean for reporting some information. There might be >>>>> some slight impact on the accuracy of the consumer metrics, but should be >>>>> almost ignorable because the partition discoverer is quite inactive >>>>> compared with the actual consumer. >>>>> >>>>> Thanks, >>>>> >>>>> Jiangjie (Becket) Qin >>>>> >>>>> On Mon, Mar 16, 2020 at 12:44 AM Rong Rong >>>>> wrote: >>>>> >>>>>> We also had seen this issue before running Flink apps in a shared >>>>>> cluster environment. >>>>>> >>>>>> Basically, Kafka is trying to register a JMX MBean[1] for application >>>>>> monitoring. >>>>>> This is only a WARN suggesting that you are registering more than one >>>>>> MBean with the same client id "consumer-1", it should not affect your >>>>>> normal application behavior. >>>>>> >>>>>> This is most likely occurring if you have more than one Kafka >>>>>> consumer within the same JVM, are you using a session cluster[2]? can you >>>>>> share more on your application configuration including parallelism and >>>>>> slot >>>>>> configs? >>>>>> Also based on the log, you are not configuring the "client.id" >>>>>> correctly. which config key are you using? could you also share your fill >>>>>> Kafka properties map? >>>>>> >>>>>> >>>>>> -- >>>>>> Rong >>>>>> >>>>>> [1] https://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html >>>>>> [2] >>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#flink-yarn-session >>>>>>
Re: KafkaConsumer keeps getting InstanceAlreadyExistsException
Actually it might be better to create another ticket, FLINK-8093 was mainly complaining about the JMX bean collision when there are multiple tasks running in the same TM. Jiangjie (Becket) Qin On Tue, Mar 17, 2020 at 6:33 PM Becket Qin wrote: > Hi Till, > > It looks FLINK-8093 <https://issues.apache.org/jira/browse/FLINK-8093> reports > the same issue, although the reported information is not exactly correct, > as this should not cause the producer to fail. I'll take care of the ticket. > > Thanks, > > Jiangjie (Becket) Qin > > On Tue, Mar 17, 2020 at 6:00 PM Till Rohrmann > wrote: > >> @Becket do we already have a JIRA ticket to track this effort? >> >> Cheers, >> Till >> >> On Mon, Mar 16, 2020 at 4:07 AM Becket Qin wrote: >> >>> Hi Sidney, >>> >>> The WARN logging you saw was from the AbstractPartitionDiscoverer which >>> is created by FlinkKafkaConsumer itself. It has an internal consumer which >>> shares the client.id of the actual consumer fetching data. This is a >>> bug that we should fix. >>> >>> As Rong said, this won't affect the normal operation of the consumer. It >>> is just an AppInfo MBean for reporting some information. There might be >>> some slight impact on the accuracy of the consumer metrics, but should be >>> almost ignorable because the partition discoverer is quite inactive >>> compared with the actual consumer. >>> >>> Thanks, >>> >>> Jiangjie (Becket) Qin >>> >>> On Mon, Mar 16, 2020 at 12:44 AM Rong Rong wrote: >>> >>>> We also had seen this issue before running Flink apps in a shared >>>> cluster environment. >>>> >>>> Basically, Kafka is trying to register a JMX MBean[1] for application >>>> monitoring. >>>> This is only a WARN suggesting that you are registering more than one >>>> MBean with the same client id "consumer-1", it should not affect your >>>> normal application behavior. >>>> >>>> This is most likely occurring if you have more than one Kafka consumer >>>> within the same JVM, are you using a session cluster[2]? can you share more >>>> on your application configuration including parallelism and slot configs? >>>> Also based on the log, you are not configuring the "client.id" >>>> correctly. which config key are you using? could you also share your fill >>>> Kafka properties map? >>>> >>>> >>>> -- >>>> Rong >>>> >>>> [1] https://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html >>>> [2] >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#flink-yarn-session >>>> >>>> On Sun, Mar 15, 2020 at 8:28 AM Sidney Feiner < >>>> sidney.fei...@startapp.com> wrote: >>>> >>>>> Hey, >>>>> I've been using Flink for a while now without any problems when >>>>> running apps with a FlinkKafkaConsumer. >>>>> All my apps have the same overall logic (consume from kafka -> >>>>> transform event -> write to file) and the only way they differ from each >>>>> other is the topic they read (remaining kafka config remains identical) >>>>> and >>>>> the way they transform the event. >>>>> But suddenly, I've been starting to get the following error: >>>>> >>>>> >>>>> 2020-03-15 12:13:56,911 WARN >>>>> org.apache.kafka.common.utils.AppInfoParser - Error >>>>> registering AppInfo mbean >>>>> javax.management.InstanceAlreadyExistsException: >>>>> kafka.consumer:type=app-info,id=consumer-1 >>>>>at >>>>> com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) >>>>>at >>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) >>>>> >>>>>at >>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) >>>>> >>>>>at >>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) >>>>> >>>>>at >>>>> com.sun.jmx.interceptor.Def
Re: KafkaConsumer keeps getting InstanceAlreadyExistsException
Hi Till, It looks FLINK-8093 <https://issues.apache.org/jira/browse/FLINK-8093> reports the same issue, although the reported information is not exactly correct, as this should not cause the producer to fail. I'll take care of the ticket. Thanks, Jiangjie (Becket) Qin On Tue, Mar 17, 2020 at 6:00 PM Till Rohrmann wrote: > @Becket do we already have a JIRA ticket to track this effort? > > Cheers, > Till > > On Mon, Mar 16, 2020 at 4:07 AM Becket Qin wrote: > >> Hi Sidney, >> >> The WARN logging you saw was from the AbstractPartitionDiscoverer which >> is created by FlinkKafkaConsumer itself. It has an internal consumer which >> shares the client.id of the actual consumer fetching data. This is a bug >> that we should fix. >> >> As Rong said, this won't affect the normal operation of the consumer. It >> is just an AppInfo MBean for reporting some information. There might be >> some slight impact on the accuracy of the consumer metrics, but should be >> almost ignorable because the partition discoverer is quite inactive >> compared with the actual consumer. >> >> Thanks, >> >> Jiangjie (Becket) Qin >> >> On Mon, Mar 16, 2020 at 12:44 AM Rong Rong wrote: >> >>> We also had seen this issue before running Flink apps in a shared >>> cluster environment. >>> >>> Basically, Kafka is trying to register a JMX MBean[1] for application >>> monitoring. >>> This is only a WARN suggesting that you are registering more than one >>> MBean with the same client id "consumer-1", it should not affect your >>> normal application behavior. >>> >>> This is most likely occurring if you have more than one Kafka consumer >>> within the same JVM, are you using a session cluster[2]? can you share more >>> on your application configuration including parallelism and slot configs? >>> Also based on the log, you are not configuring the "client.id" >>> correctly. which config key are you using? could you also share your fill >>> Kafka properties map? >>> >>> >>> -- >>> Rong >>> >>> [1] https://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html >>> [2] >>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#flink-yarn-session >>> >>> On Sun, Mar 15, 2020 at 8:28 AM Sidney Feiner < >>> sidney.fei...@startapp.com> wrote: >>> >>>> Hey, >>>> I've been using Flink for a while now without any problems when running >>>> apps with a FlinkKafkaConsumer. >>>> All my apps have the same overall logic (consume from kafka -> >>>> transform event -> write to file) and the only way they differ from each >>>> other is the topic they read (remaining kafka config remains identical) and >>>> the way they transform the event. >>>> But suddenly, I've been starting to get the following error: >>>> >>>> >>>> 2020-03-15 12:13:56,911 WARN >>>> org.apache.kafka.common.utils.AppInfoParser - Error >>>> registering AppInfo mbean >>>> javax.management.InstanceAlreadyExistsException: >>>> kafka.consumer:type=app-info,id=consumer-1 >>>>at >>>> com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) >>>>at >>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) >>>> >>>>at >>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) >>>> >>>>at >>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) >>>> >>>>at >>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) >>>> >>>>at >>>> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) >>>> >>>>at >>>> org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:62) >>>> >>>>at >>>> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:805) >>>> >>>>at >>>> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:659) >&
Re: KafkaConsumer keeps getting InstanceAlreadyExistsException
Hi Sidney, The WARN logging you saw was from the AbstractPartitionDiscoverer which is created by FlinkKafkaConsumer itself. It has an internal consumer which shares the client.id of the actual consumer fetching data. This is a bug that we should fix. As Rong said, this won't affect the normal operation of the consumer. It is just an AppInfo MBean for reporting some information. There might be some slight impact on the accuracy of the consumer metrics, but should be almost ignorable because the partition discoverer is quite inactive compared with the actual consumer. Thanks, Jiangjie (Becket) Qin On Mon, Mar 16, 2020 at 12:44 AM Rong Rong wrote: > We also had seen this issue before running Flink apps in a shared cluster > environment. > > Basically, Kafka is trying to register a JMX MBean[1] for application > monitoring. > This is only a WARN suggesting that you are registering more than one > MBean with the same client id "consumer-1", it should not affect your > normal application behavior. > > This is most likely occurring if you have more than one Kafka consumer > within the same JVM, are you using a session cluster[2]? can you share more > on your application configuration including parallelism and slot configs? > Also based on the log, you are not configuring the "client.id" correctly. > which config key are you using? could you also share your fill Kafka > properties map? > > > -- > Rong > > [1] https://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#flink-yarn-session > > On Sun, Mar 15, 2020 at 8:28 AM Sidney Feiner > wrote: > >> Hey, >> I've been using Flink for a while now without any problems when running >> apps with a FlinkKafkaConsumer. >> All my apps have the same overall logic (consume from kafka -> transform >> event -> write to file) and the only way they differ from each other is the >> topic they read (remaining kafka config remains identical) and the way they >> transform the event. >> But suddenly, I've been starting to get the following error: >> >> >> 2020-03-15 12:13:56,911 WARN org.apache.kafka.common.utils.AppInfoParser >> - Error registering AppInfo mbean >> javax.management.InstanceAlreadyExistsException: >> kafka.consumer:type=app-info,id=consumer-1 >>at >> com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) >>at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) >> >>at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) >> >>at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) >> >>at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) >> >>at >> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) >> >>at >> org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:62) >> >>at >> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:805) >> >>at >> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:659) >> >>at >> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:639) >> >>at >> org.apache.flink.streaming.connectors.kafka.internal.KafkaPartitionDiscoverer.initializeConnections(KafkaPartitionDiscoverer.java:58) >> >>at >> org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.open(AbstractPartitionDiscoverer.java:94) >> >>at >> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:505) >> >>at >> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) >> >>at >> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) >> >>at >> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:552) >> >>at >> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:416) >> >>at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) >>at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) >>at java.lang.Thread.run(Thread.java:748) >> >> >> I've tried setting the "client.id" on my consumer to a random UUID, >> making sure I don't have any duplicates but that didn't help either. >> Any idea what could be causing this? >> >> Thanks 🙂 >> >> *Sidney Feiner* */* Data Platform Developer >> M: +972.528197720 */* Skype: sidney.feiner.startapp >> >> [image: emailsignature] >> >>
Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1
+1 (binding) - verified signature - Ran word count example successfully. Thanks, Jiangjie (Becket) Qin On Wed, Feb 12, 2020 at 1:29 PM Jark Wu wrote: > +1 > > - checked/verified signatures and hashes > - Pip installed the package successfully: pip install > apache-flink-1.9.2.tar.gz > - Run word count example successfully through the documentation [1]. > > Best, > Jark > > [1]: > > https://ci.apache.org/projects/flink/flink-docs-release-1.9/tutorials/python_table_api.html > > On Tue, 11 Feb 2020 at 22:00, Hequn Cheng wrote: > > > +1 (non-binding) > > > > - Check signature and checksum. > > - Install package successfully with Pip under Python 3.7.4. > > - Run wordcount example successfully under Python 3.7.4. > > > > Best, Hequn > > > > On Tue, Feb 11, 2020 at 12:17 PM Dian Fu wrote: > > > > > +1 (non-binding) > > > > > > - Verified the signature and checksum > > > - Pip installed the package successfully: pip install > > > apache-flink-1.9.2.tar.gz > > > - Run word count example successfully. > > > > > > Regards, > > > Dian > > > > > > 在 2020年2月11日，上午11:44，jincheng sun 写道： > > > > > > > > > +1 (binding) > > > > > > - Install the PyFlink by `pip install` [SUCCESS] > > > - Run word_count in both command line and IDE [SUCCESS] > > > > > > Best, > > > Jincheng > > > > > > > > > > > > Wei Zhong 于2020年2月11日周二 上午11:17写道： > > > > > >> Hi, > > >> > > >> Thanks for driving this, Jincheng. > > >> > > >> +1 (non-binding) > > >> > > >> - Verified signatures and checksums. > > >> - Verified README.md and setup.py. > > >> - Run `pip install apache-flink-1.9.2.tar.gz` in Python 2.7.15 and > > Python > > >> 3.7.5 successfully. > > >> - Start local pyflink shell in Python 2.7.15 and Python 3.7.5 via > > >> `pyflink-shell.sh local` and try the examples in the help message, run > > well > > >> and no exception. > > >> - Try a word count example in IDE with Python 2.7.15 and Python 3.7.5, > > >> run well and no exception. > > >> > > >> Best, > > >> Wei > > >> > > >> > > >> 在 2020年2月10日，19:12，jincheng sun 写道： > > >> > > >> Hi everyone, > > >> > > >> Please review and vote on the release candidate #1 for the PyFlink > > >> version 1.9.2, as follows: > > >> > > >> [ ] +1, Approve the release > > >> [ ] -1, Do not approve the release (please provide specific comments) > > >> > > >> The complete staging area is available for your review, which > includes: > > >> > > >> * the official Apache binary convenience releases to be deployed to > > >> dist.apache.org [1], which are signed with the key with fingerprint > > >> 8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E [2] and built from source > code > > [3]. > > >> > > >> The vote will be open for at least 72 hours. It is adopted by majority > > >> approval, with at least 3 PMC affirmative votes. > > >> > > >> Thanks, > > >> Jincheng > > >> > > >> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.2-rc1/ > > >> [2] https://dist.apache.org/repos/dist/release/flink/KEYS > > >> [3] https://github.com/apache/flink/tree/release-1.9.2 > > >> > > >> > > > > > >
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Hi Thomas, Event time alignment is absolutely one of the important considerations of FLIP-27. That said, we are not implementing that in FLIP-27, but just make sure such feature can be easily added in the future. The design was to make the communication between SplitEnumerator and SourceReader quite extensible so we can put more features on top of it. More specifically, we will have two tiers of events flowing between the SourceReader and Enumerator - OperatorEvent and SourceEvent. The OperatorEvents are "lower" level events that are not visible to the user provided SourceReader / SplitEnumerator directly. Instead they are handled by the SourceOperator / SourceCoordinator that run SourceReader / SplitEnumerator. The SourceEvents are a subclass of OperatorEvents. They are "higher" level events that each connector implementation can have their own events so the SourceReader and SplitEnumerator can exchange arbitrary information. The event time alignment can be implemented on top of this communication mechanism: 1. Each SourceReader periodically reports the current watermark to the SplitEnumerator via a SourceEvent. 2. The SplitEnumerator collects the global watermark frontier and decide to apply some flow control to certain splits by sending another SourceEvent to relevant SourceReader. 3. The SourceReaders will then refrain from reading the throttled splits. This is just one implementation, the SplitEnumerator may also just broadcast the current global watermark to each SourceReader and let themselves make decision. With some API change, there is also a possibility to make this alignment handled at the SourceOperator / SourceCoordinator level. But in any case, after FLIP-27 such feature addition should be fairly easy. FLIP-27 is already quite a huge change by itself. I think there will be at least 10K lines of additions. Because of that, it might worth confine the scope a little bit. So I do not plan to implement event time alignment in FLIP-27. But we did think about it and I think it can be implemented easily afterwards. Thanks, Jiangjie (Becket) Qin On Mon, Nov 4, 2019 at 10:18 PM Thomas Weise wrote: > Hi Becket, > > Thanks for the reply, it is good to know that there is activity on FLIP-27. > > A while ago I was wondering if event time alignment is on the radar [1], > can you please clarify that? > > There is a parallel discussion of adding it to the existing Kafka consumer > [2], could you please take a look and help figuring out the path forward? > > Thanks, > Thomas > > [1] > https://lists.apache.org/thread.html/c7336f2594c482de0580ecf2472a6b3e5a0fe2b7dd9258b159f2@%3Cdev.flink.apache.org%3E > [2] https://issues.apache.org/jira/browse/FLINK-12675 > > > > On Mon, Nov 4, 2019 at 5:48 AM Becket Qin wrote: > >> Hi Steven and Thomas, >> >> Sorry about missing the update of FLIP-27. >> >> I am working on the implementation of FLIP-27 at this point. It is about >> 70% done. Right now I am integrating the source coordinator to the job >> master. Hopefully I can get the basics of Kafka connector work from end to >> end by this Sunday. Then I'll update FLIP-27 wiki page. >> >> Thanks, >> >> Jiangjie (Becket) Qin >> >> >> On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise wrote: >> >>> Is there any activity on FLIP-27 that would make it relevant for 1.10 >>> release? >>> >>> Thanks Gary for the update, it provides excellent visibility on current >>> activity and what we can expect with the release. >>> >>> >>> On Fri, Nov 1, 2019 at 1:52 PM Steven Wu wrote: >>> >>>> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info >>>> from update #1. >>>> >>>> - FLIP-27: Refactor Source Interface [20] >>>> - FLIP accepted. Implementation is in progress. >>>> >>>> >>>> >>>> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao wrote: >>>> >>>> > Hi community, >>>> > >>>> > Because we have approximately one month of development time left >>>> until the >>>> > targeted Flink 1.10 feature freeze, we thought now would be a good >>>> time to >>>> > give another progress update. Below we have included a list of the >>>> ongoing >>>> > efforts that have made progress since our last release progress update >>>> > [1]. As >>>> > always, if you are working on something that is not included here, >>>> feel >>>> > free >>>> > to use this thread to share your progress. >>>> > >>>> > - Support Java
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Hi Steven and Thomas, Sorry about missing the update of FLIP-27. I am working on the implementation of FLIP-27 at this point. It is about 70% done. Right now I am integrating the source coordinator to the job master. Hopefully I can get the basics of Kafka connector work from end to end by this Sunday. Then I'll update FLIP-27 wiki page. Thanks, Jiangjie (Becket) Qin On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise wrote: > Is there any activity on FLIP-27 that would make it relevant for 1.10 > release? > > Thanks Gary for the update, it provides excellent visibility on current > activity and what we can expect with the release. > > > On Fri, Nov 1, 2019 at 1:52 PM Steven Wu wrote: > >> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info >> from update #1. >> >> - FLIP-27: Refactor Source Interface [20] >> - FLIP accepted. Implementation is in progress. >> >> >> >> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao wrote: >> >> > Hi community, >> > >> > Because we have approximately one month of development time left until >> the >> > targeted Flink 1.10 feature freeze, we thought now would be a good time >> to >> > give another progress update. Below we have included a list of the >> ongoing >> > efforts that have made progress since our last release progress update >> > [1]. As >> > always, if you are working on something that is not included here, feel >> > free >> > to use this thread to share your progress. >> > >> > - Support Java 11 [2] >> > - Implementation is in progress (18/21 subtasks resolved) >> > >> > - Table API improvements >> > - Full Data Type Support in Planner [3] >> > - Implementing (1/8 subtasks resolved) >> > - FLIP-66 Support Time Attribute in SQL DDL [4] >> > - Implementation is in progress (1/7 subtasks resolved). >> > - FLIP-70 Support Computed Column [5] >> > - FLIP voting [6] >> > - FLIP-63 Rework Table Partition Support [7] >> > - Implementation is in progress (3/15 subtasks resolved). >> > - FLIP-51 Rework of Expression Design [8] >> > - Implementation is in progress (2/12 subtasks resolved). >> > - FLIP-64 Support for Temporary Objects in Table Module [9] >> > - Implementation is in progress >> > >> > - Hive compatibility completion (DDL/UDF) to support full Hive >> integration >> > - FLIP-57 Rework FunctionCatalog [10] >> > - Implementation is in progress (6/9 subtasks resolved) >> > - FLIP-68 Extend Core Table System with Modular Plugins [11] >> > - Implementation is in progress (2/8 subtasks resolved) >> > >> > - Finer grained resource management >> > - FLIP-49: Unified Memory Configuration for TaskExecutors [12] >> > - Implementation is in progress (6/10 subtasks resolved) >> > - FLIP-53: Fine Grained Operator Resource Management [13] >> > - Implementation is in progress (1/9 subtasks resolved) >> > >> > - Finish scheduler re-architecture [14] >> > - Integration tests are being enabled for new scheduler >> > >> > - Executor/Client refactoring [15] >> > - FLIP-81: Executor-related new ConfigOptions [16] >> > - done >> > - FLIP-73: Introducing Executors for job submission [17] >> > - Implementation is in progress >> > >> > - FLIP-36 Support Interactive Programming [18] >> > - Is built on top of FLIP-67 [19], which has been accepted >> > - Implementation in progress >> > >> > - FLIP-58: Flink Python User-Defined Stateless Function for Table [20] >> > - Implementation is in progress (12/22 subtask resolved) >> > - FLIP-50: Spill-able Heap Keyed State Backend [21] >> > - Implementation is in progress (2/11 subtasks resolved) >> > >> > - RocksDB Backend Memory Control [22] >> > - FLIP for resource management on state backend will be opened soon >> > - Write Buffer Manager will be backported to FRocksDB due to >> > performance regression [23] in new RocksDB versions >> > >> > - Unaligned Checkpoints >> > - FLIP-76 [24] was published and received positive feedback >> > - Implementation is in progress >> > >> > - Separate framework and user class loader in per-job mode [25] >> > - First PR is almost done. Remaining PRs will be ready next week >
Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink
+1 to adding Stateful Function to Flink. It is a very useful addition to the Flink ecosystem. Given this is essentially a new top-level / first-citizen API of Flink, it seems better to have it the Flink core repo. This will also avoid letting this important new API to be blocked on potential problems of maintaining multiple different repositories. Thanks, Jiangjie (Becket) Qin On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng wrote: > Hi Stephan, > > Big +1 for adding this to Apache Flink! > > As for the problem of whether this should be added to the Flink main > repository, from my side, I prefer to put it in the main repository. Not > only Stateful Functions shares very close relations with the current Flink, > but also other libs or modules in Flink can make use of it the other way > round in the future. At that time the Flink API stack would also be changed > a bit and this would be cool. > > Best, Hequn > > On Sat, Oct 12, 2019 at 9:16 PM Biao Liu wrote: > >> Hi Stehpan, >> >> +1 for having Stateful Functions in Flink. >> >> Before discussing which repository it should belong, I was wondering if >> we have reached an agreement of "splitting flink repository" as Piotr >> mentioned or not. It seems that it's just no more further discussion. >> It's OK for me to add it to core repository. After all almost everything >> is in core repository now. But if we decide to split the core repository >> someday, I tend to create a separate repository for Stateful Functions. It >> might be good time to take the first step of splitting. >> >> Thanks, >> Biao /'bɪ.aʊ/ >> >> >> >> On Sat, 12 Oct 2019 at 19:31, Yu Li wrote: >> >>> Hi Stephan, >>> >>> Big +1 for adding stateful functions to Flink. I believe a lot of user >>> would be interested to try this out and I could imagine how this could >>> contribute to reduce the TCO for business requiring both streaming >>> processing and stateful functions. >>> >>> And my 2 cents is to put it into flink core repository since I could see >>> a tight connection between this library and flink state. >>> >>> Best Regards, >>> Yu >>> >>> >>> On Sat, 12 Oct 2019 at 17:31, jincheng sun >>> wrote: >>> >>>> Hi Stephan, >>>> >>>> bit +1 for adding this great features to Apache Flink. >>>> >>>> Regarding where we should place it, put it into Flink core repository >>>> or create a separate repository? I prefer put it into main repository and >>>> looking forward the more detail discussion for this decision. >>>> >>>> Best, >>>> Jincheng >>>> >>>> >>>> Jingsong Li 于2019年10月12日周六 上午11:32写道： >>>> >>>>> Hi Stephan, >>>>> >>>>> big +1 for this contribution. It provides another user interface that >>>>> is easy to use and popular at this time. these functions, It's hard for >>>>> users to write in SQL/TableApi, while using DataStream is too complex. >>>>> (We've done some stateFun kind jobs using DataStream before). With >>>>> statefun, it is very easy. >>>>> >>>>> I think it's also a good opportunity to exercise Flink's core >>>>> capabilities. I looked at stateful-functions-flink briefly, it is very >>>>> interesting. I think there are many other things Flink can improve. So I >>>>> think it's a better thing to put it into Flink, and the improvement for it >>>>> will be more natural in the future. >>>>> >>>>> Best, >>>>> Jingsong Lee >>>>> >>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz < >>>>> dwysakow...@apache.org> wrote: >>>>> >>>>>> Hi Stephan, >>>>>> >>>>>> I think this is a nice library, but what I like more about it is that >>>>>> it suggests exploring different use-cases. I think it definitely makes >>>>>> sense for the Flink community to explore more lightweight applications >>>>>> that >>>>>> reuses resources. Therefore I definitely think it is a good idea for >>>>>> Flink >>>>>> community to accept this contribution and help maintaining it. >>>>>> >>>>>> Personally I'd prefer to have it in a separate repository. There were >>>>&
Re: [DISCUSS] Drop older versions of Kafka Connectors (0.9, 0.10) for Flink 1.10
Thanks for bringing this up, Stephan. I am +1 on dropping support for Kafka 0.8. It is a pretty old version and I don't think there are many users on that version now. However, for Kafka 0.9, I think there are still quite some users on that version. It might be better to keep it a little longer. Given that dropping connector support is a backwards incompatible change, technically speaking we need to bump up the major version. I am not sure if release 1.10 is a good version to remove the code. Personally I prefer to mark Kafka 0.8 connector as deprecated and remove the code base in Flink 2.x. Thanks, Jiangjie (Becket) Qin On Wed, Sep 11, 2019 at 4:26 PM Wesley Peng wrote: > > > on 2019/9/11 16:17, Stephan Ewen wrote: > > We still maintain connectors for Kafka 0.8 and 0.9 in Flink. > > I would suggest to drop those with Flink 1.10 and start supporting only > > Kafka 0.10 onwards. > > > > Are there any concerns about this, or still a significant number of > > users of these versions? > > But we still have a large scale of deployment kafka 0.9 in production. > Do you mean the new coming flink won't support kafka 0.9? > Though I understand for it, but just sounds sorry. :) > > regards. >
Re: [ANNOUNCE] Zili Chen becomes a Flink committer
Congrats, Zili! On Thu, Sep 12, 2019 at 9:39 AM Paul Lam wrote: > Congratulations Zili！ > > Best, > Paul Lam > > 在 2019年9月12日，09:34，Rong Rong 写道： > > Congratulations Zili! > > -- > Rong > > On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng wrote: > >> Congratulations! >> >> Best, Hequn >> >> On Thu, Sep 12, 2019 at 9:24 AM Jark Wu wrote: >> >>> Congratulations Zili! >>> >>> Best, >>> Jark >>> >>> On Wed, 11 Sep 2019 at 23:06, wrote: >>> >>> > Congratulations, Zili. >>> > >>> > >>> > >>> > Best, >>> > >>> > Xingcan >>> > >>> > >>> > >>> > *From:* SHI Xiaogang >>> > *Sent:* Wednesday, September 11, 2019 7:43 AM >>> > *To:* Guowei Ma >>> > *Cc:* Fabian Hueske ; Biao Liu >> >; >>> > Oytun Tez ; bupt_ljy ; dev < >>> > d...@flink.apache.org>; user ; Till Rohrmann < >>> > trohrm...@apache.org> >>> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer >>> > >>> > >>> > >>> > Congratulations! >>> > >>> > >>> > >>> > Regards, >>> > >>> > Xiaogang >>> > >>> > >>> > >>> > Guowei Ma 于2019年9月11日周三 下午7:07写道： >>> > >>> > Congratulations Zili ! >>> > >>> > >>> > Best, >>> > >>> > Guowei >>> > >>> > >>> > >>> > >>> > >>> > Fabian Hueske 于2019年9月11日周三 下午7:02写道： >>> > >>> > Congrats Zili Chen :-) >>> > >>> > >>> > >>> > Cheers, Fabian >>> > >>> > >>> > >>> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu < >>> mmyy1...@gmail.com>: >>> > >>> > Congrats Zili! >>> > >>> > >>> > >>> > Thanks, >>> > >>> > Biao /'bɪ.aʊ/ >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez wrote: >>> > >>> > Congratulations! >>> > >>> > >>> > >>> > --- >>> > >>> > Oytun Tez >>> > >>> > >>> > >>> > *M O T A W O R D* >>> > >>> > *The World's Fastest Human Translation Platform.* >>> > >>> > oy...@motaword.com — www.motaword.com >>> > >>> > >>> > >>> > >>> > >>> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy wrote: >>> > >>> > Congratulations! >>> > >>> > >>> > >>> > Best, >>> > >>> > Jiayi Liao >>> > >>> > >>> > >>> > Original Message >>> > >>> > *Sender:* Till Rohrmann >>> > >>> > *Recipient:* dev; user >>> > >>> > *Date:* Wednesday, Sep 11, 2019 17:22 >>> > >>> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer >>> > >>> > >>> > >>> > Hi everyone, >>> > >>> > >>> > >>> > I'm very happy to announce that Zili Chen (some of you might also know >>> > him as Tison Kun) accepted the offer of the Flink PMC to become a >>> committer >>> > of the Flink project. >>> > >>> > >>> > >>> > Zili Chen has been an active community member for almost 16 months now. >>> > He helped pushing the Flip-6 effort over the finish line, ported a lot >>> of >>> > legacy code tests, removed a good part of the legacy code, contributed >>> > numerous fixes, is involved in the Flink's client API refactoring, >>> drives >>> > the refactoring of Flink's HighAvailabilityServices and much more. Zili >>> > Chen also helped the community by PR reviews, reporting Flink issues, >>> > answering user mails and being very active on the dev mailing list. >>> > >>> > >>> > >>> > Congratulations Zili Chen! >>> > >>> > >>> > >>> > Best, Till >>> > >>> > (on behalf of the Flink PMC) >>> > >>> > >>> >> >
Re: Kafka consumer behavior with same group Id, 2 instances of apps in separate cluster?
Thanks for the explanation Ashish. Glad you made it work with custom source. I guess your application is probably stateless. If so, another option might be having a geo-distributed Flink deployment. That means there will be TM in different datacenter to form a single Flink cluster. This will also come with failover if one of the TM is down. I am not sure if anyone have tried this. It is probably a heavier solution than using Kafka to do the failover, but the good thing is that you may also do some stateful processing if you have a globally accessible storage for the state backup. Thanks, Jiangjie (Becket) Qin On Wed, Sep 4, 2019 at 11:00 AM Ashish Pokharel wrote: > Thanks Becket, > > Sorry for delayed response. That’s what I thought as well. I built a hacky > custom source today directly using Kafka client which was able to join > consumer group etc. which works as I expected but not sure about production > readiness for something like that :) > > The need arises because of (1) Business continuity needs (2) Some of the > pipelines we are building are close to network edge and need to run on > nodes where we are not allowed to create cluster (yea - let’s not get into > that can of security related worms :)). We will get there at some point but > for now we are trying to support business continuity on those edge nodes by > not actually forming a cluster but using “walled garden” individual Flink > server. I fully understand this is not ideal. And all of this started > because some of the work we were doing with Logstash needed to be migrated > out as Logstash wasn’t able to keep up with data rates unless we put some > ridiculous number of servers. In essence, we have pre-approved constraints > to connect to Kafka and southbound interfaces using Logstash, which we need > to replace for some datasets as they are massive for Logstash to keep up > with. > > Hope that explains a bit where our head is at. > > Thanks, Ashish > > On Aug 29, 2019, at 11:40 AM, Becket Qin wrote: > > Hi Ashish, > > You are right. Flink does not use Kafka based group management. So if you > have two clusters consuming the same topic, they will not divide the > partitions. The cross cluster HA is not quite possible at this point. It > would be good to know the reason you want to have such HA and see if Flink > meets you requirement in another way. > > Thanks, > > Jiangjie (Becket) Qin > > On Thu, Aug 29, 2019 at 9:19 PM ashish pok wrote: > >> Looks like Flink is using “assign” partitions instead of “subscribe” >> which will not allow participating in a group if I read the code correctly. >> >> Has anyone solved this type of problem in past of active-active HA across >> 2 clusters using Kafka? >> >> >> - Ashish >> >> On Wednesday, August 28, 2019, 6:52 PM, ashish pok >> wrote: >> >> All, >> >> I was wondering what the expected default behavior is when same app is >> deployed in 2 separate clusters but with same group Id. In theory idea was >> to create active-active across separate clusters but it seems like both >> apps are getting all the data from Kafka. >> >> Anyone else has tried something similar or have an insight on expected >> behavior? I was expecting to see partial data on both apps and to get all >> data in one app if other was turned off. >> >> Thanks in advance, >> >> - Ashish >> >> >
Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction
Hi Tony, >From the symptom it is not quite clear to me what may cause this issue. Supposedly the TransactionCoordinator is independent of the active controller, so bouncing the active controller should not have special impact on the transactions (at least not every time). If this is stably reproducible, is it possible to turn on debug level logging on kafka.coordinator.transaction.TransactionCoordinator to see what does the broker say? Thanks, Jiangjie (Becket) Qin On Thu, Aug 29, 2019 at 3:55 PM Tony Wei wrote: > Hi, > > Has anyone run into the same problem? I have updated my producer > transaction timeout to 1.5 hours, > but the problem sill happened when I restarted broker with active > controller. It might not due to the > problem that checkpoint duration is too long causing transaction timeout. > I had no more clue to find out > what's wrong about my kafka producer. Could someone help me please? > > Best, > Tony Wei > > Fabian Hueske 於 2019年8月16日 週五 下午4:10寫道： > >> Hi Tony, >> >> I'm sorry I cannot help you with this issue, but Becket (in CC) might >> have an idea what went wrong here. >> >> Best, Fabian >> >> Am Mi., 14. Aug. 2019 um 07:00 Uhr schrieb Tony Wei < >> tony19920...@gmail.com>: >> >>> Hi, >>> >>> Currently, I was trying to update our kafka cluster with larger ` >>> transaction.max.timeout.ms`. The >>> original setting is kafka's default value (i.e. 15 minutes) and I tried >>> to set as 3 hours. >>> >>> When I was doing rolling-restart for my brokers, this exception came to >>> me on the next checkpoint >>> after I restarted the broker with active controller. >>> >>> java.lang.RuntimeException: Error while confirming checkpoint at >>>> org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:1218) at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at >>>> java.util.concurrent.FutureTask.run(FutureTask.java:266) at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>> at java.lang.Thread.run(Thread.java:748) Caused by: >>>> org.apache.flink.util.FlinkRuntimeException: Committing one of transactions >>>> failed, logging first encountered failure at >>>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.notifyCheckpointComplete(TwoPhaseCommitSinkFunction.java:296) >>>> at >>>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyCheckpointComplete(AbstractUdfStreamOperator.java:130) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:684) >>>> at org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:1213) ... 5 >>>> more Caused by: org.apache.kafka.common.errors.InvalidTxnStateException: >>>> The producer attempted a transactional operation in an invalid state >>> >>> >>> I have no idea why it happened, and I didn't find any error log from >>> brokers. Does anyone have >>> this exception before? How can I prevent from this exception when I >>> tried to restart kafka cluster? >>> Does this exception mean that I will lost data in some of these >>> transactions? >>> >>> flink cluster version: 1.8.1 >>> kafka cluster version: 1.0.1 >>> flink kafka producer version: universal >>> producer transaction timeout: 15 minutes >>> checkpoint interval: 5 minutes >>> number of concurrent checkpoint: 1 >>> max checkpoint duration before and after the exception occurred: < 2 >>> minutes >>> >>> Best, >>> Tony Wei >>> >>
Re: Kafka consumer behavior with same group Id, 2 instances of apps in separate cluster?
Hi Ashish, You are right. Flink does not use Kafka based group management. So if you have two clusters consuming the same topic, they will not divide the partitions. The cross cluster HA is not quite possible at this point. It would be good to know the reason you want to have such HA and see if Flink meets you requirement in another way. Thanks, Jiangjie (Becket) Qin On Thu, Aug 29, 2019 at 9:19 PM ashish pok wrote: > Looks like Flink is using “assign” partitions instead of “subscribe” which > will not allow participating in a group if I read the code correctly. > > Has anyone solved this type of problem in past of active-active HA across > 2 clusters using Kafka? > > > - Ashish > > On Wednesday, August 28, 2019, 6:52 PM, ashish pok > wrote: > > All, > > I was wondering what the expected default behavior is when same app is > deployed in 2 separate clusters but with same group Id. In theory idea was > to create active-active across separate clusters but it seems like both > apps are getting all the data from Kafka. > > Anyone else has tried something similar or have an insight on expected > behavior? I was expecting to see partial data on both apps and to get all > data in one app if other was turned off. > > Thanks in advance, > > - Ashish > >
Re: [ANNOUNCE] Apache Flink 1.9.0 released
Cheers!! Thanks Gordon and Kurt for driving the release! On Thu, Aug 22, 2019 at 5:36 PM 不常用邮箱 wrote: > Good news! > > Best. > -- > Louis > Email： xu_soft39211...@163.com > > On Aug 22, 2019, at 22:10, Yang Wang wrote: > > Glad to hear that. > Thanks Gordon, Kurt and everyone who had made contributions to the great > version. > > > Best, > Yang > > > Biao Liu 于2019年8月22日周四 下午9:33写道： > >> Great news! >> >> Thank your Gordon & Kurt for being the release managers! >> Thanks all contributors worked on this release! >> >> Thanks, >> Biao /'bɪ.aʊ/ >> >> >> >> On Thu, 22 Aug 2019 at 21:14, Paul Lam wrote: >> >>> Well done! Thanks to everyone who contributed to the release! >>> >>> Best, >>> Paul Lam >>> >>> Yu Li 于2019年8月22日周四 下午9:03写道： >>> Thanks for the update Gordon, and congratulations! Great thanks to all for making this release possible, especially to our release managers! Best Regards, Yu On Thu, 22 Aug 2019 at 14:55, Xintong Song wrote: > Congratulations! > Thanks Gordon and Kurt for being the release managers, and thanks all > the contributors. > > Thank you~ > Xintong Song > > > > On Thu, Aug 22, 2019 at 2:39 PM Yun Gao wrote: > >> Congratulations ! >> >> Very thanks for Gordon and Kurt for managing the release and >> very thanks for everyone for the contributions ! >> >> Best, >> Yun >> >> >> >> -- >> From:Zhu Zhu >> Send Time:2019 Aug. 22 (Thu.) 20:18 >> To:Eliza >> Cc:user >> Subject:Re: [ANNOUNCE] Apache Flink 1.9.0 released >> >> Thanks Gordon for the update. >> Congratulations that we have Flink 1.9.0 released! >> Thanks to all the contributors. >> >> Thanks, >> Zhu Zhu >> >> >> Eliza 于2019年8月22日周四 下午8:10写道： >> >> >> On 2019/8/22 星期四 下午 8:03, Tzu-Li (Gordon) Tai wrote: >> > The Apache Flink community is very happy to announce the release of >> > Apache Flink 1.9.0, which is the latest major release. >> >> Congratulations and thanks~ >> >> regards. >> >> >> >
Re: [ANNOUNCE] Hequn becomes a Flink committer
Congrats, Hequn! Well deserved! On Wed, Aug 7, 2019 at 11:16 AM Zili Chen wrote: > Congrats Hequn! > > Best, > tison. > > > Jeff Zhang 于2019年8月7日周三 下午5:14写道： > >> Congrats Hequn! >> >> Paul Lam 于2019年8月7日周三 下午5:08写道： >> >>> Congrats Hequn! Well deserved! >>> >>> Best, >>> Paul Lam >>> >>> 在 2019年8月7日，16:28，jincheng sun 写道： >>> >>> Hi everyone, >>> >>> I'm very happy to announce that Hequn accepted the offer of the Flink >>> PMC to become a committer of the Flink project. >>> >>> Hequn has been contributing to Flink for many years, mainly working on >>> SQL/Table API features. He's also frequently helping out on the user >>> mailing lists and helping check/vote the release. >>> >>> Congratulations Hequn! >>> >>> Best, Jincheng >>> (on behalf of the Flink PMC) >>> >>> >>> >> >> -- >> Best Regards >> >> Jeff Zhang >> >
Re: [DISCUSS] Create a Flink ecosystem website
[Sorry for the incomplete message. Clicked send by mistake...] I agree with Marta that it might be good to have multi-language support as a mid-term goal. Jiangjie (Becket) Qin On Sat, Jul 20, 2019 at 11:22 AM Becket Qin wrote: > The website is awesome! I really like its conciseness and yet fairly > useful information and functionalities. I cannot think of much to improve > at the moment. Just one thought, do we need an "others" category, just in > case a package does not fit into any of the current given categories? > > Thanks Robert and Daryl for the great effort. Looking forward to seeing > this get published soon!! > > I agree with Marta that > > Jiangjie (Becket) Qin > > On Sat, Jul 20, 2019 at 1:34 AM Marta Paes Moreira > wrote: > >> Hey, Robert. >> >> I will keep an eye on the overall progress and get started on the blog >> post >> to make the community announcement. Are there (mid-term) plans to >> translate/localize this website as well? It might be a point worth >> mentioning in the blogpost. >> >> Hats off to you and Daryl — this turned out amazing! >> >> Marta >> >> On Thu, Jul 18, 2019 at 10:57 AM Congxian Qiu >> wrote: >> >> > Robert and Daryl, thanks for the great work, I tried the website and >> filed >> > some issues on Github. >> > Best, >> > Congxian >> > >> > >> > Robert Metzger 于2019年7月17日周三 下午11:28写道： >> > >> >> Hey all, >> >> >> >> Daryl and I have great news to share. We are about to finish adding the >> >> basic features to the ecosystem page. >> >> We are at a stage where it is ready to be reviewed and made public. >> >> >> >> You can either check out a development instance of the ecosystem page >> >> here: https://flink-ecosystem-demo.flink-resources.org/ >> >> Or you run it locally, with the instructions from the README.md: >> >> https://github.com/sorahn/flink-ecosystem >> >> >> >> Please report all issues you find here: >> >> https://github.com/sorahn/flink-ecosystem/issues or in this thread. >> >> >> >> The next steps in this project are the following: >> >> - We fix all issues reported through this testing >> >> - We set up the site on the INFRA resources Becket has secured [1], do >> >> some further testing (including email notifications) and pre-fill the >> page >> >> with some packages. >> >> - We set up a packages.flink.apache.org or flink.apache.org/packages >> >> domain >> >> - We announce the packages through a short blog post >> >> >> >> Happy testing! >> >> >> >> Best, >> >> Robert >> >> >> >> [1] https://issues.apache.org/jira/browse/INFRA-18010 >> >> >> >> >> >> On Thu, Apr 25, 2019 at 6:23 AM Becket Qin >> wrote: >> >> >> >>> Thanks for the update, Robert. Looking forward to the website. If >> there >> >>> is already a list of software we need to run the website, we can ask >> Apache >> >>> infra team to prepare the VM for us, as that may also take some time. >> >>> >> >>> On Wed, Apr 24, 2019 at 11:57 PM Robert Metzger >> >>> wrote: >> >>> >> >>>> Hey all, >> >>>> >> >>>> quick update on this project: The frontend and backend code have been >> >>>> put together into this repository: >> >>>> https://github.com/sorahn/flink-ecosystem >> >>>> We also just agreed on an API specification, and will now work on >> >>>> finishing the backend. >> >>>> >> >>>> It will probably take a few more weeks for this to finish, but we are >> >>>> making progress :) >> >>>> >> >>>> Best, >> >>>> Robert >> >>>> >> >>>> >> >>>> On Mon, Apr 15, 2019 at 11:18 AM Robert Metzger > > >> >>>> wrote: >> >>>> >> >>>>> Hey Daryl, >> >>>>> >> >>>>> thanks a lot for posting a link to this first prototype on the >> mailing >> >>>>> list! I really like it! >> >>>>> >> >>>>> Becket: Our plan forward is that Congxian is implementing the >> backend >> >>>>> for the website.
Re: [DISCUSS] Create a Flink ecosystem website
The website is awesome! I really like its conciseness and yet fairly useful information and functionalities. I cannot think of much to improve at the moment. Just one thought, do we need an "others" category, just in case a package does not fit into any of the current given categories? Thanks Robert and Daryl for the great effort. Looking forward to seeing this get published soon!! I agree with Marta that Jiangjie (Becket) Qin On Sat, Jul 20, 2019 at 1:34 AM Marta Paes Moreira wrote: > Hey, Robert. > > I will keep an eye on the overall progress and get started on the blog post > to make the community announcement. Are there (mid-term) plans to > translate/localize this website as well? It might be a point worth > mentioning in the blogpost. > > Hats off to you and Daryl — this turned out amazing! > > Marta > > On Thu, Jul 18, 2019 at 10:57 AM Congxian Qiu > wrote: > > > Robert and Daryl, thanks for the great work, I tried the website and > filed > > some issues on Github. > > Best, > > Congxian > > > > > > Robert Metzger 于2019年7月17日周三 下午11:28写道： > > > >> Hey all, > >> > >> Daryl and I have great news to share. We are about to finish adding the > >> basic features to the ecosystem page. > >> We are at a stage where it is ready to be reviewed and made public. > >> > >> You can either check out a development instance of the ecosystem page > >> here: https://flink-ecosystem-demo.flink-resources.org/ > >> Or you run it locally, with the instructions from the README.md: > >> https://github.com/sorahn/flink-ecosystem > >> > >> Please report all issues you find here: > >> https://github.com/sorahn/flink-ecosystem/issues or in this thread. > >> > >> The next steps in this project are the following: > >> - We fix all issues reported through this testing > >> - We set up the site on the INFRA resources Becket has secured [1], do > >> some further testing (including email notifications) and pre-fill the > page > >> with some packages. > >> - We set up a packages.flink.apache.org or flink.apache.org/packages > >> domain > >> - We announce the packages through a short blog post > >> > >> Happy testing! > >> > >> Best, > >> Robert > >> > >> [1] https://issues.apache.org/jira/browse/INFRA-18010 > >> > >> > >> On Thu, Apr 25, 2019 at 6:23 AM Becket Qin > wrote: > >> > >>> Thanks for the update, Robert. Looking forward to the website. If there > >>> is already a list of software we need to run the website, we can ask > Apache > >>> infra team to prepare the VM for us, as that may also take some time. > >>> > >>> On Wed, Apr 24, 2019 at 11:57 PM Robert Metzger > >>> wrote: > >>> > >>>> Hey all, > >>>> > >>>> quick update on this project: The frontend and backend code have been > >>>> put together into this repository: > >>>> https://github.com/sorahn/flink-ecosystem > >>>> We also just agreed on an API specification, and will now work on > >>>> finishing the backend. > >>>> > >>>> It will probably take a few more weeks for this to finish, but we are > >>>> making progress :) > >>>> > >>>> Best, > >>>> Robert > >>>> > >>>> > >>>> On Mon, Apr 15, 2019 at 11:18 AM Robert Metzger > >>>> wrote: > >>>> > >>>>> Hey Daryl, > >>>>> > >>>>> thanks a lot for posting a link to this first prototype on the > mailing > >>>>> list! I really like it! > >>>>> > >>>>> Becket: Our plan forward is that Congxian is implementing the backend > >>>>> for the website. He has already started with the work, but needs at > least > >>>>> one more week. > >>>>> > >>>>> > >>>>> [Re-sending this email because the first one was blocked on dev@f.a.o > ] > >>>>> > >>>>> > >>>>> On Mon, Apr 15, 2019 at 7:59 AM Becket Qin > >>>>> wrote: > >>>>> > >>>>>> Hi Daryl, > >>>>>> > >>>>>> Thanks a lot for the update. The site looks awesome! This is a great > >>>>>> progress. I really like the conciseness of GUI. > >>>&g
Re: [ANNOUNCE] Rong Rong becomes a Flink committer
Congrats, Rong! On Fri, Jul 12, 2019 at 1:13 AM Xingcan Cui wrote: > Congrats Rong! > > Best, > Xingcan > > On Jul 11, 2019, at 1:08 PM, Shuyi Chen wrote: > > Congratulations, Rong! > > On Thu, Jul 11, 2019 at 8:26 AM Yu Li wrote: > >> Congratulations Rong! >> >> Best Regards, >> Yu >> >> >> On Thu, 11 Jul 2019 at 22:54, zhijiang >> wrote: >> >>> Congratulations Rong! >>> >>> Best, >>> Zhijiang >>> >>> -- >>> From:Kurt Young >>> Send Time:2019年7月11日(星期四) 22:54 >>> To:Kostas Kloudas >>> Cc:Jark Wu ; Fabian Hueske ; dev < >>> d...@flink.apache.org>; user >>> Subject:Re: [ANNOUNCE] Rong Rong becomes a Flink committer >>> >>> Congratulations Rong! >>> >>> Best, >>> Kurt >>> >>> >>> On Thu, Jul 11, 2019 at 10:53 PM Kostas Kloudas >>> wrote: >>> Congratulations Rong! >>> >>> On Thu, Jul 11, 2019 at 4:40 PM Jark Wu wrote: >>> Congratulations Rong Rong! >>> Welcome on board! >>> >>> On Thu, 11 Jul 2019 at 22:25, Fabian Hueske wrote: >>> Hi everyone, >>> >>> I'm very happy to announce that Rong Rong accepted the offer of the >>> Flink PMC to become a committer of the Flink project. >>> >>> Rong has been contributing to Flink for many years, mainly working on >>> SQL and Yarn security features. He's also frequently helping out on the >>> user@f.a.o mailing lists. >>> >>> Congratulations Rong! >>> >>> Best, Fabian >>> (on behalf of the Flink PMC) >>> >>> >>> >
Re: [DISCUSS] Deprecate previous Python APIs
+1 on deprecating the old Python API in 1.9 release. Thanks, Jiangjie (Becket) Qin On Wed, Jun 12, 2019 at 9:07 AM Dian Fu wrote: > +1 for this proposal. > > Regards, > Dian > > 在 2019年6月12日，上午8:24，jincheng sun 写道： > > big +1 for the proposal. > > We will soon complete all the Python API functional development of the 1.9 > release, the development of UDFs will be carried out. After the support of > UDFs is completed, it will be very natural to support Datastream API. > > If all of us agree with this proposal, I believe that for the 1.10 > release, it is possible to complete support both UDFs and DataStream API. > And we will do our best to make the 1.10 release that contains the Python > DataStream API support. > > So, great thanks to @Stephan for this proposal! > > Best, > Jincheng > > Zili Chen 于2019年6月11日周二 下午10:56写道： > >> +1 >> >> Best, >> tison. >> >> >> zhijiang 于2019年6月11日周二 下午10:52写道： >> >>> It is reasonable as stephan explained. +1 from my side! >>> >>> -- >>> From:Jeff Zhang >>> Send Time:2019年6月11日(星期二) 22:11 >>> To:Stephan Ewen >>> Cc:user ; dev >>> Subject:Re: [DISCUSS] Deprecate previous Python APIs >>> >>> +1 >>> >>> Stephan Ewen 于2019年6月11日周二 下午9:30写道： >>> >>> > Hi all! >>> > >>> > I would suggest to deprecating the existing python APIs for DataSet and >>> > DataStream API with the 1.9 release. >>> > >>> > Background is that there is a new Python API under development. >>> > The new Python API is initially against the Table API. Flink 1.9 will >>> >>> > support Table API programs without UDFs, 1.10 is planned to support UDFs. >>> > Future versions would support also the DataStream API. >>> > >>> > In the long term, Flink should have one Python API for DataStream and >>> >>> > Table APIs. We should not maintain multiple different implementations and >>> > confuse users that way. >>> >>> > Given that the existing Python APIs are a bit limited and not under active >>> > development, I would suggest to deprecate them in favor of the new API. >>> > >>> > Best, >>> > Stephan >>> > >>> > >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >>> >>> >>> >
Re: [SURVEY] Usage of flink-ml and [DISCUSS] Delete flink-ml
+1 for removal. Personally I'd prefer marking it as deprecated and remove the module in the next release, just to follow the established procedure. And +1 on removing the `flink-libraries/flink-ml-uber` as well. Thanks, Jiangjie (Becket) Qin On Mon, May 27, 2019 at 5:07 PM jincheng sun wrote: > +1 for remove it! > > And we also plan to delete the `flink-libraries/flink-ml-uber`, right? > > Best, > Jincheng > > Rong Rong 于2019年5月24日周五 上午1:18写道： > >> +1 for the deletion. >> >> Also I think it also might be a good idea to update the roadmap for the >> plan of removal/development since we've reached the consensus on FLIP-39. >> >> Thanks, >> Rong >> >> >> On Wed, May 22, 2019 at 8:26 AM Shaoxuan Wang >> wrote: >> >>> Hi Chesnay, >>> Yes, you are right. There is not any active commit planned for the >>> legacy Flink-ml package. It does not matter delete it now or later. I will >>> open a PR and remove it. >>> >>> Shaoxuan >>> >>> On Wed, May 22, 2019 at 7:05 PM Chesnay Schepler >>> wrote: >>> >>>> I believe we can remove it regardless since users could just use the >>>> 1.8 >>>> version against future releases. >>>> >>>> Generally speaking, any library/connector that is no longer actively >>>> developed can be removed from the project as existing users can always >>>> rely on previous versions, which should continue to work by virtue of >>>> working against @Stable APIs. >>>> >>>> On 22/05/2019 12:08, Shaoxuan Wang wrote: >>>> > Hi Flink community, >>>> > >>>> > We plan to delete/deprecate the legacy flink-libraries/flink-ml >>>> package in >>>> > Flink1.9, and replace it with the new flink-ml interface proposed in >>>> FLIP39 >>>> > (FLINK-12470). >>>> > Before we remove this package, I want to reach out to you and ask if >>>> there >>>> > is any active project still uses this package. Please respond to this >>>> > thread and outline how you use flink-libraries/flink-ml. >>>> > Depending on the replies of activity and adoption >>>> > of flink-libraries/flink-ml, we will decide to either delete this >>>> package >>>> > in Flink1.9 or deprecate it for now & remove it in the next release >>>> after >>>> > 1.9. >>>> > >>>> > Thanks for your attention and help! >>>> > >>>> > Regards, >>>> > Shaoxuan >>>> > >>>> >>>>
Re: [Discuss] Add JobListener (hook) in flink job lifecycle
Thanks for the proposal, Jeff. Adding a listener to allow users handle events during the job lifecycle makes a lot of sense to me. Here are my two cents. * How do user specify the listener? * It is not quite clear to me whether we consider ClusterClient as a public interface? From what I understand ClusterClient is not a public interface right now. In contrast, ExecutionEnvironment is the de facto interface for administrative work. After job submission, it is essentially bound to a job as an administrative handle. Given this current state, personally I feel acceptable to have the listener registered to the ExecutionEnvironment. * Where should the listener run? * If the listener runs on the client side, the client have to be always connected to the Flink cluster. This does not quite work if the Job is a streaming job. Should we provide the option to run the listener in JobMaster as well? * What should be reported to the Listener? * Besides the proposed APIs, does it make sense to also report events such as failover? * What can the listeners do on notifications? * If the listeners are expected to do anything on the job, should some helper class to manipulate the jobs be passed to the listener method? Otherwise users may not be able to easily take action. Thanks, Jiangjie (Becket) Qin On Wed, Apr 24, 2019 at 2:43 PM Jeff Zhang wrote: > Hi Till, > > IMHO, allow adding hooks involves 2 steps. > 1. Provide hook interface, and call these hook in flink (ClusterClient) at > the right place. This should be done by framework (flink) > 2. Implement new hook implementation and add/register them into > framework(flink) > > What I am doing is step 1 which should be done by flink, step 2 is done by > users. But IIUC, your suggestion of using custom ClusterClient seems mixing > these 2 steps together. Say I'd like to add new hooks, I have to implement > a new custom ClusterClient, add new hooks and call them in the custom > ClusterClient at the right place. > This doesn't make sense to me. For a user who want to add hooks, he is not > supposed to understand the mechanism of ClusterClient, and should not touch > ClusterClient. What do you think ? > > > > > Till Rohrmann 于2019年4月23日周二 下午4:24写道： > >> I think we should not expose the ClusterClient configuration via the >> ExecutionEnvironment (env.getClusterClient().addJobListener) because this >> is effectively the same as exposing the JobListener interface directly on >> the ExecutionEnvironment. Instead I think it could be possible to provide a >> ClusterClient factory which is picked up from the Configuration or some >> other mechanism for example. That way it would not need to be exposed via >> the ExecutionEnvironment at all. >> >> Cheers, >> Till >> >> On Fri, Apr 19, 2019 at 11:12 AM Jeff Zhang wrote: >> >>> >>> The ExecutionEnvironment is usually used by the user who writes the >>> code and this person (I assume) would not be really interested in these >>> callbacks. >>> >>> Usually ExecutionEnvironment is used by the user who write the code, but >>> it doesn't needs to be created and configured by this person. e.g. in >>> Zeppelin notebook, ExecutionEnvironment is created by Zeppelin, user just >>> use ExecutionEnvironment to write flink program. You are right that the >>> end user would not be interested in these callback, but the third party >>> library that integrate with zeppelin would be interested in these callbacks. >>> >>> >>> In your case, it could be sufficient to offer some hooks for the >>> ClusterClient or being able to provide a custom ClusterClient. >>> >>> Actually in my initial PR (https://github.com/apache/flink/pull/8190), >>> I do pass JobListener to ClusterClient and invoke it there. >>> But IMHO, ClusterClient is not supposed be a public api for users. >>> Instead JobClient is the public api that user should use to control job. So >>> adding hooks to ClusterClient directly and provide a custom ClusterClient >>> doesn't make sense to me. IIUC, you are suggesting the following approach >>> env.getClusterClient().addJobListener(jobListener) >>> but I don't see its benefit compared to this. >>> env.addJobListener(jobListener) >>> >>> Overall, I think adding hooks is orthogonal with fine grained job >>> control. And I agree that we should refactor the flink client component, >>> but I don't think it would affect the JobListener interface. What do you >>> think ? >>> >>> >>> >>> >>> Till Rohrmann 于2019年4月18日周四 下午8:57写道： >>> >&
Re: [DISCUSS] Create a Flink ecosystem website
Thanks for the update, Robert. Looking forward to the website. If there is already a list of software we need to run the website, we can ask Apache infra team to prepare the VM for us, as that may also take some time. On Wed, Apr 24, 2019 at 11:57 PM Robert Metzger wrote: > Hey all, > > quick update on this project: The frontend and backend code have been put > together into this repository: https://github.com/sorahn/flink-ecosystem > We also just agreed on an API specification, and will now work on > finishing the backend. > > It will probably take a few more weeks for this to finish, but we are > making progress :) > > Best, > Robert > > > On Mon, Apr 15, 2019 at 11:18 AM Robert Metzger > wrote: > >> Hey Daryl, >> >> thanks a lot for posting a link to this first prototype on the mailing >> list! I really like it! >> >> Becket: Our plan forward is that Congxian is implementing the backend for >> the website. He has already started with the work, but needs at least one >> more week. >> >> >> [Re-sending this email because the first one was blocked on dev@f.a.o] >> >> >> On Mon, Apr 15, 2019 at 7:59 AM Becket Qin wrote: >> >>> Hi Daryl, >>> >>> Thanks a lot for the update. The site looks awesome! This is a great >>> progress. I really like the conciseness of GUI. >>> >>> One minor suggestion is that for the same library, there might be >>> multiple versions compatible with different Flink versions. It would be >>> good to show that somewhere in the project page as it seems important to >>> the users. >>> >>> BTW, will you share the plan to move forward? Would additional hands >>> help? >>> >>> Thanks, >>> >>> Jiangjie (Becket) Qin >>> >>> On Sat, Apr 13, 2019 at 7:10 PM Daryl Roberts >>> wrote: >>> >>>> > Shall we add a guide page to show people how to publish their >>>> projects to the website? The exact rules can be discussed and drafted in a >>>> separate email thread IMO >>>> >>>> This is a good idea. (Both the guise, and separate thread), I think >>>> once there is an actual packed in place we’ll be in a lot better position >>>> to discuss this. >>>> >>>> > The "Log in with Github" link doesn't seem to work yet. Will it only >>>> allow login for admins and publishers, or for everyone? >>>> >>>> Correct, all the oauth stuff requires a real server. We are currently >>>> just faking everything. >>>> >>>> I will add a mock-login page (username/password that just accepts >>>> anything and displays whatever username you type in) so we can see the >>>> add-comment field and add-packages page once they exist. >>>> >>>> >>>>>>>>>>>> >>>>>>>>>>>>
Re: [DISCUSS] Create a Flink ecosystem website
Thanks for the update Robert! Looking forward to the prototype! On Thu, Mar 21, 2019 at 10:07 PM Robert Metzger wrote: > Quick summary of our call: > Daryl will soon start with a front end, build against a very simple > mock-backend. > Congxian will start implementing the Spring-based backend early April. > > As soon as the first prototype of the UI is ready, we'll share it here for > feedback. > > On Thu, Mar 21, 2019 at 10:08 AM Robert Metzger > wrote: > > > Okay, great. > > > > Congxian Qiu, Daryl and I have a kick-off call later today at 2pm CET, > 9pm > > China time about the design of the ecosystem page (see: > > https://github.com/rmetzger/flink-community-tools/issues/4) > > Please let me know if others want to join as well, I can add them to the > > invite. > > > > On Wed, Mar 20, 2019 at 4:10 AM Becket Qin wrote: > > > >> I agree. We can start with english-only and see how it goes. The > comments > >> and descriptions can always be multi-lingual but that is up to the > package > >> owners. > >> > >> On Tue, Mar 19, 2019 at 6:07 PM Robert Metzger > >> wrote: > >> > >>> Thanks. > >>> > >>> Do we actually want this page to be multi-language? > >>> > >>> I propose to make the website english-only, but maybe consider allowing > >>> comments in different languages. > >>> If we would make it multi-language, then we might have problems with > >>> people submitting packages in non-english languages. > >>> > >>> > >>> > >>> On Tue, Mar 19, 2019 at 2:42 AM Becket Qin > wrote: > >>> > >>>> Done. The writeup looks great! > >>>> > >>>> On Mon, Mar 18, 2019 at 9:09 PM Robert Metzger > >>>> wrote: > >>>> > >>>>> Nice, really good news on the INFRA front! > >>>>> I think the hardware specs sound reasonable. And a periodic backup of > >>>>> the website's database to Infra's backup solution sounds reasonable > too. > >>>>> > >>>>> Can you accept and review my proposal for the website? > >>>>> > >>>>> > >>>>> On Sat, Mar 16, 2019 at 3:47 PM Becket Qin > >>>>> wrote: > >>>>> > >>>>>> > > >>>>>> > I have a very capable and motivated frontend developer who would > be > >>>>>> > willing to implement what I've mocked in my proposal. > >>>>>> > >>>>>> > >>>>>> That is awesome! > >>>>>> > >>>>>> I created a Jira ticket[1] to Apache Infra and got the reply. It > >>>>>> looks that > >>>>>> Apache infra team could provide a decent VM. The last piece is how > to > >>>>>> ensure the data is persisted so we won't lose the project info / > user > >>>>>> feedbacks when the VM is down. If Apache infra does not provide a > >>>>>> persistent storage for DB backup, we can always ask for multiple VMs > >>>>>> and do > >>>>>> the fault tolerance by ourselves. It seems we can almost say the > >>>>>> hardware > >>>>>> side is also ready. > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Jiangjie (Becket) Qin > >>>>>> > >>>>>> [1] https://issues.apache.org/jira/browse/INFRA-18010 > >>>>>> > >>>>>> On Fri, Mar 15, 2019 at 5:39 PM Robert Metzger > > >>>>>> wrote: > >>>>>> > >>>>>> > Thank you for reaching out to Infra and the ember client. > >>>>>> > When I first saw the Ember repository, I thought it is the whole > >>>>>> thing > >>>>>> > (frontend and backend), but while testing it, I realized it is > >>>>>> "only" the > >>>>>> > frontend. I'm not sure if it makes sense to adjust the Ember > >>>>>> observer > >>>>>> > client, or just write a simple UI from scratch. > >>>>>> > I have a very capable and motivated frontend developer who would > be > >>>>>> > willing to implement wh
Re: [DISCUSS] Create a Flink ecosystem website
I agree. We can start with english-only and see how it goes. The comments and descriptions can always be multi-lingual but that is up to the package owners. On Tue, Mar 19, 2019 at 6:07 PM Robert Metzger wrote: > Thanks. > > Do we actually want this page to be multi-language? > > I propose to make the website english-only, but maybe consider allowing > comments in different languages. > If we would make it multi-language, then we might have problems with > people submitting packages in non-english languages. > > > > On Tue, Mar 19, 2019 at 2:42 AM Becket Qin wrote: > >> Done. The writeup looks great! >> >> On Mon, Mar 18, 2019 at 9:09 PM Robert Metzger >> wrote: >> >>> Nice, really good news on the INFRA front! >>> I think the hardware specs sound reasonable. And a periodic backup of >>> the website's database to Infra's backup solution sounds reasonable too. >>> >>> Can you accept and review my proposal for the website? >>> >>> >>> On Sat, Mar 16, 2019 at 3:47 PM Becket Qin wrote: >>> >>>> > >>>> > I have a very capable and motivated frontend developer who would be >>>> > willing to implement what I've mocked in my proposal. >>>> >>>> >>>> That is awesome! >>>> >>>> I created a Jira ticket[1] to Apache Infra and got the reply. It looks >>>> that >>>> Apache infra team could provide a decent VM. The last piece is how to >>>> ensure the data is persisted so we won't lose the project info / user >>>> feedbacks when the VM is down. If Apache infra does not provide a >>>> persistent storage for DB backup, we can always ask for multiple VMs >>>> and do >>>> the fault tolerance by ourselves. It seems we can almost say the >>>> hardware >>>> side is also ready. >>>> >>>> Thanks, >>>> >>>> Jiangjie (Becket) Qin >>>> >>>> [1] https://issues.apache.org/jira/browse/INFRA-18010 >>>> >>>> On Fri, Mar 15, 2019 at 5:39 PM Robert Metzger >>>> wrote: >>>> >>>> > Thank you for reaching out to Infra and the ember client. >>>> > When I first saw the Ember repository, I thought it is the whole thing >>>> > (frontend and backend), but while testing it, I realized it is "only" >>>> the >>>> > frontend. I'm not sure if it makes sense to adjust the Ember observer >>>> > client, or just write a simple UI from scratch. >>>> > I have a very capable and motivated frontend developer who would be >>>> > willing to implement what I've mocked in my proposal. >>>> > In addition, I found somebody (Congxian Qiu) who seems to be eager to >>>> help >>>> > with this project for the backend: >>>> > https://github.com/rmetzger/flink-community-tools/issues/4 >>>> > >>>> > For Infra: I made the same experience when asking for more GitHub >>>> > permissions for "flinkbot": They didn't respond on their mailing >>>> list, only >>>> > on Jira. >>>> > >>>> > >>>> > >>>> > On Thu, Mar 14, 2019 at 2:45 PM Becket Qin >>>> wrote: >>>> > >>>> >> Thanks for writing up the specifications. >>>> >> >>>> >> Regarding the website source code, Austin found a website[1] whose >>>> >> frontend code[2] is available publicly. It lacks some support (e.g >>>> login), >>>> >> but it is still a good starting point. One thing is that I did not >>>> find a >>>> >> License statement for that source code. I'll reach out to the author >>>> to see >>>> >> if they have any concern over our usage. >>>> >> >>>> >> Apache Infra has not replied to my email regarding some details >>>> about the >>>> >> VM. I'll open an infra Jira ticket tomorrow if there is still no >>>> response. >>>> >> >>>> >> Thanks, >>>> >> >>>> >> Jiangjie (Becket) Qin >>>> >> >>>> >> [1] https://emberobserver.com/ >>>> >> [2] https://github.com/emberobserver/client >>>> >> >>>> >> >>>> >> >>>> >> On Thu, Mar
Re: [DISCUSS] Create a Flink ecosystem website
Done. The writeup looks great! On Mon, Mar 18, 2019 at 9:09 PM Robert Metzger wrote: > Nice, really good news on the INFRA front! > I think the hardware specs sound reasonable. And a periodic backup of the > website's database to Infra's backup solution sounds reasonable too. > > Can you accept and review my proposal for the website? > > > On Sat, Mar 16, 2019 at 3:47 PM Becket Qin wrote: > >> > >> > I have a very capable and motivated frontend developer who would be >> > willing to implement what I've mocked in my proposal. >> >> >> That is awesome! >> >> I created a Jira ticket[1] to Apache Infra and got the reply. It looks >> that >> Apache infra team could provide a decent VM. The last piece is how to >> ensure the data is persisted so we won't lose the project info / user >> feedbacks when the VM is down. If Apache infra does not provide a >> persistent storage for DB backup, we can always ask for multiple VMs and >> do >> the fault tolerance by ourselves. It seems we can almost say the hardware >> side is also ready. >> >> Thanks, >> >> Jiangjie (Becket) Qin >> >> [1] https://issues.apache.org/jira/browse/INFRA-18010 >> >> On Fri, Mar 15, 2019 at 5:39 PM Robert Metzger >> wrote: >> >> > Thank you for reaching out to Infra and the ember client. >> > When I first saw the Ember repository, I thought it is the whole thing >> > (frontend and backend), but while testing it, I realized it is "only" >> the >> > frontend. I'm not sure if it makes sense to adjust the Ember observer >> > client, or just write a simple UI from scratch. >> > I have a very capable and motivated frontend developer who would be >> > willing to implement what I've mocked in my proposal. >> > In addition, I found somebody (Congxian Qiu) who seems to be eager to >> help >> > with this project for the backend: >> > https://github.com/rmetzger/flink-community-tools/issues/4 >> > >> > For Infra: I made the same experience when asking for more GitHub >> > permissions for "flinkbot": They didn't respond on their mailing list, >> only >> > on Jira. >> > >> > >> > >> > On Thu, Mar 14, 2019 at 2:45 PM Becket Qin >> wrote: >> > >> >> Thanks for writing up the specifications. >> >> >> >> Regarding the website source code, Austin found a website[1] whose >> >> frontend code[2] is available publicly. It lacks some support (e.g >> login), >> >> but it is still a good starting point. One thing is that I did not >> find a >> >> License statement for that source code. I'll reach out to the author >> to see >> >> if they have any concern over our usage. >> >> >> >> Apache Infra has not replied to my email regarding some details about >> the >> >> VM. I'll open an infra Jira ticket tomorrow if there is still no >> response. >> >> >> >> Thanks, >> >> >> >> Jiangjie (Becket) Qin >> >> >> >> [1] https://emberobserver.com/ >> >> [2] https://github.com/emberobserver/client >> >> >> >> >> >> >> >> On Thu, Mar 14, 2019 at 1:35 AM Robert Metzger >> >> wrote: >> >> >> >>> @Bowen: I agree. Confluent Hub looks nicer, but it is on their company >> >>> website. I guess the likelihood that they give out code from their >> company >> >>> website is fairly low. >> >>> @Nils: Beam's page is similar to our Ecosystem page, which we'll >> >>> reactivate as part of this PR: >> >>> https://github.com/apache/flink-web/pull/187 >> >>> >> >>> Spark-packages.org did not respond to my request. >> >>> I will propose a short specification in Becket's initial document. >> >>> >> >>> >> >>> On Mon, Mar 11, 2019 at 11:38 AM Niels Basjes >> wrote: >> >>> >> >>>> Hi, >> >>>> >> >>>> The Beam project has something in this area that is simply a page >> >>>> within their documentation website: >> >>>> https://beam.apache.org/documentation/sdks/java-thirdparty/ >> >>>> >> >>>> Niels Basjes >> >>>> >> >>>> On Fri, Mar 8, 2019 at 11:39 PM Bowen Li >>
Re: [DISCUSS] Create a Flink ecosystem website
> > I have a very capable and motivated frontend developer who would be > willing to implement what I've mocked in my proposal. That is awesome! I created a Jira ticket[1] to Apache Infra and got the reply. It looks that Apache infra team could provide a decent VM. The last piece is how to ensure the data is persisted so we won't lose the project info / user feedbacks when the VM is down. If Apache infra does not provide a persistent storage for DB backup, we can always ask for multiple VMs and do the fault tolerance by ourselves. It seems we can almost say the hardware side is also ready. Thanks, Jiangjie (Becket) Qin [1] https://issues.apache.org/jira/browse/INFRA-18010 On Fri, Mar 15, 2019 at 5:39 PM Robert Metzger wrote: > Thank you for reaching out to Infra and the ember client. > When I first saw the Ember repository, I thought it is the whole thing > (frontend and backend), but while testing it, I realized it is "only" the > frontend. I'm not sure if it makes sense to adjust the Ember observer > client, or just write a simple UI from scratch. > I have a very capable and motivated frontend developer who would be > willing to implement what I've mocked in my proposal. > In addition, I found somebody (Congxian Qiu) who seems to be eager to help > with this project for the backend: > https://github.com/rmetzger/flink-community-tools/issues/4 > > For Infra: I made the same experience when asking for more GitHub > permissions for "flinkbot": They didn't respond on their mailing list, only > on Jira. > > > > On Thu, Mar 14, 2019 at 2:45 PM Becket Qin wrote: > >> Thanks for writing up the specifications. >> >> Regarding the website source code, Austin found a website[1] whose >> frontend code[2] is available publicly. It lacks some support (e.g login), >> but it is still a good starting point. One thing is that I did not find a >> License statement for that source code. I'll reach out to the author to see >> if they have any concern over our usage. >> >> Apache Infra has not replied to my email regarding some details about the >> VM. I'll open an infra Jira ticket tomorrow if there is still no response. >> >> Thanks, >> >> Jiangjie (Becket) Qin >> >> [1] https://emberobserver.com/ >> [2] https://github.com/emberobserver/client >> >> >> >> On Thu, Mar 14, 2019 at 1:35 AM Robert Metzger >> wrote: >> >>> @Bowen: I agree. Confluent Hub looks nicer, but it is on their company >>> website. I guess the likelihood that they give out code from their company >>> website is fairly low. >>> @Nils: Beam's page is similar to our Ecosystem page, which we'll >>> reactivate as part of this PR: >>> https://github.com/apache/flink-web/pull/187 >>> >>> Spark-packages.org did not respond to my request. >>> I will propose a short specification in Becket's initial document. >>> >>> >>> On Mon, Mar 11, 2019 at 11:38 AM Niels Basjes wrote: >>> >>>> Hi, >>>> >>>> The Beam project has something in this area that is simply a page >>>> within their documentation website: >>>> https://beam.apache.org/documentation/sdks/java-thirdparty/ >>>> >>>> Niels Basjes >>>> >>>> On Fri, Mar 8, 2019 at 11:39 PM Bowen Li wrote: >>>> > >>>> > Confluent hub for Kafka is another good example of this kind. I >>>> personally like it over the spark site. May worth checking it out with >>>> Kafka folks >>>> > >>>> > On Thu, Mar 7, 2019 at 6:06 AM Becket Qin >>>> wrote: >>>> >> >>>> >> Absolutely! Thanks for the pointer. I'll submit a PR to update the >>>> >> ecosystem page and the navigation. >>>> >> >>>> >> Thanks, >>>> >> >>>> >> Jiangjie (Becket) Qin >>>> >> >>>> >> On Thu, Mar 7, 2019 at 8:47 PM Robert Metzger >>>> wrote: >>>> >> >>>> >> > Okay. I will reach out to spark-packages.org and see if they are >>>> willing >>>> >> > to share. >>>> >> > >>>> >> > Do you want to raise a PR to update the ecosystem page (maybe sync >>>> with >>>> >> > the "Software Projects" listed here: >>>> >> > https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink) >>>> and >>>> >> &
Re: [DISCUSS] Create a Flink ecosystem website
Thanks for writing up the specifications. Regarding the website source code, Austin found a website[1] whose frontend code[2] is available publicly. It lacks some support (e.g login), but it is still a good starting point. One thing is that I did not find a License statement for that source code. I'll reach out to the author to see if they have any concern over our usage. Apache Infra has not replied to my email regarding some details about the VM. I'll open an infra Jira ticket tomorrow if there is still no response. Thanks, Jiangjie (Becket) Qin [1] https://emberobserver.com/ [2] https://github.com/emberobserver/client On Thu, Mar 14, 2019 at 1:35 AM Robert Metzger wrote: > @Bowen: I agree. Confluent Hub looks nicer, but it is on their company > website. I guess the likelihood that they give out code from their company > website is fairly low. > @Nils: Beam's page is similar to our Ecosystem page, which we'll > reactivate as part of this PR: > https://github.com/apache/flink-web/pull/187 > > Spark-packages.org did not respond to my request. > I will propose a short specification in Becket's initial document. > > > On Mon, Mar 11, 2019 at 11:38 AM Niels Basjes wrote: > >> Hi, >> >> The Beam project has something in this area that is simply a page >> within their documentation website: >> https://beam.apache.org/documentation/sdks/java-thirdparty/ >> >> Niels Basjes >> >> On Fri, Mar 8, 2019 at 11:39 PM Bowen Li wrote: >> > >> > Confluent hub for Kafka is another good example of this kind. I >> personally like it over the spark site. May worth checking it out with >> Kafka folks >> > >> > On Thu, Mar 7, 2019 at 6:06 AM Becket Qin wrote: >> >> >> >> Absolutely! Thanks for the pointer. I'll submit a PR to update the >> >> ecosystem page and the navigation. >> >> >> >> Thanks, >> >> >> >> Jiangjie (Becket) Qin >> >> >> >> On Thu, Mar 7, 2019 at 8:47 PM Robert Metzger >> wrote: >> >> >> >> > Okay. I will reach out to spark-packages.org and see if they are >> willing >> >> > to share. >> >> > >> >> > Do you want to raise a PR to update the ecosystem page (maybe sync >> with >> >> > the "Software Projects" listed here: >> >> > https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink) >> and >> >> > link it in the navigation? >> >> > >> >> > Best, >> >> > Robert >> >> > >> >> > >> >> > On Thu, Mar 7, 2019 at 10:13 AM Becket Qin >> wrote: >> >> > >> >> >> Hi Robert, >> >> >> >> >> >> I think it at least worths checking if spark-packages.org owners >> are >> >> >> willing to share. Thanks for volunteering to write the requirement >> >> >> descriptions! In any case, that will be very helpful. >> >> >> >> >> >> Since a static page has almost no cost, and we will need it to >> redirect >> >> >> to the dynamic site anyways, how about we first do that while >> working on >> >> >> the dynamic website? >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Jiangjie (Becket) Qin >> >> >> >> >> >> On Thu, Mar 7, 2019 at 4:59 AM Ufuk Celebi wrote: >> >> >> >> >> >>> I like Shaoxuan's idea to keep this a static site first. We could >> then >> >> >>> iterate on this and make it a dynamic thing. Of course, if we have >> the >> >> >>> resources in the community to quickly start with a dynamic site, >> I'm >> >> >>> not apposed. >> >> >>> >> >> >>> – Ufuk >> >> >>> >> >> >>> On Wed, Mar 6, 2019 at 2:31 PM Robert Metzger > > >> >> >>> wrote: >> >> >>> > >> >> >>> > Awesome! Thanks a lot for looking into this Becket! The VMs >> hosted by >> >> >>> Infra >> >> >>> > look suitable. >> >> >>> > >> >> >>> > @Shaoxuan: There is actually already a static page. It used to be >> >> >>> linked, >> >> >>> > but has been removed from the navigation bar for some reason. >> This is
Re: [DISCUSS] Create a Flink ecosystem website
Absolutely! Thanks for the pointer. I'll submit a PR to update the ecosystem page and the navigation. Thanks, Jiangjie (Becket) Qin On Thu, Mar 7, 2019 at 8:47 PM Robert Metzger wrote: > Okay. I will reach out to spark-packages.org and see if they are willing > to share. > > Do you want to raise a PR to update the ecosystem page (maybe sync with > the "Software Projects" listed here: > https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink) and > link it in the navigation? > > Best, > Robert > > > On Thu, Mar 7, 2019 at 10:13 AM Becket Qin wrote: > >> Hi Robert, >> >> I think it at least worths checking if spark-packages.org owners are >> willing to share. Thanks for volunteering to write the requirement >> descriptions! In any case, that will be very helpful. >> >> Since a static page has almost no cost, and we will need it to redirect >> to the dynamic site anyways, how about we first do that while working on >> the dynamic website? >> >> Thanks, >> >> Jiangjie (Becket) Qin >> >> On Thu, Mar 7, 2019 at 4:59 AM Ufuk Celebi wrote: >> >>> I like Shaoxuan's idea to keep this a static site first. We could then >>> iterate on this and make it a dynamic thing. Of course, if we have the >>> resources in the community to quickly start with a dynamic site, I'm >>> not apposed. >>> >>> – Ufuk >>> >>> On Wed, Mar 6, 2019 at 2:31 PM Robert Metzger >>> wrote: >>> > >>> > Awesome! Thanks a lot for looking into this Becket! The VMs hosted by >>> Infra >>> > look suitable. >>> > >>> > @Shaoxuan: There is actually already a static page. It used to be >>> linked, >>> > but has been removed from the navigation bar for some reason. This is >>> the >>> > page: https://flink.apache.org/ecosystem.html >>> > We could update the page and add it back to the navigation bar for the >>> > coming weeks. What do you think? >>> > >>> > I would actually like to push for a dynamic page right away. >>> > >>> > I know it's kind of a bold move, but how do you feel about sending the >>> > owners of spark-packages.org a short note, if they are interested in >>> > sharing the source? We could maintain the code together in a public >>> repo. >>> > If they are not interested in sharing, or we decide not to ask in the >>> first >>> > place, I'm happy to write down a short description of the requirements, >>> > maybe some mockups. We could then see if we find somebody here in the >>> > community who's willing to implement it. >>> > Given the number of people who are eager to contribute, I believe we >>> will >>> > be able to find somebody pretty soon. >>> > >>> > >>> > On Wed, Mar 6, 2019 at 3:49 AM Becket Qin >>> wrote: >>> > >>> > > Forgot to provide the link... >>> > > >>> > > [1] https://www.apache.org/dev/services.html#blogs (Apache infra >>> services) >>> > > [2] https://www.apache.org/dev/freebsd-jails (FreeBSD Jail provided >>> by >>> > > Apache Infra) >>> > > >>> > > On Wed, Mar 6, 2019 at 10:46 AM Becket Qin >>> wrote: >>> > > >>> > >> Hi Robert, >>> > >> >>> > >> Thanks for the feedback. These are good points. We should absolutely >>> > >> shoot for a dynamic website to support more interactions in the >>> community. >>> > >> There might be a few things to solve: >>> > >> 1. The website code itself. An open source solution would be great. >>> TBH, >>> > >> I do not have much experience on building a website. It'll be great >>> if >>> > >> someone could help comment on the solution here. >>> > >> 2. The hardware to host the website. Apache Infra provides a few >>> > >> services[1] that Apache projects can leverage. I did not see >>> database >>> > >> service, but maybe we can run a simple MySQL db in FreeBSD jail[2]. >>> > >> >>> > >> @Bowen & vino, thanks for the positive feedback! >>> > >> >>> > >> @Shaoxuan Wang >>> > >> Thanks for the suggestion. That sounds reasonable to me. We >>> probably need >>> > >
Re: [DISCUSS] Create a Flink ecosystem website
Hi Robert, I think it at least worths checking if spark-packages.org owners are willing to share. Thanks for volunteering to write the requirement descriptions! In any case, that will be very helpful. Since a static page has almost no cost, and we will need it to redirect to the dynamic site anyways, how about we first do that while working on the dynamic website? Thanks, Jiangjie (Becket) Qin On Thu, Mar 7, 2019 at 4:59 AM Ufuk Celebi wrote: > I like Shaoxuan's idea to keep this a static site first. We could then > iterate on this and make it a dynamic thing. Of course, if we have the > resources in the community to quickly start with a dynamic site, I'm > not apposed. > > – Ufuk > > On Wed, Mar 6, 2019 at 2:31 PM Robert Metzger wrote: > > > > Awesome! Thanks a lot for looking into this Becket! The VMs hosted by > Infra > > look suitable. > > > > @Shaoxuan: There is actually already a static page. It used to be linked, > > but has been removed from the navigation bar for some reason. This is the > > page: https://flink.apache.org/ecosystem.html > > We could update the page and add it back to the navigation bar for the > > coming weeks. What do you think? > > > > I would actually like to push for a dynamic page right away. > > > > I know it's kind of a bold move, but how do you feel about sending the > > owners of spark-packages.org a short note, if they are interested in > > sharing the source? We could maintain the code together in a public repo. > > If they are not interested in sharing, or we decide not to ask in the > first > > place, I'm happy to write down a short description of the requirements, > > maybe some mockups. We could then see if we find somebody here in the > > community who's willing to implement it. > > Given the number of people who are eager to contribute, I believe we will > > be able to find somebody pretty soon. > > > > > > On Wed, Mar 6, 2019 at 3:49 AM Becket Qin wrote: > > > > > Forgot to provide the link... > > > > > > [1] https://www.apache.org/dev/services.html#blogs (Apache infra > services) > > > [2] https://www.apache.org/dev/freebsd-jails (FreeBSD Jail provided by > > > Apache Infra) > > > > > > On Wed, Mar 6, 2019 at 10:46 AM Becket Qin > wrote: > > > > > >> Hi Robert, > > >> > > >> Thanks for the feedback. These are good points. We should absolutely > > >> shoot for a dynamic website to support more interactions in the > community. > > >> There might be a few things to solve: > > >> 1. The website code itself. An open source solution would be great. > TBH, > > >> I do not have much experience on building a website. It'll be great if > > >> someone could help comment on the solution here. > > >> 2. The hardware to host the website. Apache Infra provides a few > > >> services[1] that Apache projects can leverage. I did not see database > > >> service, but maybe we can run a simple MySQL db in FreeBSD jail[2]. > > >> > > >> @Bowen & vino, thanks for the positive feedback! > > >> > > >> @Shaoxuan Wang > > >> Thanks for the suggestion. That sounds reasonable to me. We probably > need > > >> a page in the Flink official site anyways, even just provide links it > to > > >> the ecosystem website. So listing the connectors in that static page > seems > > >> something we could start with while we are working on the dynamic > pages. > > >> > > >> Thanks, > > >> > > >> Jiangjie (Becket) Qin > > >> > > >> On Wed, Mar 6, 2019 at 10:40 AM Shaoxuan Wang > > >> wrote: > > >> > > >>> Hi Becket and Robert, > > >>> > > >>> I like this idea! Let us roll this out with Flink connectors at the > > >>> first beginning. We can start with a static page, and upgrade it > when we > > >>> find a better solution for dynamic one with rich functions. > > >>> > > >>> Regards, > > >>> Shaoxuan > > >>> > > >>> > > >>> On Wed, Mar 6, 2019 at 1:36 AM Robert Metzger > > >>> wrote: > > >>> > > >>>> Hey Becket, > > >>>> > > >>>> This is a great idea! > > >>>> For this to be successful, we need to make sure the page is placed > > >>>> prominently so that the people submitting something w
Re: [DISCUSS] Create a Flink ecosystem website
Forgot to provide the link... [1] https://www.apache.org/dev/services.html#blogs (Apache infra services) [2] https://www.apache.org/dev/freebsd-jails (FreeBSD Jail provided by Apache Infra) On Wed, Mar 6, 2019 at 10:46 AM Becket Qin wrote: > Hi Robert, > > Thanks for the feedback. These are good points. We should absolutely shoot > for a dynamic website to support more interactions in the community. There > might be a few things to solve: > 1. The website code itself. An open source solution would be great. TBH, I > do not have much experience on building a website. It'll be great if > someone could help comment on the solution here. > 2. The hardware to host the website. Apache Infra provides a few > services[1] that Apache projects can leverage. I did not see database > service, but maybe we can run a simple MySQL db in FreeBSD jail[2]. > > @Bowen & vino, thanks for the positive feedback! > > @Shaoxuan Wang > Thanks for the suggestion. That sounds reasonable to me. We probably need > a page in the Flink official site anyways, even just provide links it to > the ecosystem website. So listing the connectors in that static page seems > something we could start with while we are working on the dynamic pages. > > Thanks, > > Jiangjie (Becket) Qin > > On Wed, Mar 6, 2019 at 10:40 AM Shaoxuan Wang wrote: > >> Hi Becket and Robert, >> >> I like this idea! Let us roll this out with Flink connectors at the >> first beginning. We can start with a static page, and upgrade it when we >> find a better solution for dynamic one with rich functions. >> >> Regards, >> Shaoxuan >> >> >> On Wed, Mar 6, 2019 at 1:36 AM Robert Metzger >> wrote: >> >>> Hey Becket, >>> >>> This is a great idea! >>> For this to be successful, we need to make sure the page is placed >>> prominently so that the people submitting something will get attention for >>> their contributions. >>> I think a dynamic site would probably be better, if we want features >>> such as up and downvoting or comments. >>> I would also like this to be hosted on Apache infra, and endorsed by the >>> community. >>> >>> Does anybody here know any existing software that we could use? >>> The only think I was able to find is AUR: https://aur.archlinux.org/ >>> (which is a community packages site for Arch Linux. The source code of this >>> portal is open source, but the layout and structure is not an ideal fit for >>> our requirements) >>> >>> Best, >>> Robert >>> >>> >>> >>> On Tue, Mar 5, 2019 at 12:03 PM Becket Qin wrote: >>> >>>> Hi folks, >>>> >>>> I would like to start a discussion thread about creating a Flink >>>> ecosystem website. The website aims to help contributors who have developed >>>> projects around Flink share their work with the community. >>>> >>>> Please see the following doc for more details. >>>> >>>> https://docs.google.com/document/d/12oCItoLbKrLGuwEUFcCfigezIR2hW3925j1hh3kGp4A/edit# >>>> >>>> Thanks, >>>> >>>> Jiangjie (Becket) Qin >>>> >>>
Re: [DISCUSS] Create a Flink ecosystem website
Hi Robert, Thanks for the feedback. These are good points. We should absolutely shoot for a dynamic website to support more interactions in the community. There might be a few things to solve: 1. The website code itself. An open source solution would be great. TBH, I do not have much experience on building a website. It'll be great if someone could help comment on the solution here. 2. The hardware to host the website. Apache Infra provides a few services[1] that Apache projects can leverage. I did not see database service, but maybe we can run a simple MySQL db in FreeBSD jail[2]. @Bowen & vino, thanks for the positive feedback! @Shaoxuan Wang Thanks for the suggestion. That sounds reasonable to me. We probably need a page in the Flink official site anyways, even just provide links it to the ecosystem website. So listing the connectors in that static page seems something we could start with while we are working on the dynamic pages. Thanks, Jiangjie (Becket) Qin On Wed, Mar 6, 2019 at 10:40 AM Shaoxuan Wang wrote: > Hi Becket and Robert, > > I like this idea! Let us roll this out with Flink connectors at the first > beginning. We can start with a static page, and upgrade it when we find a > better solution for dynamic one with rich functions. > > Regards, > Shaoxuan > > > On Wed, Mar 6, 2019 at 1:36 AM Robert Metzger wrote: > >> Hey Becket, >> >> This is a great idea! >> For this to be successful, we need to make sure the page is placed >> prominently so that the people submitting something will get attention for >> their contributions. >> I think a dynamic site would probably be better, if we want features such >> as up and downvoting or comments. >> I would also like this to be hosted on Apache infra, and endorsed by the >> community. >> >> Does anybody here know any existing software that we could use? >> The only think I was able to find is AUR: https://aur.archlinux.org/ >> (which is a community packages site for Arch Linux. The source code of this >> portal is open source, but the layout and structure is not an ideal fit for >> our requirements) >> >> Best, >> Robert >> >> >> >> On Tue, Mar 5, 2019 at 12:03 PM Becket Qin wrote: >> >>> Hi folks, >>> >>> I would like to start a discussion thread about creating a Flink >>> ecosystem website. The website aims to help contributors who have developed >>> projects around Flink share their work with the community. >>> >>> Please see the following doc for more details. >>> >>> https://docs.google.com/document/d/12oCItoLbKrLGuwEUFcCfigezIR2hW3925j1hh3kGp4A/edit# >>> >>> Thanks, >>> >>> Jiangjie (Becket) Qin >>> >>
[DISCUSS] Create a Flink ecosystem website
Hi folks, I would like to start a discussion thread about creating a Flink ecosystem website. The website aims to help contributors who have developed projects around Flink share their work with the community. Please see the following doc for more details. https://docs.google.com/document/d/12oCItoLbKrLGuwEUFcCfigezIR2hW3925j1hh3kGp4A/edit# Thanks, Jiangjie (Becket) Qin
Re: [DISCUSS] Towards a leaner flink-dist
Hi Chesnay, Thanks for the proposal. +1 for make the distribution thinner. Meanwhile, it would be useful to have all the peripheral libraries/jars hosted somewhere so users can download them from a centralized place. We can also encourage the community to contribute their libraries, such as connectors and other pluggables, to the same place (maybe a separate category), so the community can share the commonly used libraries as well. Thanks, Jiangjie (Becket) Qin On Sat, Jan 26, 2019 at 2:49 PM Hequn Cheng wrote: > Hi Chesnay, > > Thanks a lot for the proposal! +1 for a leaner flink-dist and improve the > "Download" page. > I think a leaner flink-dist would be very helpful. If we bundle all jars > into a single one, this will easily cause class conflict problem. > > Best, > Hequn > > > On Fri, Jan 25, 2019 at 2:48 PM jincheng sun > wrote: > > > Hi Chesnay, > > > > Thank you for the proposal. And i like it very much. > > > > +1 for the leaner distribution. > > > > About improve the "Download" page, I think we can add the connectors > > download link in the "Optional components" section which @Timo Walther > > mentioned above. > > > > > > Regards, > > Jincheng > > > > Chesnay Schepler 于2019年1月18日周五 下午5:59写道： > > > >> Hello, > >> > >> the binary distribution that we release by now contains quite a lot of > >> optional components, including various filesystems, metric reporters and > >> libraries. Most users will only use a fraction of these, and as such > >> pretty much only increase the size of flink-dist. > >> > >> With Flink growing more and more in scope I don't believe it to be > >> feasible to ship everything we have with every distribution, and instead > >> suggest more of a "pick-what-you-need" model, where flink-dist is rather > >> lean and additional components are downloaded separately and added by > >> the user. > >> > >> This would primarily affect the /opt directory, but could also be > >> extended to cover flink-dist. For example, the yarn and mesos code could > >> be spliced out into separate jars that could be added to lib manually. > >> > >> Let me know what you think. > >> > >> Regards, > >> > >> Chesnay > >> > >> >
