Re: The Questions about Submitting PR

[Roc Marshal]

Till,  
Thank you for your answers.

Best regards,
Roc


| |
Roc Marshal
|
|
邮箱：flin...@126.com
|

签名由 网易邮箱大师 定制

On 05/05/2020 14:25, Till Rohrmann wrote:
Hi Roc,

afaik it can take a bit of time until the status of the PR is updated in
accordance with the AZP results.

As of a couple of days ago, we no longer use Travis for running CI. That's
why you don't see it anymore.

Cheers,
Till

On Tue, May 5, 2020 at 1:59 AM Roc Marshal  wrote:

> Hi,all.
> I have two questions about submitting pr
> When I submit PR and trigger azure test, it always shows the status of
> pending, but the details page is in the status of success. In addition, my
> PR has no information about Travis testing. What is the cause of this
> phenomenon?
> Thank you for your attention.
>
>
> Best,
> Roc Marshal
>
>
>
>
> | |
> Roc Marshal
> |
> |
> 邮箱：flin...@126.com
> |
>
> 签名由 网易邮箱大师 定制

[jira] [Created] (FLINK-17511) "RocksDB Memory Management end-to-end test" fails with "Current block cache usage 202123272 larger than expected memory limit 200000000"

[Robert Metzger (Jira)]

Robert Metzger created FLINK-17511:
--

 Summary: "RocksDB Memory Management end-to-end test" fails with 
"Current block cache usage 202123272 larger than expected memory limit 
2"
 Key: FLINK-17511
 URL: https://issues.apache.org/jira/browse/FLINK-17511
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends, Tests
Affects Versions: 1.11.0
Reporter: Robert Metzger


CI: https://travis-ci.org/github/apache/flink/jobs/683049297

{code}
Waiting for job to process up to 1 records, current progress: 9751 records 
...
Waiting for job to process up to 1 records, current progress: 9852 records 
...
Waiting for job to process up to 1 records, current progress: 9852 records 
...
Waiting for job to process up to 1 records, current progress: 9926 records 
...
Cancelling job d5a8e2c5a382b51573934c91b7b7ac9b.
Cancelled job d5a8e2c5a382b51573934c91b7b7ac9b.
[INFO] Current block cache usage for RocksDB instance in slot was 152229488
[INFO] Current block cache usage for RocksDB instance in slot was 202123272
[ERROR] Current block cache usage 202123272 larger than expected memory limit 
2
[FAIL] Test script contains errors.
Checking for errors...
No errors in log files.
Checking for exceptions...
No exceptions in log files.
Checking for non-empty .out files...
No non-empty .out files.
[FAIL] 'RocksDB Memory Management end-to-end test' failed after 8 minutes and 
20 seconds! Test exited with exit code 1
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: The Questions about Submitting PR

[Till Rohrmann]

Hi Roc,

afaik it can take a bit of time until the status of the PR is updated in
accordance with the AZP results.

As of a couple of days ago, we no longer use Travis for running CI. That's
why you don't see it anymore.

Cheers,
Till

On Tue, May 5, 2020 at 1:59 AM Roc Marshal  wrote:

> Hi,all.
> I have two questions about submitting pr
> When I submit PR and trigger azure test, it always shows the status of
> pending, but the details page is in the status of success. In addition, my
> PR has no information about Travis testing. What is the cause of this
> phenomenon?
> Thank you for your attention.
>
>
> Best,
> Roc Marshal
>
>
>
>
> | |
> Roc Marshal
> |
> |
> 邮箱：flin...@126.com
> |
>
> 签名由 网易邮箱大师 定制

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Till Rohrmann]

Thanks for all the feedback. I will open the PRs now which change the
notification settings of the above-mentioned repositories.

Cheers,
Till

On Tue, May 5, 2020 at 6:26 AM Hequn Cheng  wrote:

> +1, thanks a lot for driving this.
>
> Best, Hequn
>
> On Tue, May 5, 2020 at 11:56 AM Dian Fu  wrote:
>
> > +1
> >
> > Regards,
> > Dian
> >
> > > 在 2020年5月5日，上午9:58，Yangze Guo  写道：
> > >
> > > +1
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Tue, May 5, 2020 at 6:14 AM Thomas Weise  wrote:
> > >>
> > >> +1
> > >>
> > >>
> > >> On Mon, May 4, 2020 at 10:02 AM Marta Paes Moreira <
> ma...@ververica.com
> > >
> > >> wrote:
> > >>
> > >>> +1, this is quite annoying and distracting.
> > >>>
> > >>> Marta
> > >>>
> > >>> On Mon, May 4, 2020 at 6:27 PM Yu Li  wrote:
> > >>>
> >  +1
> > 
> >  Best Regards,
> >  Yu
> > 
> > 
> >  On Tue, 5 May 2020 at 00:21, Konstantin Knauf 
> > wrote:
> > 
> > > Yes, please.
> > >
> > > On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz <
> > >>> dwysakow...@apache.org>
> > > wrote:
> > >
> > >> +1
> > >>
> > >> Yes, please. I've also observed a lot of noise in the past days.
> > >>
> > >> Best,
> > >>
> > >> Dawid
> > >>
> > >> On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> > >>> +1
> > >>>
> > >>> All the recent new repos, flink-statefun / flink-statefun-docker
> /
> > >>> flink-training etc. are also sending notifications to issues@.
> > >>>
> > >>> Gordon
> > >>>
> > >>>
> > >>> On Mon, May 4, 2020, 11:44 PM Till Rohrmann <
> trohrm...@apache.org>
> > >> wrote:
> > >>>
> >  Hi everyone,
> > 
> >  due to some changes on the ASF side, we are now seeing issue and
> >  pull
> >  request notifications for the flink-web [1] and flink-shaded [2]
> >  repo
> > > on
> >  dev@flink.apache.org. I think this is not ideal since the dev
> ML
> > >>> is
> > >> much
> >  more noisy now.
> > 
> >  I would propose to send these notifications to
> > > iss...@flink.apache.org
> > >> as
> >  we are currently doing it for the Flink main repo [3].
> > 
> >  What do you think?
> > 
> >  [1] https://github.com/apache/flink-web
> >  [2] https://github.com/apache/flink-shaded
> >  [3] https://gitbox.apache.org/schemes.cgi?flink
> > 
> >  Cheers,
> >  Till
> > 
> > >>
> > >>
> > >
> > > --
> > >
> > > Konstantin Knauf
> > >
> > > https://twitter.com/snntrable
> > >
> > > https://github.com/knaufk
> > >
> > 
> > >>>
> >
> >
>

[GitHub] [flink-web] carp84 commented on a change in pull request #330: Add Apache Flink release 1.10.1

[GitBox]

carp84 commented on a change in pull request #330:
URL: https://github.com/apache/flink-web/pull/330#discussion_r419883635



##
File path: _config.yml
##
@@ -192,6 +192,10 @@ component_releases:
 
 release_archive:
 flink:
+  -
+version_short: "1.10"

Review comment:
   Thanks for the review. The quotes are for preventing issues like 
reported in FLINK-16663, FYI.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-17510) StreamingKafkaITCase. testKafka timeouts on downloading Kafka

[Robert Metzger (Jira)]

Robert Metzger created FLINK-17510:
--

 Summary: StreamingKafkaITCase. testKafka timeouts on downloading 
Kafka
 Key: FLINK-17510
 URL: https://issues.apache.org/jira/browse/FLINK-17510
 Project: Flink
  Issue Type: Bug
  Components: Build System / Azure Pipelines, Connectors / Kafka, Tests
Reporter: Robert Metzger


CI: 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=585&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5

{code}
2020-05-05T00:06:49.7268716Z [INFO] 
---
2020-05-05T00:06:49.7268938Z [INFO]  T E S T S
2020-05-05T00:06:49.7269282Z [INFO] 
---
2020-05-05T00:06:50.5336315Z [INFO] Running 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
2020-05-05T00:11:26.8603439Z [ERROR] Tests run: 3, Failures: 0, Errors: 2, 
Skipped: 0, Time elapsed: 276.323 s <<< FAILURE! - in 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
2020-05-05T00:11:26.8604882Z [ERROR] testKafka[1: 
kafka-version:0.11.0.2](org.apache.flink.tests.util.kafka.StreamingKafkaITCase) 
 Time elapsed: 120.024 s  <<< ERROR!
2020-05-05T00:11:26.8605942Z java.io.IOException: Process ([wget, -q, -P, 
/tmp/junit2815750531595874769/downloads/1290570732, 
https://archive.apache.org/dist/kafka/0.11.0.2/kafka_2.11-0.11.0.2.tgz]) 
exceeded timeout (12) or number of retries (3).
2020-05-05T00:11:26.8606732Zat 
org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlockingWithRetry(AutoClosableProcess.java:132)
2020-05-05T00:11:26.8607321Zat 
org.apache.flink.tests.util.cache.AbstractDownloadCache.getOrDownload(AbstractDownloadCache.java:127)
2020-05-05T00:11:26.8607826Zat 
org.apache.flink.tests.util.cache.LolCache.getOrDownload(LolCache.java:31)
2020-05-05T00:11:26.8608343Zat 
org.apache.flink.tests.util.kafka.LocalStandaloneKafkaResource.setupKafkaDist(LocalStandaloneKafkaResource.java:98)
2020-05-05T00:11:26.8608892Zat 
org.apache.flink.tests.util.kafka.LocalStandaloneKafkaResource.before(LocalStandaloneKafkaResource.java:92)
2020-05-05T00:11:26.8609602Zat 
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)
2020-05-05T00:11:26.8610026Zat 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-05-05T00:11:26.8610553Zat 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-05-05T00:11:26.8610958Zat 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-05-05T00:11:26.8611388Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-05-05T00:11:26.8612214Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-05-05T00:11:26.8612706Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-05T00:11:26.8613109Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-05T00:11:26.8613551Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-05T00:11:26.8614019Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-05T00:11:26.8614442Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-05T00:11:26.8614869Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-05-05T00:11:26.8615251Zat 
org.junit.runners.Suite.runChild(Suite.java:128)
2020-05-05T00:11:26.8615654Zat 
org.junit.runners.Suite.runChild(Suite.java:27)
2020-05-05T00:11:26.8616060Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-05T00:11:26.8616465Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-05T00:11:26.8616893Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-05T00:11:26.8617893Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-05T00:11:26.8618490Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-05T00:11:26.8619056Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-05-05T00:11:26.8619589Zat 
org.junit.runners.Suite.runChild(Suite.java:128)
2020-05-05T00:11:26.8620073Zat 
org.junit.runners.Suite.runChild(Suite.java:27)
2020-05-05T00:11:26.8620745Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-05T00:11:26.8621172Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-05T00:11:26.8621585Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-05T00:11:26.8622006Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-05T00:11:26.8622403Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-05T00:11:26.8622811Zat 
org.junit.runners.ParentRunner.run(ParentRunn

Re: [DISCUSS] flink-connector-rabbitmq api changes

[seneg...@gmail.com]

@Austin in my initial implementation you get the envelope as well. I
basically pass to the interface everything i get from the RMQ client

https://github.com/senegalo/flink/blob/e67f344884b4186126c38eaa8e112d6e5cf1152e/flink-connectors/flink-connector-rabbitmq/src/main/java/org/apache/flink/streaming/connectors/rabbitmq/RMQDeliveryParser.java#L26

Regards,
Karim Mansour

On Tue, May 5, 2020 at 12:38 AM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> Thanks Aljoscha,
>
> I'm happy to take FLINK-17204
>  for now, if you're
> able
> to assign it to me, and we'll go from there?
>
> Excited to use what you come up with Karim! It also looks like FLINK-8510
>  might also have some
> ideas on getting access to more RMQ-specific data in the source.
>
> Best,
> Austin
>
> On Mon, May 4, 2020 at 6:58 AM seneg...@gmail.com 
> wrote:
>
> > Hi,
> >
> > Okay i created a ticket:
> https://issues.apache.org/jira/browse/FLINK-17502
> >
> > i will work on the modifications "keeping the old constructor" and brush
> up
> > on the contribution guides and move from there :)
> >
> > Regards,
> > Karim Mansour
> >
> > On Mon, May 4, 2020 at 10:00 AM Aljoscha Krettek 
> > wrote:
> >
> > > Yes, that's what I was proposing!
> > >
> > > @Karim If there's not already a Jira issue, please create one. You can
> > > ping me, so that I can assign you.
> > >
> > > @Austin There's a Jira component for the RMQ source, maybe you can take
> > > a stab at some of the issues there:
> > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-17204?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Connectors%2F%20RabbitMQ%22%20AND%20statusCategory%20!%3D%20Done
> > > .
> > >
> > > Best,
> > > Aljoscha
> > >
> > > On 03.05.20 16:38, seneg...@gmail.com wrote:
> > > > Hi,
> > > >
> > > > Okay so keep the current constructors as is, create new ones with
> more
> > > > granular parsing of the results. Sounds like a good plan.
> > > >
> > > > How do we proceed from here ?
> > > >
> > > > Regards,
> > > > Karim Mansour
> > > >
> > > > On Fri, May 1, 2020 at 5:03 PM Austin Cawley-Edwards <
> > > > austin.caw...@gmail.com> wrote:
> > > >
> > > >> Hey,
> > > >>
> > > >> (Switching to my personal email)
> > > >>
> > > >> Correct me if I'm wrong, but I think Aljoscha is proposing keeping
> the
> > > >> public API as is, and adding some new constructors/ custom
> > > deserialization
> > > >> schemas as was done with Kafka. Here's what I was able to find on
> that
> > > >> feature:
> > > >>
> > > >> * https://issues.apache.org/jira/browse/FLINK-8354
> > > >> *
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaDeserializationSchema.java
> > > >> *
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer011.java#L100-L114
> > > >>
> > > >> Best,
> > > >> Austin
> > > >>
> > > >> On Fri, May 1, 2020 at 6:19 AM seneg...@gmail.com <
> seneg...@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >>> Hello,
> > > >>>
> > > >>> So the proposal is to keep the current RMQSource constructors /
> > public
> > > >> api
> > > >>> as is and create new ones that gives more granular parsing ?
> > > >>>
> > > >>> Regards,
> > > >>> Karim Mansour
> > > >>>
> > > >>> On Thu, Apr 30, 2020 at 5:23 PM Austin Cawley-Edwards <
> > > >>> aus...@fintechstudios.com> wrote:
> > > >>>
> > >  Hey all + thanks Konstantin,
> > > 
> > >  Like mentioned, we also run into issues with the RMQ Source
> > > >>> inflexibility.
> > >  I think Aljoscha's idea of supporting both would be a nice way to
> > >  incorporate new changes without breaking the current API.
> > > 
> > >  We'd definitely benefit from the changes proposed here but have
> > > another
> > >  issue with the Correlation ID. When a message gets in the queue
> > > >> without a
> > >  correlation ID, the source errors and the job cannot recover,
> > > requiring
> > >  (painful) manual intervention. It would be nice to be able to
> > > >> dead-letter
> > >  these inputs from the source, but I don't think that's possible
> with
> > > >> the
> > >  current source interface (don't know too much about the source
> > > >>> specifics).
> > >  We might be able to work around this with a custom Correlation ID
> > >  extractor, as proposed by Karim.
> > > 
> > >  Also, if there are other tickets in the RMQ integrations that have
> > > gone
> > >  unmaintained, I'm also happy to chip it at maintaining them!
> > > 
> > >  Best,
> > >  Austin
> > >  
> > >  From: Konstantin Knauf 
> > >  Sent: Thursday, April 30, 2020 6:14 AM
> > >  To: dev 
> > > >>

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Hequn Cheng]

+1, thanks a lot for driving this.

Best, Hequn

On Tue, May 5, 2020 at 11:56 AM Dian Fu  wrote:

> +1
>
> Regards,
> Dian
>
> > 在 2020年5月5日，上午9:58，Yangze Guo  写道：
> >
> > +1
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, May 5, 2020 at 6:14 AM Thomas Weise  wrote:
> >>
> >> +1
> >>
> >>
> >> On Mon, May 4, 2020 at 10:02 AM Marta Paes Moreira  >
> >> wrote:
> >>
> >>> +1, this is quite annoying and distracting.
> >>>
> >>> Marta
> >>>
> >>> On Mon, May 4, 2020 at 6:27 PM Yu Li  wrote:
> >>>
>  +1
> 
>  Best Regards,
>  Yu
> 
> 
>  On Tue, 5 May 2020 at 00:21, Konstantin Knauf 
> wrote:
> 
> > Yes, please.
> >
> > On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz <
> >>> dwysakow...@apache.org>
> > wrote:
> >
> >> +1
> >>
> >> Yes, please. I've also observed a lot of noise in the past days.
> >>
> >> Best,
> >>
> >> Dawid
> >>
> >> On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> >>> +1
> >>>
> >>> All the recent new repos, flink-statefun / flink-statefun-docker /
> >>> flink-training etc. are also sending notifications to issues@.
> >>>
> >>> Gordon
> >>>
> >>>
> >>> On Mon, May 4, 2020, 11:44 PM Till Rohrmann 
> >> wrote:
> >>>
>  Hi everyone,
> 
>  due to some changes on the ASF side, we are now seeing issue and
>  pull
>  request notifications for the flink-web [1] and flink-shaded [2]
>  repo
> > on
>  dev@flink.apache.org. I think this is not ideal since the dev ML
> >>> is
> >> much
>  more noisy now.
> 
>  I would propose to send these notifications to
> > iss...@flink.apache.org
> >> as
>  we are currently doing it for the Flink main repo [3].
> 
>  What do you think?
> 
>  [1] https://github.com/apache/flink-web
>  [2] https://github.com/apache/flink-shaded
>  [3] https://gitbox.apache.org/schemes.cgi?flink
> 
>  Cheers,
>  Till
> 
> >>
> >>
> >
> > --
> >
> > Konstantin Knauf
> >
> > https://twitter.com/snntrable
> >
> > https://github.com/knaufk
> >
> 
> >>>
>
>

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Dian Fu]

+1

Regards,
Dian

> 在 2020年5月5日，上午9:58，Yangze Guo  写道：
> 
> +1
> 
> Best,
> Yangze Guo
> 
> On Tue, May 5, 2020 at 6:14 AM Thomas Weise  wrote:
>> 
>> +1
>> 
>> 
>> On Mon, May 4, 2020 at 10:02 AM Marta Paes Moreira 
>> wrote:
>> 
>>> +1, this is quite annoying and distracting.
>>> 
>>> Marta
>>> 
>>> On Mon, May 4, 2020 at 6:27 PM Yu Li  wrote:
>>> 
 +1
 
 Best Regards,
 Yu
 
 
 On Tue, 5 May 2020 at 00:21, Konstantin Knauf  wrote:
 
> Yes, please.
> 
> On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz <
>>> dwysakow...@apache.org>
> wrote:
> 
>> +1
>> 
>> Yes, please. I've also observed a lot of noise in the past days.
>> 
>> Best,
>> 
>> Dawid
>> 
>> On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
>>> +1
>>> 
>>> All the recent new repos, flink-statefun / flink-statefun-docker /
>>> flink-training etc. are also sending notifications to issues@.
>>> 
>>> Gordon
>>> 
>>> 
>>> On Mon, May 4, 2020, 11:44 PM Till Rohrmann 
>> wrote:
>>> 
 Hi everyone,
 
 due to some changes on the ASF side, we are now seeing issue and
 pull
 request notifications for the flink-web [1] and flink-shaded [2]
 repo
> on
 dev@flink.apache.org. I think this is not ideal since the dev ML
>>> is
>> much
 more noisy now.
 
 I would propose to send these notifications to
> iss...@flink.apache.org
>> as
 we are currently doing it for the Flink main repo [3].
 
 What do you think?
 
 [1] https://github.com/apache/flink-web
 [2] https://github.com/apache/flink-shaded
 [3] https://gitbox.apache.org/schemes.cgi?flink
 
 Cheers,
 Till
 
>> 
>> 
> 
> --
> 
> Konstantin Knauf
> 
> https://twitter.com/snntrable
> 
> https://github.com/knaufk
> 
 
>>> 

Re: [VOTE] Release 1.10.1, release candidate #2

[Hequn Cheng]

Thanks a lot for managing the release!

+1 (binding)

- Go through all new commits for 1.10.1 and spot no new license problems.
- Built from source archive successfully.
- Signatures and hash are correct.
- Run SocketWindowWordCount on the local cluster.
- Install Python package and run Python WordCount example.
- Reviewed website PR

Best,
Hequn

On Sun, May 3, 2020 at 9:10 PM Robert Metzger  wrote:

> Thanks a lot for addressing the issues from the last release candidate and
> creating this one!
>
> +1 (binding)
>
> - Started Flink on YARN on Google Cloud DataProc by setting
> HADOOP_CLASSPATH
> - checked staging repo
>
>
>
> On Sat, May 2, 2020 at 6:57 PM Thomas Weise  wrote:
>
> > +1 (binding)
> >
> > Checked signatures and hashes.
> >
> > Run internal benchmark applications.
> >
> > I found a regression that was actually introduced with 1.10.0, hence not
> a
> > blocker for this release:
> >
> > https://github.com/apache/flink/pull/11975
> >
> > Thanks,
> > Thomas
> >
> >
> > On Fri, May 1, 2020 at 5:37 AM Yu Li  wrote:
> >
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #2 for version 1.10.1,
> as
> > > follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release and binary convenience releases to
> > be
> > > deployed to dist.apache.org [2], which are signed with the key with
> > > fingerprint D8D3D42E84C753CA5F170BDF93C07902771AB743 [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag "release-1.10.1-rc2" [5],
> > > * website pull request listing the new release and adding announcement
> > blog
> > > post [6].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Yu
> > >
> > > [1]
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346891
> > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.10.1-rc2/
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1363/
> > > [5]
> > >
> > >
> >
> https://github.com/apache/flink/commit/f92e8a9d60ef664acd66230da43c6f0a1cd87adc
> > > [6] https://github.com/apache/flink-web/pull/330
> > >
> >
>

Re: Re: The use of state ttl incremental cleanup strategy in sql deduplication resulting in significant performance degradation

[Jark Wu]

Hi Andrey,

Thanks for the tuning ideas. I will explain the design of deduplication.

The mini-batch implementation of deduplication buffers a bundle of input
data in heap (Java Map),
when the bundle size hit the trigger size or trigger time, the buffered
data will be processed together.
So we only need to access the state once per key. This is designed for
rocksdb statebackend to reduce the
frequently accessing, (de)serialization. And yes, this may slow down the
checkpoint, but the suggested
mini-batch timeout is <= 10s. From our production experience, it doesn't
have much impact on checkpoint.

Best,
Jark

On Tue, 5 May 2020 at 06:48, Andrey Zagrebin  wrote:

> Hi lsyldliu,
>
> You can try to tune the StateTtlConfig. As the documentation suggests [1]
> the TTL incremental cleanup can decrease the per record performance. This
> is the price of the automatic cleanup.
> If the only thing, which happens mostly in your operator, is working with
> state then even checking one additional record to cleanup is two times more
> actions to do.
> Timer approach was discussed in TTL feature design. It needs an additional
> implementation and keeps more state but performs only one cleanup action
> exactly when needed so it is a performance/storage trade-off.
>
> Anyways, 20x degradation looks indeed a lot.
> As a first step, I would suggest to configure the incremental cleanup
> explicitly in `StateTtlConfigUtil#createTtlConfig` with a less entries to
> check, e.g. 1 because processFirstRow/processLastRow already access the
> state twice and do cleanup:
>
> .cleanupIncrementally(1, false)
>
>
> Also not sure but depending on the input data, finishBundle can happen
> mostly during the snapshotting which slows down taking the checkpoint.
> Could this fail the checkpoint accumulating the backpressure and slowing
> down the pipeline?
>
> Not sure why to keep the deduplication data in a Java map and in Flink
> state at the same time, why not to keep it only in Flink state and
> deduplicate on each incoming record?
>
> Best,
> Andrey
>
> [1] note 2 in
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#incremental-cleanup
>
> On Wed, Apr 29, 2020 at 11:53 AM 刘大龙  wrote:
>
> >
> >
> >
> > > -原始邮件-
> > > 发件人: "Jark Wu" 
> > > 发送时间: 2020-04-29 14:09:44 (星期三)
> > > 收件人: dev , "Yu Li" ,
> > myas...@live.com
> > > 抄送: azagre...@apache.org
> > > 主题: Re: The use of state ttl incremental cleanup strategy in sql
> > deduplication resulting in significant performance degradation
> > >
> > > Hi lsyldliu,
> > >
> > > Thanks for investigating this.
> > >
> > > First of all, if you are using mini-batch deduplication, it doesn't
> > support
> > > state ttl in 1.9. That's why the tps looks the same with 1.11 disable
> > state
> > > ttl.
> > > We just introduce state ttl for mini-batch deduplication recently.
> > >
> > > Regarding to the performance regression, it looks very surprise to me.
> > The
> > > performance is reduced by 19x when StateTtlConfig is enabled in 1.11.
> > > I don't have much experience of the underlying of StateTtlConfig. So I
> > loop
> > > in @Yu Li  @YunTang in CC who may have more insights
> > on
> > > this.
> > >
> > > For more information, we use the following StateTtlConfig [1] in blink
> > > planner:
> > >
> > > StateTtlConfig
> > >   .newBuilder(Time.milliseconds(retentionTime))
> > >   .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
> > >
>  .setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
> > >   .build();
> > >
> > >
> > > Best,
> > > Jark
> > >
> > >
> > > [1]:
> > >
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/util/StateTtlConfigUtil.java#L27
> > >
> > >
> > >
> > >
> > >
> > > On Wed, 29 Apr 2020 at 11:53, 刘大龙  wrote:
> > >
> > > > Hi, all!
> > > >
> > > > At flink master branch, we have supported state ttl  for sql
> mini-batch
> > > > deduplication using incremental cleanup strategy on heap backend,
> > refer to
> > > > FLINK-16581. Because I want to test the performance of this feature,
> > so I
> > > > compile master branch code and deploy the jar to production
> > > > environment，then run three types of tests, respectively:
> > > >
> > > >
> > > >
> > > >
> > > > flink 1.9.0 release version enable state ttl
> > > > flink 1.11-snapshot version disable state ttl
> > > > flink 1.11-snapshot version enable state ttl
> > > >
> > > >
> > > >
> > > >
> > > > The test query sql as follows:
> > > >
> > > > select order_date,
> > > > sum(price * amount - goods_all_fav_amt - virtual_money_amt +
> > > > goods_carriage_amt) as saleP,
> > > > sum(amount) as saleN,
> > > > count(distinct parent_sn) as orderN,
> > > > count(distinct user_id) as cusN
> > > >from(
> > > > select order_date, user_id,
> > > > order_type, order_status, terminal, last_update_time,
> > > > goods_all_fav_amt,
> > > > goo

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Yangze Guo]

+1

Best,
Yangze Guo

On Tue, May 5, 2020 at 6:14 AM Thomas Weise  wrote:
>
> +1
>
>
> On Mon, May 4, 2020 at 10:02 AM Marta Paes Moreira 
> wrote:
>
> > +1, this is quite annoying and distracting.
> >
> > Marta
> >
> > On Mon, May 4, 2020 at 6:27 PM Yu Li  wrote:
> >
> > > +1
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Tue, 5 May 2020 at 00:21, Konstantin Knauf  wrote:
> > >
> > > > Yes, please.
> > > >
> > > > On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz <
> > dwysakow...@apache.org>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Yes, please. I've also observed a lot of noise in the past days.
> > > > >
> > > > > Best,
> > > > >
> > > > > Dawid
> > > > >
> > > > > On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> > > > > > +1
> > > > > >
> > > > > > All the recent new repos, flink-statefun / flink-statefun-docker /
> > > > > > flink-training etc. are also sending notifications to issues@.
> > > > > >
> > > > > > Gordon
> > > > > >
> > > > > >
> > > > > > On Mon, May 4, 2020, 11:44 PM Till Rohrmann 
> > > > > wrote:
> > > > > >
> > > > > >> Hi everyone,
> > > > > >>
> > > > > >> due to some changes on the ASF side, we are now seeing issue and
> > > pull
> > > > > >> request notifications for the flink-web [1] and flink-shaded [2]
> > > repo
> > > > on
> > > > > >> dev@flink.apache.org. I think this is not ideal since the dev ML
> > is
> > > > > much
> > > > > >> more noisy now.
> > > > > >>
> > > > > >> I would propose to send these notifications to
> > > > iss...@flink.apache.org
> > > > > as
> > > > > >> we are currently doing it for the Flink main repo [3].
> > > > > >>
> > > > > >> What do you think?
> > > > > >>
> > > > > >> [1] https://github.com/apache/flink-web
> > > > > >> [2] https://github.com/apache/flink-shaded
> > > > > >> [3] https://gitbox.apache.org/schemes.cgi?flink
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Till
> > > > > >>
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Konstantin Knauf
> > > >
> > > > https://twitter.com/snntrable
> > > >
> > > > https://github.com/knaufk
> > > >
> > >
> >

The Questions about Submitting PR

[Roc Marshal]

Hi,all.
I have two questions about submitting pr
When I submit PR and trigger azure test, it always shows the status of pending, 
but the details page is in the status of success. In addition, my PR has no 
information about Travis testing. What is the cause of this phenomenon?
Thank you for your attention.


Best,
Roc Marshal




| |
Roc Marshal
|
|
邮箱：flin...@126.com
|

签名由 网易邮箱大师 定制

Re: Re: The use of state ttl incremental cleanup strategy in sql deduplication resulting in significant performance degradation

[Andrey Zagrebin]

Hi lsyldliu,

You can try to tune the StateTtlConfig. As the documentation suggests [1]
the TTL incremental cleanup can decrease the per record performance. This
is the price of the automatic cleanup.
If the only thing, which happens mostly in your operator, is working with
state then even checking one additional record to cleanup is two times more
actions to do.
Timer approach was discussed in TTL feature design. It needs an additional
implementation and keeps more state but performs only one cleanup action
exactly when needed so it is a performance/storage trade-off.

Anyways, 20x degradation looks indeed a lot.
As a first step, I would suggest to configure the incremental cleanup
explicitly in `StateTtlConfigUtil#createTtlConfig` with a less entries to
check, e.g. 1 because processFirstRow/processLastRow already access the
state twice and do cleanup:

.cleanupIncrementally(1, false)


Also not sure but depending on the input data, finishBundle can happen
mostly during the snapshotting which slows down taking the checkpoint.
Could this fail the checkpoint accumulating the backpressure and slowing
down the pipeline?

Not sure why to keep the deduplication data in a Java map and in Flink
state at the same time, why not to keep it only in Flink state and
deduplicate on each incoming record?

Best,
Andrey

[1] note 2 in
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#incremental-cleanup

On Wed, Apr 29, 2020 at 11:53 AM 刘大龙  wrote:

>
>
>
> > -原始邮件-
> > 发件人: "Jark Wu" 
> > 发送时间: 2020-04-29 14:09:44 (星期三)
> > 收件人: dev , "Yu Li" ,
> myas...@live.com
> > 抄送: azagre...@apache.org
> > 主题: Re: The use of state ttl incremental cleanup strategy in sql
> deduplication resulting in significant performance degradation
> >
> > Hi lsyldliu,
> >
> > Thanks for investigating this.
> >
> > First of all, if you are using mini-batch deduplication, it doesn't
> support
> > state ttl in 1.9. That's why the tps looks the same with 1.11 disable
> state
> > ttl.
> > We just introduce state ttl for mini-batch deduplication recently.
> >
> > Regarding to the performance regression, it looks very surprise to me.
> The
> > performance is reduced by 19x when StateTtlConfig is enabled in 1.11.
> > I don't have much experience of the underlying of StateTtlConfig. So I
> loop
> > in @Yu Li  @YunTang in CC who may have more insights
> on
> > this.
> >
> > For more information, we use the following StateTtlConfig [1] in blink
> > planner:
> >
> > StateTtlConfig
> >   .newBuilder(Time.milliseconds(retentionTime))
> >   .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
> >   .setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
> >   .build();
> >
> >
> > Best,
> > Jark
> >
> >
> > [1]:
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/util/StateTtlConfigUtil.java#L27
> >
> >
> >
> >
> >
> > On Wed, 29 Apr 2020 at 11:53, 刘大龙  wrote:
> >
> > > Hi, all!
> > >
> > > At flink master branch, we have supported state ttl  for sql mini-batch
> > > deduplication using incremental cleanup strategy on heap backend,
> refer to
> > > FLINK-16581. Because I want to test the performance of this feature,
> so I
> > > compile master branch code and deploy the jar to production
> > > environment，then run three types of tests, respectively:
> > >
> > >
> > >
> > >
> > > flink 1.9.0 release version enable state ttl
> > > flink 1.11-snapshot version disable state ttl
> > > flink 1.11-snapshot version enable state ttl
> > >
> > >
> > >
> > >
> > > The test query sql as follows:
> > >
> > > select order_date,
> > > sum(price * amount - goods_all_fav_amt - virtual_money_amt +
> > > goods_carriage_amt) as saleP,
> > > sum(amount) as saleN,
> > > count(distinct parent_sn) as orderN,
> > > count(distinct user_id) as cusN
> > >from(
> > > select order_date, user_id,
> > > order_type, order_status, terminal, last_update_time,
> > > goods_all_fav_amt,
> > > goods_carriage_amt, virtual_money_amt, price, amount,
> > > order_quality, quality_goods_cnt, acture_goods_amt
> > > from (select *, row_number() over(partition by order_id,
> > > order_goods_id order by proctime desc) as rownum from
> dm_trd_order_goods)
> > > where rownum=1
> > > and (order_type in (1,2,3,4,5) or order_status = 70)
> > > and terminal = 'shop' and price > 0)
> > > group by order_date
> > >
> > >
> > > At runtime, this query will generate two operators which include
> > > Deduplication and GroupAgg. In the test, the configuration is same,
> > > parallelism is 20, set kafka consumer from the earliest, and disable
> > > mini-batch function, The test results as follows:
> > >
> > > flink 1.9.0 enable state ttl：this test lasted 44m, flink receive 1374w
> > > records, average tps at 5200/s, Flink UI picture link back pressure,
> > > checkpoint
> > > flink 1.11

Re: [DISCUSS] flink-connector-rabbitmq api changes

[Austin Cawley-Edwards]

Thanks Aljoscha,

I'm happy to take FLINK-17204
 for now, if you're able
to assign it to me, and we'll go from there?

Excited to use what you come up with Karim! It also looks like FLINK-8510
 might also have some
ideas on getting access to more RMQ-specific data in the source.

Best,
Austin

On Mon, May 4, 2020 at 6:58 AM seneg...@gmail.com 
wrote:

> Hi,
>
> Okay i created a ticket: https://issues.apache.org/jira/browse/FLINK-17502
>
> i will work on the modifications "keeping the old constructor" and brush up
> on the contribution guides and move from there :)
>
> Regards,
> Karim Mansour
>
> On Mon, May 4, 2020 at 10:00 AM Aljoscha Krettek 
> wrote:
>
> > Yes, that's what I was proposing!
> >
> > @Karim If there's not already a Jira issue, please create one. You can
> > ping me, so that I can assign you.
> >
> > @Austin There's a Jira component for the RMQ source, maybe you can take
> > a stab at some of the issues there:
> >
> >
> https://issues.apache.org/jira/browse/FLINK-17204?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Connectors%2F%20RabbitMQ%22%20AND%20statusCategory%20!%3D%20Done
> > .
> >
> > Best,
> > Aljoscha
> >
> > On 03.05.20 16:38, seneg...@gmail.com wrote:
> > > Hi,
> > >
> > > Okay so keep the current constructors as is, create new ones with more
> > > granular parsing of the results. Sounds like a good plan.
> > >
> > > How do we proceed from here ?
> > >
> > > Regards,
> > > Karim Mansour
> > >
> > > On Fri, May 1, 2020 at 5:03 PM Austin Cawley-Edwards <
> > > austin.caw...@gmail.com> wrote:
> > >
> > >> Hey,
> > >>
> > >> (Switching to my personal email)
> > >>
> > >> Correct me if I'm wrong, but I think Aljoscha is proposing keeping the
> > >> public API as is, and adding some new constructors/ custom
> > deserialization
> > >> schemas as was done with Kafka. Here's what I was able to find on that
> > >> feature:
> > >>
> > >> * https://issues.apache.org/jira/browse/FLINK-8354
> > >> *
> > >>
> > >>
> >
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaDeserializationSchema.java
> > >> *
> > >>
> > >>
> >
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer011.java#L100-L114
> > >>
> > >> Best,
> > >> Austin
> > >>
> > >> On Fri, May 1, 2020 at 6:19 AM seneg...@gmail.com  >
> > >> wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> So the proposal is to keep the current RMQSource constructors /
> public
> > >> api
> > >>> as is and create new ones that gives more granular parsing ?
> > >>>
> > >>> Regards,
> > >>> Karim Mansour
> > >>>
> > >>> On Thu, Apr 30, 2020 at 5:23 PM Austin Cawley-Edwards <
> > >>> aus...@fintechstudios.com> wrote:
> > >>>
> >  Hey all + thanks Konstantin,
> > 
> >  Like mentioned, we also run into issues with the RMQ Source
> > >>> inflexibility.
> >  I think Aljoscha's idea of supporting both would be a nice way to
> >  incorporate new changes without breaking the current API.
> > 
> >  We'd definitely benefit from the changes proposed here but have
> > another
> >  issue with the Correlation ID. When a message gets in the queue
> > >> without a
> >  correlation ID, the source errors and the job cannot recover,
> > requiring
> >  (painful) manual intervention. It would be nice to be able to
> > >> dead-letter
> >  these inputs from the source, but I don't think that's possible with
> > >> the
> >  current source interface (don't know too much about the source
> > >>> specifics).
> >  We might be able to work around this with a custom Correlation ID
> >  extractor, as proposed by Karim.
> > 
> >  Also, if there are other tickets in the RMQ integrations that have
> > gone
> >  unmaintained, I'm also happy to chip it at maintaining them!
> > 
> >  Best,
> >  Austin
> >  
> >  From: Konstantin Knauf 
> >  Sent: Thursday, April 30, 2020 6:14 AM
> >  To: dev 
> >  Cc: Austin Cawley-Edwards 
> >  Subject: Re: [DISCUSS] flink-connector-rabbitmq api changes
> > 
> >  Hi everyone,
> > 
> >  just looping in Austin as he mentioned that they also ran into
> issues
> > >> due
> >  to the inflexibility of the RabiitMQSourcce to me yesterday.
> > 
> >  Cheers,
> > 
> >  Konstantin
> > 
> >  On Thu, Apr 30, 2020 at 11:23 AM seneg...@gmail.com >  seneg...@gmail.com> mailto:seneg...@gmail.com>>
> > >>> wrote:
> >  Hello Guys,
> > 
> >  Thanks for all the responses, i want to stress out that i didn't
> feel
> >  ignored i just thought that i forgot an important step or something.
> > 
> >  Since i am a newbie i would follow whatever route you guys would

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Thomas Weise]

+1


On Mon, May 4, 2020 at 10:02 AM Marta Paes Moreira 
wrote:

> +1, this is quite annoying and distracting.
>
> Marta
>
> On Mon, May 4, 2020 at 6:27 PM Yu Li  wrote:
>
> > +1
> >
> > Best Regards,
> > Yu
> >
> >
> > On Tue, 5 May 2020 at 00:21, Konstantin Knauf  wrote:
> >
> > > Yes, please.
> > >
> > > On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz <
> dwysakow...@apache.org>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Yes, please. I've also observed a lot of noise in the past days.
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> > > > > +1
> > > > >
> > > > > All the recent new repos, flink-statefun / flink-statefun-docker /
> > > > > flink-training etc. are also sending notifications to issues@.
> > > > >
> > > > > Gordon
> > > > >
> > > > >
> > > > > On Mon, May 4, 2020, 11:44 PM Till Rohrmann 
> > > > wrote:
> > > > >
> > > > >> Hi everyone,
> > > > >>
> > > > >> due to some changes on the ASF side, we are now seeing issue and
> > pull
> > > > >> request notifications for the flink-web [1] and flink-shaded [2]
> > repo
> > > on
> > > > >> dev@flink.apache.org. I think this is not ideal since the dev ML
> is
> > > > much
> > > > >> more noisy now.
> > > > >>
> > > > >> I would propose to send these notifications to
> > > iss...@flink.apache.org
> > > > as
> > > > >> we are currently doing it for the Flink main repo [3].
> > > > >>
> > > > >> What do you think?
> > > > >>
> > > > >> [1] https://github.com/apache/flink-web
> > > > >> [2] https://github.com/apache/flink-shaded
> > > > >> [3] https://gitbox.apache.org/schemes.cgi?flink
> > > > >>
> > > > >> Cheers,
> > > > >> Till
> > > > >>
> > > >
> > > >
> > >
> > > --
> > >
> > > Konstantin Knauf
> > >
> > > https://twitter.com/snntrable
> > >
> > > https://github.com/knaufk
> > >
> >
>

[jira] [Created] (FLINK-17509) Support OracleDialect

[Flavio Pompermaier (Jira)]

Flavio Pompermaier created FLINK-17509:
--

 Summary: Support OracleDialect
 Key: FLINK-17509
 URL: https://issues.apache.org/jira/browse/FLINK-17509
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / JDBC
Reporter: Flavio Pompermaier


Support OracleDialect in JDBCDialects



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17508) Develop OracleCatalog

[Flavio Pompermaier (Jira)]

Flavio Pompermaier created FLINK-17508:
--

 Summary: Develop OracleCatalog
 Key: FLINK-17508
 URL: https://issues.apache.org/jira/browse/FLINK-17508
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / JDBC
Reporter: Flavio Pompermaier


Similarly to https://issues.apache.org/jira/browse/FLINK-16471



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17507) Training figure program_dataflow.svg should use preferred parts of the API

[David Anderson (Jira)]

David Anderson created FLINK-17507:
--

 Summary: Training figure program_dataflow.svg should use preferred 
parts of the API
 Key: FLINK-17507
 URL: https://issues.apache.org/jira/browse/FLINK-17507
 Project: Flink
  Issue Type: Improvement
  Components: Documentation / Training
Reporter: David Anderson


It would be better if fig/program_dataflow.svg used a 
{{ProcessWindowFunction}}, rather than a {{WindowFunction}}.

It also uses a {{BucketingSink}}, which sets a bad example. 

Note that this is not a trivial edit, since it doesn't work to simply replace 
{{new BucketingSink}} with {{new StreamingFileSink}}. Something like this would 
be better:

 
{{final StreamingFileSink sink = StreamingFileSink}}
{{        .forBulkFormat(...)}}
{{        .build();}}
{{}}
{{stats.addSink(sink);}}
{{}}
Note: This figure is only used once, in the Training Overview page.
{{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Update our Roadmap

[Robert Metzger]

Hey all,
The roadmap has been updated last in September 2019. There is quite a bit
of outdated information on that page.

Is there a volunteer who wants to create a first draft for an updated
version?

On Mon, Aug 19, 2019 at 12:14 PM Robert Metzger  wrote:

> Nice, thanks! Looking forward to the PR
>
> On Mon, Aug 19, 2019 at 11:08 AM Marta Paes Moreira 
> wrote:
>
>> Hey, Robert.
>>
>> Updating the roadmap is something I can work on (and also have on my
>> radar,
>> moving forward). Already had a quick word with Stephan and he's available
>> to provide support, if needed.
>>
>> Marta
>>
>> On Sun, Aug 18, 2019 at 4:46 PM Stephan Ewen  wrote:
>>
>> > I could help with that.
>> >
>> > On Fri, Aug 16, 2019 at 2:36 PM Robert Metzger 
>> > wrote:
>> >
>> > > Flink 1.9 is feature freezed and almost released.
>> > > I guess it makes sense to update the roadmap on the website again.
>> > >
>> > > Who feels like having a good overview of what's coming up?
>> > >
>> > > On Tue, May 7, 2019 at 4:33 PM Fabian Hueske 
>> wrote:
>> > >
>> > > > Yes, that's a very good proposal Jark.
>> > > > +1
>> > > >
>> > > > Best, Fabian
>> > > >
>> > > > Am Mo., 6. Mai 2019 um 16:33 Uhr schrieb Till Rohrmann <
>> > > > trohrm...@apache.org
>> > > > >:
>> > > >
>> > > > > I think this is a good idea Jark. Putting the last update date on
>> the
>> > > > > roadmap would also force us to regularly update it.
>> > > > >
>> > > > > Cheers,
>> > > > > Till
>> > > > >
>> > > > > On Mon, May 6, 2019 at 4:14 AM Jark Wu  wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > One suggestion for the roadmap:
>> > > > > >
>> > > > > > Shall we add a `latest-update-time` to the top of Roadmap page?
>> So
>> > > that
>> > > > > > users can know this is a up-to-date Roadmap.
>> > > > > >
>> > > > > > On Thu, 2 May 2019 at 04:49, Bowen Li 
>> wrote:
>> > > > > >
>> > > > > > > +1
>> > > > > > >
>> > > > > > > On Mon, Apr 29, 2019 at 11:41 PM jincheng sun <
>> > > > > sunjincheng...@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Jeff&Fabian,
>> > > > > > > >
>> > > > > > > > I have open the PR about add Python Table API section to the
>> > > > > roadmap. I
>> > > > > > > > appreciate if you have time to look at it. :)
>> > > > > > > >
>> > > > > > > > https://github.com/apache/flink-web/pull/204
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > > Jincheng
>> > > > > > > >
>> > > > > > > > jincheng sun  于2019年4月29日周一
>> > 下午11:12写道：
>> > > > > > > >
>> > > > > > > > > Sure, I will do it！I think the python table api info
>> should
>> > in
>> > > > the
>> > > > > > > > >  roadmap! Thank you @Jeff @Fabian
>> > > > > > > > >
>> > > > > > > > > Fabian Hueske 于2019年4月29日 周一23:05写道：
>> > > > > > > > >
>> > > > > > > > >> Great, thanks Jeff and Timo!
>> > > > > > > > >>
>> > > > > > > > >> @Jincheng do you want to write a paragraph about the
>> Python
>> > > > effort
>> > > > > > and
>> > > > > > > > >> open a PR for it?
>> > > > > > > > >>
>> > > > > > > > >> I'll remove the issue about Hadoop convenience builds
>> > > > > (FLINK-11266).
>> > > > > > > > >>
>> > > > > > > > >> Best, Fabian
>> > > > > > > > >>
>> > > > > > > > >> Am Mo., 29. Apr. 2019 um 16:37 Uhr schrieb Jeff Zhang <
>> > > > > > > zjf...@gmail.com
>> > > > > > > > >:
>> > > > > > > > >>
>> > > > > > > > >>> jincheng(cc) is driving the python effort, I think he
>> can
>> > > help
>> > > > to
>> > > > > > > > >>> prepare it.
>> > > > > > > > >>>
>> > > > > > > > >>>
>> > > > > > > > >>>
>> > > > > > > > >>> Fabian Hueske  于2019年4月29日周一
>> 下午10:15写道：
>> > > > > > > > >>>
>> > > > > > > >  Hi everyone,
>> > > > > > > > 
>> > > > > > > >  Since we had no more comments on this thread, I think
>> we
>> > > > proceed
>> > > > > > to
>> > > > > > > >  update the roadmap.
>> > > > > > > > 
>> > > > > > > >  @Jeff Zhang  I agree, we should add
>> the
>> > > > > Python
>> > > > > > > >  efforts to the roadmap.
>> > > > > > > >  Do you want to prepare a short paragraph that we can
>> add
>> > to
>> > > > the
>> > > > > > > >  document?
>> > > > > > > > 
>> > > > > > > >  Best, Fabian
>> > > > > > > > 
>> > > > > > > >  Am Mi., 17. Apr. 2019 um 15:04 Uhr schrieb Jeff Zhang <
>> > > > > > > > zjf...@gmail.com
>> > > > > > > >  >:
>> > > > > > > > 
>> > > > > > > > > Hi Fabian,
>> > > > > > > > >
>> > > > > > > > > One thing missing is python api and python udf, we
>> > already
>> > > > > > > discussed
>> > > > > > > > > it in
>> > > > > > > > > community, and it is very close to reach consensus.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Fabian Hueske  于2019年4月17日周三
>> > 下午7:51写道：
>> > > > > > > > >
>> > > > > > > > > > Hi everyone,
>> > > > > > > > > >
>> > > > > > > > > > We recently added a roadmap to our project website
>> [1]
>> > > and
>> > > > > > > decided
>> > > > > > > > to
>> > > > > > > > > > update it after every release. Flink 1.8.

[GitHub] [flink-web] XBaith commented on pull request #333: [FLINK-17490] Add training page

[GitBox]

XBaith commented on pull request #333:
URL: https://github.com/apache/flink-web/pull/333#issuecomment-623640938


   Thank you @alpinegizmo .



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Marta Paes Moreira]

+1, this is quite annoying and distracting.

Marta

On Mon, May 4, 2020 at 6:27 PM Yu Li  wrote:

> +1
>
> Best Regards,
> Yu
>
>
> On Tue, 5 May 2020 at 00:21, Konstantin Knauf  wrote:
>
> > Yes, please.
> >
> > On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz 
> > wrote:
> >
> > > +1
> > >
> > > Yes, please. I've also observed a lot of noise in the past days.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> > > > +1
> > > >
> > > > All the recent new repos, flink-statefun / flink-statefun-docker /
> > > > flink-training etc. are also sending notifications to issues@.
> > > >
> > > > Gordon
> > > >
> > > >
> > > > On Mon, May 4, 2020, 11:44 PM Till Rohrmann 
> > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> due to some changes on the ASF side, we are now seeing issue and
> pull
> > > >> request notifications for the flink-web [1] and flink-shaded [2]
> repo
> > on
> > > >> dev@flink.apache.org. I think this is not ideal since the dev ML is
> > > much
> > > >> more noisy now.
> > > >>
> > > >> I would propose to send these notifications to
> > iss...@flink.apache.org
> > > as
> > > >> we are currently doing it for the Flink main repo [3].
> > > >>
> > > >> What do you think?
> > > >>
> > > >> [1] https://github.com/apache/flink-web
> > > >> [2] https://github.com/apache/flink-shaded
> > > >> [3] https://gitbox.apache.org/schemes.cgi?flink
> > > >>
> > > >> Cheers,
> > > >> Till
> > > >>
> > >
> > >
> >
> > --
> >
> > Konstantin Knauf
> >
> > https://twitter.com/snntrable
> >
> > https://github.com/knaufk
> >
>

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

[Till Rohrmann]

Thanks everyone for this lively discussion and all your thoughts.

Let me try to summarise the current state of the discussion and then let's
see how we can move it forward.

To begin with, I think everyone agrees that we want to improve Flink's user
experience. In particular, we want to improve the experience of first time
users who want to try out Flink's SQL functionality.

The problem which stands in the way of a good user experience is that the
current Flink distribution contains too few dependencies for a smooth first
time SQL experience and too many dependencies for a lean production setup.
Hence, Aljoscha proposed to create a "fat" and "slim" Flink distribution
addressing these two differing needs.

As far as the discussion goes there are two remaining discussion points.

1. How do we serve the different types of distributions?

a) Create a "fat" and "slim" distribution which is served from the Flink
web site.
b) Create a "slim" distribution which is served from the Flink web site and
have a tool (e.g. script) which can turn a slim distribution into a fat
distribution by downloading additional dependencies.

For a) speaks that it is simpler and does not require the user to execute
an additional step. The downside is that we will add another dimension to
the release matrix which will complicate the release process (see Chesnay's
last comment for more details).

For b) speaks that it is potentially the more general solution as we can
provide different options for different distributions (e.g. choosing a
connector version, required filesystems, metric reporters, etc.). The
downside is the additional step for the user and that we need such a tool
(which in itself could be quite complex).

2. What is contained in the "fat" distribution?

The current proposal is to move everything which can be moved from opt to
the plugins directory to the plugins directory (metric reporters and
filesystems). That way the user will be able to use all of these
implementations without running into dependency conflicts.

For the SQL support, Aljoscha proposed to add:

flink-avro-1.10.0.jar
flink-csv-1.10.0.jar
flink-hbase_2.11-1.10.0.jar
flink-jdbc_2.11-1.10.0.jar
flink-json-1.10.0.jar
flink-sql-connector-elasticsearch6_2.11-1.10.0.jar
flink-sql-connector-kafka_2.11-1.10.0.jar
sql-connectors-formats

How to move forward from here?

Given that the time until the feature freeze is limited I would actually
propose to follow the simplest approach which is the creation of two
distributions ("fat" & "slim"). We can still rethink this decision at a
later point and introduce a tool which allows to download a custom build
Flink distribution. At this point we could then remove the "fat" jar from
the web site. Of course, this comes at the cost of increased release
complexity but I believe that the user experience will make up for it.

For the what to include, I think we could take Aljoscha's proposal and then
see what other dependencies the most common SQL use cases require. I guess
that the SQL guys know quite precisely where the users run into problems.

I know that this solution might not be perfect (in particular wrt releases)
but I hope that everyone could live with this solution for the time being.

Feel free to add anything I might have forgotten to mention here.

Cheers,
Till

On Tue, Apr 28, 2020 at 11:43 AM Chesnay Schepler 
wrote:

> It would be good if we could nail down what a slim/fat distribution
> would look like, as there are various ideas floating around in this thread.
>
> Like, what is a "slim" distribution? Are we just emptying /opt? Removing
> everything larger than 1mb? Are we throwing out the Table API from /lib
> for a minimal streaming distribution?
> Are we going ham and remove the YARN integration from the flink-dist jar?
>
> While I can see how a fat distribution can certainly help for the
> out-of-the-box experience, I'm not so sold on the slim variant.
> If someone is capable of assembling a distribution matching to their
> use-case, do they even need a slim distribution in the first place?
>
> I really want us to stick to 1 distribution type, as I'm worried about
> the implications of 2 or FWIW any number of additional distribution types:
>
> - you need separate assemblies, including a new profile
>  - adjusting opt/plugins and making sure the examples match the
> bundled contents (e.g., no gelly/python, maybe some SQL examples if
> there are any that use a connector)
> - another 300mb uploaded to dist.apache.org + whatever the fat
> distribution grows by x3 (scala 2.11/2.12 + python)
>  - the latter naturally being susceptible to additional growth in
> the future
>  - this is also a pain for release managers since SVN likes to throw
> up if the upload is too large + it increases upload time
> - another 2 distributions to test during a release
> - another distribution type we need to test via CI
> - more content downloaded into the docker images by default
>  - unless of course we releas

[jira] [Created] (FLINK-17506) SavepointEnvironment does not honour 'io.tmp.dirs' property

[David Artiga (Jira)]

David Artiga created FLINK-17506:


 Summary: SavepointEnvironment does not honour 'io.tmp.dirs' 
property
 Key: FLINK-17506
 URL: https://issues.apache.org/jira/browse/FLINK-17506
 Project: Flink
  Issue Type: Bug
  Components: API / State Processor
Reporter: David Artiga


{{SavepointEnvironment}} [creates an 
IOManagerAsync|https://github.com/apache/flink/blob/d6439c8d0e7792961635e3e4297c3dbfb01938e3/flink-libraries/flink-state-processing-api/src/main/java/org/apache/flink/state/api/runtime/SavepointEnvironment.java#L106]
 using it's [default 
constructor|https://github.com/apache/flink/blob/d6439c8d0e7792961635e3e4297c3dbfb01938e3/flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/IOManagerAsync.java#L62],
 meaning it [uses env var 
"java.io.tmpdir"|https://github.com/apache/flink/blob/d6439c8d0e7792961635e3e4297c3dbfb01938e3/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java#L227]
 instead of values from "io.tmp.dirs" config property,



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] Updates to our Licensing wiki (was: Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #4)

[Yu Li]

Thanks Robert, I think these supplements are quite instructive and make the
whole process more clear.

nit: it surprised me a little bit that this voting thread of a canceled RC
is revived so I edited the subject in this reply.

Best Regards,
Yu


On Mon, 4 May 2020 at 22:48, Robert Metzger  wrote:

> I think I conceptually added everything Stephan mentioned into the Wiki:
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=97552147&selectedPageVersions=5&selectedPageVersions=3
> I'm looking forward to any comments or corrections.
>
> On Wed, Apr 1, 2020 at 12:20 PM Chesnay Schepler 
> wrote:
>
>> Let's add this information to the licensing wiki page.
>> https://cwiki.apache.org/confluence/display/FLINK/Licensing
>>
>>
>> On 01/04/2020 12:16, Yu Li wrote:
>> > Thanks Stephan, this is enlightening!
>> >
>> > Best Regards,
>> > Yu
>> >
>> >
>> > On Wed, 1 Apr 2020 at 16:30, Stephan Ewen  wrote:
>> >
>> >> @Yu - there is nothing magic behind the license check, but I can share
>> >> what I did there.
>> >>
>> >> Source distribution:
>> >>- This means code copied into the repo.
>> >>- The Java source code is usually fine, committers copying code
>> verbatim
>> >> usually comment on that directly.
>> >>- Most important are other files, like docs (mostly build setup, not
>> >> contents files), other html/web related code (like UIs), build files,
>> etc.
>> >> So specifically go through these files (there are usually not too
>> many).
>> >>
>> >> Binary distribution:
>> >>- To check the compatibility of licenses of transitive dependencies,
>> >> maven generated a "DEPENDENCIES" file in the Jar that lists the
>> >> dependencies by license, which is a very helpful start
>> >>- Packaging wise, projects that build shaded jars are important,
>> because
>> >> they bundle dependencies, which means
>> >>   (a) checking that the relevant licenses / notices are present
>> >>   (b) checking that they don't bundle too much (jar tf  |
>> >> grep ... | less) because transitive dependencies and mixing compile /
>> >> provided is not straightforward in maven.
>> >>
>> >> Best,
>> >> Stephan
>> >>
>> >>
>> >>
>> >> On Wed, Apr 1, 2020 at 4:47 AM Yu Li  wrote:
>> >>
>> >>> Hi Stephan,
>> >>>
>> >>> Could you also share the method of license check, so more people could
>> >>> help in future votes? And maybe adding some instructions into our
>> wiki [1]?
>> >>> I skipped the licensing check in my vote because not aware of a good
>> way to
>> >>> do it thoroughly, not sure whether I'm the only one having such
>> question
>> >>> though. Thanks.
>> >>>
>> >>> btw, I noticed the fix version of FLINK-16891-16897 are all set to
>> 2.0.0
>> >>> but I guess it should be statefun-2.0.0 instead?
>> >>>
>> >>> Best Regards,
>> >>> Yu
>> >>>
>> >>> [1] https://cwiki.apache.org/confluence/display/FLINK/Licensing
>> >>>
>> >>> On Wed, 1 Apr 2020 at 04:01, Stephan Ewen  wrote:
>> >>>
>>  I did a release check for license issues - all in all, we need a new
>> RC.
>> 
>>  The only blocker I found was the missing jquery license file.
>> 
>>  Another somewhat critical thing is that "statefun-flink-distribution"
>>  bundles many unwanted dependencies.
>> - Because the shading merges the notice files, this is not a legal
>>  issue.
>> - Because Flinks inverted classloading still uses "parent-first"
>> for
>>  all
>>  "org.apache.flink.*" classes, this does not break the system
>>  But it is unwanted behavior and makes the artifacts unnecessarily
>> large.
>> 
>>  I opened FLINK-16891 - FLINK-16897 for the issues I found.
>>  All issues are fixed in this PR:
>>  https://github.com/apache/flink-statefun/pull/85
>> 
>> 
>> 
>>  On Tue, Mar 31, 2020 at 7:17 PM Stephan Ewen 
>> wrote:
>> 
>> > I have found a few things, am preparing a joint PR to fix them.
>> >
>> > So far, only the missing jquery license would have been a release
>>  blocker.
>> > On Tue, Mar 31, 2020 at 6:24 PM Chesnay Schepler <
>> ches...@apache.org>
>> > wrote:
>> >
>> >> The jquery license is in fact missing from the master/release-1.10
>> >> branches. https://issues.apache.org/jira/browse/FLINK-16888
>> >>
>> >>
>> >> On 31/03/2020 12:18, Chesnay Schepler wrote:
>> >>> For Kafka we traditionally exclude the NOTICE file since as far as
>>  we
>> >>> can tell it is misleading anyway, see the
>> flink-sql-connector-kafka
>> >>> modules.
>> >>>
>> >>> @Robert for the Flink project the jquery license is in the source
>> at
>> >>> licenses/LICENSE.jquery
>> >>>
>> >>> I'm a bit concerned just how many licensing issues are showing up
>> in
>> >>> these RCs. I would suggest to do a proper scan of the licensing
>>  before
>> >>> opening another RC.
>> >>>
>> >>> And yes, the missing MIT license is grounds for cancellation,
>>  hence, -1.
>> >

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Yu Li]

+1

Best Regards,
Yu


On Tue, 5 May 2020 at 00:21, Konstantin Knauf  wrote:

> Yes, please.
>
> On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz 
> wrote:
>
> > +1
> >
> > Yes, please. I've also observed a lot of noise in the past days.
> >
> > Best,
> >
> > Dawid
> >
> > On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> > > +1
> > >
> > > All the recent new repos, flink-statefun / flink-statefun-docker /
> > > flink-training etc. are also sending notifications to issues@.
> > >
> > > Gordon
> > >
> > >
> > > On Mon, May 4, 2020, 11:44 PM Till Rohrmann 
> > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> due to some changes on the ASF side, we are now seeing issue and pull
> > >> request notifications for the flink-web [1] and flink-shaded [2] repo
> on
> > >> dev@flink.apache.org. I think this is not ideal since the dev ML is
> > much
> > >> more noisy now.
> > >>
> > >> I would propose to send these notifications to
> iss...@flink.apache.org
> > as
> > >> we are currently doing it for the Flink main repo [3].
> > >>
> > >> What do you think?
> > >>
> > >> [1] https://github.com/apache/flink-web
> > >> [2] https://github.com/apache/flink-shaded
> > >> [3] https://gitbox.apache.org/schemes.cgi?flink
> > >>
> > >> Cheers,
> > >> Till
> > >>
> >
> >
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Konstantin Knauf]

Yes, please.

On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz 
wrote:

> +1
>
> Yes, please. I've also observed a lot of noise in the past days.
>
> Best,
>
> Dawid
>
> On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> > +1
> >
> > All the recent new repos, flink-statefun / flink-statefun-docker /
> > flink-training etc. are also sending notifications to issues@.
> >
> > Gordon
> >
> >
> > On Mon, May 4, 2020, 11:44 PM Till Rohrmann 
> wrote:
> >
> >> Hi everyone,
> >>
> >> due to some changes on the ASF side, we are now seeing issue and pull
> >> request notifications for the flink-web [1] and flink-shaded [2] repo on
> >> dev@flink.apache.org. I think this is not ideal since the dev ML is
> much
> >> more noisy now.
> >>
> >> I would propose to send these notifications to iss...@flink.apache.org
> as
> >> we are currently doing it for the Flink main repo [3].
> >>
> >> What do you think?
> >>
> >> [1] https://github.com/apache/flink-web
> >> [2] https://github.com/apache/flink-shaded
> >> [3] https://gitbox.apache.org/schemes.cgi?flink
> >>
> >> Cheers,
> >> Till
> >>
>
>

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [DISCUSS] Have foreground processes also create log files

[Till Rohrmann]

Hi everyone,

thanks for starting this discussion Chesnay.

I think it would be nice if we also displayed the logs when starting the
process in the foreground.

The repercussions could be mitigated if the default logger configurations
would contain file rolling with a max log file size.

@Yang I think there are solutions how to redirect stdout and stderr into
separate files using tee without duplication [1].

[1] http://www.softpanorama.org/Tools/tee.shtml

Cheers,
Till

On Wed, Apr 29, 2020 at 4:28 AM Yang Wang  wrote:

> Thanks for Chesnay starting this discussion.
>
> In FLINK-17166 implementation[1], we are trying to use "tee" instead of
> introducing the stream redirection(redirect the out/err to files). However,
> a side effect is that the logging will be duplicated both in .log and .out
> files.
> Then it may consume more disk space. However it is not a very critical
> problem since we could use log4j/logback configuration to control the
> rolling
> files and max size.
>
> Also, it only happens in docker/K8s deployment. For YARN/Mesos deployment,
> the behavior is just same as before.
>
>
> [1]. https://github.com/apache/flink/pull/11839
>
> Best,
> Yang
>
> Chesnay Schepler  于2020年4月29日周三 上午12:30写道：
>
> > Currently, processes started in the foreground (like in the case of
> > Docker) output all logging/stdout directly to the console, without
> > creating any logging files.
> >
> > The downside of this approach, as outlined in FLIP-111, is that the
> > WebUI is not able to display the logs since it relies on these very
> > files to exist.
> >
> > In FLINK-17166 (part of FLIP-111) we are trying to change this such that
> > we always created .log/.out files. It seems like a reasonable change to
> > do, but it could have repercussions on existing deployments since we
> > will naturally use more disk space (logs gotta go somewhere).
> >
> > I'm curious what people think about this.
> >
> >
>

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Dawid Wysakowicz]

+1

Yes, please. I've also observed a lot of noise in the past days.

Best,

Dawid

On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> +1
>
> All the recent new repos, flink-statefun / flink-statefun-docker /
> flink-training etc. are also sending notifications to issues@.
>
> Gordon
>
>
> On Mon, May 4, 2020, 11:44 PM Till Rohrmann  wrote:
>
>> Hi everyone,
>>
>> due to some changes on the ASF side, we are now seeing issue and pull
>> request notifications for the flink-web [1] and flink-shaded [2] repo on
>> dev@flink.apache.org. I think this is not ideal since the dev ML is much
>> more noisy now.
>>
>> I would propose to send these notifications to iss...@flink.apache.org as
>> we are currently doing it for the Flink main repo [3].
>>
>> What do you think?
>>
>> [1] https://github.com/apache/flink-web
>> [2] https://github.com/apache/flink-shaded
>> [3] https://gitbox.apache.org/schemes.cgi?flink
>>
>> Cheers,
>> Till
>>



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Tzu-Li (Gordon) Tai]

+1

All the recent new repos, flink-statefun / flink-statefun-docker /
flink-training etc. are also sending notifications to issues@.

Gordon


On Mon, May 4, 2020, 11:44 PM Till Rohrmann  wrote:

> Hi everyone,
>
> due to some changes on the ASF side, we are now seeing issue and pull
> request notifications for the flink-web [1] and flink-shaded [2] repo on
> dev@flink.apache.org. I think this is not ideal since the dev ML is much
> more noisy now.
>
> I would propose to send these notifications to iss...@flink.apache.org as
> we are currently doing it for the Flink main repo [3].
>
> What do you think?
>
> [1] https://github.com/apache/flink-web
> [2] https://github.com/apache/flink-shaded
> [3] https://gitbox.apache.org/schemes.cgi?flink
>
> Cheers,
> Till
>

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Jark Wu]

Big +1 to this.

Best,
Jark

On Mon, 4 May 2020 at 23:44, Till Rohrmann  wrote:

> Hi everyone,
>
> due to some changes on the ASF side, we are now seeing issue and pull
> request notifications for the flink-web [1] and flink-shaded [2] repo on
> dev@flink.apache.org. I think this is not ideal since the dev ML is much
> more noisy now.
>
> I would propose to send these notifications to iss...@flink.apache.org as
> we are currently doing it for the Flink main repo [3].
>
> What do you think?
>
> [1] https://github.com/apache/flink-web
> [2] https://github.com/apache/flink-shaded
> [3] https://gitbox.apache.org/schemes.cgi?flink
>
> Cheers,
> Till
>

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Till Rohrmann]

For future reference here is the link describing how to configure the
notifications:
https://cwiki.apache.org/confluence/display/INFRA/.asf.yaml+features+for+git+repositories#id-.asf.yamlfeaturesforgitrepositories-Notificationsettingsforrepositories

On Mon, May 4, 2020 at 5:43 PM Till Rohrmann  wrote:

> Hi everyone,
>
> due to some changes on the ASF side, we are now seeing issue and pull
> request notifications for the flink-web [1] and flink-shaded [2] repo on
> dev@flink.apache.org. I think this is not ideal since the dev ML is much
> more noisy now.
>
> I would propose to send these notifications to iss...@flink.apache.org as
> we are currently doing it for the Flink main repo [3].
>
> What do you think?
>
> [1] https://github.com/apache/flink-web
> [2] https://github.com/apache/flink-shaded
> [3] https://gitbox.apache.org/schemes.cgi?flink
>
> Cheers,
> Till
>

[DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Till Rohrmann]

Hi everyone,

due to some changes on the ASF side, we are now seeing issue and pull
request notifications for the flink-web [1] and flink-shaded [2] repo on
dev@flink.apache.org. I think this is not ideal since the dev ML is much
more noisy now.

I would propose to send these notifications to iss...@flink.apache.org as
we are currently doing it for the Flink main repo [3].

What do you think?

[1] https://github.com/apache/flink-web
[2] https://github.com/apache/flink-shaded
[3] https://gitbox.apache.org/schemes.cgi?flink

Cheers,
Till

Re: [DISCUSS] Hierarchies in ConfigOption

[Till Rohrmann]

Hi everyone,

I like Timo's proposal to organize our configuration more hierarchical
since this is what the coding guide specifies. The benefit I see is that
config options belonging to the same concept will be found in the same
nested object. Moreover, it will be possible to split the configuration
into unrelated parts which are fed to the respective components. That way
one has a much better separation of concern since component A cannot read
the configuration of component B.

Concerning Timo's last two proposals:

If fail-on-missing-field is a common configuration shared by all formats,
then I would go with the first option:

format.kind: json
format.fail-on-missing-field: true

If fail-on-missing-field is specific for json, then one could go with

format: json
json.fail-on-missing-field: true

or

format.kind: json
format.json.fail-on-missing-field: true

Cheers,
Till


On Fri, May 1, 2020 at 11:55 AM Timo Walther  wrote:

> Hi Jark,
>
> yes, in theory every connector can design options as they like. But for
> user experience and good coding style we should be consistent in Flink
> connectors and configuration. Because implementers of new connectors
> will copy the design of existing ones.
>
> Furthermore, I could image that people in the DataStream API would also
> like to configure their connector based on options in the near future.
> It might be the case that Flink DataStream API connectors will reuse the
> ConfigOptions from Table API for consistency.
>
> I'm favoring either:
>
> format.kind = json
> format.fail-on-missing-field: true
>
> Or:
>
> format = json
> json.fail-on-missing-field: true
>
> Both are valid hierarchies.
>
> Regards,
> Timo
>
>
> On 30.04.20 17:57, Jark Wu wrote:
> > Hi Dawid,
> >
> > I just want to mention one of your response,
> >
> >> What you described with
> >> 'format' = 'csv',
> >> 'csv.allow-comments' = 'true',
> >> 'csv.ignore-parse-errors' = 'true'
> >> would not work though as the `format` prefix is mandatory in the sources
> > as only the properties with format
> >>   will be passed to the format factory in majority of cases. We already
> > have some implicit contracts.
> >
> > IIUC, in FLIP-95 and FLIP-122, the property key style are totally decided
> > by connectors, not the framework.
> > So I custom connector can define above properties, and extract the value
> of
> > 'format', i.e. 'csv', to find the format factory.
> > And extract the properties with `csv.` prefix and remove the prefix, and
> > pass the properties (e.g. 'allow-comments' = 'true')
> > into the format factory to create format.
> >
> > So there is no a strict guarantee to have a "nested JSON style"
> properties.
> > Users can still develop a custom connector with this
> > un-hierarchy properties and works well.
> >
> > 'format' = 'json',
> > 'format.fail-on-missing-field' = 'false'
> >
> > Best,
> > Jark
> >
> >
> > On Thu, 30 Apr 2020 at 14:29, Dawid Wysakowicz 
> > wrote:
> >
> >> Hi all,
> >>
> >> I'd like to start with a comment that I am ok with the current state of
> >> the FLIP-122 if there is a strong preference for it. Nevertheless I
> still
> >> like the idea of adding `type` to the `format` to have it as
> `format.type`
> >> = `json`.
> >>
> >> I wanted to clarify a few things though:
> >>
> >> @Jingsong As far as I see it most of the users copy/paste the properties
> >> from the documentation to the SQL, so I don't think additional four
> >> characters are too cumbersome. Plus if you force the additional suffix
> onto
> >> all the options of a format you introduce way more boilerplate than if
> we
> >> added the `type/kind/name`
> >>
> >> @Kurt I agree that we cannot force it, but I think it is more of a
> >> question to set standards/implicit contracts on the properties. What you
> >> described with
> >> 'format' = 'csv',
> >> 'csv.allow-comments' = 'true',
> >> 'csv.ignore-parse-errors' = 'true'
> >>
> >> would not work though as the `format` prefix is mandatory in the sources
> >> as only the properties with format will be passed to the format factory
> in
> >> majority of cases. We already have some implicit contracts.
> >>
> >> @Forward I did not necessarily get the example. Aren't json and bson two
> >> separate formats? Do you mean you can have those two at the same time?
> Why
> >> do you need to differentiate the options for each? The way I see it is:
> >>
> >> ‘format(.name)' = 'json',
> >> ‘format.fail-on-missing-field' = 'false'
> >>
> >> or
> >>
> >> ‘format(.name)' = 'bson',
> >> ‘format.fail-on-missing-field' = 'false'
> >>
> >> @Benchao I'd be fine with any of name, kind, type(this we already had in
> >> the past)
> >>
> >> Best,
> >> Dawid
> >>
> >> On 30/04/2020 04:17, Forward Xu wrote:
> >>
> >> Here I have a little doubt. At present, our json only supports the
> >> conventional json format. If we need to implement json with bson, json
> with
> >> avro, etc., how should we express it?
> >> Do you need like the following:
> >>
> >> ‘format.name' = 'json',
> >>
> >>

[GitHub] [flink-web] hequn8128 commented on a change in pull request #330: Add Apache Flink release 1.10.1

[GitBox]

hequn8128 commented on a change in pull request #330:
URL: https://github.com/apache/flink-web/pull/330#discussion_r419501350



##
File path: _config.yml
##
@@ -192,6 +192,10 @@ component_releases:
 
 release_archive:
 flink:
+  -
+version_short: "1.10"

Review comment:
   This is right but it would be better to make it consistent with others, 
i.e., remove the quotes.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #4

[Robert Metzger]

I think I conceptually added everything Stephan mentioned into the Wiki:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=97552147&selectedPageVersions=5&selectedPageVersions=3
I'm looking forward to any comments or corrections.

On Wed, Apr 1, 2020 at 12:20 PM Chesnay Schepler  wrote:

> Let's add this information to the licensing wiki page.
> https://cwiki.apache.org/confluence/display/FLINK/Licensing
>
>
> On 01/04/2020 12:16, Yu Li wrote:
> > Thanks Stephan, this is enlightening!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Wed, 1 Apr 2020 at 16:30, Stephan Ewen  wrote:
> >
> >> @Yu - there is nothing magic behind the license check, but I can share
> >> what I did there.
> >>
> >> Source distribution:
> >>- This means code copied into the repo.
> >>- The Java source code is usually fine, committers copying code
> verbatim
> >> usually comment on that directly.
> >>- Most important are other files, like docs (mostly build setup, not
> >> contents files), other html/web related code (like UIs), build files,
> etc.
> >> So specifically go through these files (there are usually not too many).
> >>
> >> Binary distribution:
> >>- To check the compatibility of licenses of transitive dependencies,
> >> maven generated a "DEPENDENCIES" file in the Jar that lists the
> >> dependencies by license, which is a very helpful start
> >>- Packaging wise, projects that build shaded jars are important,
> because
> >> they bundle dependencies, which means
> >>   (a) checking that the relevant licenses / notices are present
> >>   (b) checking that they don't bundle too much (jar tf  |
> >> grep ... | less) because transitive dependencies and mixing compile /
> >> provided is not straightforward in maven.
> >>
> >> Best,
> >> Stephan
> >>
> >>
> >>
> >> On Wed, Apr 1, 2020 at 4:47 AM Yu Li  wrote:
> >>
> >>> Hi Stephan,
> >>>
> >>> Could you also share the method of license check, so more people could
> >>> help in future votes? And maybe adding some instructions into our wiki
> [1]?
> >>> I skipped the licensing check in my vote because not aware of a good
> way to
> >>> do it thoroughly, not sure whether I'm the only one having such
> question
> >>> though. Thanks.
> >>>
> >>> btw, I noticed the fix version of FLINK-16891-16897 are all set to
> 2.0.0
> >>> but I guess it should be statefun-2.0.0 instead?
> >>>
> >>> Best Regards,
> >>> Yu
> >>>
> >>> [1] https://cwiki.apache.org/confluence/display/FLINK/Licensing
> >>>
> >>> On Wed, 1 Apr 2020 at 04:01, Stephan Ewen  wrote:
> >>>
>  I did a release check for license issues - all in all, we need a new
> RC.
> 
>  The only blocker I found was the missing jquery license file.
> 
>  Another somewhat critical thing is that "statefun-flink-distribution"
>  bundles many unwanted dependencies.
> - Because the shading merges the notice files, this is not a legal
>  issue.
> - Because Flinks inverted classloading still uses "parent-first"
> for
>  all
>  "org.apache.flink.*" classes, this does not break the system
>  But it is unwanted behavior and makes the artifacts unnecessarily
> large.
> 
>  I opened FLINK-16891 - FLINK-16897 for the issues I found.
>  All issues are fixed in this PR:
>  https://github.com/apache/flink-statefun/pull/85
> 
> 
> 
>  On Tue, Mar 31, 2020 at 7:17 PM Stephan Ewen 
> wrote:
> 
> > I have found a few things, am preparing a joint PR to fix them.
> >
> > So far, only the missing jquery license would have been a release
>  blocker.
> > On Tue, Mar 31, 2020 at 6:24 PM Chesnay Schepler  >
> > wrote:
> >
> >> The jquery license is in fact missing from the master/release-1.10
> >> branches. https://issues.apache.org/jira/browse/FLINK-16888
> >>
> >>
> >> On 31/03/2020 12:18, Chesnay Schepler wrote:
> >>> For Kafka we traditionally exclude the NOTICE file since as far as
>  we
> >>> can tell it is misleading anyway, see the flink-sql-connector-kafka
> >>> modules.
> >>>
> >>> @Robert for the Flink project the jquery license is in the source
> at
> >>> licenses/LICENSE.jquery
> >>>
> >>> I'm a bit concerned just how many licensing issues are showing up
> in
> >>> these RCs. I would suggest to do a proper scan of the licensing
>  before
> >>> opening another RC.
> >>>
> >>> And yes, the missing MIT license is grounds for cancellation,
>  hence, -1.
> >>> On 31/03/2020 11:56, Robert Metzger wrote:
>  Thanks a lot Gordon!
> 
>  Checked:
>  - files in the staging repository seem to be ok (no unexpected
>  files,
>  versions set correctly, quickstart archetype looks ok)
>  - statefun-ridesharing-example-simulator-2.0.0.jar (and
> 
> 
> /org/apache/flink/statefun-flink-distribution/2.0.0/statefun-flink-distribution-2.0.0.jar)
>  contains a 

Re: Presentation - Real World Architectural Patterns with Apache Pulsar and Flink

[Till Rohrmann]

Thanks for sharing your talk with the Flink community.

Cheers,
Till

On Tue, Apr 28, 2020 at 10:00 PM Devin Bost  wrote:

> You can also access the slide deck here:
> https://www.slideshare.net/DevinBost/realworld-pulsar-architectural-patterns
> [image: image.png]
> Devin G. Bost
>
>
> On Tue, Apr 28, 2020 at 1:09 PM Devin Bost  wrote:
>
>> If anyone missed my presentation on Real-World Architectural Patterns
>> with Apache Pulsar that covers a use case involving Apache *Flink* for
>> distributed tracing, please check out the recording here:
>> https://youtu.be/pmaCG1SHAW8
>>
>> Devin G. Bost
>>
>

[GitHub] [flink-web] morsapaes commented on pull request #333: [FLINK-17490] Add training page

[GitBox]

morsapaes commented on pull request #333:
URL: https://github.com/apache/flink-web/pull/333#issuecomment-623493486


   Thanks, David! I think this makes it more intuitive to find. 👌 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-17505) Merge small files produced by StreamingFileSink

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-17505:
--

 Summary: Merge small files produced by StreamingFileSink
 Key: FLINK-17505
 URL: https://issues.apache.org/jira/browse/FLINK-17505
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem
Affects Versions: 1.10.0
Reporter: Piotr Nowojski


This an alternative approach to FLINK-11499, to solve a problem of creating 
many small files with bulk formats in StreamingFileSink (which have to be 
rolled on checkpoint).

Merge based approach would require converting {{StreamingFileSink}} from a 
sink, to an operator, that would be working exactly as it’s working right now, 
with the same limitations (no support for arbitrary rolling policies for bulk 
formats), followed by another operator that would be tasked with merging small 
files in the background. 

In the long term we probably would like to have both merge operator and write 
ahead log solution (WAL described in FLINK-11499) as alternatives, as WAL would 
behave better if small files are more common, and merge operator could behave 
better if small files are rare (because of data skew for example).




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[RESULT][VOTE] FLIP-108: edit the Public API

[Yangze Guo]

Hi all,

The voting time for changing the Public API of FLIP-108[1] has passed.
I'm closing the vote now.

There were 3 + 1 votes, 3 of which are binding:

- Till (binding)
- Becket (binding)
- Yu Li (binding)
- Xintong Song (non-binding)


There were no -1 votes.

Thus, changes have been accepted. I'll update the FLIP doc accordingly.

[1] 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/quot-VOTE-FLIP-108-edit-the-Public-API-quot-td40995.html

Best,
Yangze Guo

[jira] [Created] (FLINK-17504) Update translation of Getting Started / Overview

[David Anderson (Jira)]

David Anderson created FLINK-17504:
--

 Summary: Update translation of Getting Started / Overview
 Key: FLINK-17504
 URL: https://issues.apache.org/jira/browse/FLINK-17504
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation
Reporter: David Anderson


getting-started/index.zh.md is out-of-date



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink-shaded] piyushnarang commented on pull request #85: [FLINK-16955] Bump Zookeeper 3.4.X to 3.4.14

[GitBox]

piyushnarang commented on pull request #85:
URL: https://github.com/apache/flink-shaded/pull/85#issuecomment-623458136


   Ok, I've already done that degree of sanity checking and ran a few of the 
tests on my machine locally. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] carp84 commented on pull request #330: Add Apache Flink release 1.10.1

[GitBox]

carp84 commented on pull request #330:
URL: https://github.com/apache/flink-web/pull/330#issuecomment-623447337


   Thanks for the review @hequn8128 . All comments addressed, PTAL. Thanks.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] carp84 commented on a change in pull request #330: Add Apache Flink release 1.10.1

[GitBox]

carp84 commented on a change in pull request #330:
URL: https://github.com/apache/flink-web/pull/330#discussion_r419413870



##
File path: _posts/2020-05-05-release-1.10.1.md
##
@@ -0,0 +1,370 @@
+---
+layout: post
+title:  "Apache Flink 1.10.1 Released"
+date:   2020-05-05 12:00:00
+categories: news
+authors:
+- liyu:
+  name: "Yu Li"
+  twitter: "LiyuApache"
+---
+
+The Apache Flink community released the first bugfix version of the Apache 
Flink 1.10 series.
+
+This release includes 143 fixes and minor improvements for Flink 1.10.0. The 
list below includes a detailed list of all fixes and improvements.

Review comment:
   good catch.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-shaded] zentol commented on pull request #85: [FLINK-16955] Bump Zookeeper 3.4.X to 3.4.14

[GitBox]

zentol commented on pull request #85:
URL: https://github.com/apache/flink-shaded/pull/85#issuecomment-623430510


   @piyushnarang There is unfortunately no _easy_ way of doing that.
   I'd suggest to modify `tools/ci/maven-utils.sh` in the Flink project to 
checkout your branch, install it, and modifying the version property of Flink.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] alpinegizmo commented on pull request #333: [FLINK-17490] Add training page

[GitBox]

alpinegizmo commented on pull request #333:
URL: https://github.com/apache/flink-web/pull/333#issuecomment-623427987


   @morsapaes @XBaith I've changed it, based on your feedback. 
   
   
![image](https://user-images.githubusercontent.com/43608/80964626-82e73480-8e11-11ea-85f6-b9d2d920bef0.png)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-17503) Make memory configuration logging more user-friendly

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-17503:
-

 Summary: Make memory configuration logging more user-friendly
 Key: FLINK-17503
 URL: https://issues.apache.org/jira/browse/FLINK-17503
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.11.0, 1.10.2


The newly introduced memory configuration logs some output:

{code}
2020-05-04 11:50:05,984 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option Key: 'taskmanager.cpu.cores' , default: null (fallback 
keys: []) required for local execution is not set, setting it to its default 
value 1.7976931348623157E308
2020-05-04 11:50:05,989 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option Key: 'taskmanager.memory.task.heap.size' , default: null 
(fallback keys: []) required for local execution is not set, setting it to its 
default value 9223372036854775807 bytes
2020-05-04 11:50:05,989 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option Key: 'taskmanager.memory.task.off-heap.size' , default: 0 
bytes (fallback keys: []) required for local execution is not set, setting it 
to its default value 9223372036854775807 bytes
2020-05-04 11:50:05,990 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option Key: 'taskmanager.memory.network.min' , default: 64 mb 
(fallback keys: [{key=taskmanager.network.memory.min, isDeprecated=true}]) 
required for local execution is not set, setting it to its default value 64 mb
2020-05-04 11:50:05,990 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option Key: 'taskmanager.memory.network.max' , default: 1 gb 
(fallback keys: [{key=taskmanager.network.memory.max, isDeprecated=true}]) 
required for local execution is not set, setting it to its default value 64 mb
2020-05-04 11:50:05,991 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option Key: 'taskmanager.memory.managed.size' , default: null 
(fallback keys: [{key=taskmanager.memory.size, isDeprecated=true}]) required 
for local execution is not set, setting it to its default value 128 mb
{code}

This logging output could be made more user-friendly the following way:

* Print only the key string of a {{ConfigOption}}, not the config option object 
with all the deprecated keys
* Skipping the lines for {{taskmanager.memory.task.heap.size}} and 
{{taskmanager.memory.task.off-heap.size}} - we don't really set them (they are 
JVM paramaters) and the printing of long max looks strange (user would have to 
know these are place holders without effect).
* Maybe similarly skipping the CPU cores value, this looks the strangest 
(double max).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Adding support for Hadoop 3 and removing flink-shaded-hadoop

[Chesnay Schepler]

@Konstantin

It is part a) since it is not used by all hadoop users (as they may be 
using their existing hadoop infrastructure or one provided by the cloud 
service), but I'd say it's mostly for maintenance reasons.


The reality is that we cannot truly maintain flink-shaded-hadoop.
There are too many different versions, with too many features, that are 
too old, to make any change with any form of guarantee that things still 
work afterwards. This will only get worse when adding another major version.
I'm currently actively avoiding making any changes because I have no way 
of properly testing the correctness.


@Ufuk

1)
Initially I thought about keeping flink-shaded-hadoop around, but there 
actually isn't really a reason to do so.
Users can build their own version in any case with the existing source 
releases.


3)

This is where things get finicky.

At the end of the day the user will have to deal with it; be it by 
creating custom Hadoop/Flink distributions, updating to later versions, 
raising hell in the Hadoop/Flink JIRAs, ...


We can maybe turn affected components into a plugin.

We should still make sure that Flink works with Hadoop, just not that it 
works against all versions in the last 5 years.


As for "expected" conflicts; like, the thing is that who knows how 
things look like in 2 years.


On 04/05/2020 12:24, Ufuk Celebi wrote:

Hey Robert and others,

overall +1 to support Hadoop 3. It would be a great to unblock Flink
support in EMR 6.0 as noted in the linked FLINK ticket.

The arguments raised against flink-shaded-hadoop make sense to me. I have a
few general questions still:

1) Will the flink-shaded-hadoop module (in apache/flink-shaded) be fully
dropped after this change? Or do you plan to keep it (allowing users to
build their own shaded Hadoop if needed)?

2) I find Stephan's ideas pretty interesting. Will there be an official
follow-up to investigate those?

3) What will we tell users that run into class loading conflicts after this
change? What are actually the "expected" conflicts we might see?

– Ufuk

PS: Robert opened a draft PR here:
https://github.com/apache/flink/pull/11983


On Sun, May 3, 2020 at 12:02 PM Konstantin Knauf  wrote:


Hi Chesnay, Hi Robert,

I have a bit of a naive question. I assume the reason for introducing
flink-shaded-hadoop were dependency conflicts between Hadoop, Flink and/or
user code. When we drop it now is it because

a) it was not worth it (value provided did not justify maintenance overhead
and issues introduced)
b) we don't think it is a problem anymore
c) prioritizes have shifted and it *now *not worth it anymore
d) something else

Cheers,

Konstantin

On Sun, Apr 26, 2020 at 10:25 PM Stephan Ewen  wrote:


Indeed, that would be the assumption, that Hadoop does not expose its
transitive libraries on its public API surface.

 From vague memory, I think that pretty much true so far. I only remember
Kinesis and Calcite as counter examples, who exposed Guava classes as

part

of the public API.
But that is definitely the "weak spot" of this approach. Plus, as with

all

custom class loaders, the fact that the Thread Context Class Loader does
not really work well any more.

On Thu, Apr 23, 2020 at 11:50 AM Chesnay Schepler 
wrote:


This would only work so long as all Hadoop APIs do not directly expose
any transitive non-hadoop dependency.
Otherwise the user code classloader might search for this transitive
dependency in lib instead of the hadoop classpath (and possibly not

find

it).

On 23/04/2020 11:34, Stephan Ewen wrote:

True, connectors built on Hadoop make this a bit more complex. That

is

also

the reason why Hadoop is on the "parent first" patterns.

Maybe this is a bit of a wild thought, but what would happen if we

had

a

"first class" notion of a Hadoop Classloader in the system, and the

user

code classloader would explicitly fall back to that one whenever a

class

whose name starts with "org.apache.hadoop" is not found? We could

also

generalize this by associating plugin loaders with class name

prefixes.

Then it would try to load from the user code jar, and if the class

was

not

found, load it from the hadoop classpath.

On Thu, Apr 23, 2020 at 10:56 AM Chesnay Schepler <

ches...@apache.org>

wrote:


although, if you can load the HADOOP_CLASSPATH as a plugin, then you

can

also load it in the user-code classloader.

On 23/04/2020 10:50, Chesnay Schepler wrote:

@Stephan I'm not aware of anyone having tried that; possibly since

we

have various connectors that require hadoop (hadoop-compat, hive,
orc/parquet/hbase, hadoop inputformats). This would require

connectors

to be loaded as plugins (or having access to the plugin

classloader)

to be feasible.

On 23/04/2020 09:59, Stephan Ewen wrote:

Hi all!

+1 for the simplification of dropping hadoop-shaded


Have we ever investigated how much work it would be to load the
HADOOP_CLASSPATH through the plugin loader? Then Hadoop's crazy
dependency
footprint would no

Re: [DISCUSS] flink-connector-rabbitmq api changes

[seneg...@gmail.com]

Hi,

Okay i created a ticket: https://issues.apache.org/jira/browse/FLINK-17502

i will work on the modifications "keeping the old constructor" and brush up
on the contribution guides and move from there :)

Regards,
Karim Mansour

On Mon, May 4, 2020 at 10:00 AM Aljoscha Krettek 
wrote:

> Yes, that's what I was proposing!
>
> @Karim If there's not already a Jira issue, please create one. You can
> ping me, so that I can assign you.
>
> @Austin There's a Jira component for the RMQ source, maybe you can take
> a stab at some of the issues there:
>
> https://issues.apache.org/jira/browse/FLINK-17204?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Connectors%2F%20RabbitMQ%22%20AND%20statusCategory%20!%3D%20Done
> .
>
> Best,
> Aljoscha
>
> On 03.05.20 16:38, seneg...@gmail.com wrote:
> > Hi,
> >
> > Okay so keep the current constructors as is, create new ones with more
> > granular parsing of the results. Sounds like a good plan.
> >
> > How do we proceed from here ?
> >
> > Regards,
> > Karim Mansour
> >
> > On Fri, May 1, 2020 at 5:03 PM Austin Cawley-Edwards <
> > austin.caw...@gmail.com> wrote:
> >
> >> Hey,
> >>
> >> (Switching to my personal email)
> >>
> >> Correct me if I'm wrong, but I think Aljoscha is proposing keeping the
> >> public API as is, and adding some new constructors/ custom
> deserialization
> >> schemas as was done with Kafka. Here's what I was able to find on that
> >> feature:
> >>
> >> * https://issues.apache.org/jira/browse/FLINK-8354
> >> *
> >>
> >>
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaDeserializationSchema.java
> >> *
> >>
> >>
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer011.java#L100-L114
> >>
> >> Best,
> >> Austin
> >>
> >> On Fri, May 1, 2020 at 6:19 AM seneg...@gmail.com 
> >> wrote:
> >>
> >>> Hello,
> >>>
> >>> So the proposal is to keep the current RMQSource constructors /  public
> >> api
> >>> as is and create new ones that gives more granular parsing ?
> >>>
> >>> Regards,
> >>> Karim Mansour
> >>>
> >>> On Thu, Apr 30, 2020 at 5:23 PM Austin Cawley-Edwards <
> >>> aus...@fintechstudios.com> wrote:
> >>>
>  Hey all + thanks Konstantin,
> 
>  Like mentioned, we also run into issues with the RMQ Source
> >>> inflexibility.
>  I think Aljoscha's idea of supporting both would be a nice way to
>  incorporate new changes without breaking the current API.
> 
>  We'd definitely benefit from the changes proposed here but have
> another
>  issue with the Correlation ID. When a message gets in the queue
> >> without a
>  correlation ID, the source errors and the job cannot recover,
> requiring
>  (painful) manual intervention. It would be nice to be able to
> >> dead-letter
>  these inputs from the source, but I don't think that's possible with
> >> the
>  current source interface (don't know too much about the source
> >>> specifics).
>  We might be able to work around this with a custom Correlation ID
>  extractor, as proposed by Karim.
> 
>  Also, if there are other tickets in the RMQ integrations that have
> gone
>  unmaintained, I'm also happy to chip it at maintaining them!
> 
>  Best,
>  Austin
>  
>  From: Konstantin Knauf 
>  Sent: Thursday, April 30, 2020 6:14 AM
>  To: dev 
>  Cc: Austin Cawley-Edwards 
>  Subject: Re: [DISCUSS] flink-connector-rabbitmq api changes
> 
>  Hi everyone,
> 
>  just looping in Austin as he mentioned that they also ran into issues
> >> due
>  to the inflexibility of the RabiitMQSourcce to me yesterday.
> 
>  Cheers,
> 
>  Konstantin
> 
>  On Thu, Apr 30, 2020 at 11:23 AM seneg...@gmail.com  seneg...@gmail.com> mailto:seneg...@gmail.com>>
> >>> wrote:
>  Hello Guys,
> 
>  Thanks for all the responses, i want to stress out that i didn't feel
>  ignored i just thought that i forgot an important step or something.
> 
>  Since i am a newbie i would follow whatever route you guys would
> >> suggest
> >>> :)
>  and i agree that the RMQ connector needs a lot of love still "which i
> >>> would
>  be happy to submit gradually"
> 
>  as for the code i have it here in the PR:
>  https://github.com/senegalo/flink/pull/1 it's not that much of a
> >> change
> >>> in
>  terms of logic but more of what is exposed.
> 
>  Let me know how you want me to proceed.
> 
>  Thanks again,
>  Karim Mansour
> 
>  On Thu, Apr 30, 2020 at 10:40 AM Aljoscha Krettek <
> aljos...@apache.org
>  >
>  wrote:
> 
> > Hi,
> >
> > I think it's good to contribute the changes to Flink directly since
> >> we
> > already have the RMQ

[jira] [Created] (FLINK-17502) More granular source parsing

[Karim Mansour (Jira)]

Karim Mansour created FLINK-17502:
-

 Summary: More granular source parsing
 Key: FLINK-17502
 URL: https://issues.apache.org/jira/browse/FLINK-17502
 Project: Flink
  Issue Type: New Feature
  Components: Connectors/ RabbitMQ
Reporter: Karim Mansour


Currently the {{RMQSource}} is extracting the body of the message which is a 
byte array and pass it to a an instance of a user implementation of the 
{{DeserializationSchema}} class to deserialize the body of the message. It also 
uses the correlation id from the message properties to deduplicate the message.

What will be done is creating a new {{RMQSource}} constructor that is instead 
of taking a implementation of a {{DeserializationSchema}} in the {{RMQSource}} 
constructor, actually have the user implement an interface that would have 
methods to extract both the correlation id and message not only from the body 
of the message but also from it's metadata and properties thus giving the 
connector much more power and flexibility.
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17501) Improve logging in AbstractServerHandler#channelRead(ChannelHandlerContext, Object)

[Gary Yao (Jira)]

Gary Yao created FLINK-17501:


 Summary: Improve logging in 
AbstractServerHandler#channelRead(ChannelHandlerContext, Object)
 Key: FLINK-17501
 URL: https://issues.apache.org/jira/browse/FLINK-17501
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Queryable State
Reporter: Gary Yao
Assignee: Gary Yao
 Fix For: 1.11.0


Improve logging in {{AbstractServerHandler#channelRead(ChannelHandlerContext, 
Object)}}. If an Error is thrown, it should be logged as early as possible. 
Currently we try to serialize and send an error response to the client before 
logging the error; this can fail and mask the original exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Adding support for Hadoop 3 and removing flink-shaded-hadoop

[Ufuk Celebi]

Hey Robert and others,

overall +1 to support Hadoop 3. It would be a great to unblock Flink
support in EMR 6.0 as noted in the linked FLINK ticket.

The arguments raised against flink-shaded-hadoop make sense to me. I have a
few general questions still:

1) Will the flink-shaded-hadoop module (in apache/flink-shaded) be fully
dropped after this change? Or do you plan to keep it (allowing users to
build their own shaded Hadoop if needed)?

2) I find Stephan's ideas pretty interesting. Will there be an official
follow-up to investigate those?

3) What will we tell users that run into class loading conflicts after this
change? What are actually the "expected" conflicts we might see?

– Ufuk

PS: Robert opened a draft PR here:
https://github.com/apache/flink/pull/11983


On Sun, May 3, 2020 at 12:02 PM Konstantin Knauf  wrote:

> Hi Chesnay, Hi Robert,
>
> I have a bit of a naive question. I assume the reason for introducing
> flink-shaded-hadoop were dependency conflicts between Hadoop, Flink and/or
> user code. When we drop it now is it because
>
> a) it was not worth it (value provided did not justify maintenance overhead
> and issues introduced)
> b) we don't think it is a problem anymore
> c) prioritizes have shifted and it *now *not worth it anymore
> d) something else
>
> Cheers,
>
> Konstantin
>
> On Sun, Apr 26, 2020 at 10:25 PM Stephan Ewen  wrote:
>
> > Indeed, that would be the assumption, that Hadoop does not expose its
> > transitive libraries on its public API surface.
> >
> > From vague memory, I think that pretty much true so far. I only remember
> > Kinesis and Calcite as counter examples, who exposed Guava classes as
> part
> > of the public API.
> > But that is definitely the "weak spot" of this approach. Plus, as with
> all
> > custom class loaders, the fact that the Thread Context Class Loader does
> > not really work well any more.
> >
> > On Thu, Apr 23, 2020 at 11:50 AM Chesnay Schepler 
> > wrote:
> >
> > > This would only work so long as all Hadoop APIs do not directly expose
> > > any transitive non-hadoop dependency.
> > > Otherwise the user code classloader might search for this transitive
> > > dependency in lib instead of the hadoop classpath (and possibly not
> find
> > > it).
> > >
> > > On 23/04/2020 11:34, Stephan Ewen wrote:
> > > > True, connectors built on Hadoop make this a bit more complex. That
> is
> > > also
> > > > the reason why Hadoop is on the "parent first" patterns.
> > > >
> > > > Maybe this is a bit of a wild thought, but what would happen if we
> had
> > a
> > > > "first class" notion of a Hadoop Classloader in the system, and the
> > user
> > > > code classloader would explicitly fall back to that one whenever a
> > class
> > > > whose name starts with "org.apache.hadoop" is not found? We could
> also
> > > > generalize this by associating plugin loaders with class name
> prefixes.
> > > >
> > > > Then it would try to load from the user code jar, and if the class
> was
> > > not
> > > > found, load it from the hadoop classpath.
> > > >
> > > > On Thu, Apr 23, 2020 at 10:56 AM Chesnay Schepler <
> ches...@apache.org>
> > > > wrote:
> > > >
> > > >> although, if you can load the HADOOP_CLASSPATH as a plugin, then you
> > can
> > > >> also load it in the user-code classloader.
> > > >>
> > > >> On 23/04/2020 10:50, Chesnay Schepler wrote:
> > > >>> @Stephan I'm not aware of anyone having tried that; possibly since
> we
> > > >>> have various connectors that require hadoop (hadoop-compat, hive,
> > > >>> orc/parquet/hbase, hadoop inputformats). This would require
> > connectors
> > > >>> to be loaded as plugins (or having access to the plugin
> classloader)
> > > >>> to be feasible.
> > > >>>
> > > >>> On 23/04/2020 09:59, Stephan Ewen wrote:
> > >  Hi all!
> > > 
> > >  +1 for the simplification of dropping hadoop-shaded
> > > 
> > > 
> > >  Have we ever investigated how much work it would be to load the
> > >  HADOOP_CLASSPATH through the plugin loader? Then Hadoop's crazy
> > >  dependency
> > >  footprint would not spoil the main classpath.
> > > 
> > >  - HDFS might be very simple, because file systems are already
> > >  Plugin aware
> > >  - Yarn would need some extra work. In essence, we would need
> to
> > >  discover
> > >  executors also through plugins
> > >  - Kerberos is the other remaining bit. We would need to switch
> > >  security
> > >  modules to ServiceLoaders (which we should do anyways) and also
> pull
> > >  them
> > >  from plugins.
> > > 
> > >  Best,
> > >  Stephan
> > > 
> > > 
> > > 
> > >  On Thu, Apr 23, 2020 at 4:05 AM Xintong Song <
> tonysong...@gmail.com
> > >
> > >  wrote:
> > > 
> > > > +1 for supporting Hadoop 3.
> > > >
> > > > I'm not familiar with the shading efforts, thus no comment on
> > > > dropping the
> > > > flink-shaded-hadoop.
> > > >
> > > >
> > > > Corr

[jira] [Created] (FLINK-17500) Deploy JobGraph form file in StandaloneClusterEntrypoint

[Ufuk Celebi (Jira)]

Ufuk Celebi created FLINK-17500:
---

 Summary: Deploy JobGraph form file in StandaloneClusterEntrypoint
 Key: FLINK-17500
 URL: https://issues.apache.org/jira/browse/FLINK-17500
 Project: Flink
  Issue Type: Wish
  Components: Deployment / Docker
Reporter: Ufuk Celebi


We have a requirement to deploy a pre-generated {{JobGraph}} from a file in 
{{StandaloneClusterEntrypoint}}.

Currently, {{StandaloneClusterEntrypoint}} only supports deployment of a Flink 
job from the class path using {{ClassPathPackagedProgramRetriever}}. 

Our desired behaviour would be as follows:

If {{internal.jobgraph-path}} is set, prepare a {{PackagedProgram}} from a 
local {{JobGraph}} file using {{FileJobGraphRetriever}}. Otherwise, deploy 
using {{ClassPathPackagedProgramRetriever}} (current behavior).

---

I understand that this requirement is pretty niche, but wanted to get feedback 
whether the Flink community would be open to supporting this nonetheless.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] flink-connector-rabbitmq api changes

[Aljoscha Krettek]

Yes, that's what I was proposing!

@Karim If there's not already a Jira issue, please create one. You can 
ping me, so that I can assign you.


@Austin There's a Jira component for the RMQ source, maybe you can take 
a stab at some of the issues there: 
https://issues.apache.org/jira/browse/FLINK-17204?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Connectors%2F%20RabbitMQ%22%20AND%20statusCategory%20!%3D%20Done.


Best,
Aljoscha

On 03.05.20 16:38, seneg...@gmail.com wrote:

Hi,

Okay so keep the current constructors as is, create new ones with more
granular parsing of the results. Sounds like a good plan.

How do we proceed from here ?

Regards,
Karim Mansour

On Fri, May 1, 2020 at 5:03 PM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:


Hey,

(Switching to my personal email)

Correct me if I'm wrong, but I think Aljoscha is proposing keeping the
public API as is, and adding some new constructors/ custom deserialization
schemas as was done with Kafka. Here's what I was able to find on that
feature:

* https://issues.apache.org/jira/browse/FLINK-8354
*

https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaDeserializationSchema.java
*

https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer011.java#L100-L114

Best,
Austin

On Fri, May 1, 2020 at 6:19 AM seneg...@gmail.com 
wrote:


Hello,

So the proposal is to keep the current RMQSource constructors /  public

api

as is and create new ones that gives more granular parsing ?

Regards,
Karim Mansour

On Thu, Apr 30, 2020 at 5:23 PM Austin Cawley-Edwards <
aus...@fintechstudios.com> wrote:


Hey all + thanks Konstantin,

Like mentioned, we also run into issues with the RMQ Source

inflexibility.

I think Aljoscha's idea of supporting both would be a nice way to
incorporate new changes without breaking the current API.

We'd definitely benefit from the changes proposed here but have another
issue with the Correlation ID. When a message gets in the queue

without a

correlation ID, the source errors and the job cannot recover, requiring
(painful) manual intervention. It would be nice to be able to

dead-letter

these inputs from the source, but I don't think that's possible with

the

current source interface (don't know too much about the source

specifics).

We might be able to work around this with a custom Correlation ID
extractor, as proposed by Karim.

Also, if there are other tickets in the RMQ integrations that have gone
unmaintained, I'm also happy to chip it at maintaining them!

Best,
Austin

From: Konstantin Knauf 
Sent: Thursday, April 30, 2020 6:14 AM
To: dev 
Cc: Austin Cawley-Edwards 
Subject: Re: [DISCUSS] flink-connector-rabbitmq api changes

Hi everyone,

just looping in Austin as he mentioned that they also ran into issues

due

to the inflexibility of the RabiitMQSourcce to me yesterday.

Cheers,

Konstantin

On Thu, Apr 30, 2020 at 11:23 AM seneg...@gmail.com mailto:seneg...@gmail.com>>

wrote:

Hello Guys,

Thanks for all the responses, i want to stress out that i didn't feel
ignored i just thought that i forgot an important step or something.

Since i am a newbie i would follow whatever route you guys would

suggest

:)

and i agree that the RMQ connector needs a lot of love still "which i

would

be happy to submit gradually"

as for the code i have it here in the PR:
https://github.com/senegalo/flink/pull/1 it's not that much of a

change

in

terms of logic but more of what is exposed.

Let me know how you want me to proceed.

Thanks again,
Karim Mansour

On Thu, Apr 30, 2020 at 10:40 AM Aljoscha Krettek mailto:aljos...@apache.org>>
wrote:


Hi,

I think it's good to contribute the changes to Flink directly since

we

already have the RMQ connector in the respository.

I would propose something similar to the Kafka connector, which takes
both the generic DeserializationSchema and a

KafkaDeserializationSchema

that is specific to Kafka and allows access to the ConsumerRecord and
therefore all the Kafka features. What do you think about that?

Best,
Aljoscha

On 30.04.20 10:26, Robert Metzger wrote:

Hey Karim,

I'm sorry that you had such a bad experience contributing to Flink,

even

though you are nicely following the rules.

You mentioned that you've implemented the proposed change already.

Could

you share a link to a branch here so that we can take a look? I can

assess

the API changes easier if I see them :)

Thanks a lot!


Best,
Robert

On Thu, Apr 30, 2020 at 8:09 AM Dawid Wysakowicz <

dwysakow...@apache.org


wrote:


Hi Karim,

Sorry you did not have the best first time experience. You

certainly

did

everything right which I definitely appreciate.

The problem in that particular case, as I see it, is that RabbitMQ

is

not very activ