Re: [VOTE] FLIP-172: Support custom transactional.id prefix in FlinkKafkaProducer

2021-07-08 Thread Dawid Wysakowicz 
+1 (binding)

Best,

Dawid

On 28/06/2021 16:00, Wenhao Ji wrote:
> Hi everyone,
>
> I would like to start a vote on FLIP-172 [1] which was discussed in
> this thread [2].
> The vote will be open for at least 72 hours until July 1 unless there
> is an objection or not enough votes.
>
> Thanks,
> Wenhao
>
> [1] 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-172%3A+Support+custom+transactional.id+prefix+in+FlinkKafkaProducer
> [2] 
> https://lists.apache.org/thread.html/r67610aa2d4dfdaf3b027b82edd1a3f46771f0d58902a4258d931e5a5%40%3Cdev.flink.apache.org%3E



OpenPGP_signature
Description: OpenPGP digital signature

Re: [VOTE] FLIP-172: Support custom transactional.id prefix in FlinkKafkaProducer

2021-07-08 Thread Yuan Mei 
+1

Best,
Yuan

On Thu, Jul 8, 2021 at 3:03 PM Dawid Wysakowicz 
wrote:

> +1 (binding)
>
> Best,
>
> Dawid
>
> On 28/06/2021 16:00, Wenhao Ji wrote:
> > Hi everyone,
> >
> > I would like to start a vote on FLIP-172 [1] which was discussed in
> > this thread [2].
> > The vote will be open for at least 72 hours until July 1 unless there
> > is an objection or not enough votes.
> >
> > Thanks,
> > Wenhao
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-172%3A+Support+custom+transactional.id+prefix+in+FlinkKafkaProducer
> > [2]
> https://lists.apache.org/thread.html/r67610aa2d4dfdaf3b027b82edd1a3f46771f0d58902a4258d931e5a5%40%3Cdev.flink.apache.org%3E
>
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-08 Thread Roman Khachatryan 
Congratulations Yuan!

Regards,
Roman

On Thu, Jul 8, 2021 at 6:02 AM Yang Wang  wrote:
>
> Congratulations Yuan!
>
> Best,
> Yang
>
> XING JIN  于2021年7月8日周四 上午11:46写道：
>
> > Congratulations Yuan~!
> >
> > Roc Marshal  于2021年7月8日周四 上午11:28写道：
> >
> > > Congratulations, Yuan!
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > At 2021-07-08 01:21:40, "Yu Li"  wrote:
> > > >Hi all,
> > > >
> > > >On behalf of the PMC, I’m very happy to announce Yuan Mei as a new Flink
> > > >committer.
> > > >
> > > >Yuan has been an active contributor for more than two years, with code
> > > >contributions on multiple components including kafka connectors,
> > > >checkpointing, state backends, etc. Besides, she has been actively
> > > involved
> > > >in community activities such as helping manage releases, discussing
> > > >questions on dev@list, supporting users and giving talks at
> > conferences.
> > > >
> > > >Please join me in congratulating Yuan for becoming a Flink committer!
> > > >
> > > >Cheers,
> > > >Yu
> > >
> >

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-08 Thread Jiayi Liao 
Congratulations Yuan!

Best,
Jiayi Liao

On Thu, Jul 8, 2021 at 3:55 PM Roman Khachatryan  wrote:

> Congratulations Yuan!
>
> Regards,
> Roman
>
> On Thu, Jul 8, 2021 at 6:02 AM Yang Wang  wrote:
> >
> > Congratulations Yuan!
> >
> > Best,
> > Yang
> >
> > XING JIN  于2021年7月8日周四 上午11:46写道：
> >
> > > Congratulations Yuan~!
> > >
> > > Roc Marshal  于2021年7月8日周四 上午11:28写道：
> > >
> > > > Congratulations, Yuan!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > At 2021-07-08 01:21:40, "Yu Li"  wrote:
> > > > >Hi all,
> > > > >
> > > > >On behalf of the PMC, I’m very happy to announce Yuan Mei as a new
> Flink
> > > > >committer.
> > > > >
> > > > >Yuan has been an active contributor for more than two years, with
> code
> > > > >contributions on multiple components including kafka connectors,
> > > > >checkpointing, state backends, etc. Besides, she has been actively
> > > > involved
> > > > >in community activities such as helping manage releases, discussing
> > > > >questions on dev@list, supporting users and giving talks at
> > > conferences.
> > > > >
> > > > >Please join me in congratulating Yuan for becoming a Flink
> committer!
> > > > >
> > > > >Cheers,
> > > > >Yu
> > > >
> > >
>

[jira] [Created] (FLINK-23305) Support semi/anti interval joins

2021-07-08 Thread Timo Walther (Jira) 
Timo Walther created FLINK-23305:


 Summary: Support semi/anti interval joins
 Key: FLINK-23305
 URL: https://issues.apache.org/jira/browse/FLINK-23305
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Planner
Reporter: Timo Walther


I don't see a reason why we shouldn't support interval joins also for semi/anti 
joins like:

{code}
SELECT *
FROM OT
WHERE EXISTS (
SELECT *
FROM TT
WHERE TT.tx = OT.tx AND
TT.isEnd = TRUE AND
TT.rowtime BETWEEN OT.rowtime AND OT.rowtime + INTERVAL '1' HOUR)
{code}

The resulting plan contains a join operation anyway but without detecting the 
interval.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23306) FlinkRelMdUniqueKeys causes exception when used with new Schema

2021-07-08 Thread Timo Walther (Jira) 
Timo Walther created FLINK-23306:


 Summary: FlinkRelMdUniqueKeys causes exception when used with new 
Schema
 Key: FLINK-23306
 URL: https://issues.apache.org/jira/browse/FLINK-23306
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Timo Walther
Assignee: Timo Walther


FlinkRelMdUniqueKeys should not use the deprecated `TableSchema`. It causes 
exceptions when e.g. {{sourceWatermark()}} is used in schema.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23307) Rowtime attributes are not properly resolved for views

2021-07-08 Thread Jira 
Ingo Bürk created FLINK-23307:
-

 Summary: Rowtime attributes are not properly resolved for views
 Key: FLINK-23307
 URL: https://issues.apache.org/jira/browse/FLINK-23307
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.13.1
Reporter: Ingo Bürk


If a view is contains a rowtime attribute from an underlying table and is then 
used in a temporal join, an error is thrown
{quote}Temporal table join currently only supports 'FOR SYSTEM_TIME AS OF' left 
table's time attribute field
{quote}
This does not happen with the generic in-memory catalog, because it copies the 
schema as-is. However, if a catalog implementation is used which persists the 
schema without the rowtime information (which is correct, since this is defined 
by the underlying table and not the view itself), the catalog can only return 
said schema for a view. This then causes this issue during planning.

Specifically, this happens in SqlCatalogViewTable#convertToRel. After the call 
to context#expandView, the rowtime attribute is correctly present, but the 
inserted cast from RelOptUtil#createCastRel throws this information away.

The following SQL reproduces the issue. Again, please note that this does NOT 
work with the default in-memory catalog:
{code:java}
CREATE TABLE A (
id INT,
ts TIMESTAMP(3),
WATERMARK FOR ts AS ts,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'datagen'
);

CREATE VIEW B AS SELECT * FROM A;

SELECT * FROM B JOIN A FOR SYSTEM_TIME AS OF B.ts ON B.id = A.id;
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23308) Performance regression on 06.07

2021-07-08 Thread Dawid Wysakowicz (Jira) 
Dawid Wysakowicz created FLINK-23308:


 Summary: Performance regression on 06.07
 Key: FLINK-23308
 URL: https://issues.apache.org/jira/browse/FLINK-23308
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 1.14.0
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.14.0


http://codespeed.dak8s.net:8000/timeline/#/?exe=1,3&ben=twoInputMapSink&env=2&revs=200&equid=off&quarts=on&extr=on
http://codespeed.dak8s.net:8000/timeline/?ben=readFileSplit&env=2



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23309) Optimize the finish bundle logic in PyFlink

2021-07-08 Thread Huang Xingbo (Jira) 
Huang Xingbo created FLINK-23309:


 Summary: Optimize the finish bundle logic in PyFlink
 Key: FLINK-23309
 URL: https://issues.apache.org/jira/browse/FLINK-23309
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Reporter: Huang Xingbo
Assignee: Huang Xingbo
 Fix For: 1.14.0


Whenever the `bundle size` or `bundle time` is reached, the data in the input 
buffer needs to be sent from the jvm to the pvm, and then waits for the pym to 
be processed and sent back to the jvm to send all the results to the downstream 
operator, which leads to a large delay in current implementation of main thread 
triggering `finish bundle`, especially when it is a small size event as small 
messages are hard to processed in pipeline.

We need to move the logic of `finish bundle` from the main thread to the 
asynchronous sending thread, so as to avoid the problem of the main thread 
being blocked.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23310) Correct the `ModifyKindSetTrait` for `GroupWindowAggregate` when the input is an update stream

2021-07-08 Thread Shuo Cheng (Jira) 
Shuo Cheng created FLINK-23310:
--

 Summary: Correct the `ModifyKindSetTrait` for 
`GroupWindowAggregate` when the input is an update stream 
 Key: FLINK-23310
 URL: https://issues.apache.org/jira/browse/FLINK-23310
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.13.0
Reporter: Shuo Cheng
 Fix For: 1.14.0


Following FLINK-22781, currently group window supports update input stream, 
just like unbounded aggregate, group window may also emit DELETE records, so 
the `ModifyKindSetTrait` for group window should be modified as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23311) Improve PojoSerializer test

2021-07-08 Thread Nico Kruber (Jira) 
Nico Kruber created FLINK-23311:
---

 Summary: Improve PojoSerializer test
 Key: FLINK-23311
 URL: https://issues.apache.org/jira/browse/FLINK-23311
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.13.1
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.14.0


While working with the PojoSerializer a bit more, I noticed a couple of minor 
things that are off in the current tests:
- the test Pojo does not take {{dumm5}} into account for {{hashCode}} and 
{{equals}}
- error messages are not so nice (and mix up the order of expected and actual 
values)

I'll create a PR for fixing these things in one go under this ticket.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23312) Use -Dfast for building e2e tests on AZP

2021-07-08 Thread Nico Kruber (Jira) 
Nico Kruber created FLINK-23312:
---

 Summary: Use -Dfast for building e2e tests on AZP
 Key: FLINK-23312
 URL: https://issues.apache.org/jira/browse/FLINK-23312
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Affects Versions: 1.13.1
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.14.0


The "e2e" builder in Azure pipelines builds Flink again on top of what the 
"compile" builder is already doing. This unnecessary duplicates a couple of 
checks that are enough to execute once and can be skipped via providing 
{{-Dfast}}.

On my local machine with 32GB RAM, 8 physical cores and a fast NVMe SSD, the 
difference is pretty big:
{code}
time mvn clean install -Dscala-2.12 -DskipTests -pl flink-dist -am
# -> 6:40 min

time mvn clean install -Dscala-2.12 -DskipTests -Dfast -pl flink-dist -am
# -> 5:40 min
{code}

Therefore, I'm proposing to add this parameter to the "e2e" builder's compile 
step.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-08 Thread Arvid Heise 
Yay!

On Thu, Jul 8, 2021 at 10:02 AM Jiayi Liao  wrote:

> Congratulations Yuan!
>
> Best,
> Jiayi Liao
>
> On Thu, Jul 8, 2021 at 3:55 PM Roman Khachatryan  wrote:
>
> > Congratulations Yuan!
> >
> > Regards,
> > Roman
> >
> > On Thu, Jul 8, 2021 at 6:02 AM Yang Wang  wrote:
> > >
> > > Congratulations Yuan!
> > >
> > > Best,
> > > Yang
> > >
> > > XING JIN  于2021年7月8日周四 上午11:46写道：
> > >
> > > > Congratulations Yuan~!
> > > >
> > > > Roc Marshal  于2021年7月8日周四 上午11:28写道：
> > > >
> > > > > Congratulations, Yuan!
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > At 2021-07-08 01:21:40, "Yu Li"  wrote:
> > > > > >Hi all,
> > > > > >
> > > > > >On behalf of the PMC, I’m very happy to announce Yuan Mei as a new
> > Flink
> > > > > >committer.
> > > > > >
> > > > > >Yuan has been an active contributor for more than two years, with
> > code
> > > > > >contributions on multiple components including kafka connectors,
> > > > > >checkpointing, state backends, etc. Besides, she has been actively
> > > > > involved
> > > > > >in community activities such as helping manage releases,
> discussing
> > > > > >questions on dev@list, supporting users and giving talks at
> > > > conferences.
> > > > > >
> > > > > >Please join me in congratulating Yuan for becoming a Flink
> > committer!
> > > > > >
> > > > > >Cheers,
> > > > > >Yu
> > > > >
> > > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-08 Thread Jingsong Li 
Congratulations Yuan!

Best,
Jingsong

On Thu, Jul 8, 2021 at 5:43 PM Arvid Heise  wrote:

> Yay!
>
> On Thu, Jul 8, 2021 at 10:02 AM Jiayi Liao 
> wrote:
>
> > Congratulations Yuan!
> >
> > Best,
> > Jiayi Liao
> >
> > On Thu, Jul 8, 2021 at 3:55 PM Roman Khachatryan 
> wrote:
> >
> > > Congratulations Yuan!
> > >
> > > Regards,
> > > Roman
> > >
> > > On Thu, Jul 8, 2021 at 6:02 AM Yang Wang 
> wrote:
> > > >
> > > > Congratulations Yuan!
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > XING JIN  于2021年7月8日周四 上午11:46写道：
> > > >
> > > > > Congratulations Yuan~!
> > > > >
> > > > > Roc Marshal  于2021年7月8日周四 上午11:28写道：
> > > > >
> > > > > > Congratulations, Yuan!
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > At 2021-07-08 01:21:40, "Yu Li"  wrote:
> > > > > > >Hi all,
> > > > > > >
> > > > > > >On behalf of the PMC, I’m very happy to announce Yuan Mei as a
> new
> > > Flink
> > > > > > >committer.
> > > > > > >
> > > > > > >Yuan has been an active contributor for more than two years,
> with
> > > code
> > > > > > >contributions on multiple components including kafka connectors,
> > > > > > >checkpointing, state backends, etc. Besides, she has been
> actively
> > > > > > involved
> > > > > > >in community activities such as helping manage releases,
> > discussing
> > > > > > >questions on dev@list, supporting users and giving talks at
> > > > > conferences.
> > > > > > >
> > > > > > >Please join me in congratulating Yuan for becoming a Flink
> > > committer!
> > > > > > >
> > > > > > >Cheers,
> > > > > > >Yu
> > > > > >
> > > > >
> > >
> >
>


-- 
Best, Jingsong Lee

Re: [DISCUSS] Incrementally deprecating the DataSet API

2021-07-08 Thread Etienne Chauchot 

Hi Timo,

Thanks for your answers, no problem with the delay, I was in vacation 
too last week :)


My comments are inline

Best,

Etienne

On 07/07/2021 16:48, Timo Walther wrote:

Hi Etienne,

sorry for the late reply due to my vacation last week.

Regarding: "support of aggregations in batch mode for DataStream API 
[...] is there a plan to solve it before the actual drop of DataSet API"


Just to clarify it again: we will not drop the DataSet API any time 
soon. So users will have enough time to update their pipelines. There 
are a couple of features missing to fully switch to DataStream API in 
batch mode. Thanks for opening an issue, this is helpful for us to 
gradually remove those barriers. They don't need to have a "Blocker" 
priority in JIRA for now.



Ok I thought the drop was sooner, no problem then.




But aggregations is a good example where we should discuss if it would 
be easier to simply switch to Table API for that. Table API has a lot 
of aggregation optimizations and can work on binary data. Also joins 
should be easier in Table API. DataStream API can be a very low-level 
API in the near future and most use cases (esp. the batch ones) should 
be possible in Table API.



Yes sure. As a matter of fact, my point was to use low level DataStream 
API in a benchmark to compare with Table API but I guess it is not a 
common user behavior.





Regarding: "Is it needed to port these Avro enhancements to new 
DataStream connectors (add a new equivalent of 
ParquetColumnarRowInputFormat but for Avro)"


We should definitely not loose functionality. The same functionality 
should be present in the new connectors. The questions is rather 
whether we need to offer a DataStream API connector or if a Table API 
connector would be nicer to use (also nicely integrated with catalogs).


So a user can use a simple CREATE TABLE statement to configure the 
connector; an easier abstraction is almost not possible. With 
`tableEnv.toDataStream(table)` you can then continue in DataStream API 
if there is still a need for it.



Yes I agree, there is no easier connector setup than CREATE TABLE, and 
with tableEnv.toDataStream(table) if one would want to stay with 
DataStream (in my bench for example) it is still possible. And by the 
way the doc for parquet (1) for example only mentions using Table API 
connector.


So I guess the table connector would be nicer for the user indeed. But 
if we use tableEnv.toDataStream(table), we would be able to produce only 
usual types like Row, Tuples or Pojos, we still need to add Avro support 
right ?


[1] 
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/formats/parquet/





Regarding: "there are parquet bugs still open on deprecated parquet 
connector"


Yes, bugs should still be fixed in 1.13.



OK




Regarading: "I've been doing TPCDS benchmarks with Flink lately"

Great to hear that :-)



And congrats again on blink performances !




Did you also see the recent discussion? A TPC-DS benchmark can further 
be improved by providing statistics. Maybe this is helpful to you:


https://lists.apache.org/thread.html/ra383c23f230ab8e7fa16ec64b4f277c267d6358d55cc8a0edc77bb63%40%3Cuser.flink.apache.org%3E 




No, I missed that thread, thanks for the pointer, I'll read it and 
comment if I have something to add.





I will prepare a blog post shortly.



Good to hear :)




Regards,
Timo



On 06.07.21 15:05, Etienne Chauchot wrote:

Hi all,

Any comments ?

cheers,

Etienne

On 25/06/2021 15:09, Etienne Chauchot wrote:

Hi everyone,

@Timo, my comments are inline for steps 2, 4 and 5, please tell me 
what you think.


Best

Etienne


On 23/06/2021 15:27, Chesnay Schepler wrote:
If we want to publicize this plan more shouldn't we have a rough 
timeline for when 2.0 is on the table?


On 6/23/2021 2:44 PM, Stephan Ewen wrote:

Thanks for writing this up, this also reflects my understanding.

I think a blog post would be nice, ideally with an explicit call for
feedback so we learn about user concerns.
A blog post has a lot more reach than an ML thread.

Best,
Stephan


On Wed, Jun 23, 2021 at 12:23 PM Timo Walther  
wrote:



Hi everyone,

I'm sending this email to make sure everyone is on the same page 
about

slowly deprecating the DataSet API.

There have been a few thoughts mentioned in presentations, offline
discussions, and JIRA issues. However, I have observed that there 
are
still some concerns or different opinions on what steps are 
necessary to

implement this change.

Let me summarize some of the steps and assumpations and let's have a
discussion about it:

Step 1: Introduce a batch mode for Table API (FLIP-32)
[DONE in 1.9]

Step 2: Introduce a batch mode for DataStream API (FLIP-134)
[DONE in 1.12]



I've been using DataSet API and I tested migrating to DataStream + 
batch mode.


I opened this (1) ticket regarding the support of aggregations in 
batch mode for DataStream API. It seems that join operation (at 
least) doe

[jira] [Created] (FLINK-23313) Reintroduce temporal table function documentation

2021-07-08 Thread Timo Walther (Jira) 
Timo Walther created FLINK-23313:


 Summary: Reintroduce temporal table function documentation
 Key: FLINK-23313
 URL: https://issues.apache.org/jira/browse/FLINK-23313
 Project: Flink
  Issue Type: Bug
  Components: Documentation, Table SQL / API
Reporter: Timo Walther


FLIP-132 introduced the new {{FOR SYSTEM_TIME AS OF}} and dropped the main 
documentation for temporal table function. This causes a lot of confusion for 
users.

First, because processing time joins are not supported yet.

Second, because a primary key might not always be present in the current 
pipeline when using Table API.

We have not deprecated `createTemporalTableFunction` and the documentation in 
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/tableapi/#join-with-temporal-table
 is not enough.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23314) State left behind for short lived keys

2021-07-08 Thread Dawid Wysakowicz (Jira) 
Dawid Wysakowicz created FLINK-23314:


 Summary: State left behind for short lived keys
 Key: FLINK-23314
 URL: https://issues.apache.org/jira/browse/FLINK-23314
 Project: Flink
  Issue Type: Bug
  Components: Library / CEP
Affects Versions: 1.12.4, 1.13.1, 1.11.3, 1.14.0
Reporter: Dawid Wysakowicz


As reported by a user in the ML if there are short-lived keys, which do not 
appear after some time, the CepOperator leaves some state behind for those keys 
leading to the state constantly growing.

https://lists.apache.org/thread.html/r23f11a652dac7921c0068fa94a3cabe0e52fadce3679c93f21aeb91e%40%3Cuser.flink.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23315) Bump log4j to 2.14.1 for version 1.13.2

2021-07-08 Thread Guilaume Kermorgant (Jira) 
Guilaume Kermorgant created FLINK-23315:
---

 Summary: Bump log4j to 2.14.1 for version 1.13.2
 Key: FLINK-23315
 URL: https://issues.apache.org/jira/browse/FLINK-23315
 Project: Flink
  Issue Type: Improvement
Reporter: Guilaume Kermorgant
 Fix For: 1.13.2


Flink 1.13 is currently [relying on log4j 
2.12.1|[https://github.com/apache/flink/blob/release-1.13/pom.xml#L110],] which 
has a [low severity 
vulnerability|[https://nvd.nist.gov/vuln/detail/CVE-2020-9488]|https://nvd.nist.gov/vuln/detail/CVE-2020-9488].]

This is fixed in Log4j 2.13.1.

Flink 1.14 will be released with Log4j 2.14.1, c.f. 
[FLINK-22407|https://issues.apache.org/jira/browse/FLINK-22407]

It would be nice for us to have it in Flink 1.13.2 as well, if the community 
thinks it's not a bad idea; this could also be a good opportunity for me to 
open a first PR in the Flink repo.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Feedback Collection Jira Bot

2021-07-08 Thread Timo Walther 

Hi everyone,

let me also post some feedback here. As someone who monitors and 
maintains the SQL component JIRA issues, I'm quite unhappy with the 
current bot behavior. I'm watching quite a few issues and SQL has a lot 
of pretty major if not critical topics. My mailbox is full of bot 
messages every day. And I also heard from other users, that they kind of 
disappointed that the bot deprioritized their valuable feedback.


I agree with Stephan and Kurt, a project like Flink has often tickets 
that span months to years. It might be just ideas where we select more 
input or bugs where we don't have an immediate solution or need to wait 
for more reports.


`stale-critical only after 14 days` is still not enought time. I know 
critical issues that are open for 6 months already and I would like to 
avoid the work of updating them every two weeks. A reminder would be 
nice though.


How about:

Contributors can only open issues with minor priority

This would mean less work for the bot because only committers can raise 
the priority. We would not start automatic deprioriziation at all and 
avoid load on the committers to raise the priority again. We could let 
the bot send a periodic comment if the priority is still necessary.


Maybe this was mentioned before, the thread is already quite long. But 
let me know what you think?


Regards,
Timo

On 05.07.21 12:15, Konstantin Knauf wrote:

Hi everyone,

ok, let's in addition try out not unassigning anyone from tickets. This
makes it the responsibility of the component maintainers to
periodically check for stale-unassigned tickets and bring them to a
resolution. We can monitor the situation (# of stale-unassigned tickets)
and if the number of open stale-unassigned tickets is ever increasing, we
need to revisit this topic.

For reference here are the tickets for the adjustments:

* https://issues.apache.org/jira/browse/FLINK-23207 (PR available)
* https://issues.apache.org/jira/browse/FLINK-23206 (blocked by INFRA)
* https://issues.apache.org/jira/browse/FLINK-23205 (merged)
* https://issues.apache.org/jira/browse/FLINK-23250 (open)

Cheers,

Konstantin

On Fri, Jul 2, 2021 at 9:43 AM Piotr Nowojski  wrote:


+1 for the unassignment remark from Stephan

Piotrek

czw., 1 lip 2021 o 12:35 Stephan Ewen  napisał(a):


It is true that the bot surfaces problems that are there (not enough
committer attention sometimes), but it also "rubs salt in the wound" of
contributors, and that is tricky.

We can try it out with the extended periods (although I think that in
reality we probably need even longer periods) and see how it goes.

One thing I would suggest is to never let the bot unassign issues. It

just

strikes me as very cold and respectless to be unassigned by a bot from an
issue in which I invested time and energy. (The committers don't even

take

the time to talk to me and explain why the contribution will not go
forward).
Unassignment should come from another person, possibly in response to a
ping from the bot. I think that makes a big difference in contributor
treatment.



On Wed, Jun 30, 2021 at 12:30 PM Till Rohrmann 
wrote:


I agree that we shouldn't discourage contributions.

For me the main idea of the bot is not to clean up the JIRA but to

improve

our communication and expectation management with the community. There

are

many things we could do but for a lot of things we don't have the time

and

capacity. Then to say at some point that we won't do something is just
being honest. This also shows when looking at the JIRA numbers of the
merged commits. We very rarely resolve tickets which are older than x

days

and if we do, then we usually create a new ticket for the problem.

The fact that we see some tickets with available pull requests go stale

is

the symptom that we don't value them to be important enough or
allocate enough time for external contributions imo. Otherwise, they

would

have gotten the required attention and been merged. In such a case,

raising

awareness by pinging the watchers of the respective ticket is probably
better than silently ignoring the PR. Also adding labels to filter for
these PRs should help to get them the required attention. But also

here,

it

happens very rarely that we actually merge a PR that is older than y

days.

Ideally we avoid this situation altogether by only assigning

contributors

to tickets for which a committer has review capacity. However, this

does

not seem to always work.

In some sense, the JIRA bot shows us the things, which fall through the
cracks, more explicitly (which is probably not different than before).

Of

course we should try to find the time periods for when to ping or
de-prioritize tickets that work best for the community.

+1 for the proposed changes (extended time periods, "Not a Priority",
default priority and fixVersion).

@Piotr, I think we have the priorities defined here [1]. Maybe it is

enough

to share the link so that everyone can check whether her assumptions

are

[jira] [Created] (FLINK-23316) There's no test for custom PartitionCommitPolicy

2021-07-08 Thread Rui Li (Jira) 
Rui Li created FLINK-23316:
--

 Summary: There's no test for custom PartitionCommitPolicy
 Key: FLINK-23316
 URL: https://issues.apache.org/jira/browse/FLINK-23316
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Ecosystem
Reporter: Rui Li






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23317) Only keep the latest checkpoint in CompletedCheckpointStore

2021-07-08 Thread Jira 
David Morávek created FLINK-23317:
-

 Summary: Only keep the latest checkpoint in 
CompletedCheckpointStore
 Key: FLINK-23317
 URL: https://issues.apache.org/jira/browse/FLINK-23317
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: David Morávek


Issue based on the discussion from FLINK-22483

We can lower the memory footprint of CompletedCheckpointStore by keeping only 
the latest checkpoint / savepoint, that will be used for recovery. We need to 
respect `preferCheckpoint`, but the sentiment of this stays the same.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23318) AkkaRpcActorTest#testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor fails on azure

2021-07-08 Thread Dawid Wysakowicz (Jira) 
Dawid Wysakowicz created FLINK-23318:


 Summary: 
AkkaRpcActorTest#testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor fails 
on azure
 Key: FLINK-23318
 URL: https://issues.apache.org/jira/browse/FLINK-23318
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.14.0
Reporter: Dawid Wysakowicz


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20163&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=5360d54c-8d94-5d85-304e-a89267eb785a&l=6023

{code}
Jul 08 11:03:13 java.lang.AssertionError: 
Jul 08 11:03:13 
Jul 08 11:03:13 Expected: is 
Jul 08 11:03:13  but: was 
Jul 08 11:03:13 at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
Jul 08 11:03:13 at org.junit.Assert.assertThat(Assert.java:964)
Jul 08 11:03:13 at org.junit.Assert.assertThat(Assert.java:930)
Jul 08 11:03:13 at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActorTest.testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor(AkkaRpcActorTest.java:375)
Jul 08 11:03:13 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
Jul 08 11:03:13 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Jul 08 11:03:13 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jul 08 11:03:13 at java.lang.reflect.Method.invoke(Method.java:498)
Jul 08 11:03:13 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
Jul 08 11:03:13 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Jul 08 11:03:13 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
Jul 08 11:03:13 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Jul 08 11:03:13 at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Jul 08 11:03:13 at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
Jul 08 11:03:13 at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
Jul 08 11:03:13 at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
Jul 08 11:03:13 at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
Jul 08 11:03:13 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
Jul 08 11:03:13 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
Jul 08 11:03:13 at 
org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
Jul 08 11:03:13 at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
Jul 08 11:03:13 at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
Jul 08 11:03:13 at 
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
Jul 08 11:03:13 at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
Jul 08 11:03:13 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Jul 08 11:03:13 at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
Jul 08 11:03:13 at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
Jul 08 11:03:13 at 
org.junit.runners.ParentRunner.run(ParentRunner.java:413)
Jul 08 11:03:13 at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
Jul 08 11:03:13 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
Jul 08 11:03:13 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
Jul 08 11:03:13 at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
Jul 08 11:03:13 at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
Jul 08 11:03:13 at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
Jul 08 11:03:13 at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
Jul 08 11:03:13 at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Jul 08 11:03:13 
Jul 08 11:03:13 [INFO] Running 
org.apache.flink.runtime.rpc.akka.TimeoutCallStackTest
Jul 08 11:03:13 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time 
elapsed: 0.107 s - in org.apache.flink.runtime.rpc.akka.TimeoutCallStackTest
Jul 08 11:03:13 [INFO] Running 
org.apache.flink.runtime.rpc.akka.AkkaRpcActorOversizedResponseMessageTest
Jul 08 11:03:13 [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time 
elapsed: 0.823 s - in org.apache.flink.runtime.rpc.akka.AkkaRpcServiceTest
Jul 08 11:03:14 [INFO]

[DISCUSS] FLIP-179: Expose Standardized Operator Metrics

2021-07-08 Thread Arvid Heise 
Dear devs,

As a continuation and generalization of FLIP-33 (Standardize Connector
Metrics) [1], we'd like to discuss how we actually expose the standardized
operator metrics to users in terms of changes to the API.

Please check out the FLIP [2] and provide feedback.

Best,

Arvid

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-179%3A+Expose+Standardized+Operator+Metrics

[jira] [Created] (FLINK-23319) Support semi/anti lookup joins

2021-07-08 Thread Timo Walther (Jira) 
Timo Walther created FLINK-23319:


 Summary: Support semi/anti lookup joins
 Key: FLINK-23319
 URL: https://issues.apache.org/jira/browse/FLINK-23319
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Planner
Reporter: Timo Walther


Similar to FLINK-23305, we should also allow semi/anti joins for lookup joins 
such as:

{code}
SELECT T.*
FROM MyTable AS T
WHERE EXISTS (
SELECT * FROM LookupTable FOR SYSTEM_TIME AS OF T.proctime AS D
WHERE T.a = D.id)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23320) Support semi/anti temporal joins

2021-07-08 Thread Timo Walther (Jira) 
Timo Walther created FLINK-23320:


 Summary: Support semi/anti temporal joins
 Key: FLINK-23320
 URL: https://issues.apache.org/jira/browse/FLINK-23320
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Planner
Reporter: Timo Walther


Similar to FLINK-23305, we should also allow semi/anti joins for temporal joins 
such as:

{code}
SELECT T.*
FROM MyTable AS T
WHERE EXISTS (
SELECT * FROM VersionedTable FOR SYSTEM_TIME AS OF T.rowtime AS D
WHERE T.a = D.id)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Incrementally deprecating the DataSet API

2021-07-08 Thread Seth Wiesman 
Hi Etienne,

The `toDataStream` method supports converting to concrete Java types, not
just Row, which can include your Avro specific-records. See example 2:

https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/data_stream_api/#examples-for-todatastream

On Thu, Jul 8, 2021 at 5:11 AM Etienne Chauchot 
wrote:

> Hi Timo,
>
> Thanks for your answers, no problem with the delay, I was in vacation
> too last week :)
>
> My comments are inline
>
> Best,
>
> Etienne
>
> On 07/07/2021 16:48, Timo Walther wrote:
> > Hi Etienne,
> >
> > sorry for the late reply due to my vacation last week.
> >
> > Regarding: "support of aggregations in batch mode for DataStream API
> > [...] is there a plan to solve it before the actual drop of DataSet API"
> >
> > Just to clarify it again: we will not drop the DataSet API any time
> > soon. So users will have enough time to update their pipelines. There
> > are a couple of features missing to fully switch to DataStream API in
> > batch mode. Thanks for opening an issue, this is helpful for us to
> > gradually remove those barriers. They don't need to have a "Blocker"
> > priority in JIRA for now.
>
>
> Ok I thought the drop was sooner, no problem then.
>
>
> >
> > But aggregations is a good example where we should discuss if it would
> > be easier to simply switch to Table API for that. Table API has a lot
> > of aggregation optimizations and can work on binary data. Also joins
> > should be easier in Table API. DataStream API can be a very low-level
> > API in the near future and most use cases (esp. the batch ones) should
> > be possible in Table API.
>
>
> Yes sure. As a matter of fact, my point was to use low level DataStream
> API in a benchmark to compare with Table API but I guess it is not a
> common user behavior.
>
>
> >
> > Regarding: "Is it needed to port these Avro enhancements to new
> > DataStream connectors (add a new equivalent of
> > ParquetColumnarRowInputFormat but for Avro)"
> >
> > We should definitely not loose functionality. The same functionality
> > should be present in the new connectors. The questions is rather
> > whether we need to offer a DataStream API connector or if a Table API
> > connector would be nicer to use (also nicely integrated with catalogs).
> >
> > So a user can use a simple CREATE TABLE statement to configure the
> > connector; an easier abstraction is almost not possible. With
> > `tableEnv.toDataStream(table)` you can then continue in DataStream API
> > if there is still a need for it.
>
>
> Yes I agree, there is no easier connector setup than CREATE TABLE, and
> with tableEnv.toDataStream(table) if one would want to stay with
> DataStream (in my bench for example) it is still possible. And by the
> way the doc for parquet (1) for example only mentions using Table API
> connector.
>
> So I guess the table connector would be nicer for the user indeed. But
> if we use tableEnv.toDataStream(table), we would be able to produce only
> usual types like Row, Tuples or Pojos, we still need to add Avro support
> right ?
>
> [1]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/formats/parquet/
>
>
> >
> > Regarding: "there are parquet bugs still open on deprecated parquet
> > connector"
> >
> > Yes, bugs should still be fixed in 1.13.
>
>
> OK
>
>
> >
> > Regarading: "I've been doing TPCDS benchmarks with Flink lately"
> >
> > Great to hear that :-)
>
>
> And congrats again on blink performances !
>
>
> >
> > Did you also see the recent discussion? A TPC-DS benchmark can further
> > be improved by providing statistics. Maybe this is helpful to you:
> >
> >
> https://lists.apache.org/thread.html/ra383c23f230ab8e7fa16ec64b4f277c267d6358d55cc8a0edc77bb63%40%3Cuser.flink.apache.org%3E
> >
>
>
> No, I missed that thread, thanks for the pointer, I'll read it and
> comment if I have something to add.
>
>
> >
> > I will prepare a blog post shortly.
>
>
> Good to hear :)
>
>
> >
> > Regards,
> > Timo
> >
> >
> >
> > On 06.07.21 15:05, Etienne Chauchot wrote:
> >> Hi all,
> >>
> >> Any comments ?
> >>
> >> cheers,
> >>
> >> Etienne
> >>
> >> On 25/06/2021 15:09, Etienne Chauchot wrote:
> >>> Hi everyone,
> >>>
> >>> @Timo, my comments are inline for steps 2, 4 and 5, please tell me
> >>> what you think.
> >>>
> >>> Best
> >>>
> >>> Etienne
> >>>
> >>>
> >>> On 23/06/2021 15:27, Chesnay Schepler wrote:
>  If we want to publicize this plan more shouldn't we have a rough
>  timeline for when 2.0 is on the table?
> 
>  On 6/23/2021 2:44 PM, Stephan Ewen wrote:
> > Thanks for writing this up, this also reflects my understanding.
> >
> > I think a blog post would be nice, ideally with an explicit call for
> > feedback so we learn about user concerns.
> > A blog post has a lot more reach than an ML thread.
> >
> > Best,
> > Stephan
> >
> >
> > On Wed, Jun 23, 2021 at 12:23 PM Timo Walther 
> > wrote:
> >
> >> Hi everyone,
> >>
>

Re: [ANNOUNCE] New PMC member: Guowei Ma

2021-07-08 Thread Steven Wu 
Awesome! Congratulations, Guowei!

On Wed, Jul 7, 2021 at 4:25 AM Jingsong Li  wrote:

> Congratulations, Guowei!
>
> Best,
> Jingsong
>
> On Wed, Jul 7, 2021 at 6:36 PM Arvid Heise  wrote:
>
> > Congratulations!
> >
> > On Wed, Jul 7, 2021 at 11:30 AM Till Rohrmann 
> > wrote:
> >
> > > Congratulations, Guowei!
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jul 7, 2021 at 9:41 AM Roman Khachatryan 
> > wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Regards,
> > > > Roman
> > > >
> > > > On Wed, Jul 7, 2021 at 8:24 AM Rui Li  wrote:
> > > > >
> > > > > Congratulations Guowei!
> > > > >
> > > > > On Wed, Jul 7, 2021 at 1:01 PM Benchao Li 
> > > wrote:
> > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > Dian Fu  于2021年7月7日周三 下午12:46写道：
> > > > > >
> > > > > > > Congratulations, Guowei!
> > > > > > >
> > > > > > > Regards,
> > > > > > > Dian
> > > > > > >
> > > > > > > > 2021年7月7日 上午10:37，Yun Gao  写道：
> > > > > > > >
> > > > > > > > Congratulations Guowei!
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > --
> > > > > > > > Sender:JING ZHANG
> > > > > > > > Date:2021/07/07 10:33:51
> > > > > > > > Recipient:dev
> > > > > > > > Theme:Re: [ANNOUNCE] New PMC member: Guowei Ma
> > > > > > > >
> > > > > > > > Congratulations,  Guowei Ma!
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > JING ZHANG
> > > > > > > >
> > > > > > > > Zakelly Lan  于2021年7月7日周三 上午10:30写道：
> > > > > > > >
> > > > > > > >> Congratulations, Guowei!
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Zakelly
> > > > > > > >>
> > > > > > > >> On Wed, Jul 7, 2021 at 10:24 AM tison  >
> > > > wrote:
> > > > > > > >>
> > > > > > > >>> Congrats! NB.
> > > > > > > >>>
> > > > > > > >>> Best,
> > > > > > > >>> tison.
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Jark Wu  于2021年7月7日周三 上午10:20写道：
> > > > > > > >>>
> > > > > > >  Congratulations Guowei!
> > > > > > > 
> > > > > > >  Best,
> > > > > > >  Jark
> > > > > > > 
> > > > > > >  On Wed, 7 Jul 2021 at 09:54, XING JIN <
> > > jinxing.co...@gmail.com>
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Congratulations, Guowei~ !
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jin
> > > > > > > >
> > > > > > > > Xintong Song  于2021年7月7日周三
> > 上午9:37写道：
> > > > > > > >
> > > > > > > >> Congratulations, Guowei~!
> > > > > > > >>
> > > > > > > >> Thank you~
> > > > > > > >>
> > > > > > > >> Xintong Song
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Jul 7, 2021 at 9:31 AM Qingsheng Ren <
> > > > renqs...@gmail.com>
> > > > > > >  wrote:
> > > > > > > >>
> > > > > > > >>> Congratulations Guowei!
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> Best Regards,
> > > > > > > >>>
> > > > > > > >>> Qingsheng Ren
> > > > > > > >>> Email: renqs...@gmail.com
> > > > > > > >>> 2021年7月7日 +0800 09:30 Leonard Xu  >，写道：
> > > > > > >  Congratulations! Guowei Ma
> > > > > > > 
> > > > > > >  Best,
> > > > > > >  Leonard
> > > > > > > 
> > > > > > > > ÔÚ 2021Äê7ÔÂ6ÈÕ£¬21:56£¬Kurt Young 
> > > > дµÀ£º
> > > > > > > >
> > > > > > > > Hi all!
> > > > > > > >
> > > > > > > > I'm very happy to announce that Guowei Ma has joined
> > the
> > > > > > > >> Flink
> > > > > > >  PMC!
> > > > > > > >
> > > > > > > > Congratulations and welcome Guowei!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Kurt
> > > > > > > 
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > > 
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best,
> > > > > > Benchao Li
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards!
> > > > > Rui Li
> > > >
> > >
> >
>
>
> --
> Best, Jingsong Lee
>

Re: Job Recovery Time on TM Lost

2021-07-08 Thread Lu Niu 
Thanks everyone! This is a great discussion!

1. Restarting takes 30s when throwing exceptions from application code
because the restart delay is 30s in config. Before lots of related config
are 30s which lead to the confusion. I redo the test with config:

FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
backoffTimeMS=1000)
heartbeat.timeout: 50
akka.ask.timeout 30 s
akka.lookup.timeout 30 s
akka.tcp.timeout 30 s
akka.watch.heartbeat.interval 30 s
akka.watch.heartbeat.pause 120 s

   Now Phase 1 drops down to 2s now and phase 2 takes 13s. The whole
restart takes 14s. Does that mean the akka timeout situation we talked
above doesn't apply to flink 1.11?

2. About flaky connection between TMs, we did notice sometimes exception as
follows:
```
TaskFoo switched from RUNNING to FAILED on
container_e02_1599158147594_156068_01_38 @
xenon-prod-001-20200316-data-slave-prod-0a0201a4.ec2.pin220.com
(dataPort=40957).
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
Connection unexpectedly closed by remote task manager '
xenon-prod-001-20200316-data-slave-prod-0a020f7a.ec2.pin220.com/10.2.15.122:33539'.
This might indicate that the remote task manager was lost.
at
org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:144)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
at
org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
at
org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelInactive(NettyMessageClientDecoderDelegate.java:97)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1416)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:912)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:816)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:331)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
```
1. It's a bit inconvenient to debug such an exception because it doesn't
report the exact container id. Right now we have to look for `
xenon-prod-001-20200316-data-slave-prod-0a020f7a.ec2.pin220.com/10.2.15.122:33539`
in JobMananger log to find that.
2. The task manager log doesn't show anything suspicious. Also, no major
GC. So it might imply a flack connection in this case.
3. Is there any short term workaround we can try? any config tuning? Also,
what's the long term solution?

Best
Lu




On Tue, Jul 6, 2021 at 11:45 PM 刘建刚  wrote:

> It is really helpful to find the lost container quickly. In our inner
> flink version, we optimize it by task's report and jobmaster's probe. When
> a task fails because of the connection, it reports to the jobmaster. The
> jobmaster will try to confirm the liveness of the unconnected
> taskmanager for certain times by config. If the jobmaster find the
> taskmanager unconnected or dead, it releases the taskmanger. This will work
> for most cases. For an unstable environment, config needs adjustment.
>
> Gen Luo  于2021年7月6日周二 下午8:41写道：
>
>> Yes, I have noticed the PR and commented there with some consideration
>> about the new option.

[jira] [Created] (FLINK-23321) Make REST Netty thread county configurable

2021-07-08 Thread Nicolas Raga (Jira) 
Nicolas Raga created FLINK-23321:


 Summary: Make REST Netty thread county configurable
 Key: FLINK-23321
 URL: https://issues.apache.org/jira/browse/FLINK-23321
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Metrics, Runtime / REST
Reporter: Nicolas Raga


Currently 
[rest.server.numThreads|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#rest-server-numthreads]
 works to create more DispatcherRestEndpoint threads which help in the purview 
of job the dispatcher functionality. However, when used for the REST server for 
monitoring of a running job, AbstractRestHandlers actually run on a netty 
thread pool we have no control over as seen 
[here|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestServerEndpoint.java#L188-L193].
 Especially when monitoring unhealthy jobs, we would want to allow creation of 
more netty-worker threads that can process incoming connection requests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23322) RMQSourceITCase.testStopWithSavepoint fails on azure due to timeout

2021-07-08 Thread Xintong Song (Jira) 
Xintong Song created FLINK-23322:


 Summary: RMQSourceITCase.testStopWithSavepoint fails on azure due 
to timeout
 Key: FLINK-23322
 URL: https://issues.apache.org/jira/browse/FLINK-23322
 Project: Flink
  Issue Type: Bug
  Components: Connectors/ RabbitMQ
Affects Versions: 1.12.4
Reporter: Xintong Song


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20196&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=03dca39c-73e8-5aaf-601d-328ae5c35f20&l=13696

{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 41.237 
s <<< FAILURE! - in 
org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
[ERROR] 
testStopWithSavepoint(org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase)
  Time elapsed: 7.609 s  <<< ERROR!
java.util.concurrent.TimeoutException
at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:77)
at 
com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:120)
at 
com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
at 
com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:502)
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:326)
at 
com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
at 
com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
at 
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
at 
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
at 
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
at 
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
at 
org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:133)
at 
org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:30)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Job Recovery Time on TM Lost

2021-07-08 Thread Gen Luo 
@刘建刚
Welcome to join the discuss and thanks for sharing your experience.

I have a minor question. In my experience, network failures in a certain
cluster usually takes a time to recovery, which can be measured as p99 to
guide configuring. So I suppose it would be better to use time than attempt
count as the configuration for confirming TM liveness. How do you think
about this? Or is the premise right according to your experience?

@Lu Niu 
> Does that mean the akka timeout situation we talked above doesn't apply
to flink 1.11?

I suppose it's true. According to the reply from Till in FLINK-23216
, it should be confirmed
that the problem is introduced by declarative resource management, which is
introduced to Flink in 1.12.

In previous versions, although JM still uses heartbeat to check TMs status,
RM will tell JM about TM lost once it is noticed by Yarn. This is much
faster than JM's heartbeat mechanism, if one uses default heartbeat
configurations. However, after 1.12 with declarative resource management,
RM will no longer tell this to JM, since it doesn't have a related
AllocationID.  So the heartbeat mechanism becomes the only way JM can know
about TM lost.

On Fri, Jul 9, 2021 at 6:34 AM Lu Niu  wrote:

> Thanks everyone! This is a great discussion!
>
> 1. Restarting takes 30s when throwing exceptions from application code
> because the restart delay is 30s in config. Before lots of related config
> are 30s which lead to the confusion. I redo the test with config:
>
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
> backoffTimeMS=1000)
> heartbeat.timeout: 50
> akka.ask.timeout 30 s
> akka.lookup.timeout 30 s
> akka.tcp.timeout 30 s
> akka.watch.heartbeat.interval 30 s
> akka.watch.heartbeat.pause 120 s
>
>Now Phase 1 drops down to 2s now and phase 2 takes 13s. The whole
> restart takes 14s. Does that mean the akka timeout situation we talked
> above doesn't apply to flink 1.11?
>
> 2. About flaky connection between TMs, we did notice sometimes exception
> as follows:
> ```
> TaskFoo switched from RUNNING to FAILED on
> container_e02_1599158147594_156068_01_38 @
> xenon-prod-001-20200316-data-slave-prod-0a0201a4.ec2.pin220.com
> (dataPort=40957).
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Connection unexpectedly closed by remote task manager '
> xenon-prod-001-20200316-data-slave-prod-0a020f7a.ec2.pin220.com/10.2.15.122:33539'.
> This might indicate that the remote task manager was lost.
> at
> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:144)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
> at
> org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelInactive(NettyMessageClientDecoderDelegate.java:97)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1416)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:912)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:816)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:331)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.S

[jira] [Created] (FLINK-23323) HAQueryableStateRocksDBBackendITCase failed due to heap OOM

2021-07-08 Thread Xintong Song (Jira) 
Xintong Song created FLINK-23323:


 Summary: HAQueryableStateRocksDBBackendITCase failed due to heap 
OOM
 Key: FLINK-23323
 URL: https://issues.apache.org/jira/browse/FLINK-23323
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Queryable State
Affects Versions: 1.14.0
Reporter: Xintong Song


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20195&view=logs&j=c91190b6-40ae-57b2-5999-31b869b0a7c1&t=43529380-51b4-5e90-5af4-2dccec0ef402&l=14431

{code}
Jul 08 21:43:22 [ERROR] Tests run: 12, Failures: 0, Errors: 9, Skipped: 1, Time 
elapsed: 246.345 s <<< FAILURE! - in 
org.apache.flink.queryablestate.itcases.HAQueryableStateRocksDBBackendITCase
Jul 08 21:43:22 [ERROR] 
testReducingState(org.apache.flink.queryablestate.itcases.HAQueryableStateRocksDBBackendITCase)
  Time elapsed: 241.454 s  <<< ERROR!
Jul 08 21:43:22 java.lang.OutOfMemoryError: Java heap space
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Lifecycle of ShuffleMaster and its Relationship with JobMaster and PartitionTracker

2021-07-08 Thread Zhu Zhu 
Thanks for starting this discussion.
Here are some of my thoughts regarding the proposal and discussions
above.

*+1 to enable ShuffleMaster to stop track partitions proactively*
In production we have encountered problems that it needs *hours* to
recover from a remote shuffle worker lost problem. Because the lost
finished partitions cannot be detected and reproduced all at once.
This improvement can help to solve this problem.


*+1 to make ShuffleMaster a cluster level component*
This helps to avoid maintain multiple clients and connections to the
same remote shuffle service. It also makes it possible to support
external cluster partitions in the future.

*+1 to enable ShuffleMaster to notify master about its internal *
*non-recoverable error*
The scheduler can keep failing or hang due to ShuffleMaster internal
non-recoverable error. Currently this kind of problem cannot be
auto-recovered and are hard to diagnose.
One question which might be out of the scope is that whether we
should do similar things for ShuffleEnvironment?


*+1 that the abstraction should be able to support different jobs to*
*use different ShuffleServices eventually*
I think the proposal does not conflict with this target.
One idea in my mind is to maintain multiple different ShuffleServices
in the Dispatcher/JobManagerSharedServices and let them be shared
between different jobs. Each job should be configured with a key which
points to a ShuffleService. The key should be used by both the scheduler
and tasks on task managers to select their respective
ShuffleMaster/ShuffleEnvironment. This will need work both on the master
and on the worker. Currently the worker will launch a ShuffleEnvironment
shared between different tasks which can be from different jobs. But only
one single ShuffleEnvironment will be created on each task manager.

Thanks,
Zhu

Yingjie Cao  于2021年7月8日周四 上午11:43写道：

> Hi,
>
> Thanks for the reply.
>
> @Guowei
> I agree that we can move forward step by step and start from the most
> important part. Apart from the two points mentioned in your reply,
> initializing and shutting down some external resources gracefully is also
> important which is a reason for the open/close method.
> About the cluster partitions and the ShuffleMasterContext, I agree that we
> can postpone handling the cluster partitions because we need to do more to
> support it, for ShuffleMasterContext, I think we still need it even we do
> not support the cluster partitions in the first step. Currently, the
> shuffle master can only access the cluster configuration, except that, I
> think we also need need the ability of handling the fatal errors occurring
> in the  ShuffleMaster gracefully by propagate the errors to the framework.
> By introducing the ShuffleMasterContext, we can give ShuffleMaster the
> ability to access both the  cluster configuration and the fatal error
> handler. Instead of passing these components directly to the ShuffleMaster,
> a ShuffleMasterContext interface can keep compatibility easily in the
> future. Even we add some new method in the future, we can offer default
> empty implementation in the interface which can keep compatibility.
> About the JobShuffleContext::getConfiguration/listPartitions methods, I
> agree that we can remove them in the first step and we can add them back
> latter. As mentioned above, we can easily keep compatibility based on the
> Context interface.
>
> @Till
> I totally agree that we should support that different jobs use different
> shuffle services and the proposed solution will support this use case
> eventually.
>
> Best,
> Yingjie
>
> Till Rohrmann  于2021年7月7日周三 下午8:15写道：
>
> > One quick comment: When developing the ShuffleService abstraction we also
> > thought that different jobs might want to use different ShuffleServices
> > depending on their workload (e.g. batch vs. streaming workload). So
> > ideally, the chosen solution here can also support this use case
> > eventually.
> >
> > Cheers,
> > Till
> >
> > On Wed, Jul 7, 2021 at 12:50 PM Guowei Ma  wrote:
> >
> > > Hi,
> > > Thank Yingjie for initiating this discussion. What I understand that
> the
> > > document[1] actually mainly discusses two issues:
> > > 1. ShuffleMaster should be at the cluster level instead of the job
> level
> > > 2. ShuffleMaster should notify PartitionTracker that some data has been
> > > lost
> > >
> > > Relatively speaking, I think the second problem is more serious.
> Because
> > > for external or remote batch shuffling services, after the machine
> > storing
> > > shuffled data goes offline, PartitionTracker needs to be notified in
> time
> > > to avoid repeated failures of the job. Therefore, it is hoped that when
> > > shuffle data goes offline due to a machine error, ShuffleMaster can
> > notify
> > > the PartitionTracker in time. This requires ShuffleMaster to notify the
> > > PartitionTracker with a handle such as JobShuffleContext.
> > >
> > > So how to pass JobShuffleContext to ShuffleMaster? There

[jira] [Created] (FLINK-23324) Postgres of JDBC Connector enable case-sensitive.

2021-07-08 Thread Ada Wong (Jira) 
Ada Wong created FLINK-23324:


 Summary: Postgres of JDBC Connector enable case-sensitive.
 Key: FLINK-23324
 URL: https://issues.apache.org/jira/browse/FLINK-23324
 Project: Flink
  Issue Type: Bug
  Components: Connectors / JDBC
Affects Versions: 1.12.4, 1.13.1
Reporter: Ada Wong


Now the PostgresDialect is case-insensitive. I think this is a bug.

https://stackoverflow.com/questions/20878932/are-postgresql-column-names-case-sensitive
https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS

Could we delete PostgresDialect#quoteIdentifier, make it using super class.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)