[jira] [Created] (FLINK-22957) Rank TTL should use enableTimeToLive of state instead of timer

2021-06-09 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-22957: Summary: Rank TTL should use enableTimeToLive of state instead of timer Key: FLINK-22957 URL: https://issues.apache.org/jira/browse/FLINK-22957 Project: Flink

[jira] [Created] (FLINK-22956) Stream Over TTL should use enableTimeToLive of state instead of timer

2021-06-09 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-22956: Summary: Stream Over TTL should use enableTimeToLive of state instead of timer Key: FLINK-22956 URL: https://issues.apache.org/jira/browse/FLINK-22956 Project: Flink

[jira] [Created] (FLINK-22955) lookup join filter push down result to mismatch function signature

2021-06-09 Thread Cooper Luan (Jira)
Cooper Luan created FLINK-22955: --- Summary: lookup join filter push down result to mismatch function signature Key: FLINK-22955 URL: https://issues.apache.org/jira/browse/FLINK-22955 Project: Flink

Re: [DISCUSS] Definition of idle partitions

2021-06-09 Thread Tzu-Li (Gordon) Tai
Forgot to provide the link to the [1] reference: [1] https://issues.apache.org/jira/browse/FLINK-5017 On Thu, Jun 10, 2021 at 11:43 AM Tzu-Li (Gordon) Tai wrote: > Hi everyone, > > Sorry for chiming in late here. > > Regarding the topic of changing the definition of StreamStatus and > changing

Re: [DISCUSS] Definition of idle partitions

2021-06-09 Thread Tzu-Li (Gordon) Tai
Hi everyone, Sorry for chiming in late here. Regarding the topic of changing the definition of StreamStatus and changing the name as well: After digging into some of the roots of this implementation [1], initially the StreamStatus was actually defined to mark "watermark idleness", and not

1

2021-06-09 Thread fengcheng

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

2021-06-09 Thread Senhong Liu
Hi Piotrek, Thanks for your feedback! 1. Why not ExternallyInducedSourceReader/ExternallyInducedSource? a. The `org.apache.flink.api.connector.source.ExternallyInducedSourceReader` and `org.apache.flink.api.connector.source.ExternallyInducedSource` seems like playing the role of checkpoint

[jira] [Created] (FLINK-22954) Don't support consuming update and delete changes when use table function that does not contain table field

2021-06-09 Thread hehuiyuan (Jira)
hehuiyuan created FLINK-22954: - Summary: Don't support consuming update and delete changes when use table function that does not contain table field Key: FLINK-22954 URL:

Re: [DISCUSS] FLIP-169: DataStream API for Fine-Grained Resource Requirements

2021-06-09 Thread Yangze Guo
Thanks all for the discussion. I've updated the FLIP accordingly, the key changes are: - Introduce SlotSharingGroup instead of ResourceSpec which contains the resource spec of slot sharing group - Introduce two interfaces for specifying the SlotSharingGroup: #slotSharingGroup(SlotSharingGroup) and

Re: [VOTE] Watermark propagation with Sink API

2021-06-09 Thread Tzu-Li (Gordon) Tai
+1 On Thu, Jun 10, 2021 at 9:04 AM jincheng sun wrote: > +1 (binding) // Sorry for the late reply. > > Best, > Jincheng > > > Piotr Nowojski 于2021年6月9日周三 下午10:04写道: > > > Thanks for driving this Eron, and sorry for causing the delay. > > > > +1 (binding) from my side > > > > Piotrek > > > >

[jira] [Created] (FLINK-22953) Using Types.LIST(Types.STRING()) in datastream result in crash

2021-06-09 Thread awayne (Jira)
awayne created FLINK-22953: -- Summary: Using Types.LIST(Types.STRING()) in datastream result in crash Key: FLINK-22953 URL: https://issues.apache.org/jira/browse/FLINK-22953 Project: Flink Issue

[jira] [Created] (FLINK-22952) docs_404_check fail on azure due to ruby version not available

2021-06-09 Thread Xintong Song (Jira)
Xintong Song created FLINK-22952: Summary: docs_404_check fail on azure due to ruby version not available Key: FLINK-22952 URL: https://issues.apache.org/jira/browse/FLINK-22952 Project: Flink

[jira] [Created] (FLINK-22951) First parameter of ROW() function must be constant

2021-06-09 Thread Fu Kai (Jira)
Fu Kai created FLINK-22951: -- Summary: First parameter of ROW() function must be constant Key: FLINK-22951 URL: https://issues.apache.org/jira/browse/FLINK-22951 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-22950) Back-pressure report final sink operator is not accurate

2021-06-09 Thread Fu Kai (Jira)
Fu Kai created FLINK-22950: -- Summary: Back-pressure report final sink operator is not accurate Key: FLINK-22950 URL: https://issues.apache.org/jira/browse/FLINK-22950 Project: Flink Issue Type: Bug

Re: [VOTE] Watermark propagation with Sink API

2021-06-09 Thread jincheng sun
+1 (binding) // Sorry for the late reply. Best, Jincheng Piotr Nowojski 于2021年6月9日周三 下午10:04写道: > Thanks for driving this Eron, and sorry for causing the delay. > > +1 (binding) from my side > > Piotrek > > wt., 8 cze 2021 o 23:48 Eron Wright > napisał(a): > > > Voting is re-open for

[jira] [Created] (FLINK-22949) java.io.InvalidClassException With Flink Kafka Beam

2021-06-09 Thread Ravikiran Borse (Jira)
Ravikiran Borse created FLINK-22949: --- Summary: java.io.InvalidClassException With Flink Kafka Beam Key: FLINK-22949 URL: https://issues.apache.org/jira/browse/FLINK-22949 Project: Flink

Re: [DISCUSS] FLIP-171: Async Sink

2021-06-09 Thread Piotr Nowojski
Hi Steffen, Thanks for writing down the proposal. Back when the new Sink API was being discussed, I was proposing to add our usual `CompletableFuture isAvailable()` pattern to make sinks non-blocking. You can see the discussion starting here [1], and continuing for a couple of more posts until

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-06-09 Thread Arvid Heise
Hi Piot, I'm fine with just doing it on the Sink. My responses were focused on the API (the how) not on the concept (the if). Just keep the methods on the different places in sync, such that it is easy to introduce a common interface later. Re name: drain is not a reinvention as it's used quite

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-06-09 Thread Piotr Nowojski
Hi, Arvid: What's the problem with providing `void flush()`/`void drain()` only in the `SinkFunction`? It would avoid the problem of typing. Why would one need to have it in the other `Rich***Function`s? For `flush()` to make sense, the entity which has this method, would need to buffer some

[jira] [Created] (FLINK-22948) Scala example for toDataStream does not compile

2021-06-09 Thread David Anderson (Jira)
David Anderson created FLINK-22948: -- Summary: Scala example for toDataStream does not compile Key: FLINK-22948 URL: https://issues.apache.org/jira/browse/FLINK-22948 Project: Flink Issue

Re: [VOTE] Watermark propagation with Sink API

2021-06-09 Thread Piotr Nowojski
Thanks for driving this Eron, and sorry for causing the delay. +1 (binding) from my side Piotrek wt., 8 cze 2021 o 23:48 Eron Wright napisał(a): > Voting is re-open for FLIP-167 as-is (without idleness support as was the > point of contention). > > On Fri, Jun 4, 2021 at 10:45 AM Eron Wright

[jira] [Created] (FLINK-22947) Reduce DataSet visibility in documentation

2021-06-09 Thread Seth Wiesman (Jira)
Seth Wiesman created FLINK-22947: Summary: Reduce DataSet visibility in documentation Key: FLINK-22947 URL: https://issues.apache.org/jira/browse/FLINK-22947 Project: Flink Issue Type:

[jira] [Created] (FLINK-22946) Network buffer deadlock introduced by unaligned checkpoint

2021-06-09 Thread Guokuai Huang (Jira)
Guokuai Huang created FLINK-22946: - Summary: Network buffer deadlock introduced by unaligned checkpoint Key: FLINK-22946 URL: https://issues.apache.org/jira/browse/FLINK-22946 Project: Flink

Re: [DISCUSS] Dashboard/HistoryServer authentication

2021-06-09 Thread Till Rohrmann
Thanks for the update Gabor. I'll take a look and respond in the document. Cheers, Till On Wed, Jun 9, 2021 at 12:59 PM Gabor Somogyi wrote: > Hi Till, > > Your proxy suggestion has been considered in-depth and updated the FLIP > accordingly. > We've considered 2 proxy implementation (Nginx

Re: [DISCUSS][Statebackend][Runtime] Changelog Statebackend Configuration Proposal

2021-06-09 Thread Yuan Mei
Thank you everyone for replying! Option 3 wins with dominating # of votes + mine. This option works as a refined version of the original proposal in FLIP-158: Generalized incremental checkpoints [1]: - Define consistent override and combination policy (flag + state backend) in different config

[jira] [Created] (FLINK-22945) StackOverflowException can happen when a large scale job is CANCELED/FAILED

2021-06-09 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-22945: --- Summary: StackOverflowException can happen when a large scale job is CANCELED/FAILED Key: FLINK-22945 URL: https://issues.apache.org/jira/browse/FLINK-22945 Project: Flink

Re: [DISCUSS] Dashboard/HistoryServer authentication

2021-06-09 Thread Gabor Somogyi
Hi Till, Your proxy suggestion has been considered in-depth and updated the FLIP accordingly. We've considered 2 proxy implementation (Nginx and Squid) but according to our analysis and testing it's not suitable for the mentioned use-cases. Please take a look at the rejected alternatives for

[jira] [Created] (FLINK-22944) Optimize writing state changes

2021-06-09 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-22944: - Summary: Optimize writing state changes Key: FLINK-22944 URL: https://issues.apache.org/jira/browse/FLINK-22944 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-06-09 Thread Arvid Heise
Hi Dawid, I see your point. I'd probably add drain only to Rich*Function where we have the type bounds. Then we still need your Flushable interface in Rich*Function<..., T> to call it efficiently but we at least avoid weird type combinations. I'll have a rethink later. The proper solution is

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-06-09 Thread Dawid Wysakowicz
Hey, @Arvid The problem with adding the "drain/flush/stopProcessing" method to RichFunction is that it is not typed with the output type. At the same time we would most likely need a way to emit records from the method. That's originally thought about adding a typed interface which honestly I

[jira] [Created] (FLINK-22943) java.lang.ClassCastException: java.time.Instant cannot be cast to java.sql.Timestamp

2021-06-09 Thread jack wang (Jira)
jack wang created FLINK-22943: - Summary: java.lang.ClassCastException: java.time.Instant cannot be cast to java.sql.Timestamp Key: FLINK-22943 URL: https://issues.apache.org/jira/browse/FLINK-22943

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-06-09 Thread Arvid Heise
I have not followed the complete discussion and can't comment on the concepts. However, I have some ideas on the API changes: 1. If it's about adding additional life-cycle methods to UDFs, we should add the flush/endOfInput to RichFunction as this is the current way to define it. At this point, I

Re: State migration scenario's

2021-06-09 Thread Yun Tang
Hi Marlo, One of the scenarios that we're trying to improve is to add or remove one field in state serializer. Users might add or remove one field during their schema evolution, state processor could help it with another offline job while state migration could help it once we restart the new

[jira] [Created] (FLINK-22942) Disable upsert into syntax in Flink SQL

2021-06-09 Thread Leonard Xu (Jira)
Leonard Xu created FLINK-22942: -- Summary: Disable upsert into syntax in Flink SQL Key: FLINK-22942 URL: https://issues.apache.org/jira/browse/FLINK-22942 Project: Flink Issue Type: Improvement

State migration scenario's

2021-06-09 Thread Marlo Ploemen
Hi community, I am looking into the data migration and schema evolution process for stateful streaming jobs. Currently, there is no orchestration support for performing these job evolutions and no in-job state migration or schema evolution syntax (as this is part of the separate state

[DISCUSS] FLIP-171: Async Sink

2021-06-09 Thread Hausmann, Steffen
Hi there, We would like to start a discussion thread on "FLIP-171: Async Sink" [1], where we propose to create a common abstraction for destinations that support async requests. This abstraction will make it easier to add destinations to Flink by implementing a lightweight shim, while it

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-06-09 Thread Till Rohrmann
Thanks for the lively discussion everyone. I have to admit that I am not really convinced that we should call the interface Flushable and the method flush. The problem is that this method should in the first place tell the operator that it should stop processing and flush all buffered data. The

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

2021-06-09 Thread Piotr Nowojski
Hi Senhong, Thanks for the proposal. I have a couple of questions. Have you seen `org.apache.flink.streaming.api.checkpoint.ExternallyInducedSource` (for the legacy SourceFunction) and `org.apache.flink.api.connector.source.ExternallyInducedSourceReader` (for FLIP-27) interfaces? They work the

[jira] [Created] (FLINK-22941) support column comment in catalogTable column schema

2021-06-09 Thread peng wang (Jira)
peng wang created FLINK-22941: - Summary: support column comment in catalogTable column schema Key: FLINK-22941 URL: https://issues.apache.org/jira/browse/FLINK-22941 Project: Flink Issue Type:

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

2021-06-09 Thread Senhong Liu
Here is some brief context about the new feature. 1. Actively checkpoint rejecting by the operator. Follow by the current checkpoint mechanism, one more preliminary step is added to help the operator determine that if it is able to take snapshots. The preliminary step is a new API provided to the

Re: [DISCUSS] Definition of idle partitions

2021-06-09 Thread Arvid Heise
Hi Eron, again to recap from the other thread: - You are right that idleness is correct with static assignment and fully active partitions. In this case, the source defines idleness. (case A) - For the more pressing use cases of idle, assigned partitions, the user defines an idleness threshold,

Re: Request to open the contributor permission!

2021-06-09 Thread Yun Tang
Hi hapihu, Welcome to Apache Flink community! You don't need to ask contributor permission for Flink JIRA issues now, and you could comment in the issue which you're interested to ask as to be assigned. You could also find more details in [1] [1]

[jira] [Created] (FLINK-22940) Make SQL column max column widh configurable

2021-06-09 Thread Svend Vanderveken (Jira)
Svend Vanderveken created FLINK-22940: - Summary: Make SQL column max column widh configurable Key: FLINK-22940 URL: https://issues.apache.org/jira/browse/FLINK-22940 Project: Flink Issue