[jira] [Created] (FLINK-19199) Add execution attempt ID to feedback channel keys

Tzu-Li (Gordon) Tai created FLINK-19199: --- Summary: Add execution attempt ID to feedback channel keys Key: FLINK-19199 URL: https://issues.apache.org/jira/browse/FLINK-19199 Project: Flink

[jira] [Created] (FLINK-19198) Invalid default value for consume-start-offset in Hive streaming

Paul Lin created FLINK-19198: Summary: Invalid default value for consume-start-offset in Hive streaming Key: FLINK-19198 URL: https://issues.apache.org/jira/browse/FLINK-19198 Project: Flink

[jira] [Created] (FLINK-19197) Add documentation for PersistedStateRegistry

Tzu-Li (Gordon) Tai created FLINK-19197: --- Summary: Add documentation for PersistedStateRegistry Key: FLINK-19197 URL: https://issues.apache.org/jira/browse/FLINK-19197 Project: Flink

Re: [DISCUSS] FLIP-143: Unified Sink API

Guowei, Thanks a lot for the proposal and starting the discussion thread. Very excited. For the big question of "Is the sink an operator or a topology?", I have a few related sub questions. * Where should we run the committers? * Is the committer parallel or single parallelism? * Can a single

[jira] [Created] (FLINK-19196) FlinkSQL aggregation function to aggregate array typed column to Map/Multiset

sam lin created FLINK-19196: --- Summary: FlinkSQL aggregation function to aggregate array typed column to Map/Multiset Key: FLINK-19196 URL: https://issues.apache.org/jira/browse/FLINK-19196 Project: Flink

Re: [VOTE] FLIP-107: Handling of metadata in SQL connectors

+1 (binding) On Thu, Sep 10, 2020 at 4:29 PM Dawid Wysakowicz wrote: > +1 (binding) > > On 10/09/2020 14:03, Aljoscha Krettek wrote: > > +1 (binding) > > > > Aljoscha > > > > On 10.09.20 13:57, Timo Walther wrote: > >> Hi all, > >> > >> after the discussion in [1], I would like to open a voting

Re: [DISCUSS] FLIP-107: Reading table columns from different parts of source records

Hi everyone, I had to add some last minute changes to FLIP-107. Jark pointed out that we might also discuss the implications to the `Schema` class from FLIP-129 as well as the LIKE clause from FLIP-110. I added another small section to the document. I just started a vote, if you have

Notification of analysis on publicly available project data

Dear PMC, I’m contacting you because your project has been selected by the ASF D committee which is leading a research project to evaluate and understand the current state of diversity in our community [1]. As part of this research, we will analyze publicly available data about your project such

[jira] [Created] (FLINK-19195) question on security vulnerabilities in flink

Miguel Costa created FLINK-19195: Summary: question on security vulnerabilities in flink Key: FLINK-19195 URL: https://issues.apache.org/jira/browse/FLINK-19195 Project: Flink Issue Type:

Re: [DISCUSS] Drop Scala 2.11

@glen Yes, we would absolutely migrate statefun. StateFun can be compiled with Scala 2.12 today, I'm not sure why it's not cross released. @aljoscha :) @mathieu Its on the roadmap but it's non-trivial and I'm not aware of anyone actively working on it. On Thu, Sep 10, 2020 at 10:09 AM Matthieu

Re: [DISCUSS] Drop Scala 2.11

That makes sense. We are using 2.12 for our production Also, for flink scala 2.12 support, it is in fact limited to scala 2.12.7. It is binary incompatible with version 2.12 above ( https://issues.apache.org/jira/browse/FLINK-12461 ) That would be great to at least move to a more recent 2.12

Re: [DISCUSS] Drop Scala 2.11

Yes! I would be in favour of this since it's blocking us from upgrading certain dependencies. I would also be in favour of dropping Scala completely but that's a different story. Aljoscha On 10.09.20 16:51, Seth Wiesman wrote: Hi Everyone, Think of this as a pre-flip, but what does

Re: [DISCUSS] Drop Scala 2.11

I'd be fine with dropping Scala 2.11. Stateful Functions currently requires 2.11, though, so that would have to be addressed as well. I'm using 2.11 in a production project now but only because of the Stateful Functions dependency. If I could use a newer Scala version that would be

[DISCUSS] Drop Scala 2.11

Hi Everyone, Think of this as a pre-flip, but what does everyone think about dropping Scala 2.11 support from Flink. The last patch release was in 2017 and in that time the scala community has released 2.13 and is working towards a 3.0 release. Apache Kafka and Spark have both dropped 2.11

[ANNOUNCE] Feature freeze for Stateful Functions 2.2.0

Hi all, Following the discussion in [1], we've finished all previously mentioned features in time (planned date was today, 10 Sep.). Therefore, I'm happy to announce feature freeze for StateFun 2.2.0, effective today. We still have a bit of documentation to work on as well as some recently

Re: add support for ScalaPB-based message-payload serialization to Stateful Functions?

Hi Galen, Welcome to the dev list! I'll make sure the right person responds to the ticket. Seth On Wed, Sep 9, 2020 at 5:27 PM Galen Warren wrote: > Hi all -- I created a ticket > regarding a proposal to > add a new message-payload

Re: [VOTE] FLIP-107: Handling of metadata in SQL connectors

+1 (binding) On 10/09/2020 14:03, Aljoscha Krettek wrote: > +1 (binding) > > Aljoscha > > On 10.09.20 13:57, Timo Walther wrote: >> Hi all, >> >> after the discussion in [1], I would like to open a voting thread for >> FLIP-107 [2] which discusses how to handle data next to the main >> payload

Re: [VOTE] FLIP-140: Introduce bounded style execution for keyed streams

+1 (binding) Aljoscha

[DISCUSS] FLIP-143: Unified Sink API

Hi, devs & users As discussed in FLIP-131[1], Flink will deprecate the DataSet API in favor of DataStream API and Table API. Users should be able to use DataStream API to write jobs that support both bounded and unbounded execution modes. However, Flink does not provide a sink API to guarantee

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Having just re-read FLIP-134, I think it mostly makes sense, though I'm not exactly looking forward to figuring out how to explain it without making it seem overly complicated. A few points: I'm a bit confused by the discussion around custom window Triggers. Yes, I agree that complex, mixed

[VOTE] FLIP-140: Introduce bounded style execution for keyed streams

Hi all, The discussion[1] seem to have concluded and therefore I'd like to start a vote on FLIP-140[2]. The vote will be open until 15th Sep (72h), unless there is an objection or not enough votes. [1]

Re: [DISCUSS] FLIP-107: Reading table columns from different parts of source records

Hi, Timo Thanks for the update I have a minor suggestion about the debezium metadata key, Could we use the original debezium key rather than import new key? debezium-json.schema => debezium-json.schema debezium-json.ingestion-timestamp => debezium-json.ts_ms

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Thanks for the explanation! Makes sense now! What do you think about adding this behaviour of WindowAssigner in streaming mode as well? I mean the behaviour of emitting at the end of a Window. I think it would make sense in the STREAM mode as well and keep the two modes more aligned. Best, Dawid

[jira] [Created] (FLINK-19194) The UDF split and split_index get wrong result

fa zheng created FLINK-19194: Summary: The UDF split and split_index get wrong result Key: FLINK-19194 URL: https://issues.apache.org/jira/browse/FLINK-19194 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19193) Upgrade migration guidelines to use stop-with-savepoint

Aljoscha Krettek created FLINK-19193: Summary: Upgrade migration guidelines to use stop-with-savepoint Key: FLINK-19193 URL: https://issues.apache.org/jira/browse/FLINK-19193 Project: Flink

Re: [VOTE] FLIP-107: Handling of metadata in SQL connectors

+1 (binding) Aljoscha On 10.09.20 13:57, Timo Walther wrote: Hi all, after the discussion in [1], I would like to open a voting thread for FLIP-107 [2] which discusses how to handle data next to the main payload (i.e. key and value formats as well as metadata in general) in SQL connectors

[VOTE] FLIP-107: Handling of metadata in SQL connectors

Hi all, after the discussion in [1], I would like to open a voting thread for FLIP-107 [2] which discusses how to handle data next to the main payload (i.e. key and value formats as well as metadata in general) in SQL connectors and DDL. The vote will be open until September 15th (72h +

[jira] [Created] (FLINK-19192) Set higher limit on the HTTP connection pool

Igal Shilman created FLINK-19192: Summary: Set higher limit on the HTTP connection pool Key: FLINK-19192 URL: https://issues.apache.org/jira/browse/FLINK-19192 Project: Flink Issue Type:

[jira] [Created] (FLINK-19191) Reduce the default number for async operations

Igal Shilman created FLINK-19191: Summary: Reduce the default number for async operations Key: FLINK-19191 URL: https://issues.apache.org/jira/browse/FLINK-19191 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-107: Reading table columns from different parts of source records

I've only been watching this from the sidelines but that latest proposal looks very good to me! Aljoscha On 10.09.20 12:20, Kurt Young wrote: The new syntax looks good to me. Best, Kurt On Thu, Sep 10, 2020 at 5:57 PM Jark Wu wrote: Hi Timo, I have one minor suggestion. Maybe the

Re: [DISCUSS] FLIP-107: Reading table columns from different parts of source records

The new syntax looks good to me. Best, Kurt On Thu, Sep 10, 2020 at 5:57 PM Jark Wu wrote: > Hi Timo, > > I have one minor suggestion. > Maybe the default data type of `timestamp` can be `TIMESTAMP(3) WITH > LOCAL TIME ZONE`, because this is the type that users want to use, this can > avoid

Re: [DISCUSS] FLIP-107: Reading table columns from different parts of source records

Hi Timo, I have one minor suggestion. Maybe the default data type of `timestamp` can be `TIMESTAMP(3) WITH LOCAL TIME ZONE`, because this is the type that users want to use, this can avoid unnecessary casting. Besides, currently, the bigint is casted to timestamp in seconds, so the implicit cast

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

On 10.09.20 11:30, Dawid Wysakowicz wrote: I am not sure about the option for ignoring the Triggers. Do you mean to ignore all the Triggers including e.g. Flink's such as CountTrigger, EventTimeTrigger etc.? Won't it effectively disable the WindowOperator whatsoever. Or even worse make it

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Thanks for the update Aljoscha! I am not sure about the option for ignoring the Triggers. Do you mean to ignore all the Triggers including e.g. Flink's such as CountTrigger, EventTimeTrigger etc.? Won't it effectively disable the WindowOperator whatsoever. Or even worse make it unusable with ever

[jira] [Created] (FLINK-19190) Use camelCase for metric names

Igal Shilman created FLINK-19190: Summary: Use camelCase for metric names Key: FLINK-19190 URL: https://issues.apache.org/jira/browse/FLINK-19190 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-19189) Enable pipelined scheduling by default

Zhu Zhu created FLINK-19189: --- Summary: Enable pipelined scheduling by default Key: FLINK-19189 URL: https://issues.apache.org/jira/browse/FLINK-19189 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-36 - Support Interactive Programming in Flink Table API

On 10.09.20 09:00, Xuannan Su wrote: How do you imagine that? Where do you distinguish between per-job and session mode? The StreamExecutionEnvironment can distinguish between per-job and session mode by the type of the PipelineExecutor, i.e, AbstractJobClusterExecutor vs

[jira] [Created] (FLINK-19188) Add a new streaming SQL examples

Timo Walther created FLINK-19188: Summary: Add a new streaming SQL examples Key: FLINK-19188 URL: https://issues.apache.org/jira/browse/FLINK-19188 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19187) Add a new basic Table API example

Timo Walther created FLINK-19187: Summary: Add a new basic Table API example Key: FLINK-19187 URL: https://issues.apache.org/jira/browse/FLINK-19187 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19186) Add Python building blocks to make sure the basic functionality of Pandas Batch Group Aggregation could work

Huang Xingbo created FLINK-19186: Summary: Add Python building blocks to make sure the basic functionality of Pandas Batch Group Aggregation could work Key: FLINK-19186 URL:

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

Thanks for driving this Timo, +1 for voting ~ Best, Danny Chan 在 2020年9月10日 +0800 PM3:54，Timo Walther ，写道： > Thanks everyone for this healthy discussion. I updated the FLIP with the > outcome. I think the result is one of the last core API refactoring and > users will be happy to have a

Re: [DISCUSS] FLIP-107: Reading table columns from different parts of source records

Thanks for driving this Timo, +1 for voting ~ Best, Danny Chan 在 2020年9月10日 +0800 PM3:47，Timo Walther ，写道： > Thanks everyone for this healthy discussion. I updated the FLIP with the > outcome. I think the result is very powerful but also very easy to > declare. Thanks for all the contributions. >

Re: [DISCUSS] FLIP-36 - Support Interactive Programming in Flink Table API

Hi Xuannan, Thanks for updating the FLIP and answering my questions. The FLIP is good to go from my side. What do others think? Regards, Timo On 10.09.20 09:00, Xuannan Su wrote: Hi Timo, Thanks for pointing out the mistake, I have made the update accordingly. And I added more information

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

Thanks everyone for this healthy discussion. I updated the FLIP with the outcome. I think the result is one of the last core API refactoring and users will be happy to have a consistent changelog support. Thanks for all the contributions. If there are no objections, I would continue with a

[jira] [Created] (FLINK-19185) Support General Python User-Defined Aggregate Function Support on Table API (FLIP-139)

Wei Zhong created FLINK-19185: - Summary: Support General Python User-Defined Aggregate Function Support on Table API (FLIP-139) Key: FLINK-19185 URL: https://issues.apache.org/jira/browse/FLINK-19185

Re: [DISCUSS] FLIP-107: Reading table columns from different parts of source records

Thanks everyone for this healthy discussion. I updated the FLIP with the outcome. I think the result is very powerful but also very easy to declare. Thanks for all the contributions. If there are no objections, I would continue with a voting. What do you think? Regards, Timo On 09.09.20

[RESULT][VOTE] FLIP-139: General Python User-Defined Aggregate Function on Table API

Hi all, The voting time for FLIP-139 has passed. I'm closing the vote now. There were 4 +1 votes, 3 of which are binding: - Jincheng (binding) - Dian (binding) - Hequn (binding) - Xingbo (non-binding) There were no disapproving votes. Thus, FLIP-139 has been accepted. Thanks everyone for

Re: [VOTE] FLIP-139: General Python User-Defined Aggregate Function on Table API

Thanks everyone for the votes! I’ll summarize the voting result in a separate email. Best, Wei > 在 2020年9月10日，10:04，Hequn Cheng 写道： > > +1 (binding) > > Best, > Hequn > > On Thu, Sep 10, 2020 at 10:03 AM Dian Fu wrote: > >> +1(binding) >> >> Regards, >> Dian >> >>> 在

[VOTE] Release 1.11.2, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 1.11.2, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

Re: [DISCUSS] FLIP-36 - Support Interactive Programming in Flink Table API

Hi Timo, Thanks for pointing out the mistake, I have made the update accordingly. And I added more information to the FLIP to address the problem you raised in the email. To sum it up: > 1. How does `ClusterPartitionDescriptor` look like? The ClusterPartitionDescriptor will include the

[jira] [Created] (FLINK-19184) Add Batch Physical Pandas Group Aggregate Rule and RelNode

Huang Xingbo created FLINK-19184: Summary: Add Batch Physical Pandas Group Aggregate Rule and RelNode Key: FLINK-19184 URL: https://issues.apache.org/jira/browse/FLINK-19184 Project: Flink