Re: [DISCUSS] Flink SQL DDL Design

Hi Shuyi and Shaoxuan, It's great to have a collaboration with you guys! Jark and I drafted an google doc [1] mainly about the DDL syntax part which summarizing the current version in Alibaba. We can continue discussing the DDL design and API changes based on your doc. Let's moving this forward!

Re: [DISCUSS] Long-term goal of making flink-table Scala-free

Hi Timo, Thanks for the great work! Moving flink-table to Java is a long-awaited things but will involve much effort. Agree with that we should make it as a long-term goal. I have read the google doc and +1 for the proposal. Here I have some questions: 1. Where should the flink-table-common

Submitting job with savepoint through StreamExecutionEnvironment

Hi, Currently it is not possible to submit a job with savepoint restore option through the execution environment. I found that while attempting to add the support to the Flink runner in Beam ( https://issues.apache.org/jira/browse/BEAM-5396) I also found

[jira] [Created] (FLINK-11024) Overwrite yarn configuration by setting configuration started with "flink.yarn." in flink-conf.yaml

liuzhaokun created FLINK-11024: -- Summary: Overwrite yarn configuration by setting configuration started with "flink.yarn." in flink-conf.yaml Key: FLINK-11024 URL: https://issues.apache.org/jira/browse/FLINK-11024

Re: [DISCUSS] Flink SQL DDL Design

Hi Shuyi, re: we should take into account the new external catalog effort in the design Definitely. We've had a much better understanding of catalogs over the past few weeks, and will be more than happy to contribute to that part. Thanks, Bowen On Wed, Nov 28, 2018 at 4:12 PM Zhang, Xuefu

Re: [DISCUSS]Enhancing flink scheduler by implementing blacklist mechanism

Thanks, Yun. If the external resource manager cannot fully obey the blacklist, then we have two choices. The first one is do nothing and using the returned resources directly. The other one is checking the returned resource, then releasing the unsatisfied resource and reallocating until success

Re: [DISCUSS]Enhancing flink scheduler by implementing blacklist mechanism

You are right. I think, at least, we need a new interface to be implemented to collect the failure information. Best, Yingjie Guowei Ma 于2018年11月28日周三 下午2:23写道： > thanks yingjie to share this doc and I think this is very important feature > for production. > > As you mentioned in your

Re: [DISCUSS]Enhancing flink scheduler by implementing blacklist mechanism

Thanks, Weihua. Your suggestions make a lot of sense to me. Currently, all blacklisted resources will be released from blacklist if there is no available resource. Maybe only releasing a portion of the blacklisted resources based on the number of slots needed and some LRU like algorithm is a

Re: [DISCUSS] Support Interactive Programming in Flink Table API

Hi Piotrek, Thanks for the explanation. Right now we are mostly thinking of the cached table as immutable. I can see the Materialized view would be useful in the future. That said, I think a simple cache mechanism is probably still needed. So to me, cache() and materialize() should be two

Re: [DISCUSS] Flink SQL DDL Design

+1 Sounds great! -- Sender:Shuyi Chen Sent at:2018 Nov 29 (Thu) 06:56 Recipient:dev Subject:Re: [DISCUSS] Flink SQL DDL Design Thanks a lot, Shaoxuan, Jack and Lin. We should definitely collaborate here, we have also our own DDL

[VOTE] Release 1.7.0, release candidate #3

Hi everyone, Please review and vote on the release candidate #3 for the version 1.7.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

Re: [DISCUSS] Flink SQL DDL Design

Thanks a lot, Shaoxuan, Jack and Lin. We should definitely collaborate here, we have also our own DDL implementation running in production for almost 2 years at Uber. With the joint experience from both companies, we can definitely make the Flink SQL DDL better. As @shaoxuan suggest, Jark can

[jira] [Created] (FLINK-11023) Update LICENSE and NOTICE files for flink-connectors

Till Rohrmann created FLINK-11023: - Summary: Update LICENSE and NOTICE files for flink-connectors Key: FLINK-11023 URL: https://issues.apache.org/jira/browse/FLINK-11023 Project: Flink Issue

[jira] [Created] (FLINK-11022) Update LICENSE and NOTICE files for older releases

Till Rohrmann created FLINK-11022: - Summary: Update LICENSE and NOTICE files for older releases Key: FLINK-11022 URL: https://issues.apache.org/jira/browse/FLINK-11022 Project: Flink Issue

[jira] [Created] (FLINK-11021) ZoneOffset objects don't appear to be serialized correctly

Christina created FLINK-11021: - Summary: ZoneOffset objects don't appear to be serialized correctly Key: FLINK-11021 URL: https://issues.apache.org/jira/browse/FLINK-11021 Project: Flink Issue

Re: [DISCUSS] Flink SQL DDL Design

Hi Shaoxuan, I think summarizing it into a google doc is a good idea. We will prepare it in the next few days. Thanks, Jark Shaoxuan Wang 于2018年11月28日周三 下午9:17写道： > Hi Lin and Jark, > Thanks for sharing those details. Can you please consider summarizing your > DDL design into a google doc. >

Re: Custom scheduler in Flink

Thanks, I'll check it out. *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler wrote: > There's no *reasonable *way to implement a custom Scheduler,

Re: Custom scheduler in Flink

There's no /reasonable /way to implement a custom Scheduler, i.e., something where can just plug in your scheduler in a nice way. For this you'll have to directly modify the source of Flink. The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be of interest, but is still in

Re: Apply for contributor permission

I've given you contributor permissions, but whether or not you are assigned to a JIRA does not affect whether the PR is reviewed or not. Committers are simply overwhelmed with the number of incoming PRs. On 28.11.2018 04:25, Paul Lam wrote: Hi! I find some bug-fix PRs I created weeks ago are

Re: [DISCUSS] Flink SQL DDL Design

Hi Lin and Jark, Thanks for sharing those details. Can you please consider summarizing your DDL design into a google doc. We can still continue the discussions on Shuyi's proposal. But having a separate google doc will be easy for the DEV to understand/comment/discuss on your proposed DDL

[jira] [Created] (FLINK-11020) Reorder joins only to eliminate cross joins

Timo Walther created FLINK-11020: Summary: Reorder joins only to eliminate cross joins Key: FLINK-11020 URL: https://issues.apache.org/jira/browse/FLINK-11020 Project: Flink Issue Type:

[jira] [Created] (FLINK-11019) Expose the queryable proxy server port in taskinfo api

Jiayi Liao created FLINK-11019: -- Summary: Expose the queryable proxy server port in taskinfo api Key: FLINK-11019 URL: https://issues.apache.org/jira/browse/FLINK-11019 Project: Flink Issue

Re: [DISCUSS] Flink SQL DDL Design

Hi Shuyi, Thanks for bringing up this discussion and the awesome work! I have left some comments in the doc. I want to share something more about the watermark definition learned from Alibaba. 1. Table should be able to accept multiple watermark definition. Because a table may have

[jira] [Created] (FLINK-11018) Flink ignores logback.xml

prathamesh created FLINK-11018: -- Summary: Flink ignores logback.xml Key: FLINK-11018 URL: https://issues.apache.org/jira/browse/FLINK-11018 Project: Flink Issue Type: Bug Components:

[jira] [Created] (FLINK-11017) Time interval for window aggregations in SQL is wrongly translated if specified with YEAR_MONTH resolution

Dawid Wysakowicz created FLINK-11017: Summary: Time interval for window aggregations in SQL is wrongly translated if specified with YEAR_MONTH resolution Key: FLINK-11017 URL:

Re: [DISCUSS] Flink SQL DDL Design

@Shuyi Thanks for the proposal! We have a simple DDL implementation (extends Calcite's parser) which been running for almost two years on production and works well. I think the most valued things we'd learned is keeping simplicity and standard compliance. Here's the approximate grammar, FYI

Re: [DISCUSS] Table API Enhancement Outline

Hi Fabian, Thank you for listing the detailed example of forcing the use of select. If I didn't make it clear before, I would like to share my thoughts about the group keys here： 1. agg/flatagg(Expression) keeps a single Expression; 2. The way to force users to use select is as follows(As

Re: [DISCUSS] Flink backward compatibility

so let's take a look... binary client compatibility: The key issue i see hasn't changed since the last time this was brought up: Clients rely on the JobGraph to submit the job which is an internal data structure. AFAIK there will also be changes made to said class soon(ish). So long as we

Custom scheduler in Flink

Hi, I want to develop a custom scheduler in Flink to be aware for which host Flink must process some task. This post shows (using Apache Storm) the kind of example I want to build ( https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b605e37e ). I

[jira] [Created] (FLINK-11016) flink chain have a bug？？？

Wong created FLINK-11016: Summary: flink chain have a bug？？？ Key: FLINK-11016 URL: https://issues.apache.org/jira/browse/FLINK-11016 Project: Flink Issue Type: Bug Components: Core

Questions about UDTF in flink SQL

Hi all, When using user-defined table function in Flink SQL, it seems that the result type of a table function must be determinstic. If I want a UDTF whose result type is determined by its input parameters, what should I do? What I want to do is like this: ``` SELECT input, f1, f2 length

回复：[DISCUSS] Proposal of external shuffle service

Hi all, I adjusted the umbrella jira [1] and corresponding google doc [2] to narrow down the scope of introducing pluggable shuffle manager architecture as the first step. Welcome further feedbacks and suggestions, then I would create specific subtasks for it to forward. [1]