Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Dawid Wysakowicz
Hi all, Just an opinion on the built-in <> temporary functions resolution and NAMING issue. I think we should not allow overriding the built-in functions, as this may pose serious issues and to be honest is rather not feasible and would require major rework. What happens if a user wants to overrid

How to calculate one day's uv every minute by SQL

2019-09-03 Thread 刘建刚
We want to calculate one day's uv and show the result every minute . We have implemented this by java code: dataStream.keyBy(dimension) .incrementWindow(Time.days(1), Time.minutes(1)) .uv(userId) The input data is big. So we use Va

[jira] [Created] (FLINK-13952) PartitionableTableSink can not work with OverwritableTableSink

2019-09-03 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-13952: Summary: PartitionableTableSink can not work with OverwritableTableSink Key: FLINK-13952 URL: https://issues.apache.org/jira/browse/FLINK-13952 Project: Flink

Re: [DISCUSS] Releasing Flink 1.8.2

2019-09-03 Thread Jark Wu
Thanks for the work Jincheng! I have moved remaining major issues to 1.8.3 except FLINK-13586. Hi @Aljoscha Krettek , is that possible to merge FLINK-13586 today? Best, Jark On Wed, 4 Sep 2019 at 10:47, jincheng sun wrote: > Thanks for the udpate Jark! > > I have add the new version 1.8.3 in

Re: [DISCUSS] META-FLIP: Sticking (or not) to a strict FLIP voting process

2019-09-03 Thread Becket Qin
Thanks for the summary, Yu. Hi all, Just want to loop back on this thread. The META-FLIP actually does not change much in the current FLIP process. It did add a few more details that are currently not explicitly written down in the FLIP wiki page. More specifically: 1. In general FLIPs should be

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Bowen Li
Hi, I agree with Xuefu that the main controversial points are mainly the two places. My thoughts on them: 1) Determinism of referencing Hive built-in functions. We can either remove Hive built-in functions from ambiguous function resolution and require users to use special syntax for their qualif

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread JingsongLee
Thank you for your wonderful points. I like timo's proposal to enrich built-in functions to flexible function modules (For example, the financial model is useful to bank system). But I agree with bowen, I don't think hive functions deserves be a function module. I think all function modules sho

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Xuefu Z
>From what I have seen, there are a couple of focal disagreements: 1. Resolution order: temp function --> flink built-in function --> catalog function vs flink built-in function --> temp function -> catalog function. 2. "External" built-in functions: how to treat built-in functions in external sys

[jira] [Created] (FLINK-13951) Unable to call limit without sort for batch mode

2019-09-03 Thread Jeff Zhang (Jira)
Jeff Zhang created FLINK-13951: -- Summary: Unable to call limit without sort for batch mode Key: FLINK-13951 URL: https://issues.apache.org/jira/browse/FLINK-13951 Project: Flink Issue Type: Impr

Re: [DISCUSS] Releasing Flink 1.8.2

2019-09-03 Thread jincheng sun
Thanks for the udpate Jark! I have add the new version 1.8.3 in JIRA, could you please remark the JIRAs(Such as FLINK-13689) which we do not want merge into the 1.8.2 release :) You are right, I think FLINK-13586 is better to be contained in 1.8.2 release! Thanks, Jincheng Jark Wu 于2019年9月4日

[jira] [Created] (FLINK-13950) should report an error if hive table partition keys are not in the last schema fields

2019-09-03 Thread Hongtao Zhang (Jira)
Hongtao Zhang created FLINK-13950: - Summary: should report an error if hive table partition keys are not in the last schema fields Key: FLINK-13950 URL: https://issues.apache.org/jira/browse/FLINK-13950

Re: [DISCUSS] Releasing Flink 1.8.2

2019-09-03 Thread Jark Wu
Hi all, I am very happy to say that all the blockers and critical issues for release 1.8.2 have been resolved! Great thanks to everyone who contribute to the release. I hope to create the first RC on Sep 05, at 10:00 UTC+8. If you find some other blocker issues for 1.8.2, please let me know befo

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Kurt Young
Thanks Timo & Bowen for the feedback. Bowen was right, my proposal is the same as Bowen's. But after thinking about it, I'm currently lean to Timo's suggestion. The reason is backward compatibility. If we follow Bowen's approach, let's say we first find function in Flink's built-in functions, and

[jira] [Created] (FLINK-13949) Delete deduplicating JobVertexDetailsInfo.VertexTaskDetail

2019-09-03 Thread lining (Jira)
lining created FLINK-13949: -- Summary: Delete deduplicating JobVertexDetailsInfo.VertexTaskDetail Key: FLINK-13949 URL: https://issues.apache.org/jira/browse/FLINK-13949 Project: Flink Issue Type: Im

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-09-03 Thread Becket Qin
Hi Timo, Dawid and Aljoscha, Thanks for clarifying the goals. It is very helpful to understand the motivation here. It would be great to add them to the FLIP wiki. I agree that the current FLIP design achieves the two goals it wants to achieve. But I am trying to see is if the current approach is

Re: [DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

2019-09-03 Thread Rong Rong
Thanks for putting together the proposal @Timo and sorry for joining the discussion thread late. I also share the same thought with Fabian on the ease-of-use front. However I was wondering if we need to start the expression design with them? One thing I can think of is: is it possible to support "

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

2019-09-03 Thread Sijie Guo
Hi Yun, Since I was the main driver behind FLINK-9641 and FLINK-9168, let me try to add more context on this. FLINK-9641 and FLINK-9168 was created for bringing Pulsar as source and sink for Flink. The integration was done with Flink 1.6.0. We sent out pull requests about a year ago and we ended

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Bowen Li
Hi all, Thanks for the feedback. Just a kindly reminder that the [Proposal] section in the google doc was updated, please take a look first and let me know if you have more questions. On Tue, Sep 3, 2019 at 4:57 PM Bowen Li wrote: > Hi Timo, > > Re> 1) We should not have the restriction "hive b

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Bowen Li
Hi Timo, Re> 1) We should not have the restriction "hive built-in functions can only > be used when current catalog is hive catalog". Switching a catalog > should only have implications on the cat.db.object resolution but not > functions. It would be quite convinient for users to use Hive built-in

[jira] [Created] (FLINK-13948) Fix loss of state for Identical Windows merging after initial merge

2019-09-03 Thread Scott Waller (Jira)
Scott Waller created FLINK-13948: Summary: Fix loss of state for Identical Windows merging after initial merge Key: FLINK-13948 URL: https://issues.apache.org/jira/browse/FLINK-13948 Project: Flink

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Bowen Li
Hi Jingsong, Re> 1.Hive built-in functions is an intermediate solution. So we should > not introduce interfaces to influence the framework. To make > Flink itself more powerful, we should implement the functions > we need to add. Yes, please see the doc. Re> 2.Non-flink built-in functions are ea

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Bowen Li
Hi Kurt, Re: > What I want to propose is we can merge #3 and #4, make them both under >"catalog" concept, by extending catalog function to make it have ability to >have built-in catalog functions. Some benefits I can see from this approach: >1. We don't have to introduce new concept like external

[jira] [Created] (FLINK-13947) Check Hive shim serialization in Hive UDF wrapper classes and test coverage

2019-09-03 Thread Xuefu Zhang (Jira)
Xuefu Zhang created FLINK-13947: --- Summary: Check Hive shim serialization in Hive UDF wrapper classes and test coverage Key: FLINK-13947 URL: https://issues.apache.org/jira/browse/FLINK-13947 Project: Fl

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

2019-09-03 Thread Yun Tang
Hi Yijie I can see that Pulsar becomes more and more popular recently and very glad to see more people willing to contribute to Flink ecosystem. Before any further discussion, would you please give some explanation of the relationship between this thread to current existing JIRAs of pulsar sour

Re: Potential block size issue with S3 binary files

2019-09-03 Thread Ken Krugler
Hi Arvid, Thanks for following up… > On Sep 2, 2019, at 3:09 AM, Arvid Heise wrote: > > Hi Ken, > > that's indeed a very odd issue that you found. I had a hard time to connect > block size with S3 in the beginning and had to dig into the code. I still > cannot fully understand why you got two

[jira] [Created] (FLINK-13946) Remove deactivated JobSession-related code.

2019-09-03 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-13946: -- Summary: Remove deactivated JobSession-related code. Key: FLINK-13946 URL: https://issues.apache.org/jira/browse/FLINK-13946 Project: Flink Issue Type: I

Re: [DISCUSS] Reducing build times

2019-09-03 Thread Arvid Heise
+1 for Azure Pipelines, had very good experiences in the past with it and the open source and payment models are much better. The upcoming Github CI/CD seems also like a promising alternative, but from the first looks, it seems like the small brother of Azure Pipeline. So, any effort going into Az

Re: [DISCUSS] Reducing build times

2019-09-03 Thread Robert Metzger
Hi all, I wanted to give a short update on this: - Arvid, Aljoscha and I have started working on a Gradle PoC, currently working on making all modules compile and test with Gradle. We've also identified some problematic areas (shading being the most obvious one) which we will analyse as part of th

Re: [DISCUSS] Repository split

2019-09-03 Thread Robert Metzger
Thanks a lot for your summary Chesnay. I agree with you that we have no consensus in the community for splitting up the repository immediately, and I agree with you that we should have a separate discussion about reducing the build time (which is already making good progress). Also, I will keep th

Re: [DISCUSS] Releasing Flink 1.8.2

2019-09-03 Thread Kostas Kloudas
Yes, I will do that Jark! Kostas On Tue, Sep 3, 2019 at 4:19 PM Jark Wu wrote: > > Thanks Kostas for the quick fixing. > > However, I find that FLINK-13940 still target to 1.8.2 as a blocker. If I > understand correctly, FLINK-13940 is aiming for a nicer and better solution > in the future. > So

[jira] [Created] (FLINK-13945) Vendor-repos Maven profile doesn't exist in flink-shaded

2019-09-03 Thread Jira
Elise Ramé created FLINK-13945: -- Summary: Vendor-repos Maven profile doesn't exist in flink-shaded Key: FLINK-13945 URL: https://issues.apache.org/jira/browse/FLINK-13945 Project: Flink Issue Ty

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread Jan Lukavský
Answers inline. On 9/3/19 4:01 PM, Till Rohrmann wrote: How so? Does your REPL add the generated classes to the system class loader? I assume the system class loader is used to load the Flink classes. No, it does not. It cannot on JDK >= 9 (or would have to hack into jdk.internal.loader.ClassLo

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

2019-09-03 Thread zhijiang
Thanks for proposing this FLIP and also +1 on my side. @Andrey Zagrebin For the point of "network memory is actually used more than shuffling", I guess that the component of queryable state is also using network/netty stack atm, which is outside of shuffling. In addition, if we only consider the

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Timo Walther
Hi Kurt, it should not affect the functions and operations we currently have in SQL. It just categorizes the available built-in functions. It is kind of an orthogonal concept to the catalog API but built-in functions deserve this special kind of treatment. CatalogFunction still fits perfectly

Re: [ANNOUNCE] Kinesis connector becomes part of Flink releases

2019-09-03 Thread vino yang
Good news! Thanks for your efforts, Bowen! Best, Vino Yu Li 于2019年9月2日周一 上午6:04写道: > Great to know, thanks for the efforts Bowen! > > And I believe it worth a release note in the original JIRA, wdyt? Thanks. > > Best Regards, > Yu > > > On Sat, 31 Aug 2019 at 11:01, Bowen Li wrote: > >> Hi all

Re: State of FLIPs

2019-09-03 Thread vino yang
Hi Chesnay, I created FLIP-44 two months ago. Originally, the discussion of local aggregation feature happened in the ML thread.[1] After many people agree with this feature, I created an umbrella issue and split into some subtasks and opened one PR. However, @Aljoscha Krettek thinks it's a big

Re: [DISCUSS] Releasing Flink 1.8.2

2019-09-03 Thread Jark Wu
Thanks Kostas for the quick fixing. However, I find that FLINK-13940 still target to 1.8.2 as a blocker. If I understand correctly, FLINK-13940 is aiming for a nicer and better solution in the future. So should we update the fixVersion of FLINK-13940? Best, Jark On Tue, 3 Sep 2019 at 21:33, Kost

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread Till Rohrmann
How so? Does your REPL add the generated classes to the system class loader? I assume the system class loader is used to load the Flink classes. Ideally, what you would like to have is the option to provide the parent class loader which is used load user code to the LocalEnvironment. This one coul

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread Jan Lukavský
On the other hand, if you say, that the contract of LocalEnvironment is to execute as if it had all classes on its class loader, then it currently breaks this contract. :-) Jan On 9/3/19 3:45 PM, Jan Lukavský wrote: Hi Till, hmm, that sounds it might work. I would have to incorporate this (

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread Jan Lukavský
Hi Till, hmm, that sounds it might work. I would have to incorporate this (either as default, or on demand) into Apache Beam. Would you see any disadvantages of this approach? Would you suggest to make this default behavior for local beam FlinkRunner? I can introduce a configuration option to

Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread Till Rohrmann
I see the problem Jan. What about the following proposal: Instead of using the LocalEnvironment for local tests you always use the RemoteEnvironment but when testing it locally you spin up a MiniCluster in the same process and initialize the RemoteEnvironment with `MiniCluster#getRestAddress`. That

Re: [DISCUSS] Releasing Flink 1.8.2

2019-09-03 Thread Kostas Kloudas
Thanks for waiting! A fix for FLINK-13940 has been merged on 1.8, 1.9 and the master under FLINK-13941. Cheers, Kostas On Tue, Sep 3, 2019 at 11:25 AM jincheng sun wrote: > > +1 FLINK-13940 is a > blocker, due to loss data is very important bu

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Kurt Young
Does this only affect the functions and operations we currently have in SQL and have no effect on tables, right? Looks like this is an orthogonal concept with Catalog? If the answer are both yes, then the catalog function will be a weird concept? Best, Kurt On Tue, Sep 3, 2019 at 8:10 PM Danny C

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

2019-09-03 Thread Till Rohrmann
Thanks for creating this FLIP and starting the vote Xintong. +1 for the proposal from my side. I agree with Stephan that we might wanna revisit some of the configuration names. If I understood it correctly, then Task Off-heap memory represents the direct memory used by the user code, right? How

Re: [DISCUSS] Flink Python User-Defined Function for Table API

2019-09-03 Thread jincheng sun
Hi Timo, Thanks for the quick reply ! :) I have added more example for #3 and #5 to the FLIP. That are great suggestions ! Regarding 2: There are two kind Serialization for CloudPickle(Which is different from Java): 1) For class and function which can be imported, CloudPickle only serialize the

Re: Flink SQL - Support Computed Columns in DDL?

2019-09-03 Thread Qi Luo
Hi Jark and Danny, Glad to hear your plan on this! One of our use cases is to define some column as rowtime (which is not of type timestamp). Computed column seems to be a natural fit for that. Thanks, Qi On Tue, Sep 3, 2019 at 7:46 PM Jark Wu wrote: > Hi Qi, > > The computed column is not fu

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-09-03 Thread Aljoscha Krettek
Hi, I think it’s important to keep in mind the original goals of this FLIP and not let the scope grow indefinitely. As I recall it, the goals are: - Extend the ConfigOption system enough to allow the Table API to configure options that are right now only available on CheckpointingOptions, Exe

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

2019-09-03 Thread Andrey Zagrebin
Thanks for starting the vote Xintong Also +1 for the proposed FLIP-49. @Stephan regarding namings: network vs shuffle. My understanding so far was that the network memory is what we basically give to Shuffle implementations and default netty implementation uses it in particular mostly for network

Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread guaishushu1...@163.com
guaishushu1...@163.com From: guaishushu1...@163.com Date: 2019-09-03 20:25 To: dev Subject: Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader guaishushu1...@163.com From: guaishushu1...@163.com Date: 2019-09-03 20:23 To: dev Subject: Re: Re: Clas

Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread guaishushu1...@163.com
guaishushu1...@163.com From: guaishushu1...@163.com Date: 2019-09-03 20:23 To: dev Subject: Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader guaishushu1...@163.com From: Jan Lukavský Date: 2019-09-03 20:17 To: dev Subject: Re: ClassLoader create

Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread guaishushu1...@163.com
guaishushu1...@163.com From: Jan Lukavský Date: 2019-09-03 20:17 To: dev Subject: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader Hi Till, the use-case is pretty much simple - I have a REPL shell in groovy, which generates classes at runtime. The actual

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread Jan Lukavský
Hi Till, the use-case is pretty much simple - I have a REPL shell in groovy, which generates classes at runtime. The actual hierarchy is therefore  system class loader -> application classloader -> repl classloader (GroovyClassLoader actually) now, when a terminal (sink) operation in the sh

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Danny Chan
The way you proposed are basically the same as what Calcite does, I think we are in the same line. Best, Danny Chan 在 2019年9月3日 +0800 PM7:57,Timo Walther ,写道: > This sounds exactly as the module approach I mentioned, no? > > Regards, > Timo > > On 03.09.19 13:42, Danny Chan wrote: > > Thanks Bowe

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-09-03 Thread Timo Walther
Hi Danny, yes, this FLIP covers all the building blocks we need also for unification of the DDL properties. Regards, Timo On 03.09.19 13:45, Danny Chan wrote: with the new SQL DDL based on properties as well as more connectors and formats coming up, unified configuration becomes more impor

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

2019-09-03 Thread Till Rohrmann
Hi Jan, I've talked with Aljoscha and Stephan offline and we concluded that we would like to avoid the usage of context class loaders if possible. The reason for this is that using the context class loader can easily mess up an otherwise clear class loader hierarchy which makes it hard to reason a

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Timo Walther
This sounds exactly as the module approach I mentioned, no? Regards, Timo On 03.09.19 13:42, Danny Chan wrote: Thanks Bowen for bring up this topic, I think it’s a useful refactoring to make our function usage more user friendly. For the topic of how to organize the builtin operators and oper

Re: Flink SQL - Support Computed Columns in DDL?

2019-09-03 Thread Jark Wu
Hi Qi, The computed column is not fully supported in 1.9. We will start a design discussion in the dev mailing list soon. Please stay tuned! Btw, could you share with us what's the case why do you want to use computed column? Best, Jark On Tue, 3 Sep 2019 at 19:25, Danny Chan wrote: > Yeah, w

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-09-03 Thread Danny Chan
> with the new SQL DDL based on properties as well as more connectors and formats coming up, unified configuration becomes more important I Cann’t agree more, do you think we can unify the config options key format here for all the DDL properties ? Best, Danny Chan 在 2019年8月16日 +0800 PM10:12,dev

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Danny Chan
Thanks Bowen for bring up this topic, I think it’s a useful refactoring to make our function usage more user friendly. For the topic of how to organize the builtin operators and operators of Hive, here is a solution from Apache Calcite, the Calcite way is to make every dialect operators a “Libr

Re: Flink SQL - Support Computed Columns in DDL?

2019-09-03 Thread Danny Chan
Yeah, we are planning to implement this feature in release-1.10, wait for our good news ! Best, Danny Chan 在 2019年9月3日 +0800 PM6:19,Qi Luo ,写道: > Hi folks, > > Computed columns in Flink SQL DDL is currently disabled in both old planner > and Blink planner (throws "Computed columns for DDL is not

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

2019-09-03 Thread Zhu Zhu
Thanks Xintong for the explanation. For question #1, I think it's good as long as DataSet job behaviors remains the same. For question #2, agreed that the resource difference is small enough(at most 1 edge diff) in current supported point-wise execution edge connection patterns. Thanks, Zhu Zhu

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

2019-09-03 Thread Stephan Ewen
+1 to the proposal in general A few things seems to be a bit put of sync with the latest discussions though. The section about JVM Parameters states that the -XX:MaxDirectMemorySize value is set to Task Off-heap Memory, Shuffle Memory and JVM Overhead. The way I understand the last discussion con

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

2019-09-03 Thread Xintong Song
Thanks for the comments, Zhu & Kurt. Andrey and I also had some discussions offline, and I would like to first post a summary of our discussion: 1. The motivation of the fraction based approach is to unify resource management for both operators with specified and unknown resource requir

Flink SQL - Support Computed Columns in DDL?

2019-09-03 Thread Qi Luo
Hi folks, Computed columns in Flink SQL DDL is currently disabled in both old planner and Blink planner (throws "Computed columns for DDL is not supported yet!" exception in SqlToOperationConverter). I searched through the JIRA but found no relevant issues. Do we have any plans to support this ni

Re: [DISCUSS] Flink Python User-Defined Function for Table API

2019-09-03 Thread Timo Walther
Hi Jincheng, thanks for your response. 2. Serializability of functions: Using some arbitrary serialization format for shipping a function to worker sounds fine to me. But once we store functions a the catalog we need to think about backwards compatibility and evolution of interfaces etc. I'm

Re: [SURVEY] Is the default restart delay of 0s causing problems?

2019-09-03 Thread Till Rohrmann
The FLIP-62 discuss thread can be found here [1]. [1] https://lists.apache.org/thread.html/9602b342602a0181fcb618581f3b12e692ed2fad98c59fd6c1caeabd@%3Cdev.flink.apache.org%3E Cheers, Till On Tue, Sep 3, 2019 at 11:13 AM Till Rohrmann wrote: > Thanks everyone for the input again. I'll then conc

[DISCUSS] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

2019-09-03 Thread Till Rohrmann
Hi everyone, I'd like to discuss changing the default restart delay for FixedDelay- and FailureRateRestartStrategy to "1 s" [1]. According to a user survey about the default value of the restart delay [2], it turned out that the current default value of "0 s" is not optimal. In practice Flink use

[jira] [Created] (FLINK-13944) Table.toAppendStream: InvalidProgramException: Table program cannot be compiled.

2019-09-03 Thread Stefano (Jira)
Stefano created FLINK-13944: --- Summary: Table.toAppendStream: InvalidProgramException: Table program cannot be compiled. Key: FLINK-13944 URL: https://issues.apache.org/jira/browse/FLINK-13944 Project: Flink

Re: [DISCUSS] Releasing Flink 1.8.2

2019-09-03 Thread jincheng sun
+1 FLINK-13940 is a blocker, due to loss data is very important bug, And great thanks for helping fix it Kostas! Best, Jincheng Kostas Kloudas 于2019年9月2日周一 下午7:20写道: > Hi all, > > I think this should be also considered a blocker > https://iss

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Timo Walther
Hi Bowen, thanks for your proposal. Here are some thoughts: 1) We should not have the restriction "hive built-in functions can only be used when current catalog is hive catalog". Switching a catalog should only have implications on the cat.db.object resolution but not functions. It would be q

Re: [SURVEY] Is the default restart delay of 0s causing problems?

2019-09-03 Thread Till Rohrmann
Thanks everyone for the input again. I'll then conclude this survey thread and start a discuss thread to set the default restart delay to 1s. @Arvid, I agree that a better documentation how to tune Flink with sane settings for certain scenarios is super helpful. However, as you've said it is somew

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-09-03 Thread Timo Walther
Hi Becket, it is definitely API but I would rather consider configuration as second-level API. Usually, a connector or plugin should have some nice builder pattern with helpful JavaDocs instead of just exposing the pure ConfigOptions. I think there is some general misunderstanding what a con

Re: Please add me as contributor

2019-09-03 Thread Jan Lukavský
Hi Dawid, thanks for the explanation. I got warning from PR [1] associated with JIRA [2]. Jan [1] https://github.com/apache/flink/pull/9579 [2] https://issues.apache.org/jira/browse/FLINK-13925 On 9/3/19 10:43 AM, Dawid Wysakowicz wrote: Hi Jan, Recently the community changed the contribu

Re: Please add me as contributor

2019-09-03 Thread Dawid Wysakowicz
Hi Jan, Recently the community changed the contribution process a bit and there are no longer contributor privileges. The jira issues are supposed to be assigned by committers that are willing to help you with getting the contribution in. Please look at the contribution guidelines[1]. Do you have

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread JingsongLee
Thanks Bowen: +1 for this. And +1 to Kurt's suggestion. My other points are: 1.Hive built-in functions is an intermediate solution. So we should not introduce interfaces to influence the framework. To make Flink itself more powerful, we should implement the functions we need to add. 2.Non-fli

Please add me as contributor

2019-09-03 Thread Jan Lukavský
Hi, I'd like to be able to assign JIRAs to myself, can I be added as contributor, please? My JIRA ID is 'janl'. Thanks,  Jan

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Kurt Young
Thanks Bowen for driving this. +1 for the general idea. It makes the function resolved behavior more clear and deterministic. Besides, the user can use all hive built-in functions, which is a great feature. I only have one comment, but maybe it may touch your design so I think it would make sense