Re: [DISCUSS] Table API / SQL internal timestamp handling

Hi Xingcan, IMO, I don't think event-time of join results could be automatically decided by system. Considering batch tables, if users want a event time window aggregation after join, user must specify the time field explicitly (T1.rowtime or T2.rowtime or the computed result of them). So in the c

[jira] [Created] (FLINK-7267) Add support for lists of hosts to connect

Hu Hailin created FLINK-7267: Summary: Add support for lists of hosts to connect Key: FLINK-7267 URL: https://issues.apache.org/jira/browse/FLINK-7267 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Table API / SQL internal timestamp handling

Hi all, @Fabian, thanks for raising this. @Radu and Jark, personally I think the timestamp field is critical for query processing and thus should be declared as (or supposed to be) NOT NULL. In addition, I think the event-time semantic of the join results should be automatically decided by the sy

Re: Make SubmittedJobGraphStore configurable

Hi Till, As far as I know there is interests of keep job graphs recoverable from shared zk hiccups. Or standalone mode with customized leader election. I plan to spend a bit time prototyping back up to Amazon S3. Will keep folks updated as along as I got happy pass going. Thanks, Chen > On J

Re: is flink' states functionality futile?

   it's a program that sinks messages only after enough items accumulated in  the buffe Yes, that’s another important aspect of how Flink’s checkpointing is used to achieve at-least-once / exactly-once delivery to

Re: [DISCUSS] Table API / SQL internal timestamp handling

Hi, Radu's concerns make sense to me, especially the null value timestamp and multi-proctime. I have also something in my mind. I would like to propose some time indicator built-in functions, e.g. ROW_TIME(Timestamp ts) will generate a event time logical attribute, PROC_TIME() will generate a pro

Re: Towards a spec for robust streaming SQL, Part 2

Thanks for the invitation Tyler. I am sure folks who worked on the calcite integration and others would be interested. On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau wrote: > +d...@apex.apache.org, since I'm told Apex has a Calcite integration as > well. If anyone on the Apex side wants to join

Re: Towards a spec for robust streaming SQL, Part 2

+d...@apex.apache.org, since I'm told Apex has a Calcite integration as well. If anyone on the Apex side wants to join in on the fun, your input would be welcomed! -Tyler On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau wrote: > Hello Flink, Calcite, and Beam dev lists! > > Linked below is the sec

RE: [DISCUSS] Table API / SQL internal timestamp handling

Hi, I think this is an interesting discussion and I would like to add some issues and give some feedback. - For supporting the join we do not only need to think of the time but also on the null values. For example if you have a LEFT (or RIGHT) JOIN between items of 2 input streams, and the sec

[jira] [Created] (FLINK-7266) Don't attempt to delete parent directory on S3

Stephan Ewen created FLINK-7266: --- Summary: Don't attempt to delete parent directory on S3 Key: FLINK-7266 URL: https://issues.apache.org/jira/browse/FLINK-7266 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-7265) FileSystems should describe their kind and consistency level

Stephan Ewen created FLINK-7265: --- Summary: FileSystems should describe their kind and consistency level Key: FLINK-7265 URL: https://issues.apache.org/jira/browse/FLINK-7265 Project: Flink Iss

Re: [DISCUSS] Release 1.3.2 planning

I did some preliminary cluster testing on 1.3-SNAPSHOT and it seems there is another issue in RocksDB with incremental checkpoints. Stefan and I are currently investigating. I'll keep you posted and we can hopefully create an RC once we hunt this one down. Best, Aljoscha > On 24. Jul 2017, at

[DISCUSS] Table API / SQL internal timestamp handling

Hi everybody, I'd like to propose and discuss some changes in the way how the Table API / SQL internally handles timestamps. The Table API is implemented on top of the DataStream API. The DataStream API hides timestamps from users in order to ensure that timestamps and watermarks are aligned. Ins

Re: Use Flink's actor system

Hi Lorenzo, apart from experimentation it's not recommended to directly use Flink's ActorSystem, because it is an implementation detail. With Flip-6 the ActorSystem will be further hidden and in the future we might implement a different RPC system not relying on Akka. Cheers, Till On Sun, Jul 16

Re: Ship files to flink on yarn-cluster mode

Hi Guy, you can use the -yarnship command line option to specify a directory whose content will be shipped to the launched yarn application. Thus, all files contained in this directory will be copied to HDFS from where they are accessible by Flink. Cheers, Till On Thu, Jul 13, 2017 at 4:40 PM, G

[jira] [Created] (FLINK-7264) travis watchdog is killed before tests

Chesnay Schepler created FLINK-7264: --- Summary: travis watchdog is killed before tests Key: FLINK-7264 URL: https://issues.apache.org/jira/browse/FLINK-7264 Project: Flink Issue Type: Bug

Re: Make SubmittedJobGraphStore configurable

If there is a need for this, then we can definitely make this configurable. The interface SubmittedJobGraphStore is already there. Cheers, Till ​ On Fri, Jul 7, 2017 at 6:32 AM, Chen Qin wrote: > Sure, > ​ I would imagine ​couple of extra lines within flink.conf > ​...​ > graphstore.type: custo

[jira] [Created] (FLINK-7263) Improve Pull Request Template

Stephan Ewen created FLINK-7263: --- Summary: Improve Pull Request Template Key: FLINK-7263 URL: https://issues.apache.org/jira/browse/FLINK-7263 Project: Flink Issue Type: Improvement C

[jira] [Created] (FLINK-7262) remove unused FallbackLibraryCacheManager

Nico Kruber created FLINK-7262: -- Summary: remove unused FallbackLibraryCacheManager Key: FLINK-7262 URL: https://issues.apache.org/jira/browse/FLINK-7262 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] A more thorough Pull Request check list and template

Fair enough. I still think it's too verbose but most of the feedback was positive, so I don't want to block this. I think it would be good to model the form with check boxes where possible. For example > The runtime per-record code paths (performance sensitive): *(yes / no / don't know)* could b

[jira] [Created] (FLINK-7261) avoid unnecessary exceptions in the logs in non-HA cases

Nico Kruber created FLINK-7261: -- Summary: avoid unnecessary exceptions in the logs in non-HA cases Key: FLINK-7261 URL: https://issues.apache.org/jira/browse/FLINK-7261 Project: Flink Issue Type

[jira] [Created] (FLINK-7260) Match not exhaustive in WindowJoinUtil.scala

Ted Yu created FLINK-7260: - Summary: Match not exhaustive in WindowJoinUtil.scala Key: FLINK-7260 URL: https://issues.apache.org/jira/browse/FLINK-7260 Project: Flink Issue Type: Bug Repo

[jira] [Created] (FLINK-7259) Match not exhaustive in TaskMonitor

Till Rohrmann created FLINK-7259: Summary: Match not exhaustive in TaskMonitor Key: FLINK-7259 URL: https://issues.apache.org/jira/browse/FLINK-7259 Project: Flink Issue Type: Bug C

Re: [DISCUSS] A more thorough Pull Request check list and template

Great! :-) If Fabian is also OK with trying it out, I would ask Stephan to open a PR for this against Flink. On Tue, Jul 25, 2017 at 8:50 AM, Chesnay Schepler wrote: > I'm still apprehensive about it, but don't mind trying it out. It's not like > it can break something. > > > On 24.07.2017 18:52