Re: [DISCUSS] Table API / SQL indicators for event and processing time

2017-03-01 Thread jincheng sun
Hi, @Timo, thanks for your replay, and congratulations on your job. @Fibian, No matter what way to achieve, as long as when the table is generated or created, identity the field attributes, that is what we want. I think at this point we are on the same page. We can go ahead. And very glad to hear T

[jira] [Created] (FLINK-5949) Flink on YARN checks for Kerberos credentials for non-Kerberos authentication methods

2017-03-01 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5949: -- Summary: Flink on YARN checks for Kerberos credentials for non-Kerberos authentication methods Key: FLINK-5949 URL: https://issues.apache.org/jira/browse/FLINK-5949

Re: [Dev] Issue related to using Flink DataSet methods

2017-03-01 Thread Pawan Manishka Gunarathna
Hi, Thanks @Fabian and @Xingcan for the explanation. @Xingcan Here I mean I have a data analytics server that has *data tables*. So my initial requirement is to make a client connector for Flink to access that* data tables*.Then I started with implementing Flink InputFormat Interface and that was

Re: Visualizing topologies

2017-03-01 Thread Ken Krugler
Hi Greg, The use case is to create a visualization of the topology. So I don’t think there’s any reason to “act on the dot file from within the user program” Regards, — Ken > On Feb 24, 2017, at 7:51am, Greg Hogan wrote: > > Ken and Fabian, > > Is the use case to generate and act on the do

[jira] [Created] (FLINK-5948) Error in Python zip_with_index documentation

2017-03-01 Thread Geoffrey Mon (JIRA)
Geoffrey Mon created FLINK-5948: --- Summary: Error in Python zip_with_index documentation Key: FLINK-5948 URL: https://issues.apache.org/jira/browse/FLINK-5948 Project: Flink Issue Type: Bug

About the ACL of Zookeeper

2017-03-01 Thread Zhangrucong
Hi: I want to use the ACL of Zookeeper. So I configure the following configurations: 1、 high-availability.zookeeper.path.root: flink234 2、 high-availability.zookeeper.client.acl: creator 3、 zookeeper.sasl.disable: false But, I use ZK client to get the ACL, the result is : [cid:image00

[jira] [Created] (FLINK-5947) NullPointerException in ContinuousProcessingTimeTrigger.clear()

2017-03-01 Thread Xiaojun Jin (JIRA)
Xiaojun Jin created FLINK-5947: -- Summary: NullPointerException in ContinuousProcessingTimeTrigger.clear() Key: FLINK-5947 URL: https://issues.apache.org/jira/browse/FLINK-5947 Project: Flink Is

[jira] [Created] (FLINK-5946) Kinesis Producer uses KPL that orphans threads that consume 100% CPU

2017-03-01 Thread Scott Kidder (JIRA)
Scott Kidder created FLINK-5946: --- Summary: Kinesis Producer uses KPL that orphans threads that consume 100% CPU Key: FLINK-5946 URL: https://issues.apache.org/jira/browse/FLINK-5946 Project: Flink

[jira] [Created] (FLINK-5945) Close function in OuterJoinOperatorBase#executeOnCollections

2017-03-01 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-5945: - Summary: Close function in OuterJoinOperatorBase#executeOnCollections Key: FLINK-5945 URL: https://issues.apache.org/jira/browse/FLINK-5945 Project: Flink Issue T

[jira] [Created] (FLINK-5944) Flink should support reading Snappy Files

2017-03-01 Thread Ilya Ganelin (JIRA)
Ilya Ganelin created FLINK-5944: --- Summary: Flink should support reading Snappy Files Key: FLINK-5944 URL: https://issues.apache.org/jira/browse/FLINK-5944 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-5943) Unprotected access to haServices in YarnFlinkApplicationMasterRunner#shutdown()

2017-03-01 Thread Ted Yu (JIRA)
Ted Yu created FLINK-5943: - Summary: Unprotected access to haServices in YarnFlinkApplicationMasterRunner#shutdown() Key: FLINK-5943 URL: https://issues.apache.org/jira/browse/FLINK-5943 Project: Flink

Re: [DISCUSS] Gelly planning for release 1.3 and roadmap

2017-03-01 Thread Greg Hogan
Flink’s stable API provides the frameworks (DataStream and DataSet). On top of these frameworks Gelly provides additional models for iterative algorithms, but there are algorithms such as Minimum Spanning Tree which do not easily map to these models (in this instance requiring nested iterations;

[jira] [Created] (FLINK-5942) Harden ZooKeeperStateHandleStore to deal with corrupted data

2017-03-01 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-5942: Summary: Harden ZooKeeperStateHandleStore to deal with corrupted data Key: FLINK-5942 URL: https://issues.apache.org/jira/browse/FLINK-5942 Project: Flink I

Re: [DISCUSS] Gelly planning for release 1.3 and roadmap

2017-03-01 Thread Greg Hogan
On Fri, Feb 24, 2017 at 1:43 PM, Vasiliki Kalavri mailto:vasilikikala...@gmail.com>> wrote: Hi Greg, On 24 February 2017 at 18:09, Greg Hogan mailto:c...@greghogan.com>> wrote: > Thanks, Vasia, for starting the discussion. > > I was expecting more changes from the recent discussion on restructuri

Re: [DISCUSS] Code style / checkstyle

2017-03-01 Thread Henry Saputra
It is actually Databricks Scala guide to help contributions to Apache Spark so not really official Spark Scala guide. The style guide feels bit more like asking people to write Scala in Java mode so I am -1 to follow the style for Apache Flink Scala if that what you are recommending. If the "unifi

[jira] [Created] (FLINK-5941) Let handlers take part in job archiving

2017-03-01 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-5941: --- Summary: Let handlers take part in job archiving Key: FLINK-5941 URL: https://issues.apache.org/jira/browse/FLINK-5941 Project: Flink Issue Type: New F

[jira] [Created] (FLINK-5940) ZooKeeperCompletedCheckpointStore cannot handle broken state handles

2017-03-01 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-5940: Summary: ZooKeeperCompletedCheckpointStore cannot handle broken state handles Key: FLINK-5940 URL: https://issues.apache.org/jira/browse/FLINK-5940 Project: Flink

Re: [Dev] Issue related to using Flink DataSet methods

2017-03-01 Thread Xingcan Cui
Hi Pawan, @Fabian was right and I thought it was stream environment. Sorry for that. What do you mean by `read the available records of my datasource`? How do you implement the nextRecord() method in DASInputFormat? Best, Xingcan On Wed, Mar 1, 2017 at 4:45 PM, Fabian Hueske wrote: > Hi Pawa

Re: [DISCUSS] Table API / SQL indicators for event and processing time

2017-03-01 Thread Fabian Hueske
Hi, @Xingcan Yes that is right. It is not (easily) possible to change the watermarks of a stream. All attributes which are used as event-time timestamps must be aligned with these watermarks. This are only attributes which are derived from the original rowtime attribute, i.e., the one that was spe

Re: [DISCUSS] Table API / SQL indicators for event and processing time

2017-03-01 Thread Timo Walther
Hi SunJincheng, our basic idea was to let the underlying API extract and handle time correctly. Extracting timestamps and assigning watermarks is a serious business. More advanced users can create TableSources and define time there (using DataStream API) and less advanced users can simply use

Re: [Dev] Issue related to using Flink DataSet methods

2017-03-01 Thread Fabian Hueske
Hi Pawan, in the DataSet API DataSet.print() will trigger the execution (you do not need to call ExecutionEnvironment.execute()). The DataSet will be printed on the standard out of the process that submits the program. This does only work for small DataSets. In general print() should only be used