[jira] [Created] (FLINK-5651) state backends: forbid Iterator#remove() from the Iterable returned by HeapListState#get()

2017-01-26 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-5651: -- Summary: state backends: forbid Iterator#remove() from the Iterable returned by HeapListState#get() Key: FLINK-5651 URL: https://issues.apache.org/jira/browse/FLINK-5651

Re: Flink 1.2.0 rc2 Bugs

2017-01-26 Thread Ufuk Celebi
Hey Scott, thanks for reporting these. Regarding the 2nd point: RC0 had an issue with the KryoSerializer preventing users to resume from 1.1 savepoints. That was fixed at the cost of breaking savepoint compatability between the RCs, but there is no way around it. Flink guarantees compatability bet

[Discuss] Declare class field's type in scala class field

2017-01-26 Thread Jinkui Shi
hi, all Scala can infer the actual type if we didn’t declare its type. There also nothing different in the byte code of java class format. It’s convenient for write the code, but hard to read. Maybe it’s time to face such bad smell code. Scala check style plugin also have such rule. We can add

RE: STREAM SQL inner queries

2017-01-26 Thread Radu Tudoran
Hi Shaoxuan, Thanks for the feedback! Regarding the proposal for relational queries that you referenced, I am a bit confused with respect to its purpose and evolution with respect to the current implementation of stream sql - is it suppose to replace this implementation, to complement itbut

Re: STREAM SQL inner queries

2017-01-26 Thread Xingcan
Hi all, I've read the document about dynamic table. Honestly, I think it's well-defined and ingeniously compromise the batch and stream. There are two questions about the design. 1) Though it's fine to take the stream as a snapshot of a dynamic table, a table is essentially a set while a stream i

Stream SQL and Dynamic tables

2017-01-26 Thread Radu Tudoran
Hi all, I have a question with respect to the scope behind the initiative behind relational queries on data streams: https://docs.google.com/document/d/1qVVt_16kdaZQ8RTfA_f4konQPW4tnl8THw6rzGUdaqU/edit# Is the approach of using dynamic tables intended to replace the implementation and mechanism

[jira] [Created] (FLINK-5652) Memory leak in AsyncDataStream

2017-01-26 Thread Dmitry Golubets (JIRA)
Dmitry Golubets created FLINK-5652: -- Summary: Memory leak in AsyncDataStream Key: FLINK-5652 URL: https://issues.apache.org/jira/browse/FLINK-5652 Project: Flink Issue Type: Bug Co

[jira] [Created] (FLINK-5653) Add processing time OVER ROWS BETWEEN x PRECEDING aggregation to SQL

2017-01-26 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-5653: Summary: Add processing time OVER ROWS BETWEEN x PRECEDING aggregation to SQL Key: FLINK-5653 URL: https://issues.apache.org/jira/browse/FLINK-5653 Project: Flink

[jira] [Created] (FLINK-5654) Add processing time OVER RANGE BETWEEN x PRECEDING aggregation to SQL

2017-01-26 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-5654: Summary: Add processing time OVER RANGE BETWEEN x PRECEDING aggregation to SQL Key: FLINK-5654 URL: https://issues.apache.org/jira/browse/FLINK-5654 Project: Flink

[jira] [Created] (FLINK-5655) Add event time OVER RANGE BETWEEN x PRECEDING aggregation to SQL

2017-01-26 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-5655: Summary: Add event time OVER RANGE BETWEEN x PRECEDING aggregation to SQL Key: FLINK-5655 URL: https://issues.apache.org/jira/browse/FLINK-5655 Project: Flink

[jira] [Created] (FLINK-5656) Add processing time OVER ROWS BETWEEN UNBOUNDED PRECEDING aggregation to SQL

2017-01-26 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-5656: Summary: Add processing time OVER ROWS BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-5656 URL: https://issues.apache.org/jira/browse/FLINK-5656 Project: F

[jira] [Created] (FLINK-5657) Add processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL

2017-01-26 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-5657: Summary: Add processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-5657 URL: https://issues.apache.org/jira/browse/FLINK-5657 Project:

[jira] [Created] (FLINK-5658) Add event time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL

2017-01-26 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-5658: Summary: Add event time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-5658 URL: https://issues.apache.org/jira/browse/FLINK-5658 Project: Flink

RE: [jira] [Created] (FLINK-5654) Add processing time OVER RANGE BETWEEN x PRECEDING aggregation to SQL

2017-01-26 Thread Radu Tudoran
Hi, Can you assign please the jira issue to me. I already can start integrating the implementation that for this. -Original Message- From: Fabian Hueske (JIRA) [mailto:j...@apache.org] Sent: Thursday, January 26, 2017 2:44 PM To: dev@flink.apache.org Subject: [jira] [Created] (FLINK-56

Re: #quesiton: documentation for knn/examples/fetching cluster numbers in red?

2017-01-26 Thread Till Rohrmann
Hi Alex, have you seen KNN's online documentation [1]? This should answer your questions. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/libs/ml/knn.html Cheers, Till On Wed, Jan 25, 2017 at 2:23 PM, Alex De Castro wrote: > Hi flinkers, > I a new flink user and have been

Re: flink-ml test

2017-01-26 Thread Till Rohrmann
Hi Anton, which build are you using? I've recently (Monday) pushed a fix for the StochasticOutlierSelectionITSuite (32e1675aa38eec4a15272d62977dfe3ddbe92401). If you have this commit and the test is still failing, then I have to look into it again. Cheers, Till On Wed, Jan 25, 2017 at 10:59 PM,

Re: #quesiton: documentation for knn/examples/fetching cluster numbers in red?

2017-01-26 Thread Alex De Castro
Thanks Till, I misread the definition of knn. It’s knn JOIN. ☺ Dyslexia… Regards, Alex On 1/26/17, 2:00 PM, "Till Rohrmann" wrote: Hi Alex, have you seen KNN's online documentation [1]? This should answer your questions. [1] https://ci.apache.org/projects/flink/flink-docs

Re: [DISCUSS] Development of SQL OVER / Table API Row Windows for streaming tables

2017-01-26 Thread Fabian Hueske
Hi everybody, I created the following JIRAs: - FLINK-5653: processing time OVER ROWS x PRECEDING - FLINK-5654: processing time OVER RANGE x PRECEDING - FLINK-5655: event time OVER RANGE x PRECEDING - FLINK-5656: processing time OVER ROWS UNBOUNDED PRECEDING - FLINK-5657: processing time OVER RAN

Re: [Discuss] Declare class field's type in scala class field

2017-01-26 Thread Till Rohrmann
Agreed. I think it's a good idea to annotate the code with as much information as possible to help other people to understand it. I think it's a good idea to include this information in the coding guidelines. But I'm not sure whether we have a dedicated Scala coding guideline. Cheers, Till On Thu

Re: [jira] [Created] (FLINK-5654) Add processing time OVER RANGE BETWEEN x PRECEDING aggregation to SQL

2017-01-26 Thread Fabian Hueske
Hi Radu, can you reply to my mail on the discussion thread and tell me you JIRA user name? I'll give you contributor permissions then. Thanks, Fabian On Thu, Jan 26, 2017 at 2:51 PM, Radu Tudoran wrote: > Hi, > > Can you assign please the jira issue to me. I already can start > integrating the

Re: Stream SQL and Dynamic tables

2017-01-26 Thread Fabian Hueske
Hi Radu, the idea is to have dynamic tables as the common ground for Table API and SQL. I don't think it is a good idea to implement and maintain 3 different relational APIs with possibly varying semantics. Actually, you can see the current status of the Table API / SQL on stream as a subset of t

[jira] [Created] (FLINK-5659) FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows

2017-01-26 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-5659: --- Summary: FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows Key: FLINK-5659 URL: https://issues.apache.org/jira/browse/FLINK-5659 Project: Flink

[jira] [Created] (FLINK-5660) Not properly cleaning PendingCheckpoints up

2017-01-26 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-5660: Summary: Not properly cleaning PendingCheckpoints up Key: FLINK-5660 URL: https://issues.apache.org/jira/browse/FLINK-5660 Project: Flink Issue Type: Bug

Re: STREAM SQL inner queries

2017-01-26 Thread Fabian Hueske
Hi everybody, thanks for the proposal Radu. If I understood it correctly, you are proposing a left join between a stream and a single value (which is compute from a stream). This makes sense and should be a common use case. However, I think some of your example queries do not return a single valu

[jira] [Created] (FLINK-5661) flink-mesos should shade its httpclient dependency away

2017-01-26 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-5661: - Summary: flink-mesos should shade its httpclient dependency away Key: FLINK-5661 URL: https://issues.apache.org/jira/browse/FLINK-5661 Project: Flink Issue

Re: TestBaseUtils refactoring

2017-01-26 Thread Stephan Ewen
@Anton - I agree, that class is pretty ugly and mostly leftover code. +1 for tearing it apart (and dropping as much code of it as possible in the process) On Thu, Jan 26, 2017 at 4:37 AM, Jark Wu wrote: > Hi Anton, > > Thanks for bringing up this discussion. > > I think TestBaseUtils is a util

[jira] [Created] (FLINK-5662) Alias in front of output fails

2017-01-26 Thread Timo Walther (JIRA)
Timo Walther created FLINK-5662: --- Summary: Alias in front of output fails Key: FLINK-5662 URL: https://issues.apache.org/jira/browse/FLINK-5662 Project: Flink Issue Type: Bug Componen

[jira] [Created] (FLINK-5663) Checkpoint fails because of closed registry

2017-01-26 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-5663: -- Summary: Checkpoint fails because of closed registry Key: FLINK-5663 URL: https://issues.apache.org/jira/browse/FLINK-5663 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-5664) RocksDBBackend logging is noisy

2017-01-26 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-5664: -- Summary: RocksDBBackend logging is noisy Key: FLINK-5664 URL: https://issues.apache.org/jira/browse/FLINK-5664 Project: Flink Issue Type: Improvement C

[jira] [Created] (FLINK-5665) Lingering files after failed checkpoint

2017-01-26 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-5665: -- Summary: Lingering files after failed checkpoint Key: FLINK-5665 URL: https://issues.apache.org/jira/browse/FLINK-5665 Project: Flink Issue Type: Bug C

[jira] [Created] (FLINK-5666) Blob files are not cleaned up from ZK storage directory

2017-01-26 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-5666: -- Summary: Blob files are not cleaned up from ZK storage directory Key: FLINK-5666 URL: https://issues.apache.org/jira/browse/FLINK-5666 Project: Flink Issue Type:

RE: [DISCUSS] Development of SQL OVER / Table API Row Windows for streaming tables

2017-01-26 Thread Radu Tudoran
Thanks for this redesign Fabian, I am interested in "- FLINK-5654: processing time OVER RANGE x PRECEDING" However, I though the issue number is https://issues.apache.org/jira/browse/FLINK-5654 am I wrong? As you proposed I will move the discussion about your remark in the comment section fo

Re: [VOTE] Release Apache Flink 1.2.0 (RC2)

2017-01-26 Thread Ufuk Celebi
I ran some tests and found the following issues: https://issues.apache.org/jira/browse/FLINK-5663: Checkpoint fails because of closed registry => This happened a couple of times for the first checkpoints after submitting a job. If it happened on every submission I would definitely make this a bloc

Re: [VOTE] Release Apache Flink 1.2.0 (RC2)

2017-01-26 Thread Till Rohrmann
Robert also found an issue that pending checkpoint files are not properly cleaned up: https://issues.apache.org/jira/browse/FLINK-5660. To my surprise, the issue was already fixed in 1.1.4 so I guess I've forgotten to forward port the fix. There is a pending PR to fix it. The fix could also be part

Re: [Discuss] Declare class field's type in scala class field

2017-01-26 Thread Bruno Aranda
Hi, Twitter's Effective Scala contains very well-known and accepted conventions for Scala: http://twitter.github.io/effectivescala/ Cheers, Bruno On Thu, 26 Jan 2017 at 14:16 Till Rohrmann wrote: > Agreed. I think it's a good idea to annotate the code with as much > information as possible t

[jira] [Created] (FLINK-5667) Possible state data loss when task fails while checkpointing

2017-01-26 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-5667: Summary: Possible state data loss when task fails while checkpointing Key: FLINK-5667 URL: https://issues.apache.org/jira/browse/FLINK-5667 Project: Flink I

Re: [VOTE] Release Apache Flink 1.2.0 (RC2)

2017-01-26 Thread Till Rohrmann
I have found another problem: Under certain circumstances Flink can lose state data by completing an invalid checkpoint. https://issues.apache.org/jira/browse/FLINK-5667. Cheers, Till On Thu, Jan 26, 2017 at 6:27 PM, Till Rohrmann wrote: > Robert also found an issue that pending checkpoint file

Re: [VOTE] Release Apache Flink 1.2.0 (RC2)

2017-01-26 Thread Stephan Ewen
@Till - I think that FLINK-5667 is a blocker Good catch finding it! On Thu, Jan 26, 2017 at 7:51 PM, Till Rohrmann wrote: > I have found another problem: Under certain circumstances Flink can lose > state data by completing an invalid checkpoint. > https://issues.apache.org/jira/browse/FLINK-56

Re: [VOTE] Release Apache Flink 1.2.0 (RC2)

2017-01-26 Thread Robert Metzger
Damn. I really hoped that this RC goes through. I propose to keep the RC2 open until we've fixed all issues mentioned here and to get some more testing feedback. On Thu, Jan 26, 2017 at 8:06 PM, Stephan Ewen wrote: > @Till - I think that FLINK-5667 is a blocker > > Good catch finding it! > >

Re: [VOTE] Release Apache Flink 1.2.0 (RC2)

2017-01-26 Thread Gyula Fóra
Hi, Aside from the issues mentioned above I have some good news as well. I have finished porting and started testing one of our major production jobs (RBea) on 1.2 and everything seems to run well so far, with savepoints, rescaling, externalized checkpoints, metrics etc. on YARN. In this job I u

[jira] [Created] (FLINK-5668) Reduce dependency on HDFS at job startup time

2017-01-26 Thread Bill Liu (JIRA)
Bill Liu created FLINK-5668: --- Summary: Reduce dependency on HDFS at job startup time Key: FLINK-5668 URL: https://issues.apache.org/jira/browse/FLINK-5668 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-5669) flink-streaming-contrib DataStreamUtils.collect in local environment mode fails when offline

2017-01-26 Thread Rick Cox (JIRA)
Rick Cox created FLINK-5669: --- Summary: flink-streaming-contrib DataStreamUtils.collect in local environment mode fails when offline Key: FLINK-5669 URL: https://issues.apache.org/jira/browse/FLINK-5669 Proj