A Question About Execution Schedule

2015-08-18 Thread MaGuoWei
Hi,guysThere are some codes in scheduleForExecution Function in Execution.java: 1. SlotAllocationFuture future = scheduler.scheduleQueued(toSchedule); 2. future.setFutureAction(new SlotAllocationFutureAction() {..}) When scheduler finds some slot between line1 and line 2 the sc

[jira] [Created] (FLINK-2545) NegativeArraySizeException while creating hash table bloom filters

2015-08-18 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-2545: - Summary: NegativeArraySizeException while creating hash table bloom filters Key: FLINK-2545 URL: https://issues.apache.org/jira/browse/FLINK-2545 Project: Flink I

Driver Test Base issue

2015-08-18 Thread Sachin Goel
There appears to be some issue in DriverTestBase. I have observed two failures recently, once in MatchTaskTest and MapTaskTest, both with the following trace: Exception in thread "Thread-154" java.lang.AssertionError: Canceling task failed: java.lang.NullPointerException at org.apache.flink.runti

Re: "main" java.lang.NoClassDefFoundError: scala/collection/Traversable

2015-08-18 Thread Stephan Ewen
Hi! This looks like your IDE setup is not quite correct. The Scala library is probably not in the classpath when you launch the program. Please check if you correctly installed your Scala Plugin, of the Scala Lib has been added to the build path in the project properties, ... Greetings, Stephan

[jira] [Created] (FLINK-2544) Some test cases using PowerMock fail with Java 8u20

2015-08-18 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-2544: Summary: Some test cases using PowerMock fail with Java 8u20 Key: FLINK-2544 URL: https://issues.apache.org/jira/browse/FLINK-2544 Project: Flink Issue Type:

Re: Some thoughts about query max && min block count in CompactingHashTable

2015-08-18 Thread huangwei (G)
Hi Stephan, Yes, you are right. This thinking doesn`t work if the "getMaxPartition()" is rare use. Greetings, Huangwei === Hi! The thinking is good. Although in this case, it is probably not beneficial to replace the list by a heap or so, due to the rare usage of that function. A

Re: Some thoughts about query max && min block count in CompactingHashTable

2015-08-18 Thread Stephan Ewen
Hi! The thinking is good. Although in this case, it is probably not beneficial to replace the list by a heap or so, due to the rare usage of that function. As you said, the cost depends on how often the "getMaxPartition()" is called. Current approach: seldom O(n) - (only on function call) Heap a

[jira] [Created] (FLINK-2543) State handling does not support deserializing classes through the UserCodeClassloader

2015-08-18 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-2543: - Summary: State handling does not support deserializing classes through the UserCodeClassloader Key: FLINK-2543 URL: https://issues.apache.org/jira/browse/FLINK-2543

Re: Nested iterations not supported?

2015-08-18 Thread Stephan Ewen
Hi Pieter! The pattern you need is common and we have code for such loops in the works. It is being delayed right now due to work on fault-tolerance / high availability, but we plan to resume with that after those issues are done. Greetings, Stephan On Tue, Aug 18, 2015 at 12:38 PM, Pieter-Jan

[jira] [Created] (FLINK-2542) It should be documented that it is required from a join key to override hashCode(), when it is not a POJO

2015-08-18 Thread Gabor Gevay (JIRA)
Gabor Gevay created FLINK-2542: -- Summary: It should be documented that it is required from a join key to override hashCode(), when it is not a POJO Key: FLINK-2542 URL: https://issues.apache.org/jira/browse/FLINK-254

[jira] [Created] (FLINK-2541) TypeComparator creation fails for T2, byte[]>

2015-08-18 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-2541: --- Summary: TypeComparator creation fails for T2, byte[]> Key: FLINK-2541 URL: https://issues.apache.org/jira/browse/FLINK-2541 Project: Flink Issue Type:

Re: Encountered "TypeComparator creation has a bug"

2015-08-18 Thread Fabian Hueske
Can you open a JIRA with instructions to reproduce the bug? Thanks, Fabian 2015-08-18 12:33 GMT+02:00 Chesnay Schepler : > While running the following job i run into an exception: > Tuple comparator creation has a bug > java.lang.IllegalArgumentException: Tuple comparator creation has a bug >

Re: Nested iterations not supported?

2015-08-18 Thread Pieter-Jan Van Aeken
Hi, Thanks for the quick response. Thinking of a workaround right now, where I simply run the same application 100 times but I would like to avoid as much duplicate work as possible. Is it possible to "broadcast" nodes and edges datasets from one execution environment to the next without passing

Encountered "TypeComparator creation has a bug"

2015-08-18 Thread Chesnay Schepler
While running the following job i run into an exception: Tuple comparator creation has a bug java.lang.IllegalArgumentException: Tuple comparator creation has a bug at org.apache.flink.api.java.typeutils.TupleTypeInfo.getNewComparator(TupleTypeInfo.java:131) ... job: ExecutionEnvironme

Re: [NOTICE] Reporting and fixing minor issues

2015-08-18 Thread Kostas Tzoumas
big +1 On Tue, Aug 18, 2015 at 10:43 AM, Till Rohrmann wrote: > +1, there is no point in arguing with Knuth. > > On Mon, Aug 17, 2015 at 1:07 AM, Henry Saputra > wrote: > > > +1 as well. > > > > This is a great follow-up from my previous email about adding details > > in JIRA, which also being

Re: Nested iterations not supported?

2015-08-18 Thread Andra Lungu
Hello Pieter, Nested iterations are indeed not supported in Flink. http://mail-archives.apache.org/mod_mbox/flink-user/201504.mbox/%3Cop.xw24u7fhf7e33m@vaio-sb%3E The problem is not in your code. On Tue, Aug 18, 2015 at 11:27 AM, Pieter-Jan Van Aeken < pieterjan.vanae...@euranova.eu> wrote: >

[jira] [Created] (FLINK-2540) LocalBufferPool.requestBuffer gets into infinite loop

2015-08-18 Thread Gabor Gevay (JIRA)
Gabor Gevay created FLINK-2540: -- Summary: LocalBufferPool.requestBuffer gets into infinite loop Key: FLINK-2540 URL: https://issues.apache.org/jira/browse/FLINK-2540 Project: Flink Issue Type: B

Some thoughts about query max && min block count in CompactingHashTable

2015-08-18 Thread huangwei (G)
Hi, I found some like the code following may affect performance( Time complexity is O(n) ): private int getMaxPartition() { int maxPartition = 0; for(InMemoryPartition p1 : this.partitions) { if(p1.getBlockCount() > maxPartition) { maxPartition = p1.getBlockCount(); }

Nested iterations not supported?

2015-08-18 Thread Pieter-Jan Van Aeken
Hello all, I am having some troubles getting nested iterations to work. The basic outline of my application looks like this : 1. create vertex dataset 2. create edge dataset 3. bulk iterate 100 times on edges { 3a. Create graph from nodes and edges 3b. Perform GatherSumApply (delta itera

re: re: re: Some problems about Flink applications

2015-08-18 Thread huangwei (G)
Hi Chesnay, Sorry for reply late. It works! Thank you! Greetings Huangwei = How about something like this: DataStream> sink = user .join(area) .onWindow(15, TimeUnit.MINUTES) .where(0) .equalTo(0) .

Re: Code style guideline for Scala

2015-08-18 Thread Chiwan Park
Creating a JIRA issue [1] is done. Regards, Chiwan Park [1] https://issues.apache.org/jira/browse/FLINK-2539 > On Aug 18, 2015, at 5:28 PM, Till Rohrmann wrote: > > Good initiative Chiwan. +1 for a more unified code style. > > On Tue, Aug 18, 2015 at 10:25 AM, Chiwan Park wrote: > >> Okay,

[jira] [Created] (FLINK-2539) More unified code style for Scala code

2015-08-18 Thread Chiwan Park (JIRA)
Chiwan Park created FLINK-2539: -- Summary: More unified code style for Scala code Key: FLINK-2539 URL: https://issues.apache.org/jira/browse/FLINK-2539 Project: Flink Issue Type: Improvement

Re: [NOTICE] Reporting and fixing minor issues

2015-08-18 Thread Till Rohrmann
+1, there is no point in arguing with Knuth. On Mon, Aug 17, 2015 at 1:07 AM, Henry Saputra wrote: > +1 as well. > > This is a great follow-up from my previous email about adding details > in JIRA, which also being echoed by Fabian. > > - Henry > > On Sun, Aug 16, 2015 at 3:45 PM, Fabian Hueske

Re: Code style guideline for Scala

2015-08-18 Thread Till Rohrmann
Good initiative Chiwan. +1 for a more unified code style. On Tue, Aug 18, 2015 at 10:25 AM, Chiwan Park wrote: > Okay, I’ll create a JIRA issue covered this topic. > > Regards, > Chiwan Park > > > On Aug 17, 2015, at 1:17 AM, Stephan Ewen wrote: > > > > +1 for formatting templates for Eclipse a

Re: Code style guideline for Scala

2015-08-18 Thread Chiwan Park
Okay, I’ll create a JIRA issue covered this topic. Regards, Chiwan Park > On Aug 17, 2015, at 1:17 AM, Stephan Ewen wrote: > > +1 for formatting templates for Eclipse and IntelliJ. > > On Sun, Aug 16, 2015 at 6:06 PM, Sachin Goel > wrote: > >> We should also write up a matching configuration