[jira] [Commented] (FLINK-2809) DataSet[Unit] doesn't work
[ https://issues.apache.org/jira/browse/FLINK-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961835#comment-14961835 ] ASF GitHub Bot commented on FLINK-2809: --- Github user aalexandrov commented on the pull request: https://github.com/apache/flink/pull/1217#issuecomment-148901986 > Not all methods without paremeters should translate to methods without parenthesis... @StephanEwen I agree with that, but I cannot understand how the `UnitTypeInfo` might cause a confusion here. The typeInformation macros are synthesized by the macro based on the inferred collection type, which means that the meaning of `()` is resolved before that. Consider the following example: ```scala // in the Scala REPL case class Foo(answer: Int) // defined class Foo def f1(): Foo = Foo(42) // f1: ()Foo def f2: Foo = Foo(42) // f2: Foo val xs = Seq(f1(), f2) // how a literate person would write it // xs: Seq[Foo] = List(Foo(42), Foo(42)) val xs = Seq(f1, f2) // how a dazed & confused person would write it, but still compiles // xs: Seq[Foo] = List(Foo(42), Foo(42)) val xs = Seq(f1, f2()) // even worse, but this breaks with a compiler exception // error: Foo does not take parameters // val xs = Seq(f1, f2()) val xs = Seq((), ()) // typing '()' without syntactic context resolves to Unit // xs: Seq[Unit] = List((), ()) ``` In all of the above situations `env.fromCollection(xs)` is (1) either going to typecheck and trigger `TypeInformation` synthesis or (2) fail with the above. Can you point to StackOverflow conversation or something similar where the issue you mention is explained with an example? > DataSet[Unit] doesn't work > -- > > Key: FLINK-2809 > URL: https://issues.apache.org/jira/browse/FLINK-2809 > Project: Flink > Issue Type: Bug > Components: Scala API >Reporter: Gabor Gevay >Assignee: Gabor Gevay >Priority: Minor > > The following code creates a DataSet\[Unit\]: > val env = ExecutionEnvironment.createLocalEnvironment() > val a = env.fromElements(1,2,3) > val b = a.map (_ => ()) > b.writeAsText("/tmp/xxx") > env.execute() > This doesn't work, because a VoidSerializer is created, which can't cope with > a BoxedUnit. See exception below. > I'm now thinking about creating a UnitSerializer class. > org.apache.flink.runtime.client.JobExecutionException: Job execution failed. > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.applyOrElse(JobManager.scala:314) > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) > at > org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:36) > at > org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:29) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at > org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.scala:29) > at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > at > org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:92) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > at akka.actor.ActorCell.invoke(ActorCell.scala:487) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) > at akka.dispatch.Mailbox.run(Mailbox.scala:221) > at akka.dispatch.Mailbox.exec(Mailbox.scala:231) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.lang.ClassCastException: scala.runtime.BoxedUnit cannot be > cast to java.lang.Void > at > org.apache.flink.api.common.typeutils.base.VoidSerializer.serialize(VoidSerializer.java:26) > at > org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:51) > at > org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:76) > at > org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:83) > at > org.apache.flink.runtime.operators.shipping.OutputCollector.collect(OutputCollector.java:65) > at > org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:177) >
[GitHub] flink pull request: [FLINK-2809] [scala-api] Added UnitTypeInfo an...
Github user aalexandrov commented on the pull request: https://github.com/apache/flink/pull/1217#issuecomment-148901986 > Not all methods without paremeters should translate to methods without parenthesis... @StephanEwen I agree with that, but I cannot understand how the `UnitTypeInfo` might cause a confusion here. The typeInformation macros are synthesized by the macro based on the inferred collection type, which means that the meaning of `()` is resolved before that. Consider the following example: ```scala // in the Scala REPL case class Foo(answer: Int) // defined class Foo def f1(): Foo = Foo(42) // f1: ()Foo def f2: Foo = Foo(42) // f2: Foo val xs = Seq(f1(), f2) // how a literate person would write it // xs: Seq[Foo] = List(Foo(42), Foo(42)) val xs = Seq(f1, f2) // how a dazed & confused person would write it, but still compiles // xs: Seq[Foo] = List(Foo(42), Foo(42)) val xs = Seq(f1, f2()) // even worse, but this breaks with a compiler exception // error: Foo does not take parameters // val xs = Seq(f1, f2()) val xs = Seq((), ()) // typing '()' without syntactic context resolves to Unit // xs: Seq[Unit] = List((), ()) ``` In all of the above situations `env.fromCollection(xs)` is (1) either going to typecheck and trigger `TypeInformation` synthesis or (2) fail with the above. Can you point to StackOverflow conversation or something similar where the issue you mention is explained with an example? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2858) Cannot build Flink Scala 2.11 with IntelliJ
[ https://issues.apache.org/jira/browse/FLINK-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961846#comment-14961846 ] Alexander Alexandrov commented on FLINK-2858: - I like the idea for a marker-based activation suggested here: http://stackoverflow.com/a/8391313 What do you think? > Cannot build Flink Scala 2.11 with IntelliJ > --- > > Key: FLINK-2858 > URL: https://issues.apache.org/jira/browse/FLINK-2858 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 0.10 >Reporter: Till Rohrmann > > If I activate the scala-2.11 profile from within IntelliJ (and thus > deactivate the scala-2.10 profile) in order to build Flink with Scala 2.11, > then Flink cannot be built. The problem is that some Scala macros cannot be > expanded because they were compiled with the wrong version (I assume 2.10). > This makes debugging tests with Scala 2.11 in IntelliJ impossible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2692) Untangle CsvInputFormat into PojoTypeCsvInputFormat and TupleTypeCsvInputFormat
[ https://issues.apache.org/jira/browse/FLINK-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961847#comment-14961847 ] Chesnay Schepler commented on FLINK-2692: - [~aljoscha] Great, that will work. Another question: the CommonCsvInputFormat contains a method setFields(int[] sourceFieldIndices, Class[] fieldTypes) . This method is not exposed in the CsvReader and only used in tests. Can it be removed? (together with setFieldsGeneric(int[] ...) in GenericCsvInputFormat) > Untangle CsvInputFormat into PojoTypeCsvInputFormat and > TupleTypeCsvInputFormat > > > Key: FLINK-2692 > URL: https://issues.apache.org/jira/browse/FLINK-2692 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Chesnay Schepler >Priority: Minor > > The {{CsvInputFormat}} currently allows to return values as a {{Tuple}} or a > {{Pojo}} type. As a consequence, the processing logic, which has to work for > both types, is overly complex. For example, the {{CsvInputFormat}} contains > fields which are only used when a Pojo is returned. Moreover, the pojo field > information are constructed by calling setter methods which have to be called > in a very specific order, otherwise they fail. E.g. one first has to call > {{setFieldTypes}} before calling {{setOrderOfPOJOFields}}, otherwise the > number of fields might be different. Furthermore, some of the methods can > only be called if the return type is a {{Pojo}} type, because they expect > that a {{PojoTypeInfo}} is present. > I think the {{CsvInputFormat}} should be refactored to make the code more > easily maintainable. I propose to split it up into a > {{PojoTypeCsvInputFormat}} and a {{TupleTypeCsvInputFormat}} which take all > the required information via their constructors instead of using the > {{setFields}} and {{setOrderOfPOJOFields}} approach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2692) Untangle CsvInputFormat into PojoTypeCsvInputFormat and TupleTypeCsvInputFormat
[ https://issues.apache.org/jira/browse/FLINK-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961848#comment-14961848 ] Aljoscha Krettek commented on FLINK-2692: - If it is only used in tests, I would think so. > Untangle CsvInputFormat into PojoTypeCsvInputFormat and > TupleTypeCsvInputFormat > > > Key: FLINK-2692 > URL: https://issues.apache.org/jira/browse/FLINK-2692 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Chesnay Schepler >Priority: Minor > > The {{CsvInputFormat}} currently allows to return values as a {{Tuple}} or a > {{Pojo}} type. As a consequence, the processing logic, which has to work for > both types, is overly complex. For example, the {{CsvInputFormat}} contains > fields which are only used when a Pojo is returned. Moreover, the pojo field > information are constructed by calling setter methods which have to be called > in a very specific order, otherwise they fail. E.g. one first has to call > {{setFieldTypes}} before calling {{setOrderOfPOJOFields}}, otherwise the > number of fields might be different. Furthermore, some of the methods can > only be called if the return type is a {{Pojo}} type, because they expect > that a {{PojoTypeInfo}} is present. > I think the {{CsvInputFormat}} should be refactored to make the code more > easily maintainable. I propose to split it up into a > {{PojoTypeCsvInputFormat}} and a {{TupleTypeCsvInputFormat}} which take all > the required information via their constructors instead of using the > {{setFields}} and {{setOrderOfPOJOFields}} approach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2692) Untangle CsvInputFormat into PojoTypeCsvInputFormat and TupleTypeCsvInputFormat
[ https://issues.apache.org/jira/browse/FLINK-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961858#comment-14961858 ] Chesnay Schepler commented on FLINK-2692: - ah my bad, its available through the Scala API's ExecutionEnvironment, so it's staying then... > Untangle CsvInputFormat into PojoTypeCsvInputFormat and > TupleTypeCsvInputFormat > > > Key: FLINK-2692 > URL: https://issues.apache.org/jira/browse/FLINK-2692 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Chesnay Schepler >Priority: Minor > > The {{CsvInputFormat}} currently allows to return values as a {{Tuple}} or a > {{Pojo}} type. As a consequence, the processing logic, which has to work for > both types, is overly complex. For example, the {{CsvInputFormat}} contains > fields which are only used when a Pojo is returned. Moreover, the pojo field > information are constructed by calling setter methods which have to be called > in a very specific order, otherwise they fail. E.g. one first has to call > {{setFieldTypes}} before calling {{setOrderOfPOJOFields}}, otherwise the > number of fields might be different. Furthermore, some of the methods can > only be called if the return type is a {{Pojo}} type, because they expect > that a {{PojoTypeInfo}} is present. > I think the {{CsvInputFormat}} should be refactored to make the code more > easily maintainable. I propose to split it up into a > {{PojoTypeCsvInputFormat}} and a {{TupleTypeCsvInputFormat}} which take all > the required information via their constructors instead of using the > {{setFields}} and {{setOrderOfPOJOFields}} approach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2411] Add graph summarization algorithm
GitHub user s1ck opened a pull request: https://github.com/apache/flink/pull/1264 [FLINK-2411] Add graph summarization algorithm * added algorithm * added integration tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/s1ck/flink FLINK-2411 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1264.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1264 commit 3ac42a6c23fc900b6451174b5b463cdba60951da Author: Martin Junghanns Date: 2015-10-17T14:59:32Z [FLINK-2411] Add graph summarization algorithm * added algorithm * added integration tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2411) Add basic graph summarization algorithm
[ https://issues.apache.org/jira/browse/FLINK-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14962006#comment-14962006 ] ASF GitHub Bot commented on FLINK-2411: --- GitHub user s1ck opened a pull request: https://github.com/apache/flink/pull/1264 [FLINK-2411] Add graph summarization algorithm * added algorithm * added integration tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/s1ck/flink FLINK-2411 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1264.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1264 commit 3ac42a6c23fc900b6451174b5b463cdba60951da Author: Martin Junghanns Date: 2015-10-17T14:59:32Z [FLINK-2411] Add graph summarization algorithm * added algorithm * added integration tests > Add basic graph summarization algorithm > --- > > Key: FLINK-2411 > URL: https://issues.apache.org/jira/browse/FLINK-2411 > Project: Flink > Issue Type: New Feature > Components: Gelly >Affects Versions: 0.10 >Reporter: Martin Junghanns >Assignee: Martin Junghanns >Priority: Minor > > Graph summarization determines a structural grouping of similar vertices and > edges to condense a graph and thus helps to uncover insights about patterns > hidden in the graph. It can be used in OLAP-style operations on the graph and > is similar to group by in SQL but on the graph structure instead of rows. > > The graph summarization operator represents every vertex group by a single > vertex in the summarized graph; edges between vertices in the summary graph > represent a group of edges between the vertex group members of the original > graph. Summarization is defined by specifying grouping keys for vertices and > edges, respectively. > One publication that presents a Map/Reduce based approach is "Pagrol: > Parallel graph olap over large-scale attributed graphs", however they > pre-compute the graph-cube before it can be analyzed. With Flink, we can give > the user an interactive way of summarizing the graph and do not need to > compute the cube beforehand. > A more complex approach focuses on summarization on graph patterns > "SynopSys: Large Graph Analytics in the SAP HANA Database Through > Summarization". > However, I want to start with a simple algorithm that summarizes the graph on > vertex and optionally edge values and additionally stores a count aggregate > at summarized vertices/edges. > Consider the following two examples (e.g., social network with users from > cities and friendships with timestamp): > > h4. Input graph: > > Vertices (id, value): > (0, Leipzig) > (1, Leipzig) > (2, Dresden) > (3, Dresden) > (4, Dresden) > (5, Berlin) > Edges (source, target, value): > (0, 1, 2014) > (1, 0, 2014) > (1, 2, 2013) > (2, 1, 2013) > (2, 3, 2014) > (3, 2, 2014) > (4, 0, 2013) > (4, 1, 2015) > (5, 2, 2015) > (5, 3, 2015) > h4. Output graph (summarized on vertex value): > Vertices (id, value, count) > (0, Leipzig, 2) // "2 users from Leipzig" > (2, Dresden, 3) // "3 users from Dresden" > (5, Berlin, 1) // "1 user from Berlin" > Edges (source, target, count) > (0, 0, 2) // "2 edges between users in Leipzig" > (0, 2, 1) // "1 edge from users in Leipzig to users in Dresden" > (2, 0, 3) // "3 edges from users in Dresden to users in Leipzig" > (2, 2, 2) // "2 edges between users in Dresden" > (5, 2, 2) // "2 edges from users in Berlin to users in Dresden" > h4. Output graph (summarized on vertex and edge value): > Vertices (id, value, count) > (0, Leipzig, 2) > (2, Dresden, 3) > (5, Berlin, 1) > Edges (source, target, value, count) > (0, 0, 2014, 2) // ... > (0, 2, 2013, 1) // ... > (2, 0, 2013, 2) // "2 edges from users in Dresden to users in Leipzig with > timestamp 2013" > (2, 0, 2015, 1) // "1 edge from users in Dresden to users in Leipzig with > timestamp 2015" > (2, 2, 2014, 2) // ... > (5, 2, 2015, 2) // ... > I've already implemented two versions of the summarization algorithm in our > own project [Gradoop|https://github.com/dbs-leipzig/gradoop], which is a > graph analytics stack on top of Hadoop + Gelly/Flink with a fixed data model. > You can see the current WIP here: > 1 [Abstract > summarization|https://github.com/dbs-leipzig/gradoop/blob/%2345_gradoop_flink/gradoop-flink/src/main/java/org/gradoop/model/impl/operators/Summarization.java] > 2 [Implementation using > cross|https://github.com/dbs-leipzig/gradoop/blob/%2345_gradoop_flink/gradoop-flink/src/main/java/org/gradoop/model/impl/operators/SummarizationCross.java] > 3 [Implementation using > joins|https://github.com/dbs-leipzig/grado
[GitHub] flink pull request: Removed broken dependency to flink-spargel.
Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/1259#issuecomment-148939135 @StephanEwen has this been merged as part of your other commit? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2841] Correcting roadmap link to point ...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1254 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2844) Remove old web interface and default to the new one
[ https://issues.apache.org/jira/browse/FLINK-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14962047#comment-14962047 ] ASF GitHub Bot commented on FLINK-2844: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1246 > Remove old web interface and default to the new one > --- > > Key: FLINK-2844 > URL: https://issues.apache.org/jira/browse/FLINK-2844 > Project: Flink > Issue Type: New Feature > Components: JobManager >Affects Versions: 0.10 >Reporter: Maximilian Michels >Assignee: Maximilian Michels > Fix For: 0.10 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2844][jobmanager] remove old web interf...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1246 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2841) Broken roadmap link in FlinkML contribution guide
[ https://issues.apache.org/jira/browse/FLINK-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14962046#comment-14962046 ] ASF GitHub Bot commented on FLINK-2841: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1254 > Broken roadmap link in FlinkML contribution guide > - > > Key: FLINK-2841 > URL: https://issues.apache.org/jira/browse/FLINK-2841 > Project: Flink > Issue Type: Bug > Components: Documentation, Machine Learning Library >Affects Versions: 0.10 >Reporter: Chiwan Park >Assignee: Saumitra Shahapure > Labels: starter > > Because the roadmap of FlinkML is moved to wiki, we need to update roadmap > link in [FlinkML contribution > guide|https://ci.apache.org/projects/flink/flink-docs-master/libs/ml/contribution_guide.html]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Removed broken dependency to flink-spargel.
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1259 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-2841) Broken roadmap link in FlinkML contribution guide
[ https://issues.apache.org/jira/browse/FLINK-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-2841. - Resolution: Fixed Fix Version/s: 010 Fixed via 580768c3075e794cec248c8bf8de97b754a41021 Thank you for the contribution! > Broken roadmap link in FlinkML contribution guide > - > > Key: FLINK-2841 > URL: https://issues.apache.org/jira/browse/FLINK-2841 > Project: Flink > Issue Type: Bug > Components: Documentation, Machine Learning Library >Affects Versions: 0.10 >Reporter: Chiwan Park >Assignee: Saumitra Shahapure > Labels: starter > Fix For: 010 > > > Because the roadmap of FlinkML is moved to wiki, we need to update roadmap > link in [FlinkML contribution > guide|https://ci.apache.org/projects/flink/flink-docs-master/libs/ml/contribution_guide.html]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-2841) Broken roadmap link in FlinkML contribution guide
[ https://issues.apache.org/jira/browse/FLINK-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-2841. --- > Broken roadmap link in FlinkML contribution guide > - > > Key: FLINK-2841 > URL: https://issues.apache.org/jira/browse/FLINK-2841 > Project: Flink > Issue Type: Bug > Components: Documentation, Machine Learning Library >Affects Versions: 0.10 >Reporter: Chiwan Park >Assignee: Saumitra Shahapure > Labels: starter > Fix For: 010 > > > Because the roadmap of FlinkML is moved to wiki, we need to update roadmap > link in [FlinkML contribution > guide|https://ci.apache.org/projects/flink/flink-docs-master/libs/ml/contribution_guide.html]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-2844) Remove old web interface and default to the new one
[ https://issues.apache.org/jira/browse/FLINK-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-2844. - Resolution: Fixed Done in 45931095180f4fee9426bdcfb8b713ff5da0b0d2 > Remove old web interface and default to the new one > --- > > Key: FLINK-2844 > URL: https://issues.apache.org/jira/browse/FLINK-2844 > Project: Flink > Issue Type: New Feature > Components: JobManager >Affects Versions: 0.10 >Reporter: Maximilian Michels >Assignee: Maximilian Michels > Fix For: 0.10 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-2863) Kafka producer does not fail in case of write failure
[ https://issues.apache.org/jira/browse/FLINK-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-2863. - Resolution: Fixed Done in a8eeb3bb1314d6c6cc19001181539d74bc25f419 > Kafka producer does not fail in case of write failure > - > > Key: FLINK-2863 > URL: https://issues.apache.org/jira/browse/FLINK-2863 > Project: Flink > Issue Type: Bug > Components: Kafka Connector >Affects Versions: 0.10 >Reporter: Stephan Ewen >Assignee: Stephan Ewen > Fix For: 0.10 > > > The async producer used in the Kafka connector only logs errors, but does not > fail the program in case of an error. > I will change it such that it fails by default on error and add a flag for > the "lenient" mode that only logs failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-2844) Remove old web interface and default to the new one
[ https://issues.apache.org/jira/browse/FLINK-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-2844. --- > Remove old web interface and default to the new one > --- > > Key: FLINK-2844 > URL: https://issues.apache.org/jira/browse/FLINK-2844 > Project: Flink > Issue Type: New Feature > Components: JobManager >Affects Versions: 0.10 >Reporter: Maximilian Michels >Assignee: Maximilian Michels > Fix For: 0.10 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-2863) Kafka producer does not fail in case of write failure
[ https://issues.apache.org/jira/browse/FLINK-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-2863. --- > Kafka producer does not fail in case of write failure > - > > Key: FLINK-2863 > URL: https://issues.apache.org/jira/browse/FLINK-2863 > Project: Flink > Issue Type: Bug > Components: Kafka Connector >Affects Versions: 0.10 >Reporter: Stephan Ewen >Assignee: Stephan Ewen > Fix For: 0.10 > > > The async producer used in the Kafka connector only logs errors, but does not > fail the program in case of an error. > I will change it such that it fails by default on error and add a flag for > the "lenient" mode that only logs failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2864] Make State of General-Purpose Win...
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1265 [FLINK-2864] Make State of General-Purpose Window Operators Fault-Tolerant This adds method state() on Trigger context that should be used to create an OperatorState to deal with fault-tolerant state. WindowAssigner now has a method getWindowSerializer() that is used to get a TypeSerializer for the Windows that it assigns. The Serializer for the Key is retrieved from the input KeyedStream and the serializer for the input elements is already available. During checkpointing all currently in-flight windows (per key, per window) are serialized using the TypeSerializers. The state that is accessible in Triggers using state() is kept in a HashMap, this is serialized using java serialization. This introduces the restriction that the element must be Serializable when using DeltaTrigger. I did not yet take the step of integrating triggers with the operator-provided key-value state since this would require state to be very dynamic and also allow deletion of state. @StephanEwen could you please have a look at how the state checkpoint/restore is implemented. Also the triggers have a state interface that differs from the state interface that user functions have, for now. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink window-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1265.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1265 commit fb5733f52df8eabfd88f51fb39f83930f27befbd Author: Aljoscha Krettek Date: 2015-10-11T09:37:29Z [FLINK-2864] Make State of General-Purpose Window Operators Fault-Tolerant This adds method state() on Trigger context that should be used to create an OperatorState to deal with fault-tolerant state. WindowAssigner now has a method getWindowSerializer() that is used to get a TypeSerializer for the Windows that it assigns. The Serializer for the Key is retrieved from the input KeyedStream and the serializer for the input elements is already available. During checkpointing all currently in-fligh windows (per key, per window) are serialized using the TypeSerializers. The state that is accessible in Triggers using state() is kept in a HashMap, this is serialized using java serialization. commit 07d96f5cc41ae70006699c3f0a6986252565e3df Author: Aljoscha Krettek Date: 2015-10-17T11:35:24Z Replace Trigger.onTime by Trigger.onProcessingTime/onEventTime This also renames WatermarkTrigger to EventTimeTrigger and ContinuousWatermarkTrigger to ContinuousEventTimeTrigger. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2864) Make State of General-Purpose Window Operators Fault-Tolerant
[ https://issues.apache.org/jira/browse/FLINK-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14962073#comment-14962073 ] ASF GitHub Bot commented on FLINK-2864: --- GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1265 [FLINK-2864] Make State of General-Purpose Window Operators Fault-Tolerant This adds method state() on Trigger context that should be used to create an OperatorState to deal with fault-tolerant state. WindowAssigner now has a method getWindowSerializer() that is used to get a TypeSerializer for the Windows that it assigns. The Serializer for the Key is retrieved from the input KeyedStream and the serializer for the input elements is already available. During checkpointing all currently in-flight windows (per key, per window) are serialized using the TypeSerializers. The state that is accessible in Triggers using state() is kept in a HashMap, this is serialized using java serialization. This introduces the restriction that the element must be Serializable when using DeltaTrigger. I did not yet take the step of integrating triggers with the operator-provided key-value state since this would require state to be very dynamic and also allow deletion of state. @StephanEwen could you please have a look at how the state checkpoint/restore is implemented. Also the triggers have a state interface that differs from the state interface that user functions have, for now. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink window-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1265.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1265 commit fb5733f52df8eabfd88f51fb39f83930f27befbd Author: Aljoscha Krettek Date: 2015-10-11T09:37:29Z [FLINK-2864] Make State of General-Purpose Window Operators Fault-Tolerant This adds method state() on Trigger context that should be used to create an OperatorState to deal with fault-tolerant state. WindowAssigner now has a method getWindowSerializer() that is used to get a TypeSerializer for the Windows that it assigns. The Serializer for the Key is retrieved from the input KeyedStream and the serializer for the input elements is already available. During checkpointing all currently in-fligh windows (per key, per window) are serialized using the TypeSerializers. The state that is accessible in Triggers using state() is kept in a HashMap, this is serialized using java serialization. commit 07d96f5cc41ae70006699c3f0a6986252565e3df Author: Aljoscha Krettek Date: 2015-10-17T11:35:24Z Replace Trigger.onTime by Trigger.onProcessingTime/onEventTime This also renames WatermarkTrigger to EventTimeTrigger and ContinuousWatermarkTrigger to ContinuousEventTimeTrigger. > Make State of General-Purpose Window Operators Fault-Tolerant > - > > Key: FLINK-2864 > URL: https://issues.apache.org/jira/browse/FLINK-2864 > Project: Flink > Issue Type: Sub-task > Components: Streaming >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek > Fix For: 0.10 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Fix Avro getter/setter recognition
Github user ukarlsson commented on the pull request: https://github.com/apache/flink/pull/1252#issuecomment-148949103 Hello, thanks for your feedback, yes I will add these things. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---