[jira] [Commented] (FLINK-3657) Change access of DataSetUtils.countElements() to 'public'
[ https://issues.apache.org/jira/browse/FLINK-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222578#comment-15222578 ] ASF GitHub Bot commented on FLINK-3657: --- Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/1829#issuecomment-204614791 @mbalassi : There is already request from @zentol and me to clean up the PR, so would love to get this merge excluding the refactor parts. Thanks. I think we should merge this to master and 1.1.0, the question is whether this should go to 1.0.1 release, which @fhueske and @uce think should not. > Change access of DataSetUtils.countElements() to 'public' > -- > > Key: FLINK-3657 > URL: https://issues.apache.org/jira/browse/FLINK-3657 > Project: Flink > Issue Type: Improvement > Components: DataSet API >Affects Versions: 1.0.0 >Reporter: Suneel Marthi >Assignee: Suneel Marthi >Priority: Minor > Fix For: 1.0.1 > > > The access of DatasetUtils.countElements() is presently 'private', change > that to be 'public'. We happened to be replicating the functionality in our > project and realized the method already existed in Flink. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: FLINK-3657: Change access of DataSetUtils.coun...
Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/1829#issuecomment-204614791 @mbalassi : There is already request from @zentol and me to clean up the PR, so would love to get this merge excluding the refactor parts. Thanks. I think we should merge this to master and 1.1.0, the question is whether this should go to 1.0.1 release, which @fhueske and @uce think should not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3664) Create a method to easily Summarize a DataSet
[
https://issues.apache.org/jira/browse/FLINK-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222460#comment-15222460
]
Fabian Hueske commented on FLINK-3664:
--
Hi Todd, thanks for sharing your work so far. I briefly skimmed the linked
commit and it looks pretty good to me. My only comments right now would be to
add a few more tests, but that's on your to do list anyways.
I'm on vacation right now and will return in a few days. It would be nice if
you could continue with your work and open a PR when you think it is ready.
Unless nobody else has had a detailed look, I'll review it when I'm back.
Btw. this work is also relevant in the context of the currently on-going
efforts to add support for SQL as it allows to gather basic statistics on a
DataSet.
Thanks, Fabian
> Create a method to easily Summarize a DataSet
> -
>
> Key: FLINK-3664
> URL: https://issues.apache.org/jira/browse/FLINK-3664
> Project: Flink
> Issue Type: Improvement
>Reporter: Todd Lisonbee
> Attachments: DataSet-Summary-Design-March2016-v1.txt
>
>
> Here is an example:
> {code}
> /**
> * Summarize a DataSet of Tuples by collecting single pass statistics for all
> columns
> */
> public Tuple summarize()
> Dataset> input = // [...]
> Tuple3 summary
> = input.summarize()
> summary.getField(0).stddev()
> summary.getField(1).maxStringLength()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2946) Add orderBy() to Table API
[
https://issues.apache.org/jira/browse/FLINK-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222431#comment-15222431
]
Fabian Hueske commented on FLINK-2946:
--
Hi Dawid,
at the moment, the Table API does not allow to configure the parallelism of the
execution plan. All Table API operators (incl. sorting) should be executed with
the default parallelism as defined in the {{ExecutionEnvironment}}. Hence, you
are right. We should not look at the parallelism of the preceding task but at
the parallelism of the environment.
You can get the {{ExecutionEnvironment}} from the input {{DataSet}} by calling
{{getExecutionEnvironment}}. The {{ExecutionEnvironment}} has a method
{{getParallelism}}. In case the parallelism was not explicitly defined
({{getParallelism()}} returns {{-1}}) we should add the range partitioner.
> Add orderBy() to Table API
> --
>
> Key: FLINK-2946
> URL: https://issues.apache.org/jira/browse/FLINK-2946
> Project: Flink
> Issue Type: New Feature
> Components: Table API
>Reporter: Timo Walther
>Assignee: Dawid Wysakowicz
>
> In order to implement a FLINK-2099 prototype that uses the Table APIs code
> generation facilities, the Table API needs a sorting feature.
> I would implement it the next days. Ideas how to implement such a sorting
> feature are very welcome. Is there any more efficient way instead of
> {{.sortPartition(...).setParallism(1)}}? Is it better to sort locally on the
> nodes first and finally sort on one node afterwards?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: FLINK-3657: Change access of DataSetUtils.coun...
Github user mbalassi commented on the pull request: https://github.com/apache/flink/pull/1829#issuecomment-204587443 I think @smarthi has a point that the functionality is a nice fit for `DataSetUtils` and he has properly added it to the scala API and covered the tests. It is true that it has some minor refactoring obfuscating the core PR, but besides that LGTM. If noone thinks otherwise I would merge this Tuesday morning with optionally excluding the refactor parts if it is considered that bothersome. Unfortunately for the Mahout release I think the policy is as @fhueske suggests that we should reserve new features for the next, 1.1.0 release line. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3657) Change access of DataSetUtils.countElements() to 'public'
[ https://issues.apache.org/jira/browse/FLINK-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222423#comment-15222423 ] ASF GitHub Bot commented on FLINK-3657: --- Github user mbalassi commented on the pull request: https://github.com/apache/flink/pull/1829#issuecomment-204587443 I think @smarthi has a point that the functionality is a nice fit for `DataSetUtils` and he has properly added it to the scala API and covered the tests. It is true that it has some minor refactoring obfuscating the core PR, but besides that LGTM. If noone thinks otherwise I would merge this Tuesday morning with optionally excluding the refactor parts if it is considered that bothersome. Unfortunately for the Mahout release I think the policy is as @fhueske suggests that we should reserve new features for the next, 1.1.0 release line. > Change access of DataSetUtils.countElements() to 'public' > -- > > Key: FLINK-3657 > URL: https://issues.apache.org/jira/browse/FLINK-3657 > Project: Flink > Issue Type: Improvement > Components: DataSet API >Affects Versions: 1.0.0 >Reporter: Suneel Marthi >Assignee: Suneel Marthi >Priority: Minor > Fix For: 1.0.1 > > > The access of DatasetUtils.countElements() is presently 'private', change > that to be 'public'. We happened to be replicating the functionality in our > project and realized the method already existed in Flink. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-3691) Extend AvroInputFormat to support Avro GenericRecord
[ https://issues.apache.org/jira/browse/FLINK-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gna Phetsarath updated FLINK-3691: -- Priority: Minor (was: Major) Component/s: Core I already have a working version, I need to create a branch and then a PR with all of the necessary tests. Should be able to get to this next week, if there are no objections. > Extend AvroInputFormat to support Avro GenericRecord > > > Key: FLINK-3691 > URL: https://issues.apache.org/jira/browse/FLINK-3691 > Project: Flink > Issue Type: Improvement > Components: Core >Reporter: Gna Phetsarath >Priority: Minor > > Extend AvroInputFormat to support Avro GenericRecord: > Sometimes the calling application will have access to the schema but not the > implementing class for Avro files. > Using a GenericDatumReader will allow the AvroInput format to read records in > the Avro Generic Record Format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3691) Extend AvroInputFormat to support Avro GenericRecord
Gna Phetsarath created FLINK-3691: - Summary: Extend AvroInputFormat to support Avro GenericRecord Key: FLINK-3691 URL: https://issues.apache.org/jira/browse/FLINK-3691 Project: Flink Issue Type: Improvement Reporter: Gna Phetsarath Extend AvroInputFormat to support Avro GenericRecord: Sometimes the calling application will have access to the schema but not the implementing class for Avro files. Using a GenericDatumReader will allow the AvroInput format to read records in the Avro Generic Record Format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3690) Create documentation on the new ResourceManager component
[ https://issues.apache.org/jira/browse/FLINK-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1520#comment-1520 ] Henry Saputra commented on FLINK-3690: -- Thanks for the additional info, Max. From the diagram I thought there is an option to deploy ResourceManager as separate component. > Create documentation on the new ResourceManager component > - > > Key: FLINK-3690 > URL: https://issues.apache.org/jira/browse/FLINK-3690 > Project: Flink > Issue Type: Sub-task > Components: Documentation, ResourceManager >Affects Versions: 1.1.0 >Reporter: Henry Saputra >Assignee: Maximilian Michels > Fix For: 1.1.0 > > > Need proper documentation for the new ResourceManager and how it will impact > deployment in different supported modes. > Also, we have been very good adding new docs for our internal in the wiki [1] > so would like that to happen for people evaluating Flink. > [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3519) Subclasses of Tuples don't work if the declared type of a DataSet is not the descendant
[
https://issues.apache.org/jira/browse/FLINK-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222156#comment-15222156
]
ASF GitHub Bot commented on FLINK-3519:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1724#issuecomment-204517362
Thanks, I like the docs now!
Sorry for nitpicking, but could you
- Make the bullet list in the docs use the html tags
(``)? It does not render properly in the docs otherwise.
- Write references to for example `DataSet` as
`DataStream`/`DataSet`? I think the DataStream API is used
more by now, so would be nice to have it in there.
> Subclasses of Tuples don't work if the declared type of a DataSet is not the
> descendant
> ---
>
> Key: FLINK-3519
> URL: https://issues.apache.org/jira/browse/FLINK-3519
> Project: Flink
> Issue Type: Bug
> Components: Type Serialization System
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
>Assignee: Gabor Gevay
>Priority: Minor
>
> If I have a subclass of TupleN, then objects of this type will turn into
> TupleNs when I try to use them in a DataSet.
> For example, if I have a class like this:
> {code}
> public static class Foo extends Tuple1 {
> public short a;
> public Foo() {}
> public Foo(int f0, int a) {
> this.f0 = f0;
> this.a = (short)a;
> }
> @Override
> public String toString() {
> return "(" + f0 + ", " + a + ")";
> }
> }
> {code}
> And then I do this:
> {code}
> env.fromElements(0,0,0).map(new MapFunction>() {
> @Override
> public Tuple1 map(Integer value) throws Exception {
> return new Foo(5, 6);
> }
> }).print();
> {code}
> Then I don't have Foos in the output, but only Tuples:
> {code}
> (5)
> (5)
> (5)
> {code}
> The problem is caused by the TupleSerializer not caring about subclasses at
> all. I guess the reason for this is performance: we don't want to deal with
> writing and reading subclass tags when we have Tuples.
> I see three options for solving this:
> 1. Add subclass tags to the TupleSerializer: This is not really an option,
> because we don't want to loose performance.
> 2. Document this behavior in the javadoc of the Tuple classes.
> 3. Make the Tuple types final: this would be the clean solution, but it is
> API breaking, and the first victim would be Gelly: the Vertex and Edge types
> extend from tuples. (Note that the issue doesn't appear there, because the
> DataSets there always have the type of the descendant class.)
> When deciding between 2. and 3., an important point to note is that if you
> have your class extend from a Tuple type instead of just adding the f0, f1,
> ... fields manually in the hopes of getting the performance boost associated
> with Tuples, then you are out of luck: the PojoSerializer will kick in anyway
> when the declared types of your DataSets are the descendant type.
> If someone knows about a good reason to extend from a Tuple class, then
> please comment.
> For 2., this is a suggested wording for the javadoc of the Tuple classes:
> Warning: Please don't subclass Tuple classes, but if you do, then be sure to
> always declare the element type of your DataSets to your descendant type.
> (That is, if you have a "class A extends Tuple2", then don't use instances of
> A in a DataSet, but use DataSet.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3519] [core] Add warning about subclass...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1724#issuecomment-204517362 Thanks, I like the docs now! Sorry for nitpicking, but could you - Make the bullet list in the docs use the html tags (``)? It does not render properly in the docs otherwise. - Write references to for example `DataSet` as `DataStream`/`DataSet`? I think the DataStream API is used more by now, so would be nice to have it in there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3560) Examples shouldn't always print usage statement
[ https://issues.apache.org/jira/browse/FLINK-3560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222147#comment-15222147 ] ASF GitHub Bot commented on FLINK-3560: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1752#issuecomment-204510235 From my side, +1 for removing the `ProgramDescription` interface. I think it is a mismatch with the `main()` method based approach, because it needs to instantiate the class, while the general entry point is static. That does not go well together in my opinion. People will write programs with no default constructor. Previously, with the `Program` interface, such a constructor was also mandatory, it did fit better there. > Examples shouldn't always print usage statement > --- > > Key: FLINK-3560 > URL: https://issues.apache.org/jira/browse/FLINK-3560 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 1.0.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Minor > > At the moment all our examples print a usage statement no matter whether the > parameters have been provided or not. This can be confusing for people > because an usage statement is usually only printed if one has specified a > wrong parameter or if a parameter is missing. > I propose to remove the unchecked printing of the usage statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3560] [examples] Remove unchecked outpu...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1752#issuecomment-204510235 From my side, +1 for removing the `ProgramDescription` interface. I think it is a mismatch with the `main()` method based approach, because it needs to instantiate the class, while the general entry point is static. That does not go well together in my opinion. People will write programs with no default constructor. Previously, with the `Program` interface, such a constructor was also mandatory, it did fit better there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3428) Add fixed time trailing timestamp/watermark extractor
[ https://issues.apache.org/jira/browse/FLINK-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222140#comment-15222140 ] ASF GitHub Bot commented on FLINK-3428: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1764#issuecomment-204508742 The code looks good now. - How about we move all the convenience timestamp extractors / watermark generators into a separate packe `.functions.timestamps`? **Docs** I am a bit unsure about the docs. They seem to get confusing now. What is a mechanism (periodic / punctuated), what is just a convenience implementation (Ascending, BoundedOutOfOrderness). How about leaving the docs page on watermark generation as it is and add another sub-page that lists pre-defined watermark generators. We would list all the future watermark generators there: - ascending timestamps - bounded out-of-orderness - historgam based > Add fixed time trailing timestamp/watermark extractor > - > > Key: FLINK-3428 > URL: https://issues.apache.org/jira/browse/FLINK-3428 > Project: Flink > Issue Type: Improvement >Reporter: Robert Metzger >Assignee: Kostas Kloudas > > Flink currently provides only one build-in timestamp extractor, which assumes > strictly ascending timestamps. In real world use cases, timestamps are almost > never strictly ascending. > Therefore, I propose to provide an utility watermark extractor which is > generating watermarks with a fixed-time trailing. > The implementation should keep track of the highest event-time seen so far > and subtract a fixed amount of time from that event time. > This way, users can for example specify that the watermarks should always > "lag behind" 10 minutes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: FLINK-3428: Adds a fixed time trailing waterma...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1764#issuecomment-204508742 The code looks good now. - How about we move all the convenience timestamp extractors / watermark generators into a separate packe `.functions.timestamps`? **Docs** I am a bit unsure about the docs. They seem to get confusing now. What is a mechanism (periodic / punctuated), what is just a convenience implementation (Ascending, BoundedOutOfOrderness). How about leaving the docs page on watermark generation as it is and add another sub-page that lists pre-defined watermark generators. We would list all the future watermark generators there: - ascending timestamps - bounded out-of-orderness - historgam based --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-3689) JobManager blocks cluster shutdown when not connected to ResourceManager
[ https://issues.apache.org/jira/browse/FLINK-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels closed FLINK-3689. - Resolution: Fixed Fixed in c2f2122c0ba0d0b16eb59aa3ad7958c72fb486b7 > JobManager blocks cluster shutdown when not connected to ResourceManager > > > Key: FLINK-3689 > URL: https://issues.apache.org/jira/browse/FLINK-3689 > Project: Flink > Issue Type: Bug > Components: JobManager, ResourceManager >Affects Versions: 1.1.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Minor > Fix For: 1.1.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3689] fix shutdown of JM when RM is not...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1847 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3689) JobManager blocks cluster shutdown when not connected to ResourceManager
[ https://issues.apache.org/jira/browse/FLINK-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222136#comment-15222136 ] ASF GitHub Bot commented on FLINK-3689: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1847 > JobManager blocks cluster shutdown when not connected to ResourceManager > > > Key: FLINK-3689 > URL: https://issues.apache.org/jira/browse/FLINK-3689 > Project: Flink > Issue Type: Bug > Components: JobManager, ResourceManager >Affects Versions: 1.1.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Minor > Fix For: 1.1.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1159) Case style anonymous functions not supported by Scala API
[
https://issues.apache.org/jira/browse/FLINK-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222095#comment-15222095
]
ASF GitHub Bot commented on FLINK-1159:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1704#issuecomment-204500365
Looks pretty good. I noticed three remaining things:
- Sometimes, the package and the directory trees are different (example:
`OnCoGroupDataSetTest.scala`). While Scala allows that, we usually keep them in
sync.
- The classes that implement the "withX" functions sometimes have context
bounds that I think they should not require (see inline comment above). Would
be great to remove them, because removing them later makes it API breaking.
- We need to decide how to label this for the future: stable, or
evolving. I would suggest to use `@PublicEvolving` on all the involved classes
for now.
> Case style anonymous functions not supported by Scala API
> -
>
> Key: FLINK-1159
> URL: https://issues.apache.org/jira/browse/FLINK-1159
> Project: Flink
> Issue Type: Bug
> Components: Scala API
>Reporter: Till Rohrmann
>Assignee: Stefano Baghino
>
> In Scala it is very common to define anonymous functions of the following form
> {code}
> {
> case foo: Bar => foobar(foo)
> case _ => throw new RuntimeException()
> }
> {code}
> These case style anonymous functions are not supported yet by the Scala API.
> Thus, one has to write redundant code to name the function parameter.
> What works is the following pattern, but it is not intuitive for someone
> coming from Scala:
> {code}
> dataset.map{
> _ match{
> case foo:Bar => ...
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1159] Case style anonymous functions no...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1704#issuecomment-204500365 Looks pretty good. I noticed three remaining things: - Sometimes, the package and the directory trees are different (example: `OnCoGroupDataSetTest.scala`). While Scala allows that, we usually keep them in sync. - The classes that implement the "withX" functions sometimes have context bounds that I think they should not require (see inline comment above). Would be great to remove them, because removing them later makes it API breaking. - We need to decide how to label this for the future: stable, or evolving. I would suggest to use `@PublicEvolving` on all the involved classes for now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3667) Generalize client<->cluster communication
[
https://issues.apache.org/jira/browse/FLINK-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222084#comment-15222084
]
Maximilian Michels commented on FLINK-3667:
---
The clients don't communicate directly with the ResourceManager. The JobManager
remains the proxy for client communication. The general protocol for job
submission or metrics collection won't change. Essentially, the proposed
changes won't affect users, unless they rely directly on the {{Client}} class
which will change API. However, the CLI client or the RemoteEnvironment won't
be affected.
> Generalize client<->cluster communication
> -
>
> Key: FLINK-3667
> URL: https://issues.apache.org/jira/browse/FLINK-3667
> Project: Flink
> Issue Type: Improvement
> Components: YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>
> Here are some notes I took when inspecting the client<->cluster classes with
> regard to future integration of other resource management frameworks in
> addition to Yarn (e.g. Mesos).
> {noformat}
> 1 Cluster Client Abstraction
>
> 1.1 Status Quo
> ──
> 1.1.1 FlinkYarnClient
> ╌
> • Holds the cluster configuration (Flink-specific and Yarn-specific)
> • Contains the deploy() method to deploy the cluster
> • Creates the Hadoop Yarn client
> • Receives the initial job manager address
> • Bootstraps the FlinkYarnCluster
> 1.1.2 FlinkYarnCluster
> ╌╌
> • Wrapper around the Hadoop Yarn client
> • Queries cluster for status updates
> • Life time methods to start and shutdown the cluster
> • Flink specific features like shutdown after job completion
> 1.1.3 ApplicationClient
> ╌╌╌
> • Acts as a middle-man for asynchronous cluster communication
> • Designed to communicate with Yarn, not used in Standalone mode
> 1.1.4 CliFrontend
> ╌
> • Deeply integrated with FlinkYarnClient and FlinkYarnCluster
> • Constantly distinguishes between Yarn and Standalone mode
> • Would be nice to have a general abstraction in place
> 1.1.5 Client
>
> • Job submission and Job related actions, agnostic of resource framework
> 1.2 Proposal
>
> 1.2.1 ClusterConfig (before: AbstractFlinkYarnClient)
> ╌
> • Extensible cluster-agnostic config
> • May be extended by specific cluster, e.g. YarnClusterConfig
> 1.2.2 ClusterClient (before: AbstractFlinkYarnClient)
> ╌
> • Deals with cluster (RM) specific communication
> • Exposes framework agnostic information
> • YarnClusterClient, MesosClusterClient, StandaloneClusterClient
> 1.2.3 FlinkCluster (before: AbstractFlinkYarnCluster)
> ╌
> • Basic interface to communicate with a running cluster
> • Receives the ClusterClient for cluster-specific communication
> • Should not have to care about the specific implementations of the
> client
> 1.2.4 ApplicationClient
> ╌╌╌
> • Can be changed to work cluster-agnostic (first steps already in
> FLINK-3543)
> 1.2.5 CliFrontend
> ╌
> • CliFrontend does never have to differentiate between different
> cluster types after it has determined which cluster class to load.
> • Base class handles framework agnostic command line arguments
> • Pluggables for Yarn, Mesos handle specific commands
> {noformat}
> I would like to create/refactor the affected classes to set us up for a more
> flexible client side resource management abstraction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1159) Case style anonymous functions not supported by Scala API
[
https://issues.apache.org/jira/browse/FLINK-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222081#comment-15222081
]
ASF GitHub Bot commented on FLINK-1159:
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1704#discussion_r58244518
--- Diff:
flink-scala/src/main/scala/org/apache/flink/api/scala/extensions/impl/acceptPartialFunctions/OnDataSet.scala
---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.scala.extensions.impl.acceptPartialFunctions
+
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.scala.{DataSet, GroupedDataSet}
+
+import scala.reflect.ClassTag
+
+/**
+ * Wraps a data set, allowing to use anonymous partial functions to
+ * perform extraction of items in a tuple, case class instance or
collection
+ *
+ * @param ds The wrapped data set
+ * @tparam T The type of the data set items, for which the type
information must be known
+ */
+class OnDataSet[T: TypeInformation](ds: DataSet[T]) {
--- End diff --
I think you do not need the `TypeInformation` context bound here. Functions
should never need the implicit type info for the input (that is contained in
the DataSet already), only ever for the return type.
> Case style anonymous functions not supported by Scala API
> -
>
> Key: FLINK-1159
> URL: https://issues.apache.org/jira/browse/FLINK-1159
> Project: Flink
> Issue Type: Bug
> Components: Scala API
>Reporter: Till Rohrmann
>Assignee: Stefano Baghino
>
> In Scala it is very common to define anonymous functions of the following form
> {code}
> {
> case foo: Bar => foobar(foo)
> case _ => throw new RuntimeException()
> }
> {code}
> These case style anonymous functions are not supported yet by the Scala API.
> Thus, one has to write redundant code to name the function parameter.
> What works is the following pattern, but it is not intuitive for someone
> coming from Scala:
> {code}
> dataset.map{
> _ match{
> case foo:Bar => ...
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1159] Case style anonymous functions no...
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1704#discussion_r58244518
--- Diff:
flink-scala/src/main/scala/org/apache/flink/api/scala/extensions/impl/acceptPartialFunctions/OnDataSet.scala
---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.scala.extensions.impl.acceptPartialFunctions
+
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.scala.{DataSet, GroupedDataSet}
+
+import scala.reflect.ClassTag
+
+/**
+ * Wraps a data set, allowing to use anonymous partial functions to
+ * perform extraction of items in a tuple, case class instance or
collection
+ *
+ * @param ds The wrapped data set
+ * @tparam T The type of the data set items, for which the type
information must be known
+ */
+class OnDataSet[T: TypeInformation](ds: DataSet[T]) {
--- End diff --
I think you do not need the `TypeInformation` context bound here. Functions
should never need the implicit type info for the input (that is contained in
the DataSet already), only ever for the return type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-3690) Create documentation on the new ResourceManager component
[ https://issues.apache.org/jira/browse/FLINK-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222073#comment-15222073 ] Maximilian Michels commented on FLINK-3690: --- The Resource Manager component is transparent to users and doesn't affect how they interact with Flink clusters. The Resource Manager is a generalization of the old Yarn implementation to integrate other resource managers like Mesos. The old Yarn support has been reimplemented using the Resource Manager. Nothing changed from the outside. Even the existing Yarn tests remained almost untouched. +1 For adding a section in the internals to explain the reasoning behind this new component. > Create documentation on the new ResourceManager component > - > > Key: FLINK-3690 > URL: https://issues.apache.org/jira/browse/FLINK-3690 > Project: Flink > Issue Type: Sub-task > Components: Documentation, ResourceManager >Affects Versions: 1.1.0 >Reporter: Henry Saputra >Assignee: Maximilian Michels > Fix For: 1.1.0 > > > Need proper documentation for the new ResourceManager and how it will impact > deployment in different supported modes. > Also, we have been very good adding new docs for our internal in the wiki [1] > so would like that to happen for people evaluating Flink. > [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3635) Potential null deference in TwitterExample#SelectEnglishAndTokenizeFlatMap#flatMap
[
https://issues.apache.org/jira/browse/FLINK-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222017#comment-15222017
]
ASF GitHub Bot commented on FLINK-3635:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1845#issuecomment-204477155
Merging this...
> Potential null deference in
> TwitterExample#SelectEnglishAndTokenizeFlatMap#flatMap
> --
>
> Key: FLINK-3635
> URL: https://issues.apache.org/jira/browse/FLINK-3635
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> JsonNode jsonNode = jsonParser.readValue(value, JsonNode.class);
> if (jsonNode.has("user") &&
> jsonNode.get("user").get("lang").asText().equals("en")) {
> {code}
> jsonNode.get("user").get("lang") may return null, leading to
> NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3635] Potential null deference in Twitt...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1845#issuecomment-204477155 Merging this... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3689) JobManager blocks cluster shutdown when not connected to ResourceManager
[ https://issues.apache.org/jira/browse/FLINK-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222012#comment-15222012 ] ASF GitHub Bot commented on FLINK-3689: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1847#issuecomment-204476710 Looks good to merge... > JobManager blocks cluster shutdown when not connected to ResourceManager > > > Key: FLINK-3689 > URL: https://issues.apache.org/jira/browse/FLINK-3689 > Project: Flink > Issue Type: Bug > Components: JobManager, ResourceManager >Affects Versions: 1.1.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Minor > Fix For: 1.1.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3689] fix shutdown of JM when RM is not...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1847#issuecomment-204476710 Looks good to merge... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [docs] Add first version of "Concepts" documen...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1843#issuecomment-204467720 Manually merged in 9aa207c073f878d711ca8107627c1f9b2ac17507 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [docs] Add first version of "Concepts" documen...
Github user StephanEwen closed the pull request at: https://github.com/apache/flink/pull/1843 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3633) Job submission silently fails when using user code types
[
https://issues.apache.org/jira/browse/FLINK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221900#comment-15221900
]
ASF GitHub Bot commented on FLINK-3633:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1818
> Job submission silently fails when using user code types
>
>
> Key: FLINK-3633
> URL: https://issues.apache.org/jira/browse/FLINK-3633
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.1.0
>
>
> With the changes introduced by FLINK-3327, it is no longer possible to run
> remote Flink jobs which work on user code types. The reason is that now the
> {{ExecutionConfig}} is directly stored in the {{JobGraph}} which is sent as
> an Akka message to the {{JobManager}}. Per default, user code types are
> automatically detected and registered in the {{ExecutionConfig}}. When
> deserializing a {{JobGraph}} whose {{ExecutionConfig}} contains user code
> classes the user code class loader is consequently required. However, Akka
> does not have access to it and uses the system class loader. This causes that
> Akka silently discards the {{SubmitJob}} message which cannot be deserialized
> because of a {{ClassNotFoundException}}.
> I propose to not sent the {{ExecutionConfig}} explicitly with the
> {{JobGraph}} and, thus, to partially revert the changes to before FLINK-3327.
> Before, the {{ExectuionConfig}} was serialized into the job configuration and
> deserialized on the {{TaskManager}} using the proper user code class loader.
> In order to reproduce the problem you can submit the following job to a
> remote cluster.
> {code}
> public class Job {
> public static class CustomType {
> private final int value;
> public CustomType(int value) {
> this.value = value;
> }
> @Override
> public String toString() {
> return "CustomType(" + value + ")";
> }
> }
> public static void main(String[] args) throws Exception {
> ExecutionEnvironment env =
> ExecutionEnvironment.createRemoteEnvironment(Address, Port, PathToJar);
> DataSet input = env.fromElements(1,2,3,4,5);
> DataSet customTypes = input.map(new
> MapFunction() {
> @Override
> public CustomType map(Integer integer) throws Exception
> {
> return new CustomType(integer);
> }
> });
> customTypes.print();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-3633) Job submission silently fails when using user code types
[
https://issues.apache.org/jira/browse/FLINK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-3633.
Resolution: Fixed
Fixed via 0331882af124a1750a54b4c6a9bd13819d65f72e
> Job submission silently fails when using user code types
>
>
> Key: FLINK-3633
> URL: https://issues.apache.org/jira/browse/FLINK-3633
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.1.0
>
>
> With the changes introduced by FLINK-3327, it is no longer possible to run
> remote Flink jobs which work on user code types. The reason is that now the
> {{ExecutionConfig}} is directly stored in the {{JobGraph}} which is sent as
> an Akka message to the {{JobManager}}. Per default, user code types are
> automatically detected and registered in the {{ExecutionConfig}}. When
> deserializing a {{JobGraph}} whose {{ExecutionConfig}} contains user code
> classes the user code class loader is consequently required. However, Akka
> does not have access to it and uses the system class loader. This causes that
> Akka silently discards the {{SubmitJob}} message which cannot be deserialized
> because of a {{ClassNotFoundException}}.
> I propose to not sent the {{ExecutionConfig}} explicitly with the
> {{JobGraph}} and, thus, to partially revert the changes to before FLINK-3327.
> Before, the {{ExectuionConfig}} was serialized into the job configuration and
> deserialized on the {{TaskManager}} using the proper user code class loader.
> In order to reproduce the problem you can submit the following job to a
> remote cluster.
> {code}
> public class Job {
> public static class CustomType {
> private final int value;
> public CustomType(int value) {
> this.value = value;
> }
> @Override
> public String toString() {
> return "CustomType(" + value + ")";
> }
> }
> public static void main(String[] args) throws Exception {
> ExecutionEnvironment env =
> ExecutionEnvironment.createRemoteEnvironment(Address, Port, PathToJar);
> DataSet input = env.fromElements(1,2,3,4,5);
> DataSet customTypes = input.map(new
> MapFunction() {
> @Override
> public CustomType map(Integer integer) throws Exception
> {
> return new CustomType(integer);
> }
> });
> customTypes.print();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3633] Fix user code de/serialization in...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1818 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3633) Job submission silently fails when using user code types
[
https://issues.apache.org/jira/browse/FLINK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221896#comment-15221896
]
ASF GitHub Bot commented on FLINK-3633:
---
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1818#issuecomment-204452690
Failing test case seems to be unrelated
> Job submission silently fails when using user code types
>
>
> Key: FLINK-3633
> URL: https://issues.apache.org/jira/browse/FLINK-3633
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.1.0
>
>
> With the changes introduced by FLINK-3327, it is no longer possible to run
> remote Flink jobs which work on user code types. The reason is that now the
> {{ExecutionConfig}} is directly stored in the {{JobGraph}} which is sent as
> an Akka message to the {{JobManager}}. Per default, user code types are
> automatically detected and registered in the {{ExecutionConfig}}. When
> deserializing a {{JobGraph}} whose {{ExecutionConfig}} contains user code
> classes the user code class loader is consequently required. However, Akka
> does not have access to it and uses the system class loader. This causes that
> Akka silently discards the {{SubmitJob}} message which cannot be deserialized
> because of a {{ClassNotFoundException}}.
> I propose to not sent the {{ExecutionConfig}} explicitly with the
> {{JobGraph}} and, thus, to partially revert the changes to before FLINK-3327.
> Before, the {{ExectuionConfig}} was serialized into the job configuration and
> deserialized on the {{TaskManager}} using the proper user code class loader.
> In order to reproduce the problem you can submit the following job to a
> remote cluster.
> {code}
> public class Job {
> public static class CustomType {
> private final int value;
> public CustomType(int value) {
> this.value = value;
> }
> @Override
> public String toString() {
> return "CustomType(" + value + ")";
> }
> }
> public static void main(String[] args) throws Exception {
> ExecutionEnvironment env =
> ExecutionEnvironment.createRemoteEnvironment(Address, Port, PathToJar);
> DataSet input = env.fromElements(1,2,3,4,5);
> DataSet customTypes = input.map(new
> MapFunction() {
> @Override
> public CustomType map(Integer integer) throws Exception
> {
> return new CustomType(integer);
> }
> });
> customTypes.print();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3633] Fix user code de/serialization in...
Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/1818#issuecomment-204452690 Failing test case seems to be unrelated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-3690) Create documentation on the new ResourceManager component
Henry Saputra created FLINK-3690: Summary: Create documentation on the new ResourceManager component Key: FLINK-3690 URL: https://issues.apache.org/jira/browse/FLINK-3690 Project: Flink Issue Type: Sub-task Components: Documentation, ResourceManager Affects Versions: 1.1.0 Reporter: Henry Saputra Assignee: Maximilian Michels Need proper documentation for the new ResourceManager and how it will impact deployment in different supported modes. Also, we have been very good adding new docs for our internal in the wiki [1] so would like that to happen for people evaluating Flink. [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3667) Generalize client<->cluster communication
[
https://issues.apache.org/jira/browse/FLINK-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221846#comment-15221846
]
Henry Saputra commented on FLINK-3667:
--
Will the new structure change the way the clients communicate with
ResourceManager ?
> Generalize client<->cluster communication
> -
>
> Key: FLINK-3667
> URL: https://issues.apache.org/jira/browse/FLINK-3667
> Project: Flink
> Issue Type: Improvement
> Components: YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>
> Here are some notes I took when inspecting the client<->cluster classes with
> regard to future integration of other resource management frameworks in
> addition to Yarn (e.g. Mesos).
> {noformat}
> 1 Cluster Client Abstraction
>
> 1.1 Status Quo
> ──
> 1.1.1 FlinkYarnClient
> ╌
> • Holds the cluster configuration (Flink-specific and Yarn-specific)
> • Contains the deploy() method to deploy the cluster
> • Creates the Hadoop Yarn client
> • Receives the initial job manager address
> • Bootstraps the FlinkYarnCluster
> 1.1.2 FlinkYarnCluster
> ╌╌
> • Wrapper around the Hadoop Yarn client
> • Queries cluster for status updates
> • Life time methods to start and shutdown the cluster
> • Flink specific features like shutdown after job completion
> 1.1.3 ApplicationClient
> ╌╌╌
> • Acts as a middle-man for asynchronous cluster communication
> • Designed to communicate with Yarn, not used in Standalone mode
> 1.1.4 CliFrontend
> ╌
> • Deeply integrated with FlinkYarnClient and FlinkYarnCluster
> • Constantly distinguishes between Yarn and Standalone mode
> • Would be nice to have a general abstraction in place
> 1.1.5 Client
>
> • Job submission and Job related actions, agnostic of resource framework
> 1.2 Proposal
>
> 1.2.1 ClusterConfig (before: AbstractFlinkYarnClient)
> ╌
> • Extensible cluster-agnostic config
> • May be extended by specific cluster, e.g. YarnClusterConfig
> 1.2.2 ClusterClient (before: AbstractFlinkYarnClient)
> ╌
> • Deals with cluster (RM) specific communication
> • Exposes framework agnostic information
> • YarnClusterClient, MesosClusterClient, StandaloneClusterClient
> 1.2.3 FlinkCluster (before: AbstractFlinkYarnCluster)
> ╌
> • Basic interface to communicate with a running cluster
> • Receives the ClusterClient for cluster-specific communication
> • Should not have to care about the specific implementations of the
> client
> 1.2.4 ApplicationClient
> ╌╌╌
> • Can be changed to work cluster-agnostic (first steps already in
> FLINK-3543)
> 1.2.5 CliFrontend
> ╌
> • CliFrontend does never have to differentiate between different
> cluster types after it has determined which cluster class to load.
> • Base class handles framework agnostic command line arguments
> • Pluggables for Yarn, Mesos handle specific commands
> {noformat}
> I would like to create/refactor the affected classes to set us up for a more
> flexible client side resource management abstraction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3678) Make Flink logs directory configurable
[
https://issues.apache.org/jira/browse/FLINK-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221705#comment-15221705
]
ASF GitHub Bot commented on FLINK-3678:
---
Github user stefanobaghino commented on the pull request:
https://github.com/apache/flink/pull/1837#issuecomment-204405759
Thanks! Hope it will make it to 1.0.1. :smile:
> Make Flink logs directory configurable
> --
>
> Key: FLINK-3678
> URL: https://issues.apache.org/jira/browse/FLINK-3678
> Project: Flink
> Issue Type: Improvement
> Components: Start-Stop Scripts
>Affects Versions: 1.0.0
>Reporter: Stefano Baghino
>Assignee: Stefano Baghino
>Priority: Minor
> Fix For: 1.1.0
>
>
> Currently Flink logs are stored under {{$FLINK_HOME/log}} and the user cannot
> configure an alternative storage location. It would be nice to add a
> configuration key in the {{flink-conf.yaml}} and edit the {{bin/flink}}
> launch script accordingly to get the value if present or default to the
> current behavior if no value is provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3678] Make Flink logs directory configu...
Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1837#issuecomment-204405759 Thanks! Hope it will make it to 1.0.1. :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3678] Make Flink logs directory configu...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1837 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3678) Make Flink logs directory configurable
[
https://issues.apache.org/jira/browse/FLINK-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221703#comment-15221703
]
ASF GitHub Bot commented on FLINK-3678:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1837
> Make Flink logs directory configurable
> --
>
> Key: FLINK-3678
> URL: https://issues.apache.org/jira/browse/FLINK-3678
> Project: Flink
> Issue Type: Improvement
> Components: Start-Stop Scripts
>Affects Versions: 1.0.0
>Reporter: Stefano Baghino
>Assignee: Stefano Baghino
>Priority: Minor
> Fix For: 1.1.0
>
>
> Currently Flink logs are stored under {{$FLINK_HOME/log}} and the user cannot
> configure an alternative storage location. It would be nice to add a
> configuration key in the {{flink-conf.yaml}} and edit the {{bin/flink}}
> launch script accordingly to get the value if present or default to the
> current behavior if no value is provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-3678) Make Flink logs directory configurable
[
https://issues.apache.org/jira/browse/FLINK-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ufuk Celebi closed FLINK-3678.
--
Resolution: Fixed
Fix Version/s: (was: 1.0.1)
1.1.0
Fixed in 82b5824.
> Make Flink logs directory configurable
> --
>
> Key: FLINK-3678
> URL: https://issues.apache.org/jira/browse/FLINK-3678
> Project: Flink
> Issue Type: Improvement
> Components: Start-Stop Scripts
>Affects Versions: 1.0.0
>Reporter: Stefano Baghino
>Assignee: Stefano Baghino
>Priority: Minor
> Fix For: 1.1.0
>
>
> Currently Flink logs are stored under {{$FLINK_HOME/log}} and the user cannot
> configure an alternative storage location. It would be nice to add a
> configuration key in the {{flink-conf.yaml}} and edit the {{bin/flink}}
> launch script accordingly to get the value if present or default to the
> current behavior if no value is provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3633] Fix user code de/serialization in...
Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/1818#issuecomment-204399612 I've added an additional test case for registered user code types in the `ExecutionConfig`. I will rebase this PR and if travis gives green light, then I'll merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3633) Job submission silently fails when using user code types
[
https://issues.apache.org/jira/browse/FLINK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221695#comment-15221695
]
ASF GitHub Bot commented on FLINK-3633:
---
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1818#issuecomment-204399612
I've added an additional test case for registered user code types in the
`ExecutionConfig`. I will rebase this PR and if travis gives green light, then
I'll merge it.
> Job submission silently fails when using user code types
>
>
> Key: FLINK-3633
> URL: https://issues.apache.org/jira/browse/FLINK-3633
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.1.0
>
>
> With the changes introduced by FLINK-3327, it is no longer possible to run
> remote Flink jobs which work on user code types. The reason is that now the
> {{ExecutionConfig}} is directly stored in the {{JobGraph}} which is sent as
> an Akka message to the {{JobManager}}. Per default, user code types are
> automatically detected and registered in the {{ExecutionConfig}}. When
> deserializing a {{JobGraph}} whose {{ExecutionConfig}} contains user code
> classes the user code class loader is consequently required. However, Akka
> does not have access to it and uses the system class loader. This causes that
> Akka silently discards the {{SubmitJob}} message which cannot be deserialized
> because of a {{ClassNotFoundException}}.
> I propose to not sent the {{ExecutionConfig}} explicitly with the
> {{JobGraph}} and, thus, to partially revert the changes to before FLINK-3327.
> Before, the {{ExectuionConfig}} was serialized into the job configuration and
> deserialized on the {{TaskManager}} using the proper user code class loader.
> In order to reproduce the problem you can submit the following job to a
> remote cluster.
> {code}
> public class Job {
> public static class CustomType {
> private final int value;
> public CustomType(int value) {
> this.value = value;
> }
> @Override
> public String toString() {
> return "CustomType(" + value + ")";
> }
> }
> public static void main(String[] args) throws Exception {
> ExecutionEnvironment env =
> ExecutionEnvironment.createRemoteEnvironment(Address, Port, PathToJar);
> DataSet input = env.fromElements(1,2,3,4,5);
> DataSet customTypes = input.map(new
> MapFunction() {
> @Override
> public CustomType map(Integer integer) throws Exception
> {
> return new CustomType(integer);
> }
> });
> customTypes.print();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3669) WindowOperator registers a lot of timers at StreamTask
[ https://issues.apache.org/jira/browse/FLINK-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221682#comment-15221682 ] Aljoscha Krettek commented on FLINK-3669: - Sure, no pressure! > WindowOperator registers a lot of timers at StreamTask > -- > > Key: FLINK-3669 > URL: https://issues.apache.org/jira/browse/FLINK-3669 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.1 >Reporter: Aljoscha Krettek >Priority: Blocker > > Right now, the WindowOperator registers a timer at the StreamTask for every > processing-time timer that a Trigger registers. We should combine several > registered trigger timers to only register one low-level timer (timer > coalescing). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3669) WindowOperator registers a lot of timers at StreamTask
[ https://issues.apache.org/jira/browse/FLINK-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221677#comment-15221677 ] Konstantin Knauf commented on FLINK-3669: - I would like to work on it, but not today. I will give it a try during the weekend, if that's alright. > WindowOperator registers a lot of timers at StreamTask > -- > > Key: FLINK-3669 > URL: https://issues.apache.org/jira/browse/FLINK-3669 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.1 >Reporter: Aljoscha Krettek >Priority: Blocker > > Right now, the WindowOperator registers a timer at the StreamTask for every > processing-time timer that a Trigger registers. We should combine several > registered trigger timers to only register one low-level timer (timer > coalescing). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-3633) Job submission silently fails when using user code types
[
https://issues.apache.org/jira/browse/FLINK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-3633:
-
Description:
With the changes introduced by FLINK-3327, it is no longer possible to run
remote Flink jobs which work on user code types. The reason is that now the
{{ExecutionConfig}} is directly stored in the {{JobGraph}} which is sent as an
Akka message to the {{JobManager}}. Per default, user code types are
automatically detected and registered in the {{ExecutionConfig}}. When
deserializing a {{JobGraph}} whose {{ExecutionConfig}} contains user code
classes the user code class loader is consequently required. However, Akka does
not have access to it and uses the system class loader. This causes that Akka
silently discards the {{SubmitJob}} message which cannot be deserialized
because of a {{ClassNotFoundException}}.
I propose to not sent the {{ExecutionConfig}} explicitly with the {{JobGraph}}
and, thus, to partially revert the changes to before FLINK-3327. Before, the
{{ExectuionConfig}} was serialized into the job configuration and deserialized
on the {{TaskManager}} using the proper user code class loader.
In order to reproduce the problem you can submit the following job to a remote
cluster.
{code}
public class Job {
public static class CustomType {
private final int value;
public CustomType(int value) {
this.value = value;
}
@Override
public String toString() {
return "CustomType(" + value + ")";
}
}
public static void main(String[] args) throws Exception {
ExecutionEnvironment env =
ExecutionEnvironment.createRemoteEnvironment(Address, Port, PathToJar);
DataSet input = env.fromElements(1,2,3,4,5);
DataSet customTypes = input.map(new
MapFunction() {
@Override
public CustomType map(Integer integer) throws Exception
{
return new CustomType(integer);
}
});
customTypes.print();
}
}
{code}
was:
With the changes introduced by FLINK-3327, it is no longer possible to run
remote Flink jobs which work on user code types. The reason is that now the
{{ExecutionConfig}} is directly stored in the {{JobGraph}} which is sent as an
Akka message to the {{JobManager}}. Per default, user code types are
automatically detected and registered in the {{ExecutionConfig}}. When
deserializing a {{JobGraph}} whose {{ExecutionConfig}} contains user code
classes the user code class loader is consequently required. However, Akka does
not have access to it and uses the system class loader. This causes that Akka
silently discards the {{SubmitJob}} message which cannot be deserialized
because of a {{ClassNotFoundException}}.
I propose to not sent the {{ExecutionConfig}} explicitly with the {{JobGraph}}
and, thus, to partially revert the changes to before FLINK-3327. Before, the
{{ExectuionConfig}} was serialized into the job configuration and deserialized
on the {{TaskManager}} using the proper user code class loader.
In order to reproduce the problem you can submit the following job to a remote
cluster.
{code}
public class Job {
public static class CustomType {
private final int value;
public CustomType(int value) {
this.value = value;
}
@Override
public String toString() {
return "CustomType(" + value + ")";
}
}
public static void main(String[] args) throws Exception {
ExecutionEnvironment env =
ExecutionEnvironment.createRemoteEnvironment(Address, Port, PathToJar);
env.getConfig().disableAutoTypeRegistration();
DataSet input = env.fromElements(1,2,3,4,5);
DataSet customTypes = input.map(new
MapFunction() {
@Override
public CustomType map(Integer integer) throws Exception
{
return new CustomType(integer);
}
});
customTypes.print();
}
}
{code}
> Job submission silently fails when using user code types
>
>
> Key: FLINK-3633
> URL: https://issues.apache.org/jira/browse/FLINK-3633
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.1.0
>
>
> With the changes introduced by FLINK-3327, it is no longer possible to r
[GitHub] flink pull request: [FLINK-3689] fix shutdown of JM when RM is not...
GitHub user mxm opened a pull request: https://github.com/apache/flink/pull/1847 [FLINK-3689] fix shutdown of JM when RM is not available Changes also include a test case which tests this explicitly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mxm/flink FLINK-3689 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1847.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1847 commit 355240b67cadc6c984837f930cb05948fee92e3f Author: Maximilian Michels Date: 2016-04-01T12:21:50Z [FLINK-3689] fix shutdown of JM when RM is not available --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3689) JobManager blocks cluster shutdown when not connected to ResourceManager
[ https://issues.apache.org/jira/browse/FLINK-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221630#comment-15221630 ] ASF GitHub Bot commented on FLINK-3689: --- GitHub user mxm opened a pull request: https://github.com/apache/flink/pull/1847 [FLINK-3689] fix shutdown of JM when RM is not available Changes also include a test case which tests this explicitly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mxm/flink FLINK-3689 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1847.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1847 commit 355240b67cadc6c984837f930cb05948fee92e3f Author: Maximilian Michels Date: 2016-04-01T12:21:50Z [FLINK-3689] fix shutdown of JM when RM is not available > JobManager blocks cluster shutdown when not connected to ResourceManager > > > Key: FLINK-3689 > URL: https://issues.apache.org/jira/browse/FLINK-3689 > Project: Flink > Issue Type: Bug > Components: JobManager, ResourceManager >Affects Versions: 1.1.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Minor > Fix For: 1.1.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3689) JobManager blocks cluster shutdown when not connected to ResourceManager
Maximilian Michels created FLINK-3689: - Summary: JobManager blocks cluster shutdown when not connected to ResourceManager Key: FLINK-3689 URL: https://issues.apache.org/jira/browse/FLINK-3689 Project: Flink Issue Type: Bug Components: JobManager, ResourceManager Affects Versions: 1.1.0 Reporter: Maximilian Michels Assignee: Maximilian Michels Priority: Minor Fix For: 1.1.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLINK-3669) WindowOperator registers a lot of timers at StreamTask
[ https://issues.apache.org/jira/browse/FLINK-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221600#comment-15221600 ] Aljoscha Krettek edited comment on FLINK-3669 at 4/1/16 12:05 PM: -- Yes, the 3rd option would require additional documentation and care on the part of users, so I would probably not go for it. I'll try and come up with a quick prototype for 1. to see how it behaves. Unless you want to work on it. I also found another problem. The queue that is used for the triggers turns out to be a bottleneck because removing items from it is O\(n\), so with very large key ranges this becomes very slow. was (Author: aljoscha): Yes, the 3rd option would require additional documentation and care on the part of users, so I would probably not go for it. I'll try and come up with a quick prototype for 1. to see how it behaves. Unless you want to work on it. I also found another problem. The queue that is used for the triggers turns out to be a bottleneck because removing items from it is O(n), so with very large key ranges this becomes very slow. > WindowOperator registers a lot of timers at StreamTask > -- > > Key: FLINK-3669 > URL: https://issues.apache.org/jira/browse/FLINK-3669 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.1 >Reporter: Aljoscha Krettek >Priority: Blocker > > Right now, the WindowOperator registers a timer at the StreamTask for every > processing-time timer that a Trigger registers. We should combine several > registered trigger timers to only register one low-level timer (timer > coalescing). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3669) WindowOperator registers a lot of timers at StreamTask
[ https://issues.apache.org/jira/browse/FLINK-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221600#comment-15221600 ] Aljoscha Krettek commented on FLINK-3669: - Yes, the 3rd option would require additional documentation and care on the part of users, so I would probably not go for it. I'll try and come up with a quick prototype for 1. to see how it behaves. Unless you want to work on it. I also found another problem. The queue that is used for the triggers turns out to be a bottleneck because removing items from it is O(n), so with very large key ranges this becomes very slow. > WindowOperator registers a lot of timers at StreamTask > -- > > Key: FLINK-3669 > URL: https://issues.apache.org/jira/browse/FLINK-3669 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.1 >Reporter: Aljoscha Krettek >Priority: Blocker > > Right now, the WindowOperator registers a timer at the StreamTask for every > processing-time timer that a Trigger registers. We should combine several > registered trigger timers to only register one low-level timer (timer > coalescing). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3633) Job submission silently fails when using user code types
[
https://issues.apache.org/jira/browse/FLINK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221598#comment-15221598
]
ASF GitHub Bot commented on FLINK-3633:
---
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1818#issuecomment-204368378
That is a good idea @uce :-) I'll add a corresponding test for the
`ExecutionConfig`.
> Job submission silently fails when using user code types
>
>
> Key: FLINK-3633
> URL: https://issues.apache.org/jira/browse/FLINK-3633
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.1.0
>
>
> With the changes introduced by FLINK-3327, it is no longer possible to run
> remote Flink jobs which work on user code types. The reason is that now the
> {{ExecutionConfig}} is directly stored in the {{JobGraph}} which is sent as
> an Akka message to the {{JobManager}}. Per default, user code types are
> automatically detected and registered in the {{ExecutionConfig}}. When
> deserializing a {{JobGraph}} whose {{ExecutionConfig}} contains user code
> classes the user code class loader is consequently required. However, Akka
> does not have access to it and uses the system class loader. This causes that
> Akka silently discards the {{SubmitJob}} message which cannot be deserialized
> because of a {{ClassNotFoundException}}.
> I propose to not sent the {{ExecutionConfig}} explicitly with the
> {{JobGraph}} and, thus, to partially revert the changes to before FLINK-3327.
> Before, the {{ExectuionConfig}} was serialized into the job configuration and
> deserialized on the {{TaskManager}} using the proper user code class loader.
> In order to reproduce the problem you can submit the following job to a
> remote cluster.
> {code}
> public class Job {
> public static class CustomType {
> private final int value;
> public CustomType(int value) {
> this.value = value;
> }
> @Override
> public String toString() {
> return "CustomType(" + value + ")";
> }
> }
> public static void main(String[] args) throws Exception {
> ExecutionEnvironment env =
> ExecutionEnvironment.createRemoteEnvironment(Address, Port, PathToJar);
> env.getConfig().disableAutoTypeRegistration();
> DataSet input = env.fromElements(1,2,3,4,5);
> DataSet customTypes = input.map(new
> MapFunction() {
> @Override
> public CustomType map(Integer integer) throws Exception
> {
> return new CustomType(integer);
> }
> });
> customTypes.print();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3633] Fix user code de/serialization in...
Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/1818#issuecomment-204368378 That is a good idea @uce :-) I'll add a corresponding test for the `ExecutionConfig`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-3688) ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is called and TimeCharacteristic = ProcessingTime
[
https://issues.apache.org/jira/browse/FLINK-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-3688:
Assignee: Konstantin Knauf
> ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is
> called and TimeCharacteristic = ProcessingTime
>
>
> Key: FLINK-3688
> URL: https://issues.apache.org/jira/browse/FLINK-3688
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Critical
>
> Hi,
> when using {{TimeCharacteristics.ProcessingTime}} a ClassCastException is
> thrown in {{StreamRecordSerializer}} when
> {{WindowOperator.processWatermark()}} is called from
> {{WindowOperator.trigger()}}, i.e. whenever a ProcessingTimeTimer is
> triggered.
> The problem seems to be that {{processWatermark()}} is also called in
> {{trigger()}}, when time characteristic is ProcessingTime, but in
> {{RecordWriterOutput}} {{enableWatermarkMultiplexing}} is {{false}} and the
> {{TypeSerializer}} is a {{StreamRecordSerializer}}, which ultimately leads to
> the ClassCastException. Do you agree?
> If this is indeed a bug, there several possible solutions.
> # Only calling {{processWatermark()}} in {{trigger()}}, when
> TimeCharacteristic is EventTime
> # Not calling {{processWatermark()}} in {{trigger()}} at all, instead wait
> for the next watermark to trigger the EventTimeTimers with a timestamp behind
> the current watermark. This is, of course, a trade off.
> # Using {{MultiplexingStreamRecordSerializer}} all the time, but I have no
> idea what the side effect of this change would be. I assume there is a reason
> for existence of the StreamRecordSerializer ;)
> StackTrace:
> {quote}
> TimerException\{java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord\}
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:716)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:93)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.emitWatermark(OperatorChain.java:370)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processWatermark(WindowOperator.java:293)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.trigger(WindowOperator.java:323)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:710)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamRecordSerializer.serialize(StreamRecordSerializer.java:42)
> at
> org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:56)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:79)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.broadcastEmit(RecordWriter.java:109)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.broadcastEmit(StreamRecordWriter.java:95)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:90)
> ... 11 more
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3688) ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is called and TimeCharacteristic = ProcessingTime
[
https://issues.apache.org/jira/browse/FLINK-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221597#comment-15221597
]
Aljoscha Krettek commented on FLINK-3688:
-
Exactly, that's the reason. If we did that then we would invoke the trigger
another time before the call to {{registerEventTimeTimer()}} returns.
It's good to hear that you'll work on a fix, I think the solution you mentioned
(4. plus dropping the watermarks) is the best. :-) I'll assign the issue to
you.
> ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is
> called and TimeCharacteristic = ProcessingTime
>
>
> Key: FLINK-3688
> URL: https://issues.apache.org/jira/browse/FLINK-3688
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Konstantin Knauf
>Priority: Critical
>
> Hi,
> when using {{TimeCharacteristics.ProcessingTime}} a ClassCastException is
> thrown in {{StreamRecordSerializer}} when
> {{WindowOperator.processWatermark()}} is called from
> {{WindowOperator.trigger()}}, i.e. whenever a ProcessingTimeTimer is
> triggered.
> The problem seems to be that {{processWatermark()}} is also called in
> {{trigger()}}, when time characteristic is ProcessingTime, but in
> {{RecordWriterOutput}} {{enableWatermarkMultiplexing}} is {{false}} and the
> {{TypeSerializer}} is a {{StreamRecordSerializer}}, which ultimately leads to
> the ClassCastException. Do you agree?
> If this is indeed a bug, there several possible solutions.
> # Only calling {{processWatermark()}} in {{trigger()}}, when
> TimeCharacteristic is EventTime
> # Not calling {{processWatermark()}} in {{trigger()}} at all, instead wait
> for the next watermark to trigger the EventTimeTimers with a timestamp behind
> the current watermark. This is, of course, a trade off.
> # Using {{MultiplexingStreamRecordSerializer}} all the time, but I have no
> idea what the side effect of this change would be. I assume there is a reason
> for existence of the StreamRecordSerializer ;)
> StackTrace:
> {quote}
> TimerException\{java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord\}
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:716)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:93)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.emitWatermark(OperatorChain.java:370)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processWatermark(WindowOperator.java:293)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.trigger(WindowOperator.java:323)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:710)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamRecordSerializer.serialize(StreamRecordSerializer.java:42)
> at
> org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:56)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:79)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.broadcastEmit(RecordWriter.java:109)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.broadcastEmit(StreamRecordWriter.java:95)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:90)
> ... 11 more
> {quote}
--
This message was sent by Atlass
[jira] [Commented] (FLINK-3665) Range partitioning lacks support to define sort orders
[
https://issues.apache.org/jira/browse/FLINK-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221572#comment-15221572
]
Dawid Wysakowicz commented on FLINK-3665:
-
Hi as I am working on [FLINK-2946] that needs those changes I would happily
work on this either if the solution proposed by [~fhueske] is ok.
> Range partitioning lacks support to define sort orders
> --
>
> Key: FLINK-3665
> URL: https://issues.apache.org/jira/browse/FLINK-3665
> Project: Flink
> Issue Type: Improvement
> Components: DataSet API
>Affects Versions: 1.0.0
>Reporter: Fabian Hueske
> Fix For: 1.1.0
>
>
> {{DataSet.partitionByRange()}} does not allow to specify the sort order of
> fields. This is fine if range partitioning is used to reduce skewed
> partitioning.
> However, it is not sufficient if range partitioning is used to sort a data
> set in parallel.
> Since {{DataSet.partitionByRange()}} is {{@Public}} API and cannot be easily
> changed, I propose to add a method {{withOrders(Order... orders)}} to
> {{PartitionOperator}}. The method should throw an exception if the
> partitioning method of {{PartitionOperator}} is not range partitioning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2946) Add orderBy() to Table API
[
https://issues.apache.org/jira/browse/FLINK-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221567#comment-15221567
]
Dawid Wysakowicz commented on FLINK-2946:
-
I still have some problems with range partitioning and parallelism.
* First of all the {{org.apache.flink.api.java.DataSet}} that I get from
{{translateToPlan}} does not have the method getParallelism. But that's a minor
issue.
* I am not sure how to extract the eventual parallelism of the input and if I
need to do this. Let's take this as example:
{code}
val env = ExecutionEnvironment.getExecutionEnvironment
env.setParallelism(1)
val t = env.fromElements((1, 3, "Third"), (1, 2, "Fourth"), (1, 4,
"Second"),
(2, 1, "Sixth"), (1, 5, "First"), (1, 1, "Fifth")).setParallelism(4)
.toTable.orderBy('_1.asc, '_2.desc)
{code}
The dataset then looks like(the numbers in brackets is parallelism of
operator): DataSource(4) -> MapOperator(-1) -> here I must apply either
SortOperator or PartitionOperator -> SortOperator.
On what parallelism shall I decide if the PartitionOperator should be applied?
What should be the parallelism of PartitionOperator?(By default it is the one
from ExecutionEnvironment)
Hope I stated my problems clearly.
> Add orderBy() to Table API
> --
>
> Key: FLINK-2946
> URL: https://issues.apache.org/jira/browse/FLINK-2946
> Project: Flink
> Issue Type: New Feature
> Components: Table API
>Reporter: Timo Walther
>Assignee: Dawid Wysakowicz
>
> In order to implement a FLINK-2099 prototype that uses the Table APIs code
> generation facilities, the Table API needs a sorting feature.
> I would implement it the next days. Ideas how to implement such a sorting
> feature are very welcome. Is there any more efficient way instead of
> {{.sortPartition(...).setParallism(1)}}? Is it better to sort locally on the
> nodes first and finally sort on one node afterwards?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3579) Improve String concatenation
[
https://issues.apache.org/jira/browse/FLINK-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221548#comment-15221548
]
ASF GitHub Bot commented on FLINK-3579:
---
Github user ramkrish86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1821#discussion_r58191031
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/expressions/ExpressionParser.scala
---
@@ -59,6 +59,8 @@ object ExpressionParser extends JavaTokenParsers with
PackratParsers {
Literal(str.toInt)
} else if (str.endsWith("f") | str.endsWith("F")) {
Literal(str.toFloat)
+} else if (str.endsWith(".")) {
--- End diff --
@tillrohrmann
This '.' related change I think I need more time to understand how this
parser works. So for this PR can i only handle the string concat part. I have
ensured that 42+a+b or a + b + 42 works fine.
> Improve String concatenation
>
>
> Key: FLINK-3579
> URL: https://issues.apache.org/jira/browse/FLINK-3579
> Project: Flink
> Issue Type: Bug
> Components: Table API
>Reporter: Timo Walther
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Concatenation of a String and non-String does not work properly.
> e.g. {{f0 + 42}} leads to RelBuilder Exception
> ExpressionParser does not like {{f0 + 42.cast(STRING)}} either.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: FLINK-3579 Improve String concatenation (Ram)
Github user ramkrish86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1821#discussion_r58191031
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/expressions/ExpressionParser.scala
---
@@ -59,6 +59,8 @@ object ExpressionParser extends JavaTokenParsers with
PackratParsers {
Literal(str.toInt)
} else if (str.endsWith("f") | str.endsWith("F")) {
Literal(str.toFloat)
+} else if (str.endsWith(".")) {
--- End diff --
@tillrohrmann
This '.' related change I think I need more time to understand how this
parser works. So for this PR can i only handle the string concat part. I have
ensured that 42+a+b or a + b + 42 works fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-3688) ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is called and TimeCharacteristic = ProcessingTime
[
https://issues.apache.org/jira/browse/FLINK-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221514#comment-15221514
]
Konstantin Knauf commented on FLINK-3688:
-
>From what I know, watermarks should just not be emitted if ProcessingTime is
>used,right? If so, 5. seems to be work-around. So I think the 1th, 2th, 4th
>option are the best, of which I would pefer the 4th, because it is the least
>intrusive.
We could still drop watermarks in {{StreamRecordSerializer}} as kind of a
saftey net for similar problems.
Beside: What is the reason, that {{processWatermark()}} is not directly called
from {{registerEventTimeTimer}} if the timestamp is behind the watermark? I
suppose, it's because of
{quote}
This method is guaranteed to not be called concurrently with other methods of
the operator.
{quote}
Not redirecting via the TriggerTask would be cleanest way, I think.
I would like to implement some fix today, as this is naturally a blocker for
us. Depending on the discussion I will make a pull request during the weekend,
or so.
> ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is
> called and TimeCharacteristic = ProcessingTime
>
>
> Key: FLINK-3688
> URL: https://issues.apache.org/jira/browse/FLINK-3688
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Konstantin Knauf
>Priority: Critical
>
> Hi,
> when using {{TimeCharacteristics.ProcessingTime}} a ClassCastException is
> thrown in {{StreamRecordSerializer}} when
> {{WindowOperator.processWatermark()}} is called from
> {{WindowOperator.trigger()}}, i.e. whenever a ProcessingTimeTimer is
> triggered.
> The problem seems to be that {{processWatermark()}} is also called in
> {{trigger()}}, when time characteristic is ProcessingTime, but in
> {{RecordWriterOutput}} {{enableWatermarkMultiplexing}} is {{false}} and the
> {{TypeSerializer}} is a {{StreamRecordSerializer}}, which ultimately leads to
> the ClassCastException. Do you agree?
> If this is indeed a bug, there several possible solutions.
> # Only calling {{processWatermark()}} in {{trigger()}}, when
> TimeCharacteristic is EventTime
> # Not calling {{processWatermark()}} in {{trigger()}} at all, instead wait
> for the next watermark to trigger the EventTimeTimers with a timestamp behind
> the current watermark. This is, of course, a trade off.
> # Using {{MultiplexingStreamRecordSerializer}} all the time, but I have no
> idea what the side effect of this change would be. I assume there is a reason
> for existence of the StreamRecordSerializer ;)
> StackTrace:
> {quote}
> TimerException\{java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord\}
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:716)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:93)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.emitWatermark(OperatorChain.java:370)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processWatermark(WindowOperator.java:293)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.trigger(WindowOperator.java:323)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:710)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamRecordSerializer.serialize(StreamRecordSerializer.java:42)
> at
> org.apache.flink.runtime.plugable.SerializationDelegate.writ
[jira] [Commented] (FLINK-3688) ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is called and TimeCharacteristic = ProcessingTime
[
https://issues.apache.org/jira/browse/FLINK-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221496#comment-15221496
]
Aljoscha Krettek commented on FLINK-3688:
-
And, as [~StephanEwen] suggested, we could also simply drop watermarks in
{{StreamRecordSerializer}} instead of throwing an exception.
> ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is
> called and TimeCharacteristic = ProcessingTime
>
>
> Key: FLINK-3688
> URL: https://issues.apache.org/jira/browse/FLINK-3688
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Konstantin Knauf
>Priority: Critical
>
> Hi,
> when using {{TimeCharacteristics.ProcessingTime}} a ClassCastException is
> thrown in {{StreamRecordSerializer}} when
> {{WindowOperator.processWatermark()}} is called from
> {{WindowOperator.trigger()}}, i.e. whenever a ProcessingTimeTimer is
> triggered.
> The problem seems to be that {{processWatermark()}} is also called in
> {{trigger()}}, when time characteristic is ProcessingTime, but in
> {{RecordWriterOutput}} {{enableWatermarkMultiplexing}} is {{false}} and the
> {{TypeSerializer}} is a {{StreamRecordSerializer}}, which ultimately leads to
> the ClassCastException. Do you agree?
> If this is indeed a bug, there several possible solutions.
> # Only calling {{processWatermark()}} in {{trigger()}}, when
> TimeCharacteristic is EventTime
> # Not calling {{processWatermark()}} in {{trigger()}} at all, instead wait
> for the next watermark to trigger the EventTimeTimers with a timestamp behind
> the current watermark. This is, of course, a trade off.
> # Using {{MultiplexingStreamRecordSerializer}} all the time, but I have no
> idea what the side effect of this change would be. I assume there is a reason
> for existence of the StreamRecordSerializer ;)
> StackTrace:
> {quote}
> TimerException\{java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord\}
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:716)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:93)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.emitWatermark(OperatorChain.java:370)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processWatermark(WindowOperator.java:293)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.trigger(WindowOperator.java:323)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:710)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamRecordSerializer.serialize(StreamRecordSerializer.java:42)
> at
> org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:56)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:79)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.broadcastEmit(RecordWriter.java:109)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.broadcastEmit(StreamRecordWriter.java:95)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:90)
> ... 11 more
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3688) ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is called and TimeCharacteristic = ProcessingTime
[
https://issues.apache.org/jira/browse/FLINK-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221486#comment-15221486
]
Aljoscha Krettek commented on FLINK-3688:
-
Yes, this is indeed a bug. I have a fourth option:
Move the logic for checking the watermark timers out of {{processWatermark()}},
call this new method from {{trigger()}}, this way we don't emit a watermark.
But I also see that having the normal {{StreamRecordSerialzer}} and the
{{MultiplexingStreamRecordSerializer}} often leads to subtle problems.
Initially it was introduced out of performance concerns by some folks.
> ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is
> called and TimeCharacteristic = ProcessingTime
>
>
> Key: FLINK-3688
> URL: https://issues.apache.org/jira/browse/FLINK-3688
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Konstantin Knauf
>Priority: Critical
>
> Hi,
> when using {{TimeCharacteristics.ProcessingTime}} a ClassCastException is
> thrown in {{StreamRecordSerializer}} when
> {{WindowOperator.processWatermark()}} is called from
> {{WindowOperator.trigger()}}, i.e. whenever a ProcessingTimeTimer is
> triggered.
> The problem seems to be that {{processWatermark()}} is also called in
> {{trigger()}}, when time characteristic is ProcessingTime, but in
> {{RecordWriterOutput}} {{enableWatermarkMultiplexing}} is {{false}} and the
> {{TypeSerializer}} is a {{StreamRecordSerializer}}, which ultimately leads to
> the ClassCastException. Do you agree?
> If this is indeed a bug, there several possible solutions.
> # Only calling {{processWatermark()}} in {{trigger()}}, when
> TimeCharacteristic is EventTime
> # Not calling {{processWatermark()}} in {{trigger()}} at all, instead wait
> for the next watermark to trigger the EventTimeTimers with a timestamp behind
> the current watermark. This is, of course, a trade off.
> # Using {{MultiplexingStreamRecordSerializer}} all the time, but I have no
> idea what the side effect of this change would be. I assume there is a reason
> for existence of the StreamRecordSerializer ;)
> StackTrace:
> {quote}
> TimerException\{java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord\}
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:716)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:93)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.emitWatermark(OperatorChain.java:370)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processWatermark(WindowOperator.java:293)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.trigger(WindowOperator.java:323)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:710)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamRecordSerializer.serialize(StreamRecordSerializer.java:42)
> at
> org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:56)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:79)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.broadcastEmit(RecordWriter.java:109)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.broadcastEmit(StreamRecordWriter.java:95)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOu
[jira] [Commented] (FLINK-1159) Case style anonymous functions not supported by Scala API
[
https://issues.apache.org/jira/browse/FLINK-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221478#comment-15221478
]
ASF GitHub Bot commented on FLINK-1159:
---
Github user stefanobaghino commented on the pull request:
https://github.com/apache/flink/pull/1704#issuecomment-204334976
Note on the tests of the streaming extensions: I couldn't find a more
specific class than `SingleOutputStreamOperator[_]`, thus assertion may not be
very meaningful. However, I'd leave them in place to make sure further work on
the extension won't break them at compile time.
> Case style anonymous functions not supported by Scala API
> -
>
> Key: FLINK-1159
> URL: https://issues.apache.org/jira/browse/FLINK-1159
> Project: Flink
> Issue Type: Bug
> Components: Scala API
>Reporter: Till Rohrmann
>Assignee: Stefano Baghino
>
> In Scala it is very common to define anonymous functions of the following form
> {code}
> {
> case foo: Bar => foobar(foo)
> case _ => throw new RuntimeException()
> }
> {code}
> These case style anonymous functions are not supported yet by the Scala API.
> Thus, one has to write redundant code to name the function parameter.
> What works is the following pattern, but it is not intuitive for someone
> coming from Scala:
> {code}
> dataset.map{
> _ match{
> case foo:Bar => ...
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1159] Case style anonymous functions no...
Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1704#issuecomment-204334976 Note on the tests of the streaming extensions: I couldn't find a more specific class than `SingleOutputStreamOperator[_]`, thus assertion may not be very meaningful. However, I'd leave them in place to make sure further work on the extension won't break them at compile time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2392) Instable test in flink-yarn-tests
[
https://issues.apache.org/jira/browse/FLINK-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221467#comment-15221467
]
Maximilian Michels commented on FLINK-2392:
---
Ok great, then it's probably not due to the recent changes ;)
> Instable test in flink-yarn-tests
> -
>
> Key: FLINK-2392
> URL: https://issues.apache.org/jira/browse/FLINK-2392
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 0.10.0
>Reporter: Matthias J. Sax
>Assignee: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.0.0
>
>
> The test YARNSessionFIFOITCase fails from time to time on an irregular basis.
> For example see: https://travis-ci.org/apache/flink/jobs/72019690
> {noformat}
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 205.163 sec
> <<< FAILURE! - in org.apache.flink.yarn.YARNSessionFIFOITCase
> perJobYarnClusterWithParallelism(org.apache.flink.yarn.YARNSessionFIFOITCase)
> Time elapsed: 60.651 sec <<< FAILURE!
> java.lang.AssertionError: During the timeout period of 60 seconds the
> expected string did not show up
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.apache.flink.yarn.YarnTestBase.runWithArgs(YarnTestBase.java:478)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.perJobYarnClusterWithParallelism(YARNSessionFIFOITCase.java:435)
> Results :
> Failed tests:
>
> YARNSessionFIFOITCase.perJobYarnClusterWithParallelism:435->YarnTestBase.runWithArgs:478
> During the timeout period of 60 seconds the expected string did not show up
> {noformat}
> Another error case is this (see
> https://travis-ci.org/mjsax/flink/jobs/77313444)
> {noformat}
> Tests run: 12, Failures: 3, Errors: 0, Skipped: 2, Time elapsed: 182.008 sec
> <<< FAILURE! - in org.apache.flink.yarn.YARNSessionFIFOITCase
> testTaskManagerFailure(org.apache.flink.yarn.YARNSessionFIFOITCase) Time
> elapsed: 27.356 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> testNonexistingQueue(org.apache.flink.yarn.YARNSessionFIFOITCase) Time
> elapsed: 17.421 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> testJavaAPI(org.apache.flink.yarn.YARNSessionFIFOITCase) Time elapsed:
> 11.984 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> {noformat}
> Furthermore, this build failed too:
> https://travis-ci.org/apache/flink/jobs/77313450
> (no error, but Travis terminated to due no progress for 300 seconds ->
> deadlock?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2392) Instable test in flink-yarn-tests
[
https://issues.apache.org/jira/browse/FLINK-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221460#comment-15221460
]
Matthias J. Sax commented on FLINK-2392:
Yes. See post above from03/Dec/15
> Instable test in flink-yarn-tests
> -
>
> Key: FLINK-2392
> URL: https://issues.apache.org/jira/browse/FLINK-2392
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 0.10.0
>Reporter: Matthias J. Sax
>Assignee: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.0.0
>
>
> The test YARNSessionFIFOITCase fails from time to time on an irregular basis.
> For example see: https://travis-ci.org/apache/flink/jobs/72019690
> {noformat}
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 205.163 sec
> <<< FAILURE! - in org.apache.flink.yarn.YARNSessionFIFOITCase
> perJobYarnClusterWithParallelism(org.apache.flink.yarn.YARNSessionFIFOITCase)
> Time elapsed: 60.651 sec <<< FAILURE!
> java.lang.AssertionError: During the timeout period of 60 seconds the
> expected string did not show up
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.apache.flink.yarn.YarnTestBase.runWithArgs(YarnTestBase.java:478)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.perJobYarnClusterWithParallelism(YARNSessionFIFOITCase.java:435)
> Results :
> Failed tests:
>
> YARNSessionFIFOITCase.perJobYarnClusterWithParallelism:435->YarnTestBase.runWithArgs:478
> During the timeout period of 60 seconds the expected string did not show up
> {noformat}
> Another error case is this (see
> https://travis-ci.org/mjsax/flink/jobs/77313444)
> {noformat}
> Tests run: 12, Failures: 3, Errors: 0, Skipped: 2, Time elapsed: 182.008 sec
> <<< FAILURE! - in org.apache.flink.yarn.YARNSessionFIFOITCase
> testTaskManagerFailure(org.apache.flink.yarn.YARNSessionFIFOITCase) Time
> elapsed: 27.356 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> testNonexistingQueue(org.apache.flink.yarn.YARNSessionFIFOITCase) Time
> elapsed: 17.421 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> testJavaAPI(org.apache.flink.yarn.YARNSessionFIFOITCase) Time elapsed:
> 11.984 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> {noformat}
> Furthermore, this build failed too:
> https://travis-ci.org/apache/flink/jobs/77313450
> (no error, but Travis terminated to due no progress for 300 seconds ->
> deadlock?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3688) ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is called and TimeCharacteristic = ProcessingTime
Konstantin Knauf created FLINK-3688:
---
Summary: ClassCastException in StreamRecordSerializer when
WindowOperator.trigger() is called and TimeCharacteristic = ProcessingTime
Key: FLINK-3688
URL: https://issues.apache.org/jira/browse/FLINK-3688
Project: Flink
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Konstantin Knauf
Priority: Critical
Hi,
when using {{TimeCharacteristics.ProcessingTime}} a ClassCastException is
thrown in {{StreamRecordSerializer}} when {{WindowOperator.processWatermark()}}
is called from {{WindowOperator.trigger()}}, i.e. whenever a
ProcessingTimeTimer is triggered.
The problem seems to be that {{processWatermark()}} is also called in
{{trigger()}}, when time characteristic is ProcessingTime, but in
{{RecordWriterOutput}} {{enableWatermarkMultiplexing}} is {{false}} and the
{{TypeSerializer}} is a {{StreamRecordSerializer}}, which ultimately leads to
the ClassCastException. Do you agree?
If this is indeed a bug, there several possible solutions.
# Only calling {{processWatermark()}} in {{trigger()}}, when TimeCharacteristic
is EventTime
# Not calling {{processWatermark()}} in {{trigger()}} at all, instead wait for
the next watermark to trigger the EventTimeTimers with a timestamp behind the
current watermark. This is, of course, a trade off.
# Using {{MultiplexingStreamRecordSerializer}} all the time, but I have no idea
what the side effect of this change would be. I assume there is a reason for
existence of the StreamRecordSerializer ;)
StackTrace:
{quote}
TimerException\{java.lang.RuntimeException:
org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
org.apache.flink.streaming.runtime.streamrecord.StreamRecord\}
at
org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:716)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException:
org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
org.apache.flink.streaming.runtime.streamrecord.StreamRecord
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:93)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.emitWatermark(OperatorChain.java:370)
at
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processWatermark(WindowOperator.java:293)
at
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.trigger(WindowOperator.java:323)
at
org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:710)
... 7 more
Caused by: java.lang.ClassCastException:
org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
org.apache.flink.streaming.runtime.streamrecord.StreamRecord
at
org.apache.flink.streaming.runtime.streamrecord.StreamRecordSerializer.serialize(StreamRecordSerializer.java:42)
at
org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:56)
at
org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:79)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.broadcastEmit(RecordWriter.java:109)
at
org.apache.flink.streaming.runtime.io.StreamRecordWriter.broadcastEmit(StreamRecordWriter.java:95)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:90)
... 11 more
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-3687) org.apache.flink.runtime.net.ConnectionUtilsTest fails
[
https://issues.apache.org/jira/browse/FLINK-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolaas Steenbergen updated FLINK-3687:
Summary: org.apache.flink.runtime.net.ConnectionUtilsTest fails (was:
runtime.leaderelection.ZooKeeperLeaderElectionTest fails)
> org.apache.flink.runtime.net.ConnectionUtilsTest fails
> --
>
> Key: FLINK-3687
> URL: https://issues.apache.org/jira/browse/FLINK-3687
> Project: Flink
> Issue Type: Bug
>Reporter: Nikolaas Steenbergen
>
> {code}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.51 sec <<<
> FAILURE! - in org.apache.flink.runtime.net.ConnectionUtilsTest
> testReturnLocalHostAddressUsingHeuristics(org.apache.flink.runtime.net.ConnectionUtilsTest)
> Time elapsed: 0.504 sec <<< FAILURE!
> java.lang.AssertionError:
> expected:
> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.flink.runtime.net.ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics(ConnectionUtilsTest.java:59)
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.554 sec -
> in org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionTest
> Results :
> Failed tests:
> ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics:59
> expected:
> but was:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3687) org.apache.flink.runtime.net.ConnectionUtilsTest fails
[
https://issues.apache.org/jira/browse/FLINK-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221412#comment-15221412
]
Nikolaas Steenbergen commented on FLINK-3687:
-
you are right, my bad ;)
> org.apache.flink.runtime.net.ConnectionUtilsTest fails
> --
>
> Key: FLINK-3687
> URL: https://issues.apache.org/jira/browse/FLINK-3687
> Project: Flink
> Issue Type: Bug
>Reporter: Nikolaas Steenbergen
>
> {code}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.51 sec <<<
> FAILURE! - in org.apache.flink.runtime.net.ConnectionUtilsTest
> testReturnLocalHostAddressUsingHeuristics(org.apache.flink.runtime.net.ConnectionUtilsTest)
> Time elapsed: 0.504 sec <<< FAILURE!
> java.lang.AssertionError:
> expected:
> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.flink.runtime.net.ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics(ConnectionUtilsTest.java:59)
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.554 sec -
> in org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionTest
> Results :
> Failed tests:
> ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics:59
> expected:
> but was:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3687) runtime.leaderelection.ZooKeeperLeaderElectionTest fails
[
https://issues.apache.org/jira/browse/FLINK-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221404#comment-15221404
]
Till Rohrmann commented on FLINK-3687:
--
Shouldn't that be the {{org.apache.flink.runtime.net.ConnectionUtilsTest}}
which failed?
> runtime.leaderelection.ZooKeeperLeaderElectionTest fails
>
>
> Key: FLINK-3687
> URL: https://issues.apache.org/jira/browse/FLINK-3687
> Project: Flink
> Issue Type: Bug
>Reporter: Nikolaas Steenbergen
>
> {code}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.51 sec <<<
> FAILURE! - in org.apache.flink.runtime.net.ConnectionUtilsTest
> testReturnLocalHostAddressUsingHeuristics(org.apache.flink.runtime.net.ConnectionUtilsTest)
> Time elapsed: 0.504 sec <<< FAILURE!
> java.lang.AssertionError:
> expected:
> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.flink.runtime.net.ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics(ConnectionUtilsTest.java:59)
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.554 sec -
> in org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionTest
> Results :
> Failed tests:
> ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics:59
> expected:
> but was:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-3687) runtime.leaderelection.ZooKeeperLeaderElectionTest fails
[
https://issues.apache.org/jira/browse/FLINK-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolaas Steenbergen updated FLINK-3687:
Description:
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.51 sec <<<
FAILURE! - in org.apache.flink.runtime.net.ConnectionUtilsTest
testReturnLocalHostAddressUsingHeuristics(org.apache.flink.runtime.net.ConnectionUtilsTest)
Time elapsed: 0.504 sec <<< FAILURE!
java.lang.AssertionError:
expected: but
was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.flink.runtime.net.ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics(ConnectionUtilsTest.java:59)
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.554 sec - in
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionTest
Results :
Failed tests:
ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics:59
expected: but
was:
{code}
was:
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.51 sec <<<
FAILURE! - in org.apache.flink.runtime.net.ConnectionUtilsTest
testReturnLocalHostAddressUsingHeuristics(org.apache.flink.runtime.net.ConnectionUtilsTest)
Time elapsed: 0.504 sec <<< FAILURE!
java.lang.AssertionError:
expected: but
was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.flink.runtime.net.ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics(ConnectionUtilsTest.java:59)
{code}
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.554 sec - in
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionTest
Results :
Failed tests:
ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics:59
expected: but
was:
{code}
> runtime.leaderelection.ZooKeeperLeaderElectionTest fails
>
>
> Key: FLINK-3687
> URL: https://issues.apache.org/jira/browse/FLINK-3687
> Project: Flink
> Issue Type: Bug
>Reporter: Nikolaas Steenbergen
>
> {code}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.51 sec <<<
> FAILURE! - in org.apache.flink.runtime.net.ConnectionUtilsTest
> testReturnLocalHostAddressUsingHeuristics(org.apache.flink.runtime.net.ConnectionUtilsTest)
> Time elapsed: 0.504 sec <<< FAILURE!
> java.lang.AssertionError:
> expected:
> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.flink.runtime.net.ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics(ConnectionUtilsTest.java:59)
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.554 sec -
> in org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionTest
> Results :
> Failed tests:
> ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics:59
> expected:
> but was:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3687) runtime.leaderelection.ZooKeeperLeaderElectionTest fails
Nikolaas Steenbergen created FLINK-3687:
---
Summary: runtime.leaderelection.ZooKeeperLeaderElectionTest fails
Key: FLINK-3687
URL: https://issues.apache.org/jira/browse/FLINK-3687
Project: Flink
Issue Type: Bug
Reporter: Nikolaas Steenbergen
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.51 sec <<<
FAILURE! - in org.apache.flink.runtime.net.ConnectionUtilsTest
testReturnLocalHostAddressUsingHeuristics(org.apache.flink.runtime.net.ConnectionUtilsTest)
Time elapsed: 0.504 sec <<< FAILURE!
java.lang.AssertionError:
expected: but
was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.flink.runtime.net.ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics(ConnectionUtilsTest.java:59)
{code}
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.554 sec - in
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionTest
Results :
Failed tests:
ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics:59
expected: but
was:
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-3686) resourcemanager.ClusterShutdownITCase sometimes fails
[
https://issues.apache.org/jira/browse/FLINK-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolaas Steenbergen updated FLINK-3686:
Description:
resourcemanager.ClusterShutdownITCase sometimes fails
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.47 sec <<<
FAILURE! - in org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase
testClusterShutdown(org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase)
Time elapsed: 30.063 sec <<< FAILURE!
java.lang.AssertionError: assertion failed: timeout (29897828577 nanoseconds)
during expectMsgClass waiting for class
org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful
at scala.Predef$.assert(Predef.scala:179)
at
akka.testkit.TestKitBase$class.expectMsgClass_internal(TestKit.scala:423)
at akka.testkit.TestKitBase$class.expectMsgClass(TestKit.scala:410)
at akka.testkit.TestKit.expectMsgClass(TestKit.scala:718)
at akka.testkit.JavaTestKit.expectMsgClass(JavaTestKit.java:397)
at
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.run(ClusterShutdownITCase.java:92)
at akka.testkit.JavaTestKit$Within$1.apply(JavaTestKit.java:232)
at akka.testkit.TestKitBase$class.within(TestKit.scala:296)
at akka.testkit.TestKit.within(TestKit.scala:718)
at akka.testkit.TestKitBase$class.within(TestKit.scala:310)
at akka.testkit.TestKit.within(TestKit.scala:718)
at akka.testkit.JavaTestKit$Within.(JavaTestKit.java:230)
at
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.(ClusterShutdownITCase.java:72)
at
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1.(ClusterShutdownITCase.java:72)
at
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase.testClusterShutdown(ClusterShutdownITCase.java:71)
Results :
Failed tests:
ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout
(29897828577 nanoseconds) during expectMsgClass waiting for class
org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful
{code}
was:
resourcemanager.ClusterShutdownITCase sometimes fails
~~~Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.47 sec
<<< FAILURE! - in org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase
testClusterShutdown(org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase)
Time elapsed: 30.063 sec <<< FAILURE!
java.lang.AssertionError: assertion failed: timeout (29897828577 nanoseconds)
during expectMsgClass waiting for class
org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful
at scala.Predef$.assert(Predef.scala:179)
at
akka.testkit.TestKitBase$class.expectMsgClass_internal(TestKit.scala:423)
at akka.testkit.TestKitBase$class.expectMsgClass(TestKit.scala:410)
at akka.testkit.TestKit.expectMsgClass(TestKit.scala:718)
at akka.testkit.JavaTestKit.expectMsgClass(JavaTestKit.java:397)
at
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.run(ClusterShutdownITCase.java:92)
at akka.testkit.JavaTestKit$Within$1.apply(JavaTestKit.java:232)
at akka.testkit.TestKitBase$class.within(TestKit.scala:296)
at akka.testkit.TestKit.within(TestKit.scala:718)
at akka.testkit.TestKitBase$class.within(TestKit.scala:310)
at akka.testkit.TestKit.within(TestKit.scala:718)
at akka.testkit.JavaTestKit$Within.(JavaTestKit.java:230)
at
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.(ClusterShutdownITCase.java:72)
at
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1.(ClusterShutdownITCase.java:72)
at
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase.testClusterShutdown(ClusterShutdownITCase.java:71)
Results :
Failed tests:
ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout
(29897828577 nanoseconds) during expectMsgClass waiting for class
org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful~~~
> resourcemanager.ClusterShutdownITCase sometimes fails
> -
>
> Key: FLINK-3686
> URL: https://issues.apache.org/jira/browse/FLINK-3686
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Nikolaas Steenbergen
>
> resourcemanager.ClusterShutdownITCase sometimes fails
> {code}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.47 sec <<<
> FAILURE! - in org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase
> testClusterShutdown(org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase)
> Time elapsed: 30.063 sec <<< FAILURE!
> java.lang.AssertionError: assertion failed: timeout (29897828577 nanoseconds)
> during
[jira] [Commented] (FLINK-2392) Instable test in flink-yarn-tests
[
https://issues.apache.org/jira/browse/FLINK-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221397#comment-15221397
]
Maximilian Michels commented on FLINK-2392:
---
Have you seen this one before? Might be related to the recent ResourceManager
YARN changes.
> Instable test in flink-yarn-tests
> -
>
> Key: FLINK-2392
> URL: https://issues.apache.org/jira/browse/FLINK-2392
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 0.10.0
>Reporter: Matthias J. Sax
>Assignee: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.0.0
>
>
> The test YARNSessionFIFOITCase fails from time to time on an irregular basis.
> For example see: https://travis-ci.org/apache/flink/jobs/72019690
> {noformat}
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 205.163 sec
> <<< FAILURE! - in org.apache.flink.yarn.YARNSessionFIFOITCase
> perJobYarnClusterWithParallelism(org.apache.flink.yarn.YARNSessionFIFOITCase)
> Time elapsed: 60.651 sec <<< FAILURE!
> java.lang.AssertionError: During the timeout period of 60 seconds the
> expected string did not show up
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.apache.flink.yarn.YarnTestBase.runWithArgs(YarnTestBase.java:478)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.perJobYarnClusterWithParallelism(YARNSessionFIFOITCase.java:435)
> Results :
> Failed tests:
>
> YARNSessionFIFOITCase.perJobYarnClusterWithParallelism:435->YarnTestBase.runWithArgs:478
> During the timeout period of 60 seconds the expected string did not show up
> {noformat}
> Another error case is this (see
> https://travis-ci.org/mjsax/flink/jobs/77313444)
> {noformat}
> Tests run: 12, Failures: 3, Errors: 0, Skipped: 2, Time elapsed: 182.008 sec
> <<< FAILURE! - in org.apache.flink.yarn.YARNSessionFIFOITCase
> testTaskManagerFailure(org.apache.flink.yarn.YARNSessionFIFOITCase) Time
> elapsed: 27.356 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> testNonexistingQueue(org.apache.flink.yarn.YARNSessionFIFOITCase) Time
> elapsed: 17.421 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> testJavaAPI(org.apache.flink.yarn.YARNSessionFIFOITCase) Time elapsed:
> 11.984 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> {noformat}
> Furthermore, this build failed too:
> https://travis-ci.org/apache/flink/jobs/77313450
> (no error, but Travis terminated to due no progress for 300 seconds ->
> deadlock?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-3686) resourcemanager.ClusterShutdownITCase sometimes fails
[ https://issues.apache.org/jira/browse/FLINK-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolaas Steenbergen updated FLINK-3686: Description: resourcemanager.ClusterShutdownITCase sometimes fails ~~~Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.47 sec <<< FAILURE! - in org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase testClusterShutdown(org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase) Time elapsed: 30.063 sec <<< FAILURE! java.lang.AssertionError: assertion failed: timeout (29897828577 nanoseconds) during expectMsgClass waiting for class org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful at scala.Predef$.assert(Predef.scala:179) at akka.testkit.TestKitBase$class.expectMsgClass_internal(TestKit.scala:423) at akka.testkit.TestKitBase$class.expectMsgClass(TestKit.scala:410) at akka.testkit.TestKit.expectMsgClass(TestKit.scala:718) at akka.testkit.JavaTestKit.expectMsgClass(JavaTestKit.java:397) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.run(ClusterShutdownITCase.java:92) at akka.testkit.JavaTestKit$Within$1.apply(JavaTestKit.java:232) at akka.testkit.TestKitBase$class.within(TestKit.scala:296) at akka.testkit.TestKit.within(TestKit.scala:718) at akka.testkit.TestKitBase$class.within(TestKit.scala:310) at akka.testkit.TestKit.within(TestKit.scala:718) at akka.testkit.JavaTestKit$Within.(JavaTestKit.java:230) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.(ClusterShutdownITCase.java:72) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1.(ClusterShutdownITCase.java:72) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase.testClusterShutdown(ClusterShutdownITCase.java:71) Results : Failed tests: ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout (29897828577 nanoseconds) during expectMsgClass waiting for class org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful~~~ was: resourcemanager.ClusterShutdownITCase sometimes fails ```Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.47 sec <<< FAILURE! - in org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase testClusterShutdown(org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase) Time elapsed: 30.063 sec <<< FAILURE! java.lang.AssertionError: assertion failed: timeout (29897828577 nanoseconds) during expectMsgClass waiting for class org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful at scala.Predef$.assert(Predef.scala:179) at akka.testkit.TestKitBase$class.expectMsgClass_internal(TestKit.scala:423) at akka.testkit.TestKitBase$class.expectMsgClass(TestKit.scala:410) at akka.testkit.TestKit.expectMsgClass(TestKit.scala:718) at akka.testkit.JavaTestKit.expectMsgClass(JavaTestKit.java:397) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.run(ClusterShutdownITCase.java:92) at akka.testkit.JavaTestKit$Within$1.apply(JavaTestKit.java:232) at akka.testkit.TestKitBase$class.within(TestKit.scala:296) at akka.testkit.TestKit.within(TestKit.scala:718) at akka.testkit.TestKitBase$class.within(TestKit.scala:310) at akka.testkit.TestKit.within(TestKit.scala:718) at akka.testkit.JavaTestKit$Within.(JavaTestKit.java:230) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.(ClusterShutdownITCase.java:72) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1.(ClusterShutdownITCase.java:72) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase.testClusterShutdown(ClusterShutdownITCase.java:71) Results : Failed tests: ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout (29897828577 nanoseconds) during expectMsgClass waiting for class org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful``` > resourcemanager.ClusterShutdownITCase sometimes fails > - > > Key: FLINK-3686 > URL: https://issues.apache.org/jira/browse/FLINK-3686 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Nikolaas Steenbergen > > resourcemanager.ClusterShutdownITCase sometimes fails > ~~~Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.47 sec > <<< FAILURE! - in > org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase > testClusterShutdown(org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase) > Time elapsed: 30.063 sec <<< FAILURE! > java.lang.AssertionError: assertion failed: timeout (29897828577 nanoseconds) > during expectMsgCl
[jira] [Created] (FLINK-3686) resourcemanager.ClusterShutdownITCase sometimes fails
Nikolaas Steenbergen created FLINK-3686: --- Summary: resourcemanager.ClusterShutdownITCase sometimes fails Key: FLINK-3686 URL: https://issues.apache.org/jira/browse/FLINK-3686 Project: Flink Issue Type: Bug Components: Tests Reporter: Nikolaas Steenbergen resourcemanager.ClusterShutdownITCase sometimes fails ```Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.47 sec <<< FAILURE! - in org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase testClusterShutdown(org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase) Time elapsed: 30.063 sec <<< FAILURE! java.lang.AssertionError: assertion failed: timeout (29897828577 nanoseconds) during expectMsgClass waiting for class org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful at scala.Predef$.assert(Predef.scala:179) at akka.testkit.TestKitBase$class.expectMsgClass_internal(TestKit.scala:423) at akka.testkit.TestKitBase$class.expectMsgClass(TestKit.scala:410) at akka.testkit.TestKit.expectMsgClass(TestKit.scala:718) at akka.testkit.JavaTestKit.expectMsgClass(JavaTestKit.java:397) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.run(ClusterShutdownITCase.java:92) at akka.testkit.JavaTestKit$Within$1.apply(JavaTestKit.java:232) at akka.testkit.TestKitBase$class.within(TestKit.scala:296) at akka.testkit.TestKit.within(TestKit.scala:718) at akka.testkit.TestKitBase$class.within(TestKit.scala:310) at akka.testkit.TestKit.within(TestKit.scala:718) at akka.testkit.JavaTestKit$Within.(JavaTestKit.java:230) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.(ClusterShutdownITCase.java:72) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1.(ClusterShutdownITCase.java:72) at org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase.testClusterShutdown(ClusterShutdownITCase.java:71) Results : Failed tests: ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout (29897828577 nanoseconds) during expectMsgClass waiting for class org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3635) Potential null deference in TwitterExample#SelectEnglishAndTokenizeFlatMap#flatMap
[
https://issues.apache.org/jira/browse/FLINK-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221385#comment-15221385
]
ASF GitHub Bot commented on FLINK-3635:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1845#issuecomment-204310736
Looks good, +1 to merge
> Potential null deference in
> TwitterExample#SelectEnglishAndTokenizeFlatMap#flatMap
> --
>
> Key: FLINK-3635
> URL: https://issues.apache.org/jira/browse/FLINK-3635
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> JsonNode jsonNode = jsonParser.readValue(value, JsonNode.class);
> if (jsonNode.has("user") &&
> jsonNode.get("user").get("lang").asText().equals("en")) {
> {code}
> jsonNode.get("user").get("lang") may return null, leading to
> NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3635] Potential null deference in Twitt...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1845#issuecomment-204310736 Looks good, +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3541] [Kafka Connector] Clean up workar...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1846#issuecomment-204310567 This seems to change the behavior. Are you sure that the current workaround (catching the `NullPointerException`) is not needed any more? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3541) Clean up workaround in FlinkKafkaConsumer09
[
https://issues.apache.org/jira/browse/FLINK-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221382#comment-15221382
]
ASF GitHub Bot commented on FLINK-3541:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1846#issuecomment-204310567
This seems to change the behavior. Are you sure that the current workaround
(catching the `NullPointerException`) is not needed any more?
> Clean up workaround in FlinkKafkaConsumer09
>
>
> Key: FLINK-3541
> URL: https://issues.apache.org/jira/browse/FLINK-3541
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Priority: Minor
>
> In the current {{FlinkKafkaConsumer09}} implementation, we repeatedly start a
> new {{KafkaConsumer}} if the method {{KafkaConsumer.partitionsFor}} returns a
> NPE. This is due to a bug with the Kafka version 0.9.0.0. See
> https://issues.apache.org/jira/browse/KAFKA-2880. The code can be found in
> the constructor of {{FlinkKafkaConsumer09.java:208}}.
> However, the problem is marked as fixed for version 0.9.0.1, which we also
> use for the flink-connector-kafka. Therefore, we should be able to get rid of
> the workaround.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3541) Clean up workaround in FlinkKafkaConsumer09
[
https://issues.apache.org/jira/browse/FLINK-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221380#comment-15221380
]
ASF GitHub Bot commented on FLINK-3541:
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1846#discussion_r58177554
--- Diff:
flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java
---
@@ -36,6 +36,7 @@
import org.apache.kafka.common.PartitionInfo;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.errors.WakeupException;
++import org.apache.kafka.common.KafkaException;
--- End diff --
Extra `+`. See also CI logs...
> Clean up workaround in FlinkKafkaConsumer09
>
>
> Key: FLINK-3541
> URL: https://issues.apache.org/jira/browse/FLINK-3541
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Priority: Minor
>
> In the current {{FlinkKafkaConsumer09}} implementation, we repeatedly start a
> new {{KafkaConsumer}} if the method {{KafkaConsumer.partitionsFor}} returns a
> NPE. This is due to a bug with the Kafka version 0.9.0.0. See
> https://issues.apache.org/jira/browse/KAFKA-2880. The code can be found in
> the constructor of {{FlinkKafkaConsumer09.java:208}}.
> However, the problem is marked as fixed for version 0.9.0.1, which we also
> use for the flink-connector-kafka. Therefore, we should be able to get rid of
> the workaround.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3541] [Kafka Connector] Clean up workar...
Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/1846#discussion_r58177554 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java --- @@ -36,6 +36,7 @@ import org.apache.kafka.common.PartitionInfo; import org.apache.kafka.common.TopicPartition; import org.apache.kafka.common.errors.WakeupException; ++import org.apache.kafka.common.KafkaException; --- End diff -- Extra `+`. See also CI logs... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3678) Make Flink logs directory configurable
[
https://issues.apache.org/jira/browse/FLINK-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221375#comment-15221375
]
ASF GitHub Bot commented on FLINK-3678:
---
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/1837#issuecomment-204309933
Yes, I'm going to merge it later today.
> Make Flink logs directory configurable
> --
>
> Key: FLINK-3678
> URL: https://issues.apache.org/jira/browse/FLINK-3678
> Project: Flink
> Issue Type: Improvement
> Components: Start-Stop Scripts
>Affects Versions: 1.0.0
>Reporter: Stefano Baghino
>Assignee: Stefano Baghino
>Priority: Minor
> Fix For: 1.0.1
>
>
> Currently Flink logs are stored under {{$FLINK_HOME/log}} and the user cannot
> configure an alternative storage location. It would be nice to add a
> configuration key in the {{flink-conf.yaml}} and edit the {{bin/flink}}
> launch script accordingly to get the value if present or default to the
> current behavior if no value is provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3678] Make Flink logs directory configu...
Github user uce commented on the pull request: https://github.com/apache/flink/pull/1837#issuecomment-204309933 Yes, I'm going to merge it later today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3678) Make Flink logs directory configurable
[
https://issues.apache.org/jira/browse/FLINK-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221339#comment-15221339
]
ASF GitHub Bot commented on FLINK-3678:
---
Github user stefanobaghino commented on the pull request:
https://github.com/apache/flink/pull/1837#issuecomment-204301276
I think this is ready.
> Make Flink logs directory configurable
> --
>
> Key: FLINK-3678
> URL: https://issues.apache.org/jira/browse/FLINK-3678
> Project: Flink
> Issue Type: Improvement
> Components: Start-Stop Scripts
>Affects Versions: 1.0.0
>Reporter: Stefano Baghino
>Assignee: Stefano Baghino
>Priority: Minor
> Fix For: 1.0.1
>
>
> Currently Flink logs are stored under {{$FLINK_HOME/log}} and the user cannot
> configure an alternative storage location. It would be nice to add a
> configuration key in the {{flink-conf.yaml}} and edit the {{bin/flink}}
> launch script accordingly to get the value if present or default to the
> current behavior if no value is provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3678] Make Flink logs directory configu...
Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1837#issuecomment-204301276 I think this is ready. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2392) Instable test in flink-yarn-tests
[
https://issues.apache.org/jira/browse/FLINK-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221299#comment-15221299
]
Matthias J. Sax commented on FLINK-2392:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/119950627/log.txt
> Instable test in flink-yarn-tests
> -
>
> Key: FLINK-2392
> URL: https://issues.apache.org/jira/browse/FLINK-2392
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 0.10.0
>Reporter: Matthias J. Sax
>Assignee: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.0.0
>
>
> The test YARNSessionFIFOITCase fails from time to time on an irregular basis.
> For example see: https://travis-ci.org/apache/flink/jobs/72019690
> {noformat}
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 205.163 sec
> <<< FAILURE! - in org.apache.flink.yarn.YARNSessionFIFOITCase
> perJobYarnClusterWithParallelism(org.apache.flink.yarn.YARNSessionFIFOITCase)
> Time elapsed: 60.651 sec <<< FAILURE!
> java.lang.AssertionError: During the timeout period of 60 seconds the
> expected string did not show up
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.apache.flink.yarn.YarnTestBase.runWithArgs(YarnTestBase.java:478)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.perJobYarnClusterWithParallelism(YARNSessionFIFOITCase.java:435)
> Results :
> Failed tests:
>
> YARNSessionFIFOITCase.perJobYarnClusterWithParallelism:435->YarnTestBase.runWithArgs:478
> During the timeout period of 60 seconds the expected string did not show up
> {noformat}
> Another error case is this (see
> https://travis-ci.org/mjsax/flink/jobs/77313444)
> {noformat}
> Tests run: 12, Failures: 3, Errors: 0, Skipped: 2, Time elapsed: 182.008 sec
> <<< FAILURE! - in org.apache.flink.yarn.YARNSessionFIFOITCase
> testTaskManagerFailure(org.apache.flink.yarn.YARNSessionFIFOITCase) Time
> elapsed: 27.356 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> testNonexistingQueue(org.apache.flink.yarn.YARNSessionFIFOITCase) Time
> elapsed: 17.421 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> testJavaAPI(org.apache.flink.yarn.YARNSessionFIFOITCase) Time elapsed:
> 11.984 sec <<< FAILURE!
> java.lang.AssertionError: Found a file
> /home/travis/build/mjsax/flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1440595422559_0007/container_1440595422559_0007_01_03/taskmanager.log
> with a prohibited string: [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:294)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:94)
> {noformat}
> Furthermore, this build failed too:
> https://travis-ci.org/apache/flink/jobs/77313450
> (no error, but Travis terminated to due no progress for 300 seconds ->
> deadlock?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
