Hi guys, I ran into this again while playing with savepoint/restore parallelism:
https://issues.apache.org/jira/browse/FLINK-7595 https://github.com/apache/flink/pull/4651 Anyone has some idea about the status of this PR or were we planning to postpone this to 1.5? Thanks, Gyula Fabian Hueske <fhue...@gmail.com> ezt írta (időpont: 2017. nov. 29., Sze, 13:10): > OK, the situation is the following: > > The test class (org.apache.flink.yarn.UtilsTest) implements a Hadoop > interface (Container) that was extended in Hadoop 2.9.0 by a getter and > setter. > By adding the methods, we can compile Flink for Hadoop 2.9.0. However, the > getter/setter add a dependency on a class that was also added in Hadoop > 2.9.0. > Therefore, the implementation is not backwards compatible with Hadoop > versions < 2.9.0. > > Not sure how we can fix the problem. We would need two version of the class > that are chosen based on the Hadoop version. Do we have something like that > somewhere else? > > Since this is only a problem in a test class, Flink 1.4.0 might still work > very well with Hadoop 2.9.0. > However, this has not been tested AFAIK. > > Cheers, Fabian > > 2017-11-29 12:47 GMT+01:00 Fabian Hueske <fhue...@gmail.com>: > > > I just tried to build the release-1.4 branch for Hadoop 2.9.0 (released a > > few days ago) and got a compilation failure in a test class. > > > > Right now, I'm assessing how much we need to fix to support Hadoop 2.9.0. > > I'll report later. > > > > Best, Fabian > > > > 2017-11-29 11:16 GMT+01:00 Aljoscha Krettek <aljos...@apache.org>: > > > >> Agreed, this is a regression compared to the previous functionality. I > >> updated the issue to "Blocker". > >> > >> > On 29. Nov 2017, at 10:01, Gyula Fóra <gyula.f...@gmail.com> wrote: > >> > > >> > Hi all, > >> > > >> > I have found the following issue: > >> > https://issues.apache.org/jira/browse/FLINK-8165 > >> > > >> > I would say this is a blocker (I personally pass the ParameterTool all > >> over > >> > the place in my production apps), but a pretty trivial issue to fix, > we > >> can > >> > wait a little to find other potential problems. > >> > > >> > I can submit a fix in a little bit. > >> > > >> > Cheers, > >> > Gyula > >> > > >> > Tzu-Li (Gordon) Tai <tzuli...@apache.org> ezt írta (időpont: 2017. > nov. > >> > 29., Sze, 9:23): > >> > > >> >> +1 > >> >> > >> >> Verified: > >> >> - No missing release Maven artifacts > >> >> - Staged Apache source & binary convenience releases looks good > >> >> - NOTICE / LICENSE is correct, README is sane > >> >> - Built from source (macOS, Scala 2.11, Hadoop-free & Hadoop 2.8) > >> >> - Cluster testing on AWS EMR (see release-testing-doc for > configuration > >> >> details) > >> >> - Tested Kinesis / Elasticsearch connector (no dependency clashes on > >> >> cluster execution, works locally in IDE) > >> >> > >> >> Thanks a lot for managing the release Aljoscha! > >> >> > >> >> Cheers, > >> >> Gordon > >> >> > >> >> On 28 November 2017 at 8:32:42 PM, Stefan Richter ( > >> >> s.rich...@data-artisans.com) wrote: > >> >> > >> >> +1 (non-binding) > >> >> > >> >> I tested Flink in a cluster setup on Google Cloud, YARN-per-job, > >> checked > >> >> that for all backends that HA, recovery, at-least-once, end-to-end > >> exactly > >> >> once (with Kafka11 Producer), savepoints, externalized checkpoints, > and > >> >> rescaling work correctly. > >> >> > >> >>> Am 28.11.2017 um 11:47 schrieb Aljoscha Krettek < > aljos...@apache.org > >> >: > >> >>> > >> >>> +1 > >> >>> > >> >>> Verified: > >> >>> - NOTICE and LICENSE are correct > >> >>> - source doesn't contain binaries > >> >>> - verified signatures > >> >>> - verified hashes > >> >>> - cluster testing on AWS and Cloudera VM (with Kerberos) (see > >> >> release-testing doc) > >> >>> > >> >>> > >> >>>> On 28. Nov 2017, at 11:20, Aljoscha Krettek <aljos...@apache.org> > >> >> wrote: > >> >>>> > >> >>>> Phew, thanks for the update! > >> >>>> > >> >>>>> On 28. Nov 2017, at 11:19, Gyula Fóra <gyf...@apache.org> wrote: > >> >>>>> > >> >>>>> Ok seems like I had to remove the snappy jar as it was corrupted > >> (makes > >> >>>>> total sense) :P > >> >>>>> > >> >>>>> Gyula Fóra <gyf...@apache.org> ezt írta (időpont: 2017. nov. 28., > >> K, > >> >> 11:13): > >> >>>>> > >> >>>>>> Hi Aljoscha, > >> >>>>>> > >> >>>>>> Thanks for the release candidate. I am having a hard time > building > >> >> the rc, > >> >>>>>> I seem to get this error no matter what I do: > >> >>>>>> > >> >>>>>> [ERROR] Failed to execute goal > >> >>>>>> org.apache.maven.plugins:maven-shade-plugin:2.4.1:shade > >> >> (shade-hadoop) on > >> >>>>>> project flink-shaded-hadoop2-uber: Error creating shaded jar: > >> invalid > >> >> LOC > >> >>>>>> header (bad signature) -> [Help 1] > >> >>>>>> > >> >>>>>> (Apache Maven 3.3.9) > >> >>>>>> > >> >>>>>> Any idea what I am missing? > >> >>>>>> > >> >>>>>> Thanks, > >> >>>>>> Gyula > >> >>>>>> > >> >>>>>> > >> >>>>>> Aljoscha Krettek <aljos...@apache.org> ezt írta (időpont: 2017. > >> nov. > >> >> 27., > >> >>>>>> H, 19:35): > >> >>>>>> > >> >>>>>>> Hi everyone, > >> >>>>>>> > >> >>>>>>> Please review and vote on release candidate #2 for the version > >> >> 1.4.0, as > >> >>>>>>> follows: > >> >>>>>>> [ ] +1, Approve the release > >> >>>>>>> [ ] -1, Do not approve the release (please provide specific > >> comments) > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> The complete staging area is available for your review, which > >> >> includes: > >> >>>>>>> * JIRA release notes [1], > >> >>>>>>> * the official Apache source release and binary convenience > >> releases > >> >> to > >> >>>>>>> be deployed to dist.apache.org[2], which are signed with the > key > >> >> with > >> >>>>>>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3], > >> >>>>>>> * all artifacts to be deployed to the Maven Central Repository > >> [4], > >> >>>>>>> * source code tag "release-1.4.0-rc1" [5], > >> >>>>>>> * website pull request listing the new release [6]. > >> >>>>>>> > >> >>>>>>> Please have a careful look at the website PR because I changed > >> some > >> >>>>>>> wording and we're now also releasing a binary without Hadoop > >> >> dependencies. > >> >>>>>>> > >> >>>>>>> Please use this document for coordinating testing efforts: [7] > >> >>>>>>> > >> >>>>>>> The only change between RC1 and this RC2 is that the source > >> release > >> >>>>>>> package does not include the erroneously included binary Ruby > >> >> dependencies > >> >>>>>>> of the documentation anymore. Because of this I would like to > >> >> propose a > >> >>>>>>> shorter voting time and close the vote around the time that RC1 > >> >> would have > >> >>>>>>> closed. This would mean closing by end of Wednesday. Please let > me > >> >> know if > >> >>>>>>> you disagree with this. The vote is adopted by majority > approval, > >> >> with at > >> >>>>>>> least 3 PMC affirmative votes. > >> >>>>>>> > >> >>>>>>> Thanks, > >> >>>>>>> Your friendly Release Manager > >> >>>>>>> > >> >>>>>>> [1] > >> >>>>>>> > >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje > >> ctId=12315522&version=12340533 > >> >>>>>>> [2] http://people.apache.org/~aljoscha/flink-1.4.0-rc2/ > >> >>>>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > >> >>>>>>> [4] > >> >>>>>>> > >> >> > https://repository.apache.org/content/repositories/orgapacheflink-1140 > >> >>>>>>> [5] > >> >>>>>>> > >> >> https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h= > >> ea751b7b23b23446ed3fcdeed564bbe8bf4adf9c > >> >>>>>>> [6] https://github.com/apache/flink-web/pull/95 > >> >>>>>>> [7] > >> >>>>>>> > >> >> https://docs.google.com/document/d/1HqYyrNoMSXwo8zBpZj7s39Uz > >> UdlFcFO8TRpHNZ_cl44/edit?usp=sharing > >> >>>>>>> > >> >>>>>>> Pro-tip: you can create a settings.xml file with these contents: > >> >>>>>>> > >> >>>>>>> <settings> > >> >>>>>>> <activeProfiles> > >> >>>>>>> <activeProfile>flink-1.4.0</activeProfile> > >> >>>>>>> </activeProfiles> > >> >>>>>>> <profiles> > >> >>>>>>> <profile> > >> >>>>>>> <id>flink-1.4.0</id> > >> >>>>>>> <repositories> > >> >>>>>>> <repository> > >> >>>>>>> <id>flink-1.4.0</id> > >> >>>>>>> <url> > >> >>>>>>> > >> >>>>>>> > >> >> https://repository.apache.org/content/repositories/orgapache > >> flink-1140/ > >> >>>>>>> </url> > >> >>>>>>> </repository> > >> >>>>>>> <repository> > >> >>>>>>> <id>archetype</id> > >> >>>>>>> <url> > >> >>>>>>> > >> >>>>>>> > >> >> https://repository.apache.org/content/repositories/orgapache > >> flink-1140/ > >> >>>>>>> </url> > >> >>>>>>> </repository> > >> >>>>>>> </repositories> > >> >>>>>>> </profile> > >> >>>>>>> </profiles> > >> >>>>>>> </settings> > >> >>>>>>> > >> >>>>>>> And reference that in you maven commands via --settings > >> >>>>>>> path/to/settings.xml. This is useful for creating a quickstart > >> based > >> >> on the > >> >>>>>>> staged release and for building against the staged jars. > >> >>>>>> > >> >>>>>> > >> >>>> > >> >>> > >> >> > >> >> > >> > >> > > >
