Hi Chesnay, thanks for sharing your long term plan about `flink-shaded-hadoop`! Got your points. I also think we need the JIRA for that, at the follows right time. Best Regards, Jincheng
Chesnay Schepler <ches...@apache.org> 于2019年3月18日周一 下午8:13写道: > Long term plan _is_ to move flink-shaded-hadoop to flink-shaded, I believe > there's even a JIRA for that. > > Until that is in place they _must_ have retain the flink version as > otherwise we'd be unable to change them in follow-up releases without > changing the version scheme again. > > And even after the move they will retain the flink-shaded version like all > other flink-shaded modules, for the above reason. > > On 18.03.2019 12:10, jincheng sun wrote: > > Hi Chesnay, > > The artifacts to be released do not have a SNAPSHOT suffix: >> >> https://repository.apache.org/content/repositories/orgapacheflink-1213/org/apache/flink/flink-shaded-hadoop2-uber/ > > Thank you for providing this link. It's very useful for contributors who > want to check the RC on YARN. > > My suggestion may not describe being clear, let me explain: > > 1. Since 1.8.0, Flink's release package will not contain the corresponding > Hadoop dependency, then the user has two ways to get the required hadoop > dependency: > > 1). Download the existing Hadoop version on the Flink download page. > 2). Generate the version required by the user from the source code (see > https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#hadoop-versions) > For example, version 2.6.1 is required: `mvn clean install -DskipTests > -Dhadoop.version=2.6.1`. > > 2. About how to manage the JARs release of Hadoop dependencies: > > 1). The name of Hadoop shaded version should not include Flink > version, take your link as an example: > `.../flink-shaded-hadoop2-uber/2.4.1-1.8.0/xx.jar` > `.../flink-shaded-hadoop2-uber/2.6.5-1.8.0/xx.jar` > `.../flink-shaded-hadoop2-uber/2.7.5-1.8.0/xx.jar` > `.../flink-shaded-hadoop2-uber/2.8.3-1.8.0/xx.jar` > The above version name I think it is possible to change `2.4.1-1.8.0` to > `2.4.1`. That is, the same version of `Hadoop` shade can be used in many > Flink versions, such as 2.8.3 Hadoop is not only available for Flink-1.8.0, > it can be used by Flink-1.8.x or it can be used by Flink-1.9.x. etc. > > 2). Release the shaded-Hadoop independently: > For a long-term, we can release the shaded JARs independently and move > `flink-shaded-hadoop` into `https://github.com/apache/flink-shaded`, So > I suggest that we can publish Hadoop versions independently, and share > them in multiple Flink versions. > > What do you think? > > Best, > Jincheng > > > Chesnay Schepler <ches...@apache.org> 于2019年3月18日周一 下午4:15写道: > >> We release SNAPSHOT artifacts for all module, see >> >> https://repository.apache.org/content/groups/public/org/apache/flink/flink-core/ >> . >> >> The artifacts to be released do not have a SNAPSHOT suffix: >> >> https://repository.apache.org/content/repositories/orgapacheflink-1213/org/apache/flink/flink-shaded-hadoop2-uber/ >> >> Finally, we are already adding flink-shaded-hadoop to the optional >> components section in this PR: >> https://github.com/apache/flink-web/pull/180 >> >> On 18.03.2019 08:55, jincheng sun wrote: >> > -1 >> > >> > Currently, we have released the Hadoop-related JRA as a snapshot >> > version(such as flink-shaded-hadoop2-uber/2.4.1-1.8-SNAPSHOT >> > < >> https://repository.apache.org/content/groups/public/org/apache/flink/flink-shaded-hadoop2-uber/ >> >), >> > I think we should release a stable version. >> > When testing the release code on YARN, currently user cannot find out >> the >> > Hadoop dependency. Although there is a download explanation for Hadoop >> in >> > PR [`Update Downloads page for Flink 1.8 >> > <https://github.com/apache/flink-web/pull/180/files>`], a 404 error >> occurs >> > when you click Download ( I had left detail comments in the PR). >> > >> > So, I suggest as follows: >> > >> > 1. It would be better to add the changes for >> > `downloads.html#optional-components`, add the Hadoop relation JARs >> download >> > link first. >> > 2. Then add instructions on how to get the dependencies of the >> Hadoop or >> > add the correct download link directly in the next VOTE mail, due to we >> do >> > not include Hadoop in `flink-dist`. >> > 3. Release a stable version Hadoop-related JRAs. >> > >> > Then, contributors can test it more easily on YARN. What do you think? >> > >> > Best, >> > Jincheng >> > >> > >> > Chesnay Schepler <ches...@apache.org> 于2019年3月15日周五 下午10:35写道: >> > >> >> -1 >> >> >> >> Missing dependencies in NOTICE file of flink-dist (and by extension the >> >> binary distribution). >> >> * com.data-artisans:frocksdbjni:jar:5.17.2-artisans-1.0 >> >> >> >> On 14.03.2019 13:42, Aljoscha Krettek wrote: >> >>> Hi everyone, >> >>> Please review and vote on the release candidate 2 for Flink 1.8.0, as >> >> follows: >> >>> [ ] +1, Approve the release >> >>> [ ] -1, Do not approve the release (please provide specific comments) >> >>> >> >>> >> >>> The complete staging area is available for your review, which >> includes: >> >>> * JIRA release notes [1], >> >>> * the official Apache source release and binary convenience releases >> to >> >> be deployed to dist.apache.org <http://dist.apache.org/> [2], which >> are >> >> signed with the key with fingerprint >> >> F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3], >> >>> * all artifacts to be deployed to the Maven Central Repository [4], >> >>> * source code tag "release-1.8.0-rc2" [5], >> >>> * website pull request listing the new release [6] >> >>> * website pull request adding announcement blog post [7]. >> >>> >> >>> The vote will be open for at least 72 hours. It is adopted by majority >> >> approval, with at least 3 PMC affirmative votes. >> >>> Thanks, >> >>> Aljoscha >> >>> >> >>> [1] >> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274 >> >> < >> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274 >> >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc2/ < >> >> https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc2/> >> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS < >> >> https://dist.apache.org/repos/dist/release/flink/KEYS> >> >>> [4] >> >> https://repository.apache.org/content/repositories/orgapacheflink-1213 >> < >> >> >> https://repository.apache.org/content/repositories/orgapacheflink-1210/> >> >>> [5] >> >> >> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c77a329b71e3068bfde965ae91921ad5c47246dd >> >> < >> >> >> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=2d00b1c26d7b4554707063ab0d1d6cc236cfe8a5 >> >>> [6] https://github.com/apache/flink-web/pull/180 < >> >> https://github.com/apache/flink-web/pull/180> >> >>> [7] https://github.com/apache/flink-web/pull/179 < >> >> https://github.com/apache/flink-web/pull/179> >> >>> P.S. The difference to the previous RC1 is very small, you can fetch >> the >> >> two tags and do a "git log release-1.8.0-rc1..release-1.8.0-rc2” to >> see the >> >> difference in commits. Its fixes for the issues that led to the >> >> cancellation of the previous RC plus smaller fixes. Most >> >> verification/testing that was carried out should apply as is to this >> RC. >> >> >> >> >> >> >> >> >