[jira] [Created] (MAPREDUCE-6343) JobConf.parseMaximumHeapSizeMB() fails to parse value greater than 2GB expressed in bytes
Hao Xia created MAPREDUCE-6343: -- Summary: JobConf.parseMaximumHeapSizeMB() fails to parse value greater than 2GB expressed in bytes Key: MAPREDUCE-6343 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6343 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Hao Xia It currently tries to parse the value as an integer, which blows up whenever the value is greater than 2GB and expressed in bytes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Planning Hadoop 2.6.1 release
There were several requests on the user lists [1] for a 2.6.1 release. I got many offline comments too. Planning to do a 2.6.1 release in a few weeks time. We already have a bunch of tickets committed to 2.7.1. I created a filter [2] to tracking pending tickets. We need to collectively come up with a list of critical issues. We can use the JIRA Target Version field for the same. I see some but not a whole lot of new work for this release, most of it is likely going to be pulling in critical patches from 2.7.1/2.8 etc. Thoughts? Thanks +Vinod [1] Will Hadoop 2.6.1 be released soon? http://markmail.org/thread/zlsr6prejyogdyvh [2] 2.6.1 pending tickets https://issues.apache.org/jira/issues/?filter=12331711
Re: [DISCUSS] Release numbering for stable 2.8 and beyond
My understanding was that the main reason that we labeled 2.0 alpha and 2.1
beta is that we wanted flexibility to make breaking API changes.
Is that the case now with 2.7? I.e. do we have APIs labeled as Public /
Stable that we want freedom to change in 2.8? If not, I definitely don't
think the release merits a special tag. We should just make sure its bugs
are well documented.
-Sandy
On Mon, Apr 27, 2015 at 10:45 AM, Andrew Wang
wrote:
> I think Karthik correctly identified the two reasons we might want to
> denote a release as "unstable":
>
> a) Compatibility. Whether we have freedom to break compatibility before the
> final release, i.e. what "alpha" typically means.
>
> b) Quality. As Konst says, a release can be allowed to break compatibility
> ("alpha") but itself still be a high quality release.
>
> We could try and separate out these two concerns when it comes to
> versioning, but I think in reality users only care about prod vs. non-prod,
> which is why many other projects (Linux, HBase, etc) just have two release
> lines: "stable" (compatible/good-quality) vs. "unstable"
> (unknown-compatible/unknown-quality).
>
> To this end, I don't mind what we choose, as long as the difference between
> stable and unstable is denoted by the version number. I don't like the idea
> of tagging a release as good after the fact (1). The other projects we've
> used as examples make strong statements about their stable release lines,
> and we should too. Our downstreams and end users will appreciate clarity
> from the version number.
>
> Best,
> Andrew
>
> On Sun, Apr 26, 2015 at 9:51 AM, Hitesh Shah wrote:
>
> > There are a couple of different approaches we could take.
> >
> > How about publishing/releasing bits such as “2.8.0-RC0”. Downstream
> > projects can depend on these and use them normally similar to the
> approach
> > that they would have taken with release 2.8.0 or 2.8.0-alpha. After some
> > baking ( or more RC suffixed releases), at some point, a proper 2.8.0
> > release could be made? The RC tag clearly indicates to downstream layers
> > that things will be in flux slightly. Trying to distinguish
> > incompatibilities between 2.8.0 and 2.8.1/2.8.2 ( just because 2.8.0 was
> > tagged alpha in documentation ) are likely to be a nightmare to deal with
> > especially for new features introduced in the 2.8 release ( which might
> get
> > changed slightly based on usage feedback ).
> >
> > Furthermore, considering the release history of hadoop, the likelihood
> > that 2.9.0 will show up before 2.8.2 seems to be very high.
> >
> > With respect to the proposed choice (1), I thought that HBase applied a
> > different approach. The odd-number releases were always unstable and the
> > even number releases were the stable ones. This “could" make things a bit
> > more clear for downstream users without needing to resort to modified
> > versioned numbers ( with alpha/beta tags ) or requiring users to go look
> up
> > the website to find out which particular version of 2.8.x was the first
> > stable release.
> >
> > thanks
> > — Hitesh
> >
> >
> >
> >
> >
> > On Apr 22, 2015, at 2:17 PM, Vinod Kumar Vavilapalli <
> > vino...@hortonworks.com> wrote:
> >
> > > Forking the thread.
> > >
> > > In the previous 2.7.1 thread [1], there were enough yays to my proposal
> > to wait for a bug-fix release or two before calling a 2.x release stable.
> > There were some concerns about the naming.
> > >
> > > We have two options, taking 2.8 as an example
> > > (1) Release 2.8.0, call it as an alpha in documentation and release
> > notes, wait for a 2.8.1/2.8.2 reasonably stable enough to be called as
> the
> > first stable release of 2.8.
> > > (2) Release 2.8.0-alpha, 2.8.0-beta etc before culminating in a 2.8.0
> > stable release.
> > >
> > > (1) is what I preferred first up. This is what HBase used to do, and
> far
> > beyond, in the linux kernel releases. It helps in scenarios where we are
> > forced to downgrade a release, say due to major issues. We can simply
> > announce it as not stable retroactively, change the pointers on our
> website
> > and move on.
> > >
> > > Thoughts?
> > >
> > > Thanks,
> > > +Vinod
> > >
> > > [1] http://markmail.org/thread/ogzk4phj6wsdpssu
> > >
> > > On Apr 21, 2015, at 4:59 PM, Vinod Kumar Vavilapalli <
> > vino...@hortonworks.com> wrote:
> > >
> > >>
> > >> Sure, I agree it's better to have clear guidelines and scheme. Let me
> > fork this thread about that.
> > >>
> > >> Re 2.7.0, I just forgot about the naming initially though I was clear
> > in the discussion/voting. I so had to end up calling it alpha outside of
> > the release artifact naming.
> > >>
> > >> Thanks
> > >> +Vinod
> > >>
> > >> On Apr 21, 2015, at 4:26 PM, Andrew Wang
> > wrote:
> > >>
> > >>> I would also like to support Karthik's proposal on the release thread
> > about
> > >>> version numbering. 2.7.0 being "alpha" is pretty confusing since all
> > of the
> > >>> other 2.x releases since GA have been stable. I
[jira] [Created] (MAPREDUCE-6342) Make POM project names consistent
Rohith created MAPREDUCE-6342: - Summary: Make POM project names consistent Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Release numbering for stable 2.8 and beyond
I think Karthik correctly identified the two reasons we might want to
denote a release as "unstable":
a) Compatibility. Whether we have freedom to break compatibility before the
final release, i.e. what "alpha" typically means.
b) Quality. As Konst says, a release can be allowed to break compatibility
("alpha") but itself still be a high quality release.
We could try and separate out these two concerns when it comes to
versioning, but I think in reality users only care about prod vs. non-prod,
which is why many other projects (Linux, HBase, etc) just have two release
lines: "stable" (compatible/good-quality) vs. "unstable"
(unknown-compatible/unknown-quality).
To this end, I don't mind what we choose, as long as the difference between
stable and unstable is denoted by the version number. I don't like the idea
of tagging a release as good after the fact (1). The other projects we've
used as examples make strong statements about their stable release lines,
and we should too. Our downstreams and end users will appreciate clarity
from the version number.
Best,
Andrew
On Sun, Apr 26, 2015 at 9:51 AM, Hitesh Shah wrote:
> There are a couple of different approaches we could take.
>
> How about publishing/releasing bits such as “2.8.0-RC0”. Downstream
> projects can depend on these and use them normally similar to the approach
> that they would have taken with release 2.8.0 or 2.8.0-alpha. After some
> baking ( or more RC suffixed releases), at some point, a proper 2.8.0
> release could be made? The RC tag clearly indicates to downstream layers
> that things will be in flux slightly. Trying to distinguish
> incompatibilities between 2.8.0 and 2.8.1/2.8.2 ( just because 2.8.0 was
> tagged alpha in documentation ) are likely to be a nightmare to deal with
> especially for new features introduced in the 2.8 release ( which might get
> changed slightly based on usage feedback ).
>
> Furthermore, considering the release history of hadoop, the likelihood
> that 2.9.0 will show up before 2.8.2 seems to be very high.
>
> With respect to the proposed choice (1), I thought that HBase applied a
> different approach. The odd-number releases were always unstable and the
> even number releases were the stable ones. This “could" make things a bit
> more clear for downstream users without needing to resort to modified
> versioned numbers ( with alpha/beta tags ) or requiring users to go look up
> the website to find out which particular version of 2.8.x was the first
> stable release.
>
> thanks
> — Hitesh
>
>
>
>
>
> On Apr 22, 2015, at 2:17 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
>
> > Forking the thread.
> >
> > In the previous 2.7.1 thread [1], there were enough yays to my proposal
> to wait for a bug-fix release or two before calling a 2.x release stable.
> There were some concerns about the naming.
> >
> > We have two options, taking 2.8 as an example
> > (1) Release 2.8.0, call it as an alpha in documentation and release
> notes, wait for a 2.8.1/2.8.2 reasonably stable enough to be called as the
> first stable release of 2.8.
> > (2) Release 2.8.0-alpha, 2.8.0-beta etc before culminating in a 2.8.0
> stable release.
> >
> > (1) is what I preferred first up. This is what HBase used to do, and far
> beyond, in the linux kernel releases. It helps in scenarios where we are
> forced to downgrade a release, say due to major issues. We can simply
> announce it as not stable retroactively, change the pointers on our website
> and move on.
> >
> > Thoughts?
> >
> > Thanks,
> > +Vinod
> >
> > [1] http://markmail.org/thread/ogzk4phj6wsdpssu
> >
> > On Apr 21, 2015, at 4:59 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
> >
> >>
> >> Sure, I agree it's better to have clear guidelines and scheme. Let me
> fork this thread about that.
> >>
> >> Re 2.7.0, I just forgot about the naming initially though I was clear
> in the discussion/voting. I so had to end up calling it alpha outside of
> the release artifact naming.
> >>
> >> Thanks
> >> +Vinod
> >>
> >> On Apr 21, 2015, at 4:26 PM, Andrew Wang
> wrote:
> >>
> >>> I would also like to support Karthik's proposal on the release thread
> about
> >>> version numbering. 2.7.0 being "alpha" is pretty confusing since all
> of the
> >>> other 2.x releases since GA have been stable. I think users would
> prefer a
> >>> version like "2.8.0-alpha1" instead, which is very clear and similar to
> >>> what we did for 2.0 and 2.1. Then we release 2.8.0 when we're actually
> >>> stable.
> >>>
> >>> I don't know if it's retroactively possible to do this for 2.7.0, but
> it's
> >>> something to consider for the next 2.7 alpha or beta or whatever.
> >>>
> >
>
>
Hadoop-Mapreduce-trunk - Build # 2126 - Failure
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2126/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 31993 lines...] Results : Tests in error: TestClusterMRNotification>NotificationTestCase.setUp:145->HadoopTestCase.setUp:157 » YarnRuntime Tests run: 520, Failures: 0, Errors: 1, Skipped: 11 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [ 2.714 s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [01:51 min] [INFO] hadoop-mapreduce-client-common SUCCESS [ 30.935 s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [ 4.679 s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [12:54 min] [INFO] hadoop-mapreduce-client-hs SUCCESS [05:59 min] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [ 01:45 h] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] hadoop-mapreduce-client-nativetask SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:07 h [INFO] Finished at: 2015-04-27T15:34:40+00:00 [INFO] Final Memory: 34M/779M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-mapreduce-client-jobclient: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk #2125 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 20329379 bytes Compression is 0.0% Took 6.8 sec Recording test results Updating HADOOP-11857 Updating YARN-3464 Updating MAPREDUCE-6057 Updating HADOOP-11865 Updating HADOOP-11357 Updating MAPREDUCE-6252 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 1 tests failed. REGRESSION: org.apache.hadoop.mapred.TestClusterMRNotification.testMR Error Message: java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/server/MiniYARNCluster$NodeManagerWrapper$1 Stack Trace: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/server/MiniYARNCluster$NodeManagerWrapper$1 at org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper.serviceStart(MiniYARNCluster.java:576) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:80) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:187) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:175) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:167) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:159) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:152) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:145) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:138) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:133) at org.apache.hadoop.mapred.HadoopTestCase.setUp(HadoopTestCase.java:157) at org.apache.hadoop.mapred.NotificationTestC
Hadoop-Mapreduce-trunk-Java8 - Build # 177 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/177/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 26974 lines...] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.808 sec - in org.apache.hadoop.mapred.TestReporter Results : Failed tests: TestCombineFileInputFormat.testSplitPlacementForCompressedFiles:918 expected:<2> but was:<1> TestCombineFileInputFormat.testSplitPlacement:372 expected:<2> but was:<1> Tests run: 509, Failures: 2, Errors: 0, Skipped: 11 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [ 2.091 s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [01:38 min] [INFO] hadoop-mapreduce-client-common SUCCESS [ 24.876 s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [ 4.009 s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [13:23 min] [INFO] hadoop-mapreduce-client-hs SUCCESS [07:14 min] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [ 02:00 h] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] hadoop-mapreduce-client-nativetask SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:22 h [INFO] Finished at: 2015-04-27T15:30:30+00:00 [INFO] Final Memory: 33M/192M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-mapreduce-client-jobclient: There was a timeout or other error in the fork -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk-Java8 #28 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 20341150 bytes Compression is 0.0% Took 8.4 sec Recording test results Updating HADOOP-11857 Updating YARN-3464 Updating MAPREDUCE-6057 Updating HADOOP-11865 Updating HADOOP-11357 Updating MAPREDUCE-6252 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 2 tests failed. FAILED: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles Error Message: expected:<2> but was:<1> Stack Trace: java.lang.AssertionError: expected:<2> but was:<1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:918) FAILED: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement Error Message: expected:<2> but was:<1> Stack Trace: java.lang.AssertionError: expected:<2> but was:<1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:372)
[jira] [Created] (MAPREDUCE-6340) Remove .xml and documentation references to dfs.webhdfs.enabled
Ray Chiang created MAPREDUCE-6340: - Summary: Remove .xml and documentation references to dfs.webhdfs.enabled Key: MAPREDUCE-6340 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6340 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Fix For: 3.0.0 HDFS-7985 removed any references in the code to the dfs.webhdfs.enabled property. The property should also be removed from hdfs-default.xml as well as other places. As mentioned in HDFS-7985, this is an incompatible change with branch-2, so this fix should be limited to trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
