Re: [VOTE] Release Apache Hadoop 2.8.4 (RC0)
Thanks for the work Junping! +1 (non-binding) - checked out git tag release-2.8.4-RC0 - hadoop-aws unit tests ran successfully - built from source on Mac OS X 10.13.4, java 8.0.171-oracle - deployed on a 3 node cluster (HDFS HA, Non-HA YARN) - verified pi job (yarn), teragen, terasort and teravalidate Regards, Gabor Bota On Wed, May 9, 2018 at 12:14 AM Ajay Kumar wrote: > Thanks for work on this, Junping!! > > +1 (non-binding) > - verified binary checksum > - built from source and setup 4 node cluster > - run basic hdfs command > - run wordcount, pi & TestDFSIO (read/write) > - basic check for NN UI > > Best, > Ajay > > On 5/8/18, 10:41 AM, "俊平堵" wrote: > > Hi all, > I've created the first release candidate (RC0) for Apache Hadoop > 2.8.4. This is our next maint release to follow up 2.8.3. It includes > 77 > important fixes and improvements. > > The RC artifacts are available at: > http://home.apache.org/~junping_du/hadoop-2.8.4-RC0 > > The RC tag in git is: release-2.8.4-RC0 > > The maven artifacts are available via repository.apache.org< > http://repository.apache.org> at: > > https://repository.apache.org/content/repositories/orgapachehadoop-1118 > > Please try the release and vote; the vote will run for the usual 5 > working days, ending on 5/14/2018 PST time. > > Thanks, > > Junping > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >
[jira] [Created] (HADOOP-15452) Snappy Decpmpressor met ArrayIndexOutOfBoundsException when reduce task fetch map output data
jincheng created HADOOP-15452:
-
Summary: Snappy Decpmpressor met ArrayIndexOutOfBoundsException
when reduce task fetch map output data
Key: HADOOP-15452
URL: https://issues.apache.org/jira/browse/HADOOP-15452
Project: Hadoop Common
Issue Type: Bug
Components: common
Affects Versions: 2.6.0
Environment: hadoop -2.6.0-cdh5.4.4
Reporter: jincheng
RT, when reducers tasks fetch data from mapper tasks, it met
ArrayIndexOutOfBoundsException, here is stackTrace:
{code:java}
// code placeholder
{code}
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in fetcher#1 at
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:379) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:165) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:160) Caused by:
java.lang.ArrayIndexOutOfBoundsException at
org.apache.hadoop.io.compress.snappy.SnappyDecompressor.setInput(SnappyDecompressor.java:111)
at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:104)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199) at
org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.shuffle(InMemoryMapOutput.java:98)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:549)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:346)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:202)
Anyone has ideas?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
For more details, see https://builds.apache.org/job/hadoop-trunk-win/462/ [May 8, 2018 12:41:33 PM] (msingh) HDDS-23. Remove SCMNodeAddressList from SCMRegisterRequestProto. [May 8, 2018 6:34:45 PM] (wangda) YARN-8232. RMContainer lost queue name when RM HA happens. (Hu Ziqian [May 8, 2018 8:30:36 PM] (jlowe) YARN-8207. Docker container launch use popen have risk of shell [May 8, 2018 8:49:41 PM] (billie) YARN-7894. Improve ATS response for DShell DS_CONTAINER when container [May 8, 2018 10:29:54 PM] (fabbri) HADOOP-13649 s3guard: implement time-based (TTL) expiry for [May 9, 2018 1:58:42 AM] (fabbri) HADOOP-15420 s3guard ITestS3GuardToolLocal failures in diff tests (Gabor [May 9, 2018 4:51:41 AM] (msingh) HDDS-33. Ozone : Fix the test logic in [May 9, 2018 6:30:43 AM] (rohithsharmaks) YARN-8239. [UI2] Clicking on Node Manager UI under AM container info / [May 9, 2018 6:34:31 AM] (rohithsharmaks) YARN-8264. [UI2] GPU Info tab disappears if we click any sub link under [May 9, 2018 6:47:04 AM] (rohithsharmaks) YARN-8260. [UI2] Per-application tracking URL is no longer available in -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 00m 00s) unit Specific tests: Failed junit tests : hadoop.security.authentication.util.TestZKSignerSecretProvider hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec hadoop.fs.contract.rawlocal.TestRawlocalContractAppend hadoop.fs.TestFsShellCopy hadoop.fs.TestFsShellList hadoop.fs.TestLocalFileSystem hadoop.fs.TestRawLocalFileSystemContract hadoop.fs.TestSymlinkLocalFSFileContext hadoop.fs.TestTrash hadoop.http.TestHttpServer hadoop.http.TestHttpServerLogs hadoop.io.nativeio.TestNativeIO hadoop.ipc.TestSocketFactory hadoop.metrics2.impl.TestStatsDMetrics hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal hadoop.security.TestSecurityUtil hadoop.security.TestShellBasedUnixGroupsMapping hadoop.security.token.delegation.TestZKDelegationTokenSecretManager hadoop.security.token.TestDtUtilShell hadoop.util.curator.TestChildReaper hadoop.util.TestNativeCodeLoader hadoop.util.TestNodeHealthScriptRunner hadoop.crypto.key.kms.server.TestKMS hadoop.crypto.key.kms.server.TestKMSWithZK hadoop.fs.TestResolveHdfsSymlink hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy hadoop.hdfs.crypto.TestHdfsCryptoStreams hadoop.hdfs.qjournal.client.TestQuorumJournalManager hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.qjournal.server.TestJournalNodeSync hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage hadoop.hdfs.server.datanode.TestBlockRecovery hadoop.hdfs.server.datanode.TestBlockScanner hadoop.hdfs.server.datanode.TestDataNodeFaultInjector hadoop.hdfs.server.datanode.TestDataNodeMetrics hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.datanode.TestHSync hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.mover.TestStorageMover hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.namenode.ha.TestDNFencing hadoop.hdfs.server.namenode.ha.TestHASafeMode hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints hadoop.hdfs.server.name
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/ [May 8, 2018 5:05:19 AM] (sunilg) YARN-5151. [UI2] Support kill application from new YARN UI. Contributed [May 8, 2018 6:07:03 AM] (sunilg) YARN-8251. [UI2] Clicking on Application link at the header goes to [May 8, 2018 6:58:54 AM] (rohithsharmaks) YARN-8253. HTTPS Ats v2 api call fails with 'bad HTTP parsed'. [May 8, 2018 12:41:33 PM] (msingh) HDDS-23. Remove SCMNodeAddressList from SCMRegisterRequestProto. [May 8, 2018 6:34:45 PM] (wangda) YARN-8232. RMContainer lost queue name when RM HA happens. (Hu Ziqian [May 8, 2018 8:30:36 PM] (jlowe) YARN-8207. Docker container launch use popen have risk of shell [May 8, 2018 8:49:41 PM] (billie) YARN-7894. Improve ATS response for DShell DS_CONTAINER when container [May 8, 2018 10:29:54 PM] (fabbri) HADOOP-13649 s3guard: implement time-based (TTL) expiry for -1 overall The following subsystems voted -1: asflicense findbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-hdds/common Found reliance on default encoding in org.apache.hadoop.utils.MetadataKeyFilters$KeyPrefixFilter.filterKey(byte[], byte[], byte[]):in org.apache.hadoop.utils.MetadataKeyFilters$KeyPrefixFilter.filterKey(byte[], byte[], byte[]): String.getBytes() At MetadataKeyFilters.java:[line 97] Failed junit tests : hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean hadoop.hdfs.web.TestWebHdfsTimeouts cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/diff-compile-javac-root.txt [328K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-hdds_client.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-hdds_common-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [32K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [20K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-ozone_common.txt [24K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-ozone_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/branch-findbugs-hadoop-tools_hadoop-ozone.txt [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/diff-javadoc-javadoc-root.txt [760K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [332K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/776/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [84K] https://
[jira] [Created] (HADOOP-15453) hadoop fs -count -v report "-count: Illegal option -v"
zhoutai.zt created HADOOP-15453:
---
Summary: hadoop fs -count -v report "-count: Illegal option -v"
Key: HADOOP-15453
URL: https://issues.apache.org/jira/browse/HADOOP-15453
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.7.2
Reporter: zhoutai.zt
[hadoop@Hadoop1 bin]$ ./hadoop fs -count -q -h -v SparkHis*
-count: Illegal option -v
Reading the source code, can't find the -v option.
{code:java}
// code placeholder
private static final String OPTION_QUOTA = "q";
private static final String OPTION_HUMAN = "h";
public static final String NAME = "count";
public static final String USAGE =
"[-" + OPTION_QUOTA + "] [-" + OPTION_HUMAN + "] ...";
{code}
BUT the document states the -v option.
[http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/FileSystemShell.html#count]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
RE: [VOTE] Release Apache Hadoop 2.8.4 (RC0)
Thanks for working on this, Junping! +1 (non-binding) - verified checksums - succeeded "mvn clean package -Pdist,native -Dtar -DskipTests" (CentOS 7) - started hadoop cluster with 1 master and 5 slaves - run TeraGen/TeraSort - verified Web UI (NN, RM, JobHistory, Timeline) - verified some Archival Storage operations Thanks, -Takanobu > -Original Message- > From: Gabor Bota [mailto:gabor.b...@cloudera.com] > Sent: Wednesday, May 09, 2018 5:20 PM > To: ajay.ku...@hortonworks.com > Cc: junping...@apache.org; Hadoop Common ; > Hdfs-dev ; mapreduce-...@hadoop.apache.org; > yarn-...@hadoop.apache.org > Subject: Re: [VOTE] Release Apache Hadoop 2.8.4 (RC0) > > Thanks for the work Junping! > > +1 (non-binding) > >- checked out git tag release-2.8.4-RC0 >- hadoop-aws unit tests ran successfully >- built from source on Mac OS X 10.13.4, java 8.0.171-oracle >- deployed on a 3 node cluster (HDFS HA, Non-HA YARN) >- verified pi job (yarn), teragen, terasort and teravalidate > > > Regards, > Gabor Bota > > On Wed, May 9, 2018 at 12:14 AM Ajay Kumar > wrote: > > > Thanks for work on this, Junping!! > > > > +1 (non-binding) > > - verified binary checksum > > - built from source and setup 4 node cluster > > - run basic hdfs command > > - run wordcount, pi & TestDFSIO (read/write) > > - basic check for NN UI > > > > Best, > > Ajay > > > > On 5/8/18, 10:41 AM, "俊平堵" wrote: > > > > Hi all, > > I've created the first release candidate (RC0) for Apache Hadoop > > 2.8.4. This is our next maint release to follow up 2.8.3. It includes > > 77 > > important fixes and improvements. > > > > The RC artifacts are available at: > > http://home.apache.org/~junping_du/hadoop-2.8.4-RC0 > > > > The RC tag in git is: release-2.8.4-RC0 > > > > The maven artifacts are available via repository.apache.org< > > http://repository.apache.org> at: > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-111 > 8 > > > > Please try the release and vote; the vote will run for the usual > 5 > > working days, ending on 5/14/2018 PST time. > > > > Thanks, > > > > Junping > > > > > > > > - > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
