[VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0
Hi all, As always, thanks to the many, many contributors who helped with this release! I've prepared an RC0 for 3.0.0-alpha4: http://home.apache.org/~wang/3.0.0-alpha4-RC0/ The standard 5-day vote would run until midnight on Tuesday, July 4th. Given that July 4th is a holiday in the US, I expect this vote might have to be extended, but I'd like to close the vote relatively soon after. I've done my traditional testing of a pseudo-distributed cluster with a single task pi job, which was successful. Normally my testing would end there, but I'm slightly more confident this time. At Cloudera, we've successfully packaged and deployed a snapshot from a few days ago, and run basic smoke tests. Some bugs found from this include HDFS-11956, which fixes backwards compat with Hadoop 2 clients, and the revert of HDFS-11696, which broke NN QJM HA setup. Vijay is working on a test run with a fuller test suite (the results of which we can hopefully post soon). My +1 to start, Best, Andrew
[jira] [Resolved] (HDFS-11752) getNonDfsUsed return 0 if reserved bigger than actualNonDfsUsed
[
https://issues.apache.org/jira/browse/HDFS-11752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong resolved HDFS-11752.
---
Resolution: Not A Problem
> getNonDfsUsed return 0 if reserved bigger than actualNonDfsUsed
> ---
>
> Key: HDFS-11752
> URL: https://issues.apache.org/jira/browse/HDFS-11752
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs
>Affects Versions: 2.7.1
>Reporter: maobaolong
> Labels: datanode, hdfs
> Fix For: 2.7.1
>
>
> {code}
> public long getNonDfsUsed() throws IOException {
> long actualNonDfsUsed = getActualNonDfsUsed();
> if (actualNonDfsUsed < reserved) {
> return 0L;
> }
> return actualNonDfsUsed - reserved;
> }
> {code}
> The code block above is the function to caculate nonDfsUsed, but in fact it
> will let the result to be 0L out of expect. Such as this following situation:
> du.reserved = 50G
> Disk Capacity = 2048G
> Disk Available = 2000G
> Dfs used = 30G
> usage.getUsed() = dirFile.getTotalSpace() - dirFile.getFreeSpace()
> = 2048G - 2000G
> = 48G
> getActualNonDfsUsed = usage.getUsed() - getDfsUsed()
> = 48G - 30G
> = 18G
> 18G < 50G, so the function `getNonDfsUsed` actualNonDfsUsed < reserved, and
> the NonDfsUsed will return 0, is that logic make sense?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Heads up: branching 3.0.0-alpha4, use -beta1 for new commits to trunk
Hi folks, I'm in the process of moving out all the JIRA versions from 3.0.0-alpha4 to 3.0.0-beta1 in preparation for a 3.0.0-alpha4 release. I'm hoping to get an RC up tomorrow for a vote, with possibly extending it given the holiday week in the US for July 4th. Please use the beta1 target/fix version from now on unless committing to branch-3.0.0-alpha4. We're not planning on an alpha5. Best, Andrew
[jira] [Created] (HDFS-12071) Ozone: PetaGen: Implementation of PetaGen
Nandakumar created HDFS-12071:
-
Summary: Ozone: PetaGen: Implementation of PetaGen
Key: HDFS-12071
URL: https://issues.apache.org/jira/browse/HDFS-12071
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Nandakumar
Assignee: Nandakumar
Tool to populate ozone with data for testing.
This is not a map-reduce program and this is not for benchmarking Ozone write
throughput.
It supports both online and offline modes. Default mode is offline, {{-mode}}
can be used to change the mode.
In online mode, active internet connection is required, common crawl data from
AWS will be used. Default source is
https://commoncrawl.s3.amazonaws.com/CC-MAIN-2017-17/warc.paths.gz (it contains
the path to actual data segment), user can override this using {{-source}}.
The following values are derived from URL of Common Crawl data
* Domain will be used as Volume
* URL will be used as Bucket
* FileName will be used as Key
In offline mode, the data will be random bytes and size of data will be 10 KB.
* Default number of Volumes 10, {{-numOfVolumes}} can be used to override
* Default number of Buckets per Volume 1000, {{-numOfBuckets}} can be used to
override
* Default number of Keys per Bucket 50, {{-numOfKeys}} can be used to
override
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12041) Block Storage : make the server address config more concise
[
https://issues.apache.org/jira/browse/HDFS-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang resolved HDFS-12041.
---
Resolution: Won't Fix
> Block Storage : make the server address config more concise
> ---
>
> Key: HDFS-12041
> URL: https://issues.apache.org/jira/browse/HDFS-12041
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Minor
>
> Currently there are a few places where the address are read from config like
> such
> {code}
> String cbmIPAddress = ozoneConf.get(
> DFS_CBLOCK_JSCSI_CBLOCK_SERVER_ADDRESS_KEY,
> DFS_CBLOCK_JSCSI_CBLOCK_SERVER_ADDRESS_DEFAULT
> );
> int cbmPort = ozoneConf.getInt(
> DFS_CBLOCK_JSCSI_PORT_KEY,
> DFS_CBLOCK_JSCSI_PORT_DEFAULT
> );
> {code}
> Similarly for jscsi address config. Maybe we should consider merge these to
> one single key config in form of host:port.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/ [Jun 28, 2017 5:54:25 PM] (weichiu) HDFS-11870. Add CLI cmd to enable/disable an erasure code policy. [Jun 28, 2017 8:17:58 PM] (templedf) YARN-6743. yarn.resourcemanager.zk-max-znode-size.bytes description [Jun 28, 2017 8:47:23 PM] (lei) Add -E option in 'ls' to list erasure coding policy of each file and [Jun 28, 2017 8:53:54 PM] (weichiu) HADOOP-14297. Update the documentation about the new ec codecs config [Jun 28, 2017 9:18:59 PM] (liuml07) HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if [Jun 28, 2017 10:32:04 PM] (junping_du) YARN-5311. Document graceful decommission CLI and usage. Contributed by [Jun 28, 2017 10:40:58 PM] (sunilg) YARN-6280. Introduce deselect query param to skip ResourceRequest from [Jun 29, 2017 12:33:44 AM] (aw) MAPREDUCE-6536. hadoop-pipes doesn't use maven properties for openssl [Jun 29, 2017 3:26:36 AM] (kai.zheng) HADOOP-14479. Erasurecode testcase failures with native enabled. -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestDiskFailures hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-mvninstall-root.txt [500K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-compile-root.txt [20K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-compile-root.txt [20K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-compile-root.txt [20K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-unit-hadoop-assemblies.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [468K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K]
[jira] [Created] (HDFS-12070) Failed block recovery leaves files open indefinitely and at risk for data loss
Daryn Sharp created HDFS-12070: -- Summary: Failed block recovery leaves files open indefinitely and at risk for data loss Key: HDFS-12070 URL: https://issues.apache.org/jira/browse/HDFS-12070 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Daryn Sharp Files will remain open indefinitely if block recovery fails which creates a high risk of data loss. The replication monitor will not replicate these blocks. The NN provides the primary node a list of candidate nodes for recovery which involves a 2-stage process. The primary node removes any candidates that cannot init replica recovery (essentially alive and knows about the block) to create a sync list. Stage 2 issues updates to the sync list – _but fails if any node fails_ unlike the first stage. The NN should be informed of nodes that did succeed. Manual recovery will also fail until the problematic node is temporarily stopped so a connection refused will induce the bad node to be pruned from the candidates. Recovery succeeds, the lease is released, under replication is fixed, and block is invalidated from the bad node. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12069) Ozone: Create a general abstraction for metadata store
Weiwei Yang created HDFS-12069: -- Summary: Ozone: Create a general abstraction for metadata store Key: HDFS-12069 URL: https://issues.apache.org/jira/browse/HDFS-12069 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Weiwei Yang Assignee: Weiwei Yang Create a general abstraction for metadata store so that we can plug other key value store to host ozone metadata. Currently only levelDB is implemented, we want to support RocksDB as it provides more production ready features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/ [Jun 28, 2017 6:49:09 AM] (xiao) HADOOP-14515. Addendum. Specifically configure zookeeper-related log [Jun 28, 2017 9:22:13 AM] (stevel) HADOOP-14190. Add more on S3 regions to the s3a documentation. [Jun 28, 2017 5:54:25 PM] (weichiu) HDFS-11870. Add CLI cmd to enable/disable an erasure code policy. [Jun 28, 2017 8:17:58 PM] (templedf) YARN-6743. yarn.resourcemanager.zk-max-znode-size.bytes description [Jun 28, 2017 8:47:23 PM] (lei) Add -E option in 'ls' to list erasure coding policy of each file and [Jun 28, 2017 8:53:54 PM] (weichiu) HADOOP-14297. Update the documentation about the new ec codecs config [Jun 28, 2017 9:18:59 PM] (liuml07) HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if [Jun 28, 2017 10:32:04 PM] (junping_du) YARN-5311. Document graceful decommission CLI and usage. Contributed by [Jun 28, 2017 10:40:58 PM] (sunilg) YARN-6280. Introduce deselect query param to skip ResourceRequest from [Jun 29, 2017 12:33:44 AM] (aw) MAPREDUCE-6536. hadoop-pipes doesn't use maven properties for openssl [Jun 29, 2017 3:26:36 AM] (kai.zheng) HADOOP-14479. Erasurecode testcase failures with native enabled. -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Useless object stored in variable removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:[line 642] org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache() makes inefficient use of keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:[line 719] Hard coded reference to an absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:[line 455] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus() makes inefficient use of keySet iterator instead of entrySet iterator At ContainerLocalizer.java:keySet iterator instead of entrySet iterator At ContainerLocalizer.java:[line 334] org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics is a mutable collection which should be package protected At ContainerMetrics.java:which should be package protected At ContainerMetrics.java:[line 134] Failed junit tests : hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 hadoop.fs.contract.hdfs.TestHDFSContractCreate hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestAMRMClient cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-compile-javac-root.txt [192K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-patch-shelldocs.txt [12K] whitespace:
