[jira] [Created] (HDFS-11716) Revisit delete container API
Weiwei Yang created HDFS-11716: -- Summary: Revisit delete container API Key: HDFS-11716 URL: https://issues.apache.org/jira/browse/HDFS-11716 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Weiwei Yang Assignee: Weiwei Yang Current delete container API seems can be possibly running into inconsistent state. SCM maintains a mapping of container to nodes, datanode maintains the actual container's data. When deletes a container, we need to make sure db is removed as well as the mapping in SCM also gets updated. What if the datanode failed to remove stuff for a container, do we update the mapping? We need to revisit the implementation and get these issues addressed. See more discussion [here|https://issues.apache.org/jira/browse/HDFS-11675?focusedCommentId=15987798=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15987798] -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/385/ [Apr 21, 2017 6:35:36 PM] (wang) HADOOP-14261. Some refactoring work for erasure coding raw coder. [Apr 21, 2017 8:36:31 PM] (yzhang) HDFS-11689. New exception thrown by DFSClient%isHDFSEncryptionEnabled [Apr 21, 2017 11:12:01 PM] (rkanter) MAPREDUCE-6871. Allow users to specify racks and nodes for strict [Apr 22, 2017 3:35:48 AM] (yzhang) HDFS-11402. HDFS Snapshots should capture point-in-time copies of OPEN [Apr 22, 2017 9:56:48 AM] (jzhuge) HDFS-11529. Add libHDFS API to return last exception. Contributed by [Apr 24, 2017 7:58:57 AM] (sunilg) YARN-6291. Introduce query parameters (sort, filter, etc.) for tables to [Apr 24, 2017 5:03:28 PM] (epayne) YARN-5617. AMs only intended to run one attempt can be run more than [Apr 24, 2017 6:37:52 PM] (haibochen) YARN-6500. Do not mount inaccessible cgroups directories in [Apr 24, 2017 7:33:19 PM] (stevel) HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in [Apr 24, 2017 8:17:35 PM] (wangda) YARN-6392. Add submit time to Application Summary log. (Zhihai Xu via [Apr 25, 2017 10:30:42 PM] (jlowe) HDFS-11691. Add a proper scheme to the datanode links in NN web UI. [Apr 25, 2017 11:26:09 PM] (rkanter) YARN-6423. Queue metrics doesn't work for Fair Scheduler in SLS (yufeigu [Apr 26, 2017 1:33:41 AM] (liuml07) HADOOP-12856. FileUtil.checkDest() and RawLocalFileSystem.mkdirs() to [Apr 26, 2017 1:45:56 AM] (xyao) HADOOP-14207. "dfsadmin -refreshCallQueue" fails with DecayRpcScheduler. [Apr 26, 2017 3:54:56 AM] (liuml07) HADOOP-14349. Rename ADLS CONTRACT_ENABLE_KEY. Contributed by Mingliang [Apr 26, 2017 5:06:51 AM] (jzhuge) HADOOP-14341. Support multi-line value for [Apr 26, 2017 6:34:23 AM] (aajisaka) Revert "YARN-5007. Remove deprecated constructors of MiniYARNCluster and [Apr 26, 2017 6:57:00 AM] (cdouglas) HDFS-6708. StorageType should be encoded in the block token. Contributed [Apr 26, 2017 8:43:29 AM] (aajisaka) HDFS-11417. Add datanode admin command to get the storage info. [Apr 26, 2017 6:44:59 PM] (jzhuge) HADOOP-14340. Enable KMS and HttpFS to exclude SSL ciphers. Contributed [Apr 26, 2017 6:46:55 PM] (haibochen) YARN-6510. Fix profs stat file warning caused by process names that [Apr 26, 2017 8:47:18 PM] (liuml07) HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and [Apr 27, 2017 12:28:49 AM] (shv) HDFS-11384. Balancer disperses getBlocks calls to avoid NameNode's rpc [Apr 27, 2017 6:45:33 AM] (aajisaka) MAPREDUCE-6881. Fix warnings from Spotbugs in hadoop-mapreduce. [Apr 27, 2017 7:12:27 AM] (aajisaka) HDFS-11697. Add javadoc for storage policy and erasure coding policy. [Apr 27, 2017 7:57:08 AM] (aajisaka) HADOOP-11614. Remove httpclient dependency from hadoop-openstack. [Apr 27, 2017 9:31:58 AM] (varunsaxena) YARN-6455. Enhance the timelinewriter.flush() race condition fix (Haibo [Apr 27, 2017 11:19:04 AM] (stevel) HADOOP-14346. CryptoOutputStream throws IOException if stream is closed. -1 overall The following subsystems voted -1: asflicense findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-common-project/hadoop-minikdc Possible null pointer dereference in org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called method Dereferenced at MiniKdc.java:org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called method Dereferenced at MiniKdc.java:[line 368] FindBugs : module:hadoop-common-project/hadoop-auth org.apache.hadoop.security.authentication.server.MultiSchemeAuthenticationHandler.authenticate(HttpServletRequest, HttpServletResponse) makes inefficient use of keySet iterator instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:of keySet iterator instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:[line 192] FindBugs : module:hadoop-common-project/hadoop-common org.apache.hadoop.crypto.CipherSuite.setUnknownValue(int) unconditionally sets the field unknownValue At CipherSuite.java:unknownValue At CipherSuite.java:[line 44] org.apache.hadoop.crypto.CryptoProtocolVersion.setUnknownValue(int) unconditionally sets the field unknownValue At CryptoProtocolVersion.java:unknownValue At CryptoProtocolVersion.java:[line 67] Possible null pointer dereference in org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of called method Dereferenced at FileUtil.java:org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of called method Dereferenced at FileUtil.java:[line 118] Possible null pointer dereference in
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/296/ [Apr 21, 2017 6:35:36 PM] (wang) HADOOP-14261. Some refactoring work for erasure coding raw coder. [Apr 21, 2017 8:36:31 PM] (yzhang) HDFS-11689. New exception thrown by DFSClient%isHDFSEncryptionEnabled [Apr 21, 2017 11:12:01 PM] (rkanter) MAPREDUCE-6871. Allow users to specify racks and nodes for strict [Apr 22, 2017 3:35:48 AM] (yzhang) HDFS-11402. HDFS Snapshots should capture point-in-time copies of OPEN [Apr 22, 2017 9:56:48 AM] (jzhuge) HDFS-11529. Add libHDFS API to return last exception. Contributed by [Apr 24, 2017 7:58:57 AM] (sunilg) YARN-6291. Introduce query parameters (sort, filter, etc.) for tables to [Apr 24, 2017 5:03:28 PM] (epayne) YARN-5617. AMs only intended to run one attempt can be run more than [Apr 24, 2017 6:37:52 PM] (haibochen) YARN-6500. Do not mount inaccessible cgroups directories in [Apr 24, 2017 7:33:19 PM] (stevel) HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in [Apr 24, 2017 8:17:35 PM] (wangda) YARN-6392. Add submit time to Application Summary log. (Zhihai Xu via [Apr 25, 2017 10:30:42 PM] (jlowe) HDFS-11691. Add a proper scheme to the datanode links in NN web UI. [Apr 25, 2017 11:26:09 PM] (rkanter) YARN-6423. Queue metrics doesn't work for Fair Scheduler in SLS (yufeigu [Apr 26, 2017 1:33:41 AM] (liuml07) HADOOP-12856. FileUtil.checkDest() and RawLocalFileSystem.mkdirs() to [Apr 26, 2017 1:45:56 AM] (xyao) HADOOP-14207. "dfsadmin -refreshCallQueue" fails with DecayRpcScheduler. [Apr 26, 2017 3:54:56 AM] (liuml07) HADOOP-14349. Rename ADLS CONTRACT_ENABLE_KEY. Contributed by Mingliang [Apr 26, 2017 5:06:51 AM] (jzhuge) HADOOP-14341. Support multi-line value for [Apr 26, 2017 6:34:23 AM] (aajisaka) Revert "YARN-5007. Remove deprecated constructors of MiniYARNCluster and [Apr 26, 2017 6:57:00 AM] (cdouglas) HDFS-6708. StorageType should be encoded in the block token. Contributed [Apr 26, 2017 8:43:29 AM] (aajisaka) HDFS-11417. Add datanode admin command to get the storage info. [Apr 26, 2017 6:44:59 PM] (jzhuge) HADOOP-14340. Enable KMS and HttpFS to exclude SSL ciphers. Contributed [Apr 26, 2017 6:46:55 PM] (haibochen) YARN-6510. Fix profs stat file warning caused by process names that [Apr 26, 2017 8:47:18 PM] (liuml07) HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and [Apr 27, 2017 12:28:49 AM] (shv) HDFS-11384. Balancer disperses getBlocks calls to avoid NameNode's rpc [Apr 27, 2017 6:45:33 AM] (aajisaka) MAPREDUCE-6881. Fix warnings from Spotbugs in hadoop-mapreduce. [Apr 27, 2017 7:12:27 AM] (aajisaka) HDFS-11697. Add javadoc for storage policy and erasure coding policy. [Apr 27, 2017 7:57:08 AM] (aajisaka) HADOOP-11614. Remove httpclient dependency from hadoop-openstack. [Apr 27, 2017 9:31:58 AM] (varunsaxena) YARN-6455. Enhance the timelinewriter.flush() race condition fix (Haibo [Apr 27, 2017 11:19:04 AM] (stevel) HADOOP-14346. CryptoOutputStream throws IOException if stream is closed. -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.ha.TestZKFailoverControllerStress hadoop.hdfs.TestDFSStripedOutputStreamWithFailure hadoop.hdfs.TestReadStripedFileWithMissingBlocks hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.server.blockmanagement.TestBlockManager hadoop.hdfs.server.namenode.ha.TestBootstrapStandby hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure hadoop.hdfs.TestRollingUpgrade hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.mapred.TestShuffleHandler hadoop.tools.TestHadoopArchiveLogsRunner hadoop.metrics2.impl.TestKafkaMetrics hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.yarn.client.api.impl.TestAMRMClient hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager
[jira] [Created] (HDFS-11715) Ozone: SCM : Add priority for datanode commands
Anu Engineer created HDFS-11715: --- Summary: Ozone: SCM : Add priority for datanode commands Key: HDFS-11715 URL: https://issues.apache.org/jira/browse/HDFS-11715 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Anu Engineer Assignee: Anu Engineer While reviewing HDFS-11493, [~cheersyang] commented that it would be a good idea to support priority for datanode commands send from SCM. bq. The queue seems to be time ordered, I think it will be better to support priority as well. Commands may have different priority, for example, replicate a container priority is usually higher than delete a container replica; replicate a container also may have different priorities according to the number of replicas This jira tracks that work item. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion
Kihwal Lee created HDFS-11714: - Summary: Newly added NN storage directory won't get initialized and cause space exhaustion Key: HDFS-11714 URL: https://issues.apache.org/jira/browse/HDFS-11714 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.3 Reporter: Kihwal Lee Priority: Critical When an empty namenode storage directory is detected on normal NN startup, it may not be fully initialized. The new directory is still part of "in-service" NNStrage and when a checkpoint image is uploaded, a copy will also be written there. However, the retention manager won't be able to purge old files since it is lacking a VERSION file. This causes fsimages to pile up in the directory. With a big name space, the disk will be filled in the order of days or weeks. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11713) Use MoveFileEx to allow renaming a file when the destination exists
Lukas Majercak created HDFS-11713: - Summary: Use MoveFileEx to allow renaming a file when the destination exists Key: HDFS-11713 URL: https://issues.apache.org/jira/browse/HDFS-11713 Project: Hadoop HDFS Issue Type: Bug Components: hdfs, native, rolling upgrades Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.7.3, 2.7.1 Reporter: Lukas Majercak Assignee: Lukas Majercak Fix For: 3.0.0-alpha2 The {{NativeIO.c#renameTo0}} currently uses {{MoveFile}} Windows system call, which fails when renaming a file to a destination that already exists. This makes the {{TestRollingUpgrade.testRollback}} test fail on Windows, as during that execution, a DataNode tries to rename block's meta file to a destination that exists. The proposal is to change to using {{MoveFileEx}} Windows call, and passing in {{MOVEFILE_REPLACE_EXISTING}} flag to force the renaming. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-8873) Allow the directoryScanner to be rate-limited
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang reopened HDFS-8873: - > Allow the directoryScanner to be rate-limited > - > > Key: HDFS-8873 > URL: https://issues.apache.org/jira/browse/HDFS-8873 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Daniel Templeton > Labels: 2.7.2-candidate > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, > HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch, > HDFS-8873.006.patch, HDFS-8873.007.patch, HDFS-8873.008.patch, > HDFS-8873.009.patch, HDFS-8873-branch-2.7.009.patch > > > The new 2-level directory layout can make directory scans expensive in terms > of disk seeks (see HDFS-8791) for details. > It would be good if the directoryScanner() had a configurable duty cycle that > would reduce its impact on disk performance (much like the approach in > HDFS-8617). > Without such a throttle, disks can go 100% busy for many minutes at a time > (assuming the common case of all inodes in cache but no directory blocks > cached, 64K seeks are required for full directory listing which translates to > 655 seconds) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11712) Ozone: Reuse ObjectMapper instance to improve the performance
Yiqun Lin created HDFS-11712: Summary: Ozone: Reuse ObjectMapper instance to improve the performance Key: HDFS-11712 URL: https://issues.apache.org/jira/browse/HDFS-11712 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Yiqun Lin Assignee: Yiqun Lin -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11711) DN should not delete the block On "Too many open files" Exception
Brahma Reddy Battula created HDFS-11711: --- Summary: DN should not delete the block On "Too many open files" Exception Key: HDFS-11711 URL: https://issues.apache.org/jira/browse/HDFS-11711 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula *Seen the following scenario in one of our customer environment* * while jobclient writing {{"job.xml"}} there are pipeline failures and written to only one DN. * when mapper reading the {{"job.xml"}}, DN got {{"Too many open files"}} (as system exceed limit) and block got deleted. Hence mapper failed to read and job got failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11710) hadoop-hdfs-native-client build fails in trunk after HDFS-11529
Vinayakumar B created HDFS-11710: Summary: hadoop-hdfs-native-client build fails in trunk after HDFS-11529 Key: HDFS-11710 URL: https://issues.apache.org/jira/browse/HDFS-11710 Project: Hadoop HDFS Issue Type: Bug Components: native Affects Versions: 3.0.0-alpha3 Reporter: Vinayakumar B Priority: Blocker HDFS-11529 used 'hdfsThreadDestructor()' in jni_helper.c. But this function is implemented in only "posix/thread_local_storage.c" NOT in "windows/thread_local_storage.c" Fails with following errors {noformat} [exec] hdfs.dir\RelWithDebInfo\thread_local_storage.obj /machine:x64 /debug [exec] Creating library D:/hadoop/work/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/bin/RelWithDebInfo/hdfs.lib and object D:/hadoop/work/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/bin/RelWithDebInfo/hdfs.exp [exec] jni_helper.obj : error LNK2019: unresolved external symbol hdfsThreadDestructor referenced in function getJNIEnv [D:\hadoop\work\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\hdfs.vcxproj] [exec] D:\hadoop\work\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo\hdfs.dll : fatal error LNK1120: 1 unresolved externals [D:\hadoop\work\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\hdfs.vcxproj] {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: About 2.7.4 Release
Thanks Allen for the additional information. > At one point, JIRA was configured to refuse re-opening after a release is cut. In the past, release manager closed the tickets and the process is written in the wiki: https://wiki.apache.org/hadoop/HowToRelease > 10. In JIRA, close issues resolved in the release. Disable mail notifications for this bulk change. Therefore, let's close them. On 2017/04/27 3:01, Allen Wittenauer wrote: On Apr 25, 2017, at 12:35 AM, Akira Ajisakawrote: Maybe we should create a jira to track this? I think now either way (reopen or create) is fine. Release doc maker creates change logs by fetching information from JIRA, so reopening the tickets should be avoided when a release process is in progress. Keep in mind that the release documentation is part of the build process. Users who are doing their own builds will have incomplete documentation if we keep re-opening JIRAs after a release. At one point, JIRA was configured to refuse re-opening after a release is cut. I'm not sure why it stopped doing that, but it might be time to see if we can re-enable that functionality. - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org