[jira] [Resolved] (HDFS-16238) Improve comments related to EncryptionZoneManager
[
https://issues.apache.org/jira/browse/HDFS-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HDFS-16238.
Fix Version/s: 3.4.0
Resolution: Fixed
Thanks [~vjasani] [~hexiaoqiao]for the review!
> Improve comments related to EncryptionZoneManager
> -
>
> Key: HDFS-16238
> URL: https://issues.apache.org/jira/browse/HDFS-16238
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, encryption, namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> In EncryptionZoneManager, there are some missing
> The description of the relevant comment. The purpose of this jira is to
> perfect them.
> E.g:
>/**
> * Re-encrypts the given encryption zone path. If the given path is not the
> * root of an encryption zone, an exception is thrown.
> * @param zoneIIP
> * @param keyVersionName
> * @throws IOException
> */
>List reencryptEncryptionZone(final INodesInPath zoneIIP,
>final String keyVersionName) throws IOException {
> ..
> }
> The description of zoneIIP and keyVersionName is missing here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/437/ No changes - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/644/ [Sep 29, 2021 9:48:52 AM] (noreply) HADOOP-17941. Update xerces to 2.12.1 (#3496) [Sep 29, 2021 9:54:17 AM] (noreply) HADOOP-17198. Support S3 Access Points (#3260) [Sep 29, 2021 12:42:07 PM] (noreply) HADOOP-17851. S3A to support user-specified content encoding (#3498) [Sep 30, 2021 12:25:29 AM] (noreply) HADOOP-17905. Modify Text.ensureCapacity() to efficiently max out the… (#3423) [Sep 30, 2021 9:38:53 AM] (noreply) HADOOP-17922. move to fs.s3a.encryption.algorithm - JCEKS integration (#3466) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16245) Print the number of loaded INodeDirectory when starting the NameNode service
JiangHua Zhu created HDFS-16245: --- Summary: Print the number of loaded INodeDirectory when starting the NameNode service Key: HDFS-16245 URL: https://issues.apache.org/jira/browse/HDFS-16245 Project: Hadoop HDFS Issue Type: Improvement Reporter: JiangHua Zhu When starting parallel loading of FsImage (dfs.image.parallel.load=true), you can see in the startup log that the number of loaded INodes has been printed, but it is impossible to know the specific number of loaded INodeDirectory, because it is not printed now. For example, here are some startup information: 2021-09-30 19:12:55,012 [15587]-INFO [main:FSImageFormatProtobuf$Loader@340]-The fsimage will be loaded in parallel using 4 threads 2021-09-30 19:12:55,031 [15606]-INFO [main:FSImageFormatPBINode$Loader@418]-Loading the INode section in parallel with 12 sub-sections 2021-09-30 19:12:55,034 [15609]-INFO [main:FSImageFormatPBINode$Loader@409]-Loading INodes. .. 2021-09-30 19:30:37,080 [1077655]-INFO [main:FSImageFormatPBINode$Loader@465]-Completed loading all INode sections. Loaded inodes. 2021-09-30 19:30:37,086 [1077661]-INFO [main:FSImageFormatPBINode$Loader@222]-Loading the INodeDirectory section in parallel with 12 sub-sections .. 2021-09-30 19:36:58,074 [1458649]-INFO [main:FSImageFormatPBINode$Loader@261]-Completed loading all INodeDirectory sub-sections 2021-09-30 19:36:58,076 [1458651]-INFO [main:FSImageFormatPBINode$Loader@339]-Completed update blocks map and name cache, total waiting duration 1 ms. 2021-09-30 19:36:58,111 [1458686]-INFO [main:FSImageFormatProtobuf$Loader@248]-Loaded FSImage in seconds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/ [Sep 28, 2021 4:50:01 PM] (noreply) MAPREDUCE-7363. Rename JobClientUnitTest to TestJobClients (#3487) [Sep 28, 2021 4:51:18 PM] (noreply) YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics (#3439) [Sep 29, 2021 6:46:55 AM] (noreply) YARN-10970. Standby RM should expose prom endpoint (#3480) -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.TestViewDistributedFileSystemContract hadoop.hdfs.server.namenode.TestFSImage hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.namenode.TestFsck hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes hadoop.hdfs.server.namenode.TestNamenodeStorageDirectives hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile hadoop.hdfs.server.namenode.ha.TestHAStateTransitions hadoop.hdfs.server.namenode.ha.TestEditLogTailer hadoop.hdfs.server.datanode.TestBlockScanner hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.namenode.TestAuditLogs hadoop.hdfs.web.TestWebHdfsFileSystemContract hadoop.hdfs.server.datanode.TestReadOnlySharedStorage hadoop.hdfs.server.namenode.TestFSNamesystemLockReport hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure hadoop.yarn.csi.client.TestCsiClient hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.TestDynamometerInfra cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/results-compile-javac-root.txt [364K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/results-javadoc-javadoc-root.txt [408K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [2.1M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/643/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt [12K]
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.ha.TestZKFailoverController hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/diff-compile-javac-root.txt [496K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/patch-mvnsite-root.txt [584K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/diff-patch-pylint.txt [48K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/diff-patch-shelldocs.txt [48K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/patch-javadoc-root.txt [32K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [236K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [428K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [40K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [124K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/436/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K]
Re: [DISCUSS] Migrate to Yetus Interface classification annotations
Thanks Masatake for the suggestions. I agree that until Yetus comes to final conclusion on whether to keep or drop IA/IS annotations for higher JDK versions (or fix/drop doclet support), we should hold on for now. Thanks Sean and Akira for providing the context. On Tue, 28 Sep 2021 at 6:55 PM, Masatake Iwasaki < iwasak...@oss.nttdata.co.jp> wrote: > Thanks, Akira and Viraj. > > My understanding is that we have options like > > 1. migrate org.apache.yetus:audience-annotations to Java >= 9 then > migrate Hadoop to the new org.apache.yetus:audience-annotations. > > 2. "use the Jigsaw feature to export only @Public elements to other > projects > and create javadoc by new --show-packages=exported option instead of > relying on the annotations." as mentioned by Akira[1]. > > Both require dropping Java 8 support. > > If current org.apache.yetus:audience-annotations(:0.13.0) for Java 8 no > longer evolves, > migrating to it in short term is not much useful? > > [1] > https://issues.apache.org/jira/browse/HADOOP-15304?focusedCommentId=16418072=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16418072 > > On 2021/09/28 18:38, Viraj Jasani wrote: > >> The problem comes from the removal of com.sun.tools.doclets.* packages > > > > Agree. Here is the summary > > < > https://docs.oracle.com/en/java/javase/11/docs/api/jdk.javadoc/jdk/javadoc/doclet/package-summary.html > > > > of the replacement package *jdk.javadoc.doclet*. > > Here is the migration guide > > < > https://docs.oracle.com/en/java/javase/11/docs/api/jdk.javadoc/jdk/javadoc/doclet/package-summary.html#migration > > > > for the same. > > > > On Tue, Sep 28, 2021 at 1:06 PM Akira Ajisaka > wrote: > > > >> Hi Masatake, > >> > >> The problem comes from the removal of com.sun.tools.doclets.* packages > in > >> Java 10. > >> In Apache Hadoop, I removed the doclet support for filtering javadocs > when > >> the environment is Java 10 or upper. > >> https://issues.apache.org/jira/browse/HADOOP-15304 > >> > >> Thanks, > >> Akira > >> > >> On Tue, Sep 28, 2021 at 10:27 AM Masatake Iwasaki < > >> iwasak...@oss.nttdata.co.jp> wrote: > >> > In particular, there has been an outstanding problem with doclet > >> support > >>> for filtering javadocs by annotation since JDK9 came out. > >>> > >>> Could you give me a pointer to relevant Yetus JIRA or ML thread? > >>> > >>> On 2021/09/28 1:17, Sean Busbey wrote: > I think consolidating on a common library and tooling for defining API > >>> expectations for Hadoop would be great. > > Unfortunately, the Apache Yetus community recently started a > discussion > >>> around dropping their maintenance of the audience annotations > codebase[1] > >>> due to lack of community interest. In particular, there has been an > >>> outstanding problem with doclet support for filtering javadocs by > >>> annotation since JDK9 came out. > > I think that means a necessary first step here would be to determine > if > >>> we have contributors willing to show up over in that project to get > >> things > >>> into a good state for future JDK adoption. > > > > [1]: > https://s.apache.org/ybdl6 > "[DISCUSS] Drop JDK8; audience-annotations" from d...@yetus.apache.org > > > On Sep 27, 2021, at 2:46 AM, Viraj Jasani > wrote: > > > > Since the early days, Hadoop has provided Interface classification > > annotations to represent the scope and stability for downstream > > applications to select Hadoop APIs carefully. After some time, these > > annotations (InterfaceAudience and InterfaceStability) have been > >>> migrated > > to Apache Yetus. As of today, with increasing number of Hadoop > >> ecosystem > > applications using (or starting to use) Yetus stability annotations > >> for > > their own downstreamers, we should also consider using IA/IS > >> annotations > > provided by *org.apache.yetus.audience *directly in our codebase and > >>> retire > > our *org.apache.hadoop.classification* package for the better > >>> separation of > > concern and single source. > > > > I believe we can go with this migration to maintain compatibility for > > Hadoop downstreamers: > > > > 1. In Hadoop trunk (3.4.0+ releases), replace all usages of > o.a.h.c > > stability annotations with o.a.y.a annotations. > > 2. Deprecate o.a.h.c annotations, and provide deprecation warning > >>> that > > we will remove o.a.h.c in 4.0.0 (or 5.0.0) release and the only > >>> source for > > these annotations should be o.a.y.a. > > > > Any thoughts? > > > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > >>> > >>> - > >>>
