[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046823#comment-17046823 ] Hudson commented on HDFS-15124: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18004 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18004/]) HDFS-15124. Crashing bugs in NameNode when using a valid configuration (ayushsaxena: rev cd2c6b1aac470991b9b90339ce2721ba179e7c48) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/top/TopAuditLogger.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Fix For: 3.3.0 > > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch, HDFS-15124.006.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046803#comment-17046803 ] Ayush Saxena commented on HDFS-15124: - Committed to trunk. Thanx [~ctest.team] for the contribution and [~elgoiri] for the review!!! > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch, HDFS-15124.006.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger =
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046791#comment-17046791 ] Ayush Saxena commented on HDFS-15124: - v006 LGTM +1 > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch, HDFS-15124.006.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046069#comment-17046069 ] Íñigo Goiri commented on HDFS-15124: +1 on [^HDFS-15124.006.patch]. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch, HDFS-15124.006.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045194#comment-17045194 ] Ayush Saxena commented on HDFS-15124: - Thanx [~ctest.team] for the update. The v006 LGTM [~elgoiri] can you give a check too. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch, HDFS-15124.006.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger)
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044894#comment-17044894 ] Ctest commented on HDFS-15124: -- [~ayushtkn] Thank you a lot for your help! I have submitted a new patch (006) with the unit test. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch, HDFS-15124.006.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger)
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044889#comment-17044889 ] Hadoop QA commented on HDFS-15124: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 47s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 5s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 10s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 27s{color} | {color:red} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}175m 34s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes | | | hadoop.hdfs.TestMultipleNNPortQOP | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.6 Server=19.03.6 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | HDFS-15124 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12994575/HDFS-15124.006.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7138fd446b27 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / dda00d3 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_242 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28844/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/28844/testReport/ | | Max. process+thread count | 2842 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/28844/console | | Powered by | Apache
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043171#comment-17043171 ] Ctest commented on HDFS-15124: -- [~ayushtkn] Thank you a lot! I will try to write the test and submit the patch later! > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance();
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042695#comment-17042695 ] Ayush Saxena commented on HDFS-15124: - Thanx Everyone for the work here. [~ctest.team] you can check {{TestFsNamesystem.java}} there is a {{testInitAuditLoggers()}} which already tests most of the things. You can add your case also in the same test. You can add at the end in same the same test and do something like this to repro your issue : {code:java} // Test Configuring TopAuditLogger. conf.set(DFSConfigKeys.DFS_NAMENODE_AUDIT_LOGGERS_KEY, "org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger"); fsn = new FSNamesystem(conf, fsImage); auditLoggers = fsn.getAuditLoggers(); assertEquals(1, auditLoggers.size()); assertThat(auditLoggers.get(0), instanceOf(TopAuditLogger.class)); {code} Let me know, if you intend to write test for something else and facing trouble, will try to help write one. :) > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027901#comment-17027901 ] Íñigo Goiri commented on HDFS-15124: Let's try to do it in this JIRA if it doesn't take too long. It is usually better to have complete patches. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger)
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027881#comment-17027881 ] Ctest commented on HDFS-15124: -- I can try to write a test to do this, it may take some time. Should I incorporate the test into this patch? > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger)
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027868#comment-17027868 ] Íñigo Goiri commented on HDFS-15124: Before closing this... just making sure there is no quick test we can add for this. What about initialize one with and without the TopAuditLogger and checking that in each case it does the right thing? > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Assignee: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027853#comment-17027853 ] Hadoop QA commented on HDFS-15124: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 58s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 49s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 41s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}185m 41s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDeadNodeDetection | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | HDFS-15124 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12992358/HDFS-15124.005.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9ec3af45c2b7 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / bf8686f | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_232 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28730/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/28730/testReport/ | | Max. process+thread count | 3013 (vs. ulimit of
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027790#comment-17027790 ] Ctest commented on HDFS-15124: -- [~elgoiri] Thank you a lot for pointing out this! The 004 patch has already fixed the checkstyle issue. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch, > HDFS-15124.005.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027778#comment-17027778 ] Hadoop QA commented on HDFS-15124: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 49s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 35s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 35s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 1s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}172m 19s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDeadNodeDetection | | | hadoop.hdfs.TestMultipleNNPortQOP | | | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | HDFS-15124 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12992348/HDFS-15124.004.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1e49660b6b58 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / bf8686f | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_232 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28729/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/28729/testReport/ | | Max. process+thread count | 2974 (vs. ulimit of 5500) | | modules | C:
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027151#comment-17027151 ] Hadoop QA commented on HDFS-15124: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color} | {color:red} HDFS-15124 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-15124 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12992292/HDFS-15124.004.patch | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/28727/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch, HDFS-15124.004.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027144#comment-17027144 ] Íñigo Goiri commented on HDFS-15124: Let's split the line to fix the checkstyle. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } > logger.initialize(conf); > auditLoggers.add(logger); > } catch
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027140#comment-17027140 ] Hadoop QA commented on HDFS-15124: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 43s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 47s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 167 unchanged - 0 fixed = 168 total (was 167) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 36s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}111m 12s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}175m 44s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | HDFS-15124 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12992282/HDFS-15124.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 61bf8783bffa 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a7d72c5 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_232 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/28726/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28726/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results |
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027070#comment-17027070 ] Ctest commented on HDFS-15124: -- [~elgoiri] Thank you for pointing this out. I have already uploaded a new patch for this. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch, HDFS-15124.003.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } > logger.initialize(conf); >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027055#comment-17027055 ] Íñigo Goiri commented on HDFS-15124: The tests look reasonable. We may want to fix the checkstyle. Actually, is there a chance we can do: {{code}} if (TopAuditLogger.class.getName().equals(logger.getClass().getName())) { {{code}} > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger)
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027046#comment-17027046 ] Ctest commented on HDFS-15124: -- [~elgoiri] I have already run the 4 failed test classes with my patch in the official docker image and all of them passed successfully. I feel like the failures are not about the content in the patch. Actually the content in the patch won't be executed if not setting `dfs.namenode.audit.loggers` to `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. Could you please help to check whether these failures are due to some flakiness in tests? Thank you a lot! > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017653#comment-17017653 ] Hadoop QA commented on HDFS-15124: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 41s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 11s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 166 unchanged - 0 fixed = 167 total (was 166) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 25s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 13s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}191m 22s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDeadNodeDetection | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.namenode.TestRedudantBlocks | | | hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | HDFS-15124 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12991181/HDFS-15124.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 0fb99bc6d2d0 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 263413e | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_232 | | findbugs | v3.1.0-RC1 | | checkstyle |
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017619#comment-17017619 ] Ctest commented on HDFS-15124: -- Uploaded one new patch for trunk. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch, > HDFS-15124.002.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } > logger.initialize(conf); > auditLoggers.add(logger); > } catch (RuntimeException re) { > throw re; >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017471#comment-17017471 ] Hadoop QA commented on HDFS-15124: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 51s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 29s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 166 unchanged - 0 fixed = 167 total (was 166) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 33s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}134m 8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}198m 52s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDecommission | | | hadoop.hdfs.server.namenode.snapshot.TestSnapshot | | | hadoop.hdfs.server.namenode.TestINodeAttributeProvider | | | hadoop.hdfs.TestDatanodeRegistration | | | hadoop.hdfs.server.namenode.TestRedudantBlocks | | | hadoop.hdfs.server.namenode.TestFSImage | | | hadoop.hdfs.server.namenode.TestFsck | | | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | HDFS-15124 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12991162/HDFS-15124.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 8fac4d4b27be 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a0ff42d | | maven | version: Apache Maven
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017418#comment-17017418 ] Ctest commented on HDFS-15124: -- I was using hadoop-2.10.0 and the FSNameSystem was using {code:java} public static final Log LOG = LogFactory.getLog(FSNamesystem.class);{code} [https://github.com/apache/hadoop/blob/e2f1f118e465e787d8567dfa6e2f3b72a0eb9194/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L328] Should I use the 3.x.x version to do the fix? It seems that 3.x.x is using org.slf4j.Logger but 2.x.x is using org.apache.commons.logging.Log for FSNanesystem.java > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017346#comment-17017346 ] Íñigo Goiri commented on HDFS-15124: FSNameSystem seems to be using logger, right? https://github.com/apache/hadoop/blob/a0ff42d7612e744e0bf88aa14078ea3ab6bcce49/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L373 It was done by HDFS-12552. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017329#comment-17017329 ] Ctest commented on HDFS-15124: -- I uploaded a new patch to do all these changes. [~elgoiri] I am sorry I didn't use {} style for the LOG in FSNamesystem.java because in FSNamesystem.java the LOG is `org.apache.commons.logging.Log` instead of `org.slf4j.Logger`. The `org.apache.commons.logging.Log` doesn't support {code:java} LOG.error("xxx {}", "yyy"){code} Please let me know if anything else needed. Thank you! > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch, HDFS-15124.001.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016458#comment-17016458 ] Ctest commented on HDFS-15124: -- [~elgoiri] [~weichiu] Sure. I will upload a new patch to do that. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } > logger.initialize(conf); > auditLoggers.add(logger); > } catch (RuntimeException re) { > throw re; > } catch
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016261#comment-17016261 ] Wei-Chiu Chuang commented on HDFS-15124: Please make sure you don't add TopAuditLogger twice by setting dfs.namenode.top.enabled to TopAuditLogger. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } > logger.initialize(conf); > auditLoggers.add(logger); > } catch
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016215#comment-17016215 ] Íñigo Goiri commented on HDFS-15124: Thanks [~ctest.team] for the proposal, yes, I think it makes sense. Just use the logger style with {}. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } > logger.initialize(conf); > auditLoggers.add(logger); > } catch (RuntimeException
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015638#comment-17015638 ] Hadoop QA commented on HDFS-15124: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 45s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 34s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 25s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 21s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}180m 15s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.TestRedudantBlocks | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | HDFS-15124 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12990931/HDFS-15124.000.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 778f53fdcbb8 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / c36f09d | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_232 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28675/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/28675/testReport/ | | Max. process+thread count | 3894 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output |
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015581#comment-17015581 ] Ctest commented on HDFS-15124: -- [~elgoiri] How about catch InstantiationException in initAuditLoggers(Configuration conf) It will look like: {code:java} private List initAuditLoggers(Configuration conf) { // Initialize the custom access loggers if configured. Collection alClasses = conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); List auditLoggers = Lists.newArrayList(); if (alClasses != null && !alClasses.isEmpty()) { for (String className : alClasses) { try { AuditLogger logger; if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { logger = new DefaultAuditLogger(); } else { logger = (AuditLogger) Class.forName(className).newInstance(); } logger.initialize(conf); auditLoggers.add(logger); } catch (InstantiationException e) { LOG.error("Instantiation Error For " + className); throw new RuntimeException(e); } catch (RuntimeException re) { throw re; } catch (Exception e) { throw new RuntimeException(e); } } } {code} The log error message can be refined. I can upload another patch for this. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015488#comment-17015488 ] Íñigo Goiri commented on HDFS-15124: In addition to the constructor, the exception for this case is a little verbose and not intuitive. Is there a way we can improve that too? One can follow the full stack trace but it is buried... > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782) > Caused by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger > at java.lang.Class.newInstance(Class.java:427) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.() > at java.lang.Class.getConstructor0(Class.java:3082) > at java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } >
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015478#comment-17015478 ] Ctest commented on HDFS-15124: -- [~weichiu] Thank you for your reply! Yes. The default value can also add TopAuditLogger, but most users didn't read the src code and don't know it. If the users want to use TopAuditLogger and they directly set it to TopAuditLogger (without understanding the src code), then the namenode will crash. I wrote a patch to add the default constructor for the TopAuditLogger which I think can make this part more robust. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > Attachments: HDFS-15124.000.patch > > > I am using Hadoop-2.10.0. > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > *Symptom* > *$ ./start-dfs.sh* > {code:java} > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLoggerat > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024)at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858)at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677)at > > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674)at > > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736)at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961)at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940)at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714)at > > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782)Caused > by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLoggerat > java.lang.Class.newInstance(Class.java:427)at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 moreCaused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.()at > java.lang.Class.getConstructor0(Class.java:3082)at > java.lang.Class.newInstance(Class.java:412) > ... 9 more{code} > > > *Detailed Root Cause* > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > {code:java} > /** > * An {@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > private final TopMetrics topMetrics; > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > @Override > public void initialize(Configuration conf) { > } > {code} > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > {code:java} > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if
[jira] [Commented] (HDFS-15124) Crashing bugs in NameNode when using a valid configuration for `dfs.namenode.audit.loggers`
[ https://issues.apache.org/jira/browse/HDFS-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015456#comment-17015456 ] Wei-Chiu Chuang commented on HDFS-15124: You shouldn't need to add TopAuditLogger to auditlogger configuration. As long as dfs.namenode.top.enabled is true (default), it is added. > Crashing bugs in NameNode when using a valid configuration for > `dfs.namenode.audit.loggers` > --- > > Key: HDFS-15124 > URL: https://issues.apache.org/jira/browse/HDFS-15124 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.10.0 >Reporter: Ctest >Priority: Critical > > I am using Hadoop-2.10.0. > > The configuration parameter `dfs.namenode.audit.loggers` allows `default` > (which is the default value) and > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`. > > When I use `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > namenode will not be started successfully because of an > `InstantiationException` thrown from > `org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers`. > > The root cause is that while initializing namenode, `initAuditLoggers` will > be called and it will try to call the default constructor of > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger` which doesn't > have a default constructor. Thus the `InstantiationException` exception is > thrown. > > > > Symptom > > $ ./start-dfs.sh > > > > 2019-12-18 14:05:20,670 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed.java.lang.RuntimeException: > java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLoggerat > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1024)at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:858)at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:677)at > > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674)at > > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736)at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:961)at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:940)at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714)at > > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782)Caused > by: java.lang.InstantiationException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLoggerat > java.lang.Class.newInstance(Class.java:427)at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initAuditLoggers(FSNamesystem.java:1017)... > 8 moreCaused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger.()at > java.lang.Class.getConstructor0(Class.java:3082)at > java.lang.Class.newInstance(Class.java:412) > ... 9 more > > > > > Detailed Root Cause > > There is no default constructor in > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`: > > /** > * An \{@link AuditLogger} that sends logged data directly to the metrics > * systems. It is used when the top service is used directly by the name node > */ > @InterfaceAudience.Private > public class TopAuditLogger implements AuditLogger { > public static finalLogger LOG = > LoggerFactory.getLogger(TopAuditLogger.class); > > private final TopMetrics topMetrics; > > public TopAuditLogger(TopMetrics topMetrics) { > Preconditions.checkNotNull(topMetrics, "Cannot init with a null " + > "TopMetrics"); > this.topMetrics = topMetrics; > } > > @Override > public void initialize(Configuration conf) { > } > As long as the configuration parameter `dfs.namenode.audit.loggers` is set to > `org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger`, > `initAuditLoggers` will try to call its default constructor to make a new > instance: > > private List initAuditLoggers(Configuration conf) { > // Initialize the custom access loggers if configured. > Collection alClasses = > conf.getTrimmedStringCollection(DFS_NAMENODE_AUDIT_LOGGERS_KEY); > List auditLoggers = Lists.newArrayList(); > if (alClasses != null && !alClasses.isEmpty()) { > for (String className : alClasses) { > try { > AuditLogger logger; > if (DFS_NAMENODE_DEFAULT_AUDIT_LOGGER_NAME.equals(className)) { > logger = new DefaultAuditLogger(); > } else { > logger = (AuditLogger) Class.forName(className).newInstance(); > } > logger.initialize(conf); > auditLoggers.add(logger); > } catch (RuntimeException re)