On Aug 17, 2013, at 8:20 AM, Gary Gregory wrote: > Detective Nick is one the case! :)
Thank, Gary! I don't give up. I do NOT like not knowing why something is working in an unexpected way. It means I don't know something I should. > Do those other projects use third party line for this or roll their own? Tomcat rolls its own, AFAICT. Spring and Hibernate use third-party libraries for sure. I'm going to look into what's necessary. My understanding is we would get both a startup performance boost /and/ use less memory by not loading every class to scan for annotations. Question: WHAT does Log4j scan to look for plugins? Does it scan every class in the JAR (in which case the performance and memory improvements would be minor), or does it scan every class on the entire class path (in which case the performance and memory improvements would me major)? The larger the likely improvements, the more effort we should invest it making this happen. Nick > On Aug 17, 2013, at 4:17, Nick Williams <[email protected]> wrote: > >> Solved it! >> >> And you're never gonna believe what I learned tonight...(well, maybe you >> will) >> >> I solved the error by changing this: >> >> public final class MongoDBProvider implements >> NoSQLProvider<MongoDBConnection> { >> ... >> + try { >> + if (!database.authenticate(username, >> password.toCharArray())) { >> + LOGGER.error("Failed to authenticate against >> MongoDB server. Unknown error."); >> + } >> + } catch (MongoException e) { >> + LOGGER.error("Failed to authenticate against MongoDB: " >> + e.getMessage(), e); >> + } catch (IllegalStateException e) { >> + LOGGER.error("Factory-supplied MongoDB database >> connection already authenticated with different" + >> + "credentials but lost connection."); >> + } >> ... >> } >> >> To this: >> >> public final class MongoDBProvider implements >> NoSQLProvider<MongoDBConnection> { >> ... >> + MongoDBConnection.authenticate(database, username, >> password); >> ... >> } >> >> public final class MongoDBConnection implements >> NoSQLConnection<BasicDBObject, MongoDBObject> { >> ... >> + static void authenticate(final DB database, final String username, >> final String password) { >> + try { >> + if (!database.authenticate(username, password.toCharArray())) { >> + LOGGER.error("Failed to authenticate against MongoDB >> server. Unknown error."); >> + } >> + } catch (final MongoException e) { >> + LOGGER.error("Failed to authenticate against MongoDB: " + >> e.getMessage(), e); >> + } catch (final IllegalStateException e) { >> + LOGGER.error("Factory-supplied MongoDB database connection >> already authenticated with different" + >> + "credentials but lost connection."); >> + } >> + } >> ... >> } >> >> Crazy, right!? Here's what I've learned: >> >> The errors were occurring in tests for the Log4j 1.2 API and the SLF4J >> Bridge. These tests use the core Logger which triggers plugin discovery. In >> order to scan for annotations, plugin discovery loads the MongoDBProvider, >> CouchDBProvider, and JPAAppender classes, among many others, all of which >> have transitive dependencies that are not on the classpath for running the >> unit tests for Log4j 1.2 API and SLF4J. So how did it ever work in the first >> place? >> >> As you may already know, when Java loads a class it also automatically loads >> any classes it extends or implements, any classes that are the types of >> static members of that class, any static inner classes, and any classes used >> within the static initializer. It doesn't load any other classes that class >> uses in any methods or constructors or as instance members--like >> com.mongodb.DB or javax.persistence.*--until the code that uses them >> actually executes for the first time. Because of this, we can do something >> like load the MongoDBProvider class to scan for @Plugin annotations even >> though the com.mongodb classes it uses are not on the classpath (as long as >> they aren't static members of or extended by the MongoDBProvider, that is). >> >> However, Java has a special behavior with exceptions. Because we have these >> lovely things called checked exceptions that methods must declare to be >> thrown, exceptions are naturally part of a class's interface. Thus, when >> Java loads a class it must load the exceptions the class's methods might >> throw so that it can complete the interface in memory >> (java.lang.Class.getMethod("someMethod").getExceptionTypes()). Likely for >> performance reasons, it doesn't differentiate between exceptions that are >> actually declared to be thrown and exceptions that are just used (caught). >> ANY dependent classes that are exceptions are loaded when the class loads, >> even if they're just caught exceptions. This is why this all worked until I >> started using an exception from a transitive dependency within a plugin >> class (MongoDBProvider). >> >> (Incidentally, it's also why more advanced class-scanning projects like >> Spring, Hibernate and Tomcat don't load classes using a ClassLoader just to >> scan for annotations. Instead, they inspect the byte code manually to scan >> for annotations, preventing such class loading errors during discovery >> phases and also saving memory resources since Classes aren't usually garbage >> collected. It might be worthwhile to look into doing something similar in >> Log4j plugin discovery. I don't know how much effort would be involved.) >> >> I haven't confirmed any of this with JLS documentation because no amount of >> Google searching for the combination of "class loading" and "exception" >> brings up anything other than 10,000,000 people asking questions about >> what's wrong with their classpath. I simply can't find that needle in a >> planet full of haystacks. But my thorough experimentation has some pretty >> clear results. This is exactly what's happening. >> >> Nick >> >> On Aug 17, 2013, at 12:44 AM, Ralph Goers wrote: >> >>> I'll reiterate what I wrote. Catch the RuntimeException and then do >>> >>> if (e.class.getName().equals("com.mongodb.MongoException")) { >>> LOGGER.error("..."); >>> } else { >>> throw e; >>> } >>> >>> This should give you the same behavior. >>> >>> Ralph >>> >>> On Aug 16, 2013, at 9:49 PM, Nick Williams wrote: >>> >>>> That approach concerns me. Catching RuntimeException essentially opens it >>>> up to thousands of possible exceptions that could be the cause, as opposed >>>> to looking for that exact cause. I suppose I don't have a choice, though. >>>> This apparently just isn't going to work. >>>> >>>> Definitely agreed that there is too much going on for a simple Exception >>>> class. >>>> >>>> :-/ >>>> >>>> Nick >>>> >>>> On Aug 16, 2013, at 11:45 PM, Ralph Goers wrote: >>>> >>>>> After following the chain of stuff that gets brought in via BSONObject I >>>>> still recommend the approach in my other email of just catching >>>>> RuntimeException. A bunch of other classes are being referenced, one of >>>>> which is creating a static Logger from java.util.logging. I have no idea >>>>> why that might fail but there is just way too much going on for a simple >>>>> Exception class. >>>>> >>>>> Ralph >>>>> >>>>> On Aug 16, 2013, at 9:26 PM, Nick Williams wrote: >>>>> >>>>>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/DB.java >>>>>> >>>>>> That also shows an import for org.bson.BSONObject, but the tests still >>>>>> run with just DB and no MongoException. org.bson is in the >>>>>> org.mongodb:mongo-java-driver JAR file. So, no, that's not the problem. >>>>>> There's something else... >>>>>> >>>>>> Nick >>>>>> >>>>>> On Aug 16, 2013, at 11:20 PM, Ralph Goers wrote: >>>>>> >>>>>>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/MongoException.java >>>>>>> shows an import for org.bson.BSONObject. The pom.xml for >>>>>>> mongo-java-driver doesn't contain a transitive dependency for that and >>>>>>> mvn dependency:tree on core doesn't show it. >>>>>>> >>>>>>> Ralph >>>>>>> >>>>>>> >>>>>>> On Aug 16, 2013, at 3:48 PM, Nick Williams wrote: >>>>>>> >>>>>>>> Guys, I'm having a hard time with this simple fix that should have >>>>>>>> taken five minutes. I'm getting test failures due to >>>>>>>> NoClassDefFoundErrors that shouldn't happen. >>>>>>>> >>>>>>>> Here are the tests in error: >>>>>>>> CategoryTest.setupClass:52 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testTraceWithException:415 ? NoClassDefFound >>>>>>>> com/mongodb/MongoExcep... >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testLog:459 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testRB1:295 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testRB2:314 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testRB3:334 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testTrace:388 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testAdditivity1:119 ? NoClassDefFound >>>>>>>> com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testAdditivity2:144 ? NoClassDefFound >>>>>>>> com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testAdditivity3:183 ? NoClassDefFound >>>>>>>> com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testIsTraceEnabled:443 ? NoClassDefFound >>>>>>>> com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.testExists:355 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggingTest.setupClass:44 ? NoClassDefFound com/mongodb/MongoException >>>>>>>> LoggingTest.cleanupClass:49 NullPointer >>>>>>>> >>>>>>>> Here's the code I added: >>>>>>>> >>>>>>>> try { >>>>>>>> if (!database.authenticate(username, >>>>>>>> password.toCharArray())) { >>>>>>>> LOGGER.error("Failed to authenticate against >>>>>>>> MongoDB server. Unknown error."); >>>>>>>> } >>>>>>>> } catch (MongoException e) { >>>>>>>> LOGGER.error("Failed to authenticate against >>>>>>>> MongoDB: " + e.getMessage(), e); >>>>>>>> } catch (IllegalStateException e) { >>>>>>>> LOGGER.error("Factory-supplied MongoDB database >>>>>>>> connection already authenticated with different" + >>>>>>>> "credentials but lost connection."); >>>>>>>> } >>>>>>>> >>>>>>>> Problem is, "database" is an instance of com.mongodb.DB, which is in >>>>>>>> the same JAR as com.mongodb.MongoException. If I remove this code, the >>>>>>>> tests pass. How is this possible? The DB instance is there with or >>>>>>>> without this new code, which means the JAR is on the classpath, which >>>>>>>> means MongoException should be on the classpath. >>>>>>>> >>>>>>>> Very confused... >>>>>>>> >>>>>>>> Nick >>>>>>>> >>>>>>>> On Aug 16, 2013, at 5:13 PM, Gary Gregory wrote: >>>>>>>> >>>>>>>>> Thank you for the update Nick! >>>>>>>>> :) >>>>>>>>> Gary >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Aug 16, 2013 at 5:39 PM, Nick Williams >>>>>>>>> <[email protected]> wrote: >>>>>>>>> Answers inline. >>>>>>>>> >>>>>>>>> On Aug 14, 2013, at 2:10 AM, YuCheng Ting wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I use beta8 log4j2 and wrote log4j2.xml like example in document >>>>>>>>>> (http://logging.apache.org/log4j/2.x/manual/appenders.html#NoSQLAppender >>>>>>>>>> ): >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <appenders> >>>>>>>>>> <NoSql name="databaseAppender"> >>>>>>>>>> <MongoDb databaseName="applicationDb" >>>>>>>>>> collectionName="applicationLog" >>>>>>>>>> server="mongo.example.org" >>>>>>>>>> username="loggingUser" password="abc123" /> >>>>>>>>>> </NoSql> >>>>>>>>>> </appenders> >>>>>>>>> >>>>>>>>> Yep. That's correct. >>>>>>>>> >>>>>>>>>> but I get the two exception: >>>>>>>>>> >>>>>>>>>> 1, "can't serialize class org.apache.logging.log4j.Level" exception >>>>>>>>>> in (BasicBSONEncoder.java:270), I read the code and add follow code >>>>>>>>>> in my project before logging, it gone. >>>>>>>>>> >>>>>>>>>> BSON.addEncodingHook(org.apache.logging.log4j.Level.class, new >>>>>>>>>> Transformer() { >>>>>>>>>> @Override >>>>>>>>>> public Object transform(Object o) { >>>>>>>>>> return o.toString(); >>>>>>>>>> } >>>>>>>>>> }); >>>>>>>>> >>>>>>>>> This bug was reported and fixed a few weeks ago. The fix will be in >>>>>>>>> the next version, or you can compile locally. >>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-330 >>>>>>>>> >>>>>>>>>> 2, “not authorized for insert test.log”, because my MongoDB need >>>>>>>>>> auth to write, but the the "username" and "password" attributes in >>>>>>>>>> log4j2.xml is nearly useless, after I read source code, found it NOT >>>>>>>>>> auth in >>>>>>>>>> >>>>>>>>>> org.apache.logging.log4j.core.appender.db.nosql.mongo.MongoDBProvider.createNoSQLProvider >>>>>>>>>> source code line 181 after check username and password and >>>>>>>>>> com.mongodb.DB.authenticate never be called. >>>>>>>>> >>>>>>>>> This is a bug. I'm reporting it and fixing it now. The fix will be in >>>>>>>>> the next version, or you can compile locally (after I get the change >>>>>>>>> committed, of course). >>>>>>>>> >>>>>>>>>> so I change log4j2.xml : >>>>>>>>>> >>>>>>>>>> <NoSql name="mongodb"> >>>>>>>>>> <MongoDb collectionName="log" databaseName="test" >>>>>>>>>> >>>>>>>>>> factoryClassName="com.yuchs.test.log4j.MainTest" >>>>>>>>>> factoryMethodName="getMongoClient" /> >>>>>>>>>> </NoSql> >>>>>>>>>> >>>>>>>>>> and create MongoClient and call com.mongodb.DB.authenticate method >>>>>>>>>> in com.yuchs.test.log4j.MainTest.getMongoClient. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This is my question: >>>>>>>>>> >>>>>>>>>> 1, Why not add BSON.addEncodingHook code into log4j2 project to >>>>>>>>>> avoid basic exception ? or another rule of method I don't know ? >>>>>>>>>> >>>>>>>>>> 2, Why not auth DB in log4j2 project if password and username is set >>>>>>>>>> in log4j2.xml ? or another rule of method I don't know ? >>>>>>>>>> >>>>>>>>>> Thanks everyone! >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> E-Mail: [email protected] | [email protected] >>>>>>>>> Java Persistence with Hibernate, Second Edition >>>>>>>>> JUnit in Action, Second Edition >>>>>>>>> Spring Batch in Action >>>>>>>>> Blog: http://garygregory.wordpress.com >>>>>>>>> Home: http://garygregory.com/ >>>>>>>>> Tweet! http://twitter.com/GaryGregory >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
