Keep in mind, I added the Log4j2Plugins.dat because classpath scanning was so slow. This would still be done for users who create their own plugins but don't provide a Log4j2Plugins.dat file, so finding a faster method could be worthwhile.
Ralph On Aug 17, 2013, at 8:44 AM, Nick Williams wrote: > > On Aug 17, 2013, at 10:28 AM, Remko Popma wrote: > >> As part of the build, a plugin database file is generated and included in >> the core jar. The file is called Log4j2Plugins.dat and it is located in the >> org.apache.logging.log4j.core.config.plugins package. It is in binary format >> and contains all classes that define plugins that could be found during the >> build. > > Understood. > >> >> At load time, the PluginManager class will search for all resources named >> org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat in the >> classpath. (So there may be multiple jars that each contain a database >> file.) This is pretty fast. > > Then it's unlikely we would see any performance gains of significance. > There's still the problem, though, that the plugin classes in this dat file > are actually _loaded_ eagerly, even if they are never used. My particular JVM > implementation only complained when I started using an exception from a > transitive dependency, but some other JVM (like IBM's or Azul Systems's JVMs) > might complain more aggressively. So there may still be a way we can improve > this without too much effort. > >> In addition to this, you can also provide a list of packages that contain >> custom plugins in the configuration: >> <configuration status="TRACE" packages="com.a.b,com.x.y"> ... > > It's possible this could also use improvement, once again because loading > classes that are never actually used can cause problems like this. > > N > >> I'm not too worried about the performance of this as there is no scanning or >> searching outside of these packages in the jars in the classpath. >> >> >> On Sat, Aug 17, 2013 at 11:46 PM, Nick Williams >> <[email protected]> wrote: >> >> On Aug 17, 2013, at 8:20 AM, Gary Gregory wrote: >> >> > Detective Nick is one the case! :) >> >> Thank, Gary! I don't give up. I do NOT like not knowing why something is >> working in an unexpected way. It means I don't know something I should. >> >> > Do those other projects use third party line for this or roll their own? >> >> Tomcat rolls its own, AFAICT. Spring and Hibernate use third-party libraries >> for sure. I'm going to look into what's necessary. My understanding is we >> would get both a startup performance boost /and/ use less memory by not >> loading every class to scan for annotations. >> >> Question: WHAT does Log4j scan to look for plugins? Does it scan every class >> in the JAR (in which case the performance and memory improvements would be >> minor), or does it scan every class on the entire class path (in which case >> the performance and memory improvements would me major)? The larger the >> likely improvements, the more effort we should invest it making this happen. >> >> Nick >> >> > On Aug 17, 2013, at 4:17, Nick Williams <[email protected]> >> > wrote: >> > >> >> Solved it! >> >> >> >> And you're never gonna believe what I learned tonight...(well, maybe you >> >> will) >> >> >> >> I solved the error by changing this: >> >> >> >> public final class MongoDBProvider implements >> >> NoSQLProvider<MongoDBConnection> { >> >> ... >> >> + try { >> >> + if (!database.authenticate(username, >> >> password.toCharArray())) { >> >> + LOGGER.error("Failed to authenticate against >> >> MongoDB server. Unknown error."); >> >> + } >> >> + } catch (MongoException e) { >> >> + LOGGER.error("Failed to authenticate against >> >> MongoDB: " + e.getMessage(), e); >> >> + } catch (IllegalStateException e) { >> >> + LOGGER.error("Factory-supplied MongoDB database >> >> connection already authenticated with different" + >> >> + "credentials but lost connection."); >> >> + } >> >> ... >> >> } >> >> >> >> To this: >> >> >> >> public final class MongoDBProvider implements >> >> NoSQLProvider<MongoDBConnection> { >> >> ... >> >> + MongoDBConnection.authenticate(database, username, >> >> password); >> >> ... >> >> } >> >> >> >> public final class MongoDBConnection implements >> >> NoSQLConnection<BasicDBObject, MongoDBObject> { >> >> ... >> >> + static void authenticate(final DB database, final String username, >> >> final String password) { >> >> + try { >> >> + if (!database.authenticate(username, >> >> password.toCharArray())) { >> >> + LOGGER.error("Failed to authenticate against MongoDB >> >> server. Unknown error."); >> >> + } >> >> + } catch (final MongoException e) { >> >> + LOGGER.error("Failed to authenticate against MongoDB: " + >> >> e.getMessage(), e); >> >> + } catch (final IllegalStateException e) { >> >> + LOGGER.error("Factory-supplied MongoDB database connection >> >> already authenticated with different" + >> >> + "credentials but lost connection."); >> >> + } >> >> + } >> >> ... >> >> } >> >> >> >> Crazy, right!? Here's what I've learned: >> >> >> >> The errors were occurring in tests for the Log4j 1.2 API and the SLF4J >> >> Bridge. These tests use the core Logger which triggers plugin discovery. >> >> In order to scan for annotations, plugin discovery loads the >> >> MongoDBProvider, CouchDBProvider, and JPAAppender classes, among many >> >> others, all of which have transitive dependencies that are not on the >> >> classpath for running the unit tests for Log4j 1.2 API and SLF4J. So how >> >> did it ever work in the first place? >> >> >> >> As you may already know, when Java loads a class it also automatically >> >> loads any classes it extends or implements, any classes that are the >> >> types of static members of that class, any static inner classes, and any >> >> classes used within the static initializer. It doesn't load any other >> >> classes that class uses in any methods or constructors or as instance >> >> members--like com.mongodb.DB or javax.persistence.*--until the code that >> >> uses them actually executes for the first time. Because of this, we can >> >> do something like load the MongoDBProvider class to scan for @Plugin >> >> annotations even though the com.mongodb classes it uses are not on the >> >> classpath (as long as they aren't static members of or extended by the >> >> MongoDBProvider, that is). >> >> >> >> However, Java has a special behavior with exceptions. Because we have >> >> these lovely things called checked exceptions that methods must declare >> >> to be thrown, exceptions are naturally part of a class's interface. Thus, >> >> when Java loads a class it must load the exceptions the class's methods >> >> might throw so that it can complete the interface in memory >> >> (java.lang.Class.getMethod("someMethod").getExceptionTypes()). Likely for >> >> performance reasons, it doesn't differentiate between exceptions that are >> >> actually declared to be thrown and exceptions that are just used >> >> (caught). ANY dependent classes that are exceptions are loaded when the >> >> class loads, even if they're just caught exceptions. This is why this all >> >> worked until I started using an exception from a transitive dependency >> >> within a plugin class (MongoDBProvider). >> >> >> >> (Incidentally, it's also why more advanced class-scanning projects like >> >> Spring, Hibernate and Tomcat don't load classes using a ClassLoader just >> >> to scan for annotations. Instead, they inspect the byte code manually to >> >> scan for annotations, preventing such class loading errors during >> >> discovery phases and also saving memory resources since Classes aren't >> >> usually garbage collected. It might be worthwhile to look into doing >> >> something similar in Log4j plugin discovery. I don't know how much effort >> >> would be involved.) >> >> >> >> I haven't confirmed any of this with JLS documentation because no amount >> >> of Google searching for the combination of "class loading" and >> >> "exception" brings up anything other than 10,000,000 people asking >> >> questions about what's wrong with their classpath. I simply can't find >> >> that needle in a planet full of haystacks. But my thorough >> >> experimentation has some pretty clear results. This is exactly what's >> >> happening. >> >> >> >> Nick >> >> >> >> On Aug 17, 2013, at 12:44 AM, Ralph Goers wrote: >> >> >> >>> I'll reiterate what I wrote. Catch the RuntimeException and then do >> >>> >> >>> if (e.class.getName().equals("com.mongodb.MongoException")) { >> >>> LOGGER.error("..."); >> >>> } else { >> >>> throw e; >> >>> } >> >>> >> >>> This should give you the same behavior. >> >>> >> >>> Ralph >> >>> >> >>> On Aug 16, 2013, at 9:49 PM, Nick Williams wrote: >> >>> >> >>>> That approach concerns me. Catching RuntimeException essentially opens >> >>>> it up to thousands of possible exceptions that could be the cause, as >> >>>> opposed to looking for that exact cause. I suppose I don't have a >> >>>> choice, though. This apparently just isn't going to work. >> >>>> >> >>>> Definitely agreed that there is too much going on for a simple >> >>>> Exception class. >> >>>> >> >>>> :-/ >> >>>> >> >>>> Nick >> >>>> >> >>>> On Aug 16, 2013, at 11:45 PM, Ralph Goers wrote: >> >>>> >> >>>>> After following the chain of stuff that gets brought in via BSONObject >> >>>>> I still recommend the approach in my other email of just catching >> >>>>> RuntimeException. A bunch of other classes are being referenced, one >> >>>>> of which is creating a static Logger from java.util.logging. I have no >> >>>>> idea why that might fail but there is just way too much going on for a >> >>>>> simple Exception class. >> >>>>> >> >>>>> Ralph >> >>>>> >> >>>>> On Aug 16, 2013, at 9:26 PM, Nick Williams wrote: >> >>>>> >> >>>>>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/DB.java >> >>>>>> >> >>>>>> That also shows an import for org.bson.BSONObject, but the tests >> >>>>>> still run with just DB and no MongoException. org.bson is in the >> >>>>>> org.mongodb:mongo-java-driver JAR file. So, no, that's not the >> >>>>>> problem. There's something else... >> >>>>>> >> >>>>>> Nick >> >>>>>> >> >>>>>> On Aug 16, 2013, at 11:20 PM, Ralph Goers wrote: >> >>>>>> >> >>>>>>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/MongoException.java >> >>>>>>> shows an import for org.bson.BSONObject. The pom.xml for >> >>>>>>> mongo-java-driver doesn't contain a transitive dependency for that >> >>>>>>> and mvn dependency:tree on core doesn't show it. >> >>>>>>> >> >>>>>>> Ralph >> >>>>>>> >> >>>>>>> >> >>>>>>> On Aug 16, 2013, at 3:48 PM, Nick Williams wrote: >> >>>>>>> >> >>>>>>>> Guys, I'm having a hard time with this simple fix that should have >> >>>>>>>> taken five minutes. I'm getting test failures due to >> >>>>>>>> NoClassDefFoundErrors that shouldn't happen. >> >>>>>>>> >> >>>>>>>> Here are the tests in error: >> >>>>>>>> CategoryTest.setupClass:52 ? NoClassDefFound >> >>>>>>>> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testTraceWithException:415 ? NoClassDefFound >> >>>>>>>> com/mongodb/MongoExcep... >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testLog:459 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testRB1:295 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testRB2:314 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testRB3:334 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testTrace:388 ? NoClassDefFound >> >>>>>>>> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testAdditivity1:119 ? NoClassDefFound >> >>>>>>>> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testAdditivity2:144 ? NoClassDefFound >> >>>>>>>> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testAdditivity3:183 ? NoClassDefFound >> >>>>>>>> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testIsTraceEnabled:443 ? NoClassDefFound >> >>>>>>>> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggerTest.testExists:355 ? NoClassDefFound >> >>>>>>>> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException >> >>>>>>>> LoggingTest.setupClass:44 ? NoClassDefFound >> >>>>>>>> com/mongodb/MongoException >> >>>>>>>> LoggingTest.cleanupClass:49 NullPointer >> >>>>>>>> >> >>>>>>>> Here's the code I added: >> >>>>>>>> >> >>>>>>>> try { >> >>>>>>>> if (!database.authenticate(username, >> >>>>>>>> password.toCharArray())) { >> >>>>>>>> LOGGER.error("Failed to authenticate against >> >>>>>>>> MongoDB server. Unknown error."); >> >>>>>>>> } >> >>>>>>>> } catch (MongoException e) { >> >>>>>>>> LOGGER.error("Failed to authenticate against >> >>>>>>>> MongoDB: " + e.getMessage(), e); >> >>>>>>>> } catch (IllegalStateException e) { >> >>>>>>>> LOGGER.error("Factory-supplied MongoDB database >> >>>>>>>> connection already authenticated with different" + >> >>>>>>>> "credentials but lost connection."); >> >>>>>>>> } >> >>>>>>>> >> >>>>>>>> Problem is, "database" is an instance of com.mongodb.DB, which is >> >>>>>>>> in the same JAR as com.mongodb.MongoException. If I remove this >> >>>>>>>> code, the tests pass. How is this possible? The DB instance is >> >>>>>>>> there with or without this new code, which means the JAR is on the >> >>>>>>>> classpath, which means MongoException should be on the classpath. >> >>>>>>>> >> >>>>>>>> Very confused... >> >>>>>>>> >> >>>>>>>> Nick >> >>>>>>>> >> >>>>>>>> On Aug 16, 2013, at 5:13 PM, Gary Gregory wrote: >> >>>>>>>> >> >>>>>>>>> Thank you for the update Nick! >> >>>>>>>>> :) >> >>>>>>>>> Gary >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Fri, Aug 16, 2013 at 5:39 PM, Nick Williams >> >>>>>>>>> <[email protected]> wrote: >> >>>>>>>>> Answers inline. >> >>>>>>>>> >> >>>>>>>>> On Aug 14, 2013, at 2:10 AM, YuCheng Ting wrote: >> >>>>>>>>> >> >>>>>>>>>> Hi all, >> >>>>>>>>>> >> >>>>>>>>>> I use beta8 log4j2 and wrote log4j2.xml like example in document >> >>>>>>>>>> (http://logging.apache.org/log4j/2.x/manual/appenders.html#NoSQLAppender >> >>>>>>>>>> ): >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> <appenders> >> >>>>>>>>>> <NoSql name="databaseAppender"> >> >>>>>>>>>> <MongoDb databaseName="applicationDb" >> >>>>>>>>>> collectionName="applicationLog" >> >>>>>>>>>> server="mongo.example.org" >> >>>>>>>>>> username="loggingUser" password="abc123" /> >> >>>>>>>>>> </NoSql> >> >>>>>>>>>> </appenders> >> >>>>>>>>> >> >>>>>>>>> Yep. That's correct. >> >>>>>>>>> >> >>>>>>>>>> but I get the two exception: >> >>>>>>>>>> >> >>>>>>>>>> 1, "can't serialize class org.apache.logging.log4j.Level" >> >>>>>>>>>> exception in (BasicBSONEncoder.java:270), I read the code and add >> >>>>>>>>>> follow code in my project before logging, it gone. >> >>>>>>>>>> >> >>>>>>>>>> BSON.addEncodingHook(org.apache.logging.log4j.Level.class, >> >>>>>>>>>> new Transformer() { >> >>>>>>>>>> @Override >> >>>>>>>>>> public Object transform(Object o) { >> >>>>>>>>>> return o.toString(); >> >>>>>>>>>> } >> >>>>>>>>>> }); >> >>>>>>>>> >> >>>>>>>>> This bug was reported and fixed a few weeks ago. The fix will be >> >>>>>>>>> in the next version, or you can compile locally. >> >>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-330 >> >>>>>>>>> >> >>>>>>>>>> 2, “not authorized for insert test.log”, because my MongoDB need >> >>>>>>>>>> auth to write, but the the "username" and "password" attributes >> >>>>>>>>>> in log4j2.xml is nearly useless, after I read source code, found >> >>>>>>>>>> it NOT auth in >> >>>>>>>>>> >> >>>>>>>>>> org.apache.logging.log4j.core.appender.db.nosql.mongo.MongoDBProvider.createNoSQLProvider >> >>>>>>>>>> source code line 181 after check username and password and >> >>>>>>>>>> com.mongodb.DB.authenticate never be called. >> >>>>>>>>> >> >>>>>>>>> This is a bug. I'm reporting it and fixing it now. The fix will be >> >>>>>>>>> in the next version, or you can compile locally (after I get the >> >>>>>>>>> change committed, of course). >> >>>>>>>>> >> >>>>>>>>>> so I change log4j2.xml : >> >>>>>>>>>> >> >>>>>>>>>> <NoSql name="mongodb"> >> >>>>>>>>>> <MongoDb collectionName="log" databaseName="test" >> >>>>>>>>>> >> >>>>>>>>>> factoryClassName="com.yuchs.test.log4j.MainTest" >> >>>>>>>>>> factoryMethodName="getMongoClient" /> >> >>>>>>>>>> </NoSql> >> >>>>>>>>>> >> >>>>>>>>>> and create MongoClient and call com.mongodb.DB.authenticate >> >>>>>>>>>> method in com.yuchs.test.log4j.MainTest.getMongoClient. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> This is my question: >> >>>>>>>>>> >> >>>>>>>>>> 1, Why not add BSON.addEncodingHook code into log4j2 project to >> >>>>>>>>>> avoid basic exception ? or another rule of method I don't know ? >> >>>>>>>>>> >> >>>>>>>>>> 2, Why not auth DB in log4j2 project if password and username is >> >>>>>>>>>> set in log4j2.xml ? or another rule of method I don't know ? >> >>>>>>>>>> >> >>>>>>>>>> Thanks everyone! >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> E-Mail: [email protected] | [email protected] >> >>>>>>>>> Java Persistence with Hibernate, Second Edition >> >>>>>>>>> JUnit in Action, Second Edition >> >>>>>>>>> Spring Batch in Action >> >>>>>>>>> Blog: http://garygregory.wordpress.com >> >>>>>>>>> Home: http://garygregory.com/ >> >>>>>>>>> Tweet! http://twitter.com/GaryGregory >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> >> For additional commands, e-mail: [email protected] >> >> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> > >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >
