I believe both already are documented Sent from my iPhone
On Aug 17, 2013, at 9:02 AM, Remko Popma <[email protected]> wrote: > We should document how the PluginManager class can be used to generate the > Log42Plugins.dat file, and how the packages attribute works. > Maybe in Manual > Extending Log4j > Custom Plugins? That section is currently > empty... > > > On Sun, Aug 18, 2013 at 12:57 AM, Ralph Goers <[email protected]> > wrote: >> Keep in mind, I added the Log4j2Plugins.dat because classpath scanning was >> so slow. This would still be done for users who create their own plugins but >> don't provide a Log4j2Plugins.dat file, so finding a faster method could be >> worthwhile. >> >> Ralph >> >> >> >> On Aug 17, 2013, at 8:44 AM, Nick Williams wrote: >> >>> >>> On Aug 17, 2013, at 10:28 AM, Remko Popma wrote: >>> >>>> As part of the build, a plugin database file is generated and included in >>>> the core jar. The file is called Log4j2Plugins.dat and it is located in >>>> the org.apache.logging.log4j.core.config.plugins package. It is in binary >>>> format and contains all classes that define plugins that could be found >>>> during the build. >>> >>> Understood. >>> >>>> >>>> At load time, the PluginManager class will search for all resources named >>>> org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat in the >>>> classpath. (So there may be multiple jars that each contain a database >>>> file.) This is pretty fast. >>> >>> Then it's unlikely we would see any performance gains of significance. >>> There's still the problem, though, that the plugin classes in this dat file >>> are actually _loaded_ eagerly, even if they are never used. My particular >>> JVM implementation only complained when I started using an exception from a >>> transitive dependency, but some other JVM (like IBM's or Azul Systems's >>> JVMs) might complain more aggressively. So there may still be a way we can >>> improve this without too much effort. >>> >>>> In addition to this, you can also provide a list of packages that contain >>>> custom plugins in the configuration: >>>> <configuration status="TRACE" packages="com.a.b,com.x.y"> ... >>> >>> It's possible this could also use improvement, once again because loading >>> classes that are never actually used can cause problems like this. >>> >>> N >>> >>>> I'm not too worried about the performance of this as there is no scanning >>>> or searching outside of these packages in the jars in the classpath. >>>> >>>> >>>> On Sat, Aug 17, 2013 at 11:46 PM, Nick Williams >>>> <[email protected]> wrote: >>>>> >>>>> On Aug 17, 2013, at 8:20 AM, Gary Gregory wrote: >>>>> >>>>> > Detective Nick is one the case! :) >>>>> >>>>> Thank, Gary! I don't give up. I do NOT like not knowing why something is >>>>> working in an unexpected way. It means I don't know something I should. >>>>> >>>>> > Do those other projects use third party line for this or roll their own? >>>>> >>>>> Tomcat rolls its own, AFAICT. Spring and Hibernate use third-party >>>>> libraries for sure. I'm going to look into what's necessary. My >>>>> understanding is we would get both a startup performance boost /and/ use >>>>> less memory by not loading every class to scan for annotations. >>>>> >>>>> Question: WHAT does Log4j scan to look for plugins? Does it scan every >>>>> class in the JAR (in which case the performance and memory improvements >>>>> would be minor), or does it scan every class on the entire class path (in >>>>> which case the performance and memory improvements would me major)? The >>>>> larger the likely improvements, the more effort we should invest it >>>>> making this happen. >>>>> >>>>> Nick >>>>> >>>>> > On Aug 17, 2013, at 4:17, Nick Williams <[email protected]> >>>>> > wrote: >>>>> > >>>>> >> Solved it! >>>>> >> >>>>> >> And you're never gonna believe what I learned tonight...(well, maybe >>>>> >> you will) >>>>> >> >>>>> >> I solved the error by changing this: >>>>> >> >>>>> >> public final class MongoDBProvider implements >>>>> >> NoSQLProvider<MongoDBConnection> { >>>>> >> ... >>>>> >> + try { >>>>> >> + if (!database.authenticate(username, >>>>> >> password.toCharArray())) { >>>>> >> + LOGGER.error("Failed to authenticate against >>>>> >> MongoDB server. Unknown error."); >>>>> >> + } >>>>> >> + } catch (MongoException e) { >>>>> >> + LOGGER.error("Failed to authenticate against >>>>> >> MongoDB: " + e.getMessage(), e); >>>>> >> + } catch (IllegalStateException e) { >>>>> >> + LOGGER.error("Factory-supplied MongoDB database >>>>> >> connection already authenticated with different" + >>>>> >> + "credentials but lost connection."); >>>>> >> + } >>>>> >> ... >>>>> >> } >>>>> >> >>>>> >> To this: >>>>> >> >>>>> >> public final class MongoDBProvider implements >>>>> >> NoSQLProvider<MongoDBConnection> { >>>>> >> ... >>>>> >> + MongoDBConnection.authenticate(database, username, >>>>> >> password); >>>>> >> ... >>>>> >> } >>>>> >> >>>>> >> public final class MongoDBConnection implements >>>>> >> NoSQLConnection<BasicDBObject, MongoDBObject> { >>>>> >> ... >>>>> >> + static void authenticate(final DB database, final String >>>>> >> username, final String password) { >>>>> >> + try { >>>>> >> + if (!database.authenticate(username, >>>>> >> password.toCharArray())) { >>>>> >> + LOGGER.error("Failed to authenticate against MongoDB >>>>> >> server. Unknown error."); >>>>> >> + } >>>>> >> + } catch (final MongoException e) { >>>>> >> + LOGGER.error("Failed to authenticate against MongoDB: " + >>>>> >> e.getMessage(), e); >>>>> >> + } catch (final IllegalStateException e) { >>>>> >> + LOGGER.error("Factory-supplied MongoDB database >>>>> >> connection already authenticated with different" + >>>>> >> + "credentials but lost connection."); >>>>> >> + } >>>>> >> + } >>>>> >> ... >>>>> >> } >>>>> >> >>>>> >> Crazy, right!? Here's what I've learned: >>>>> >> >>>>> >> The errors were occurring in tests for the Log4j 1.2 API and the SLF4J >>>>> >> Bridge. These tests use the core Logger which triggers plugin >>>>> >> discovery. In order to scan for annotations, plugin discovery loads >>>>> >> the MongoDBProvider, CouchDBProvider, and JPAAppender classes, among >>>>> >> many others, all of which have transitive dependencies that are not on >>>>> >> the classpath for running the unit tests for Log4j 1.2 API and SLF4J. >>>>> >> So how did it ever work in the first place? >>>>> >> >>>>> >> As you may already know, when Java loads a class it also automatically >>>>> >> loads any classes it extends or implements, any classes that are the >>>>> >> types of static members of that class, any static inner classes, and >>>>> >> any classes used within the static initializer. It doesn't load any >>>>> >> other classes that class uses in any methods or constructors or as >>>>> >> instance members--like com.mongodb.DB or javax.persistence.*--until >>>>> >> the code that uses them actually executes for the first time. Because >>>>> >> of this, we can do something like load the MongoDBProvider class to >>>>> >> scan for @Plugin annotations even though the com.mongodb classes it >>>>> >> uses are not on the classpath (as long as they aren't static members >>>>> >> of or extended by the MongoDBProvider, that is). >>>>> >> >>>>> >> However, Java has a special behavior with exceptions. Because we have >>>>> >> these lovely things called checked exceptions that methods must >>>>> >> declare to be thrown, exceptions are naturally part of a class's >>>>> >> interface. Thus, when Java loads a class it must load the exceptions >>>>> >> the class's methods might throw so that it can complete the interface >>>>> >> in memory >>>>> >> (java.lang.Class.getMethod("someMethod").getExceptionTypes()). Likely >>>>> >> for performance reasons, it doesn't differentiate between exceptions >>>>> >> that are actually declared to be thrown and exceptions that are just >>>>> >> used (caught). ANY dependent classes that are exceptions are loaded >>>>> >> when the class loads, even if they're just caught exceptions. This is >>>>> >> why this all worked until I started using an exception from a >>>>> >> transitive dependency within a plugin class (MongoDBProvider). >>>>> >> >>>>> >> (Incidentally, it's also why more advanced class-scanning projects >>>>> >> like Spring, Hibernate and Tomcat don't load classes using a >>>>> >> ClassLoader just to scan for annotations. Instead, they inspect the >>>>> >> byte code manually to scan for annotations, preventing such class >>>>> >> loading errors during discovery phases and also saving memory >>>>> >> resources since Classes aren't usually garbage collected. It might be >>>>> >> worthwhile to look into doing something similar in Log4j plugin >>>>> >> discovery. I don't know how much effort would be involved.) >>>>> >> >>>>> >> I haven't confirmed any of this with JLS documentation because no >>>>> >> amount of Google searching for the combination of "class loading" and >>>>> >> "exception" brings up anything other than 10,000,000 people asking >>>>> >> questions about what's wrong with their classpath. I simply can't find >>>>> >> that needle in a planet full of haystacks. But my thorough >>>>> >> experimentation has some pretty clear results. This is exactly what's >>>>> >> happening. >>>>> >> >>>>> >> Nick >>>>> >> >>>>> >> On Aug 17, 2013, at 12:44 AM, Ralph Goers wrote: >>>>> >> >>>>> >>> I'll reiterate what I wrote. Catch the RuntimeException and then do >>>>> >>> >>>>> >>> if (e.class.getName().equals("com.mongodb.MongoException")) { >>>>> >>> LOGGER.error("..."); >>>>> >>> } else { >>>>> >>> throw e; >>>>> >>> } >>>>> >>> >>>>> >>> This should give you the same behavior. >>>>> >>> >>>>> >>> Ralph >>>>> >>> >>>>> >>> On Aug 16, 2013, at 9:49 PM, Nick Williams wrote: >>>>> >>> >>>>> >>>> That approach concerns me. Catching RuntimeException essentially >>>>> >>>> opens it up to thousands of possible exceptions that could be the >>>>> >>>> cause, as opposed to looking for that exact cause. I suppose I don't >>>>> >>>> have a choice, though. This apparently just isn't going to work. >>>>> >>>> >>>>> >>>> Definitely agreed that there is too much going on for a simple >>>>> >>>> Exception class. >>>>> >>>> >>>>> >>>> :-/ >>>>> >>>> >>>>> >>>> Nick >>>>> >>>> >>>>> >>>> On Aug 16, 2013, at 11:45 PM, Ralph Goers wrote: >>>>> >>>> >>>>> >>>>> After following the chain of stuff that gets brought in via >>>>> >>>>> BSONObject I still recommend the approach in my other email of just >>>>> >>>>> catching RuntimeException. A bunch of other classes are being >>>>> >>>>> referenced, one of which is creating a static Logger from >>>>> >>>>> java.util.logging. I have no idea why that might fail but there is >>>>> >>>>> just way too much going on for a simple Exception class. >>>>> >>>>> >>>>> >>>>> Ralph >>>>> >>>>> >>>>> >>>>> On Aug 16, 2013, at 9:26 PM, Nick Williams wrote: >>>>> >>>>> >>>>> >>>>>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/DB.java >>>>> >>>>>> >>>>> >>>>>> That also shows an import for org.bson.BSONObject, but the tests >>>>> >>>>>> still run with just DB and no MongoException. org.bson is in the >>>>> >>>>>> org.mongodb:mongo-java-driver JAR file. So, no, that's not the >>>>> >>>>>> problem. There's something else... >>>>> >>>>>> >>>>> >>>>>> Nick >>>>> >>>>>> >>>>> >>>>>> On Aug 16, 2013, at 11:20 PM, Ralph Goers wrote: >>>>> >>>>>> >>>>> >>>>>>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/MongoException.java >>>>> >>>>>>> shows an import for org.bson.BSONObject. The pom.xml for >>>>> >>>>>>> mongo-java-driver doesn't contain a transitive dependency for >>>>> >>>>>>> that and mvn dependency:tree on core doesn't show it. >>>>> >>>>>>> >>>>> >>>>>>> Ralph >>>>> >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> On Aug 16, 2013, at 3:48 PM, Nick Williams wrote: >>>>> >>>>>>> >>>>> >>>>>>>> Guys, I'm having a hard time with this simple fix that should >>>>> >>>>>>>> have taken five minutes. I'm getting test failures due to >>>>> >>>>>>>> NoClassDefFoundErrors that shouldn't happen. >>>>> >>>>>>>> >>>>> >>>>>>>> Here are the tests in error: >>>>> >>>>>>>> CategoryTest.setupClass:52 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testTraceWithException:415 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoExcep... >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testLog:459 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testRB1:295 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testRB2:314 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testRB3:334 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testTrace:388 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testAdditivity1:119 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testAdditivity2:144 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testAdditivity3:183 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testIsTraceEnabled:443 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.testExists:355 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggingTest.setupClass:44 ? NoClassDefFound >>>>> >>>>>>>> com/mongodb/MongoException >>>>> >>>>>>>> LoggingTest.cleanupClass:49 NullPointer >>>>> >>>>>>>> >>>>> >>>>>>>> Here's the code I added: >>>>> >>>>>>>> >>>>> >>>>>>>> try { >>>>> >>>>>>>> if (!database.authenticate(username, >>>>> >>>>>>>> password.toCharArray())) { >>>>> >>>>>>>> LOGGER.error("Failed to authenticate >>>>> >>>>>>>> against MongoDB server. Unknown error."); >>>>> >>>>>>>> } >>>>> >>>>>>>> } catch (MongoException e) { >>>>> >>>>>>>> LOGGER.error("Failed to authenticate against >>>>> >>>>>>>> MongoDB: " + e.getMessage(), e); >>>>> >>>>>>>> } catch (IllegalStateException e) { >>>>> >>>>>>>> LOGGER.error("Factory-supplied MongoDB >>>>> >>>>>>>> database connection already authenticated with different" + >>>>> >>>>>>>> "credentials but lost connection."); >>>>> >>>>>>>> } >>>>> >>>>>>>> >>>>> >>>>>>>> Problem is, "database" is an instance of com.mongodb.DB, which >>>>> >>>>>>>> is in the same JAR as com.mongodb.MongoException. If I remove >>>>> >>>>>>>> this code, the tests pass. How is this possible? The DB instance >>>>> >>>>>>>> is there with or without this new code, which means the JAR is >>>>> >>>>>>>> on the classpath, which means MongoException should be on the >>>>> >>>>>>>> classpath. >>>>> >>>>>>>> >>>>> >>>>>>>> Very confused... >>>>> >>>>>>>> >>>>> >>>>>>>> Nick >>>>> >>>>>>>> >>>>> >>>>>>>> On Aug 16, 2013, at 5:13 PM, Gary Gregory wrote: >>>>> >>>>>>>> >>>>> >>>>>>>>> Thank you for the update Nick! >>>>> >>>>>>>>> :) >>>>> >>>>>>>>> Gary >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> On Fri, Aug 16, 2013 at 5:39 PM, Nick Williams >>>>> >>>>>>>>> <[email protected]> wrote: >>>>> >>>>>>>>> Answers inline. >>>>> >>>>>>>>> >>>>> >>>>>>>>> On Aug 14, 2013, at 2:10 AM, YuCheng Ting wrote: >>>>> >>>>>>>>> >>>>> >>>>>>>>>> Hi all, >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I use beta8 log4j2 and wrote log4j2.xml like example in >>>>> >>>>>>>>>> document >>>>> >>>>>>>>>> (http://logging.apache.org/log4j/2.x/manual/appenders.html#NoSQLAppender >>>>> >>>>>>>>>> ): >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> <appenders> >>>>> >>>>>>>>>> <NoSql name="databaseAppender"> >>>>> >>>>>>>>>> <MongoDb databaseName="applicationDb" >>>>> >>>>>>>>>> collectionName="applicationLog" >>>>> >>>>>>>>>> server="mongo.example.org" >>>>> >>>>>>>>>> username="loggingUser" password="abc123" /> >>>>> >>>>>>>>>> </NoSql> >>>>> >>>>>>>>>> </appenders> >>>>> >>>>>>>>> >>>>> >>>>>>>>> Yep. That's correct. >>>>> >>>>>>>>> >>>>> >>>>>>>>>> but I get the two exception: >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> 1, "can't serialize class org.apache.logging.log4j.Level" >>>>> >>>>>>>>>> exception in (BasicBSONEncoder.java:270), I read the code and >>>>> >>>>>>>>>> add follow code in my project before logging, it gone. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> BSON.addEncodingHook(org.apache.logging.log4j.Level.class, new >>>>> >>>>>>>>>> Transformer() { >>>>> >>>>>>>>>> @Override >>>>> >>>>>>>>>> public Object transform(Object o) { >>>>> >>>>>>>>>> return o.toString(); >>>>> >>>>>>>>>> } >>>>> >>>>>>>>>> }); >>>>> >>>>>>>>> >>>>> >>>>>>>>> This bug was reported and fixed a few weeks ago. The fix will >>>>> >>>>>>>>> be in the next version, or you can compile locally. >>>>> >>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-330 >>>>> >>>>>>>>> >>>>> >>>>>>>>>> 2, “not authorized for insert test.log”, because my MongoDB >>>>> >>>>>>>>>> need auth to write, but the the "username" and "password" >>>>> >>>>>>>>>> attributes in log4j2.xml is nearly useless, after I read >>>>> >>>>>>>>>> source code, found it NOT auth in >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> org.apache.logging.log4j.core.appender.db.nosql.mongo.MongoDBProvider.createNoSQLProvider >>>>> >>>>>>>>>> source code line 181 after check username and password and >>>>> >>>>>>>>>> com.mongodb.DB.authenticate never be called. >>>>> >>>>>>>>> >>>>> >>>>>>>>> This is a bug. I'm reporting it and fixing it now. The fix will >>>>> >>>>>>>>> be in the next version, or you can compile locally (after I get >>>>> >>>>>>>>> the change committed, of course). >>>>> >>>>>>>>> >>>>> >>>>>>>>>> so I change log4j2.xml : >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> <NoSql name="mongodb"> >>>>> >>>>>>>>>> <MongoDb collectionName="log" databaseName="test" >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> factoryClassName="com.yuchs.test.log4j.MainTest" >>>>> >>>>>>>>>> factoryMethodName="getMongoClient" /> >>>>> >>>>>>>>>> </NoSql> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> and create MongoClient and call com.mongodb.DB.authenticate >>>>> >>>>>>>>>> method in com.yuchs.test.log4j.MainTest.getMongoClient. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> This is my question: >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> 1, Why not add BSON.addEncodingHook code into log4j2 project >>>>> >>>>>>>>>> to avoid basic exception ? or another rule of method I don't >>>>> >>>>>>>>>> know ? >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> 2, Why not auth DB in log4j2 project if password and username >>>>> >>>>>>>>>> is set in log4j2.xml ? or another rule of method I don't know ? >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> Thanks everyone! >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> -- >>>>> >>>>>>>>> E-Mail: [email protected] | [email protected] >>>>> >>>>>>>>> Java Persistence with Hibernate, Second Edition >>>>> >>>>>>>>> JUnit in Action, Second Edition >>>>> >>>>>>>>> Spring Batch in Action >>>>> >>>>>>>>> Blog: http://garygregory.wordpress.com >>>>> >>>>>>>>> Home: http://garygregory.com/ >>>>> >>>>>>>>> Tweet! http://twitter.com/GaryGregory >>>>> >> >>>>> >> >>>>> >> --------------------------------------------------------------------- >>>>> >> To unsubscribe, e-mail: [email protected] >>>>> >> For additional commands, e-mail: [email protected] >>>>> >> >>>>> > >>>>> > --------------------------------------------------------------------- >>>>> > To unsubscribe, e-mail: [email protected] >>>>> > For additional commands, e-mail: [email protected] >>>>> > >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >
