Ah. Manual > Plugins... I missed that. I'd like to add something that explains how to use PluginManager to generate the Log42Plugins.dat file without using Maven.
On Sun, Aug 18, 2013 at 1:27 AM, Ralph Goers <[email protected]>wrote: > I believe both already are documented > > Sent from my iPhone > > On Aug 17, 2013, at 9:02 AM, Remko Popma <[email protected]> wrote: > > We should document how the PluginManager class can be used to generate the > Log42Plugins.dat file, and how the packages attribute works. > Maybe in Manual > Extending Log4j > Custom Plugins? That section is > currently empty... > > > On Sun, Aug 18, 2013 at 12:57 AM, Ralph Goers > <[email protected]>wrote: > >> Keep in mind, I added the Log4j2Plugins.dat because classpath scanning >> was so slow. This would still be done for users who create their own >> plugins but don't provide a Log4j2Plugins.dat file, so finding a faster >> method could be worthwhile. >> >> Ralph >> >> >> >> On Aug 17, 2013, at 8:44 AM, Nick Williams wrote: >> >> >> On Aug 17, 2013, at 10:28 AM, Remko Popma wrote: >> >> As part of the build, a plugin database file is generated and included in >> the core jar. The file is called Log4j2Plugins.dat and it is located in the >> org.apache.logging.log4j.core.config.plugins package. It is in binary >> format and contains all classes that define plugins that could be found >> during the build. >> >> >> Understood. >> >> >> At load time, the PluginManager class will search for all resources named >> org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat in the >> classpath. (So there may be multiple jars that each contain a database >> file.) This is pretty fast. >> >> >> Then it's unlikely we would see any performance gains of significance. >> There's still the problem, though, that the plugin classes in this dat file >> are actually _loaded_ eagerly, even if they are never used. My particular >> JVM implementation only complained when I started using an exception from a >> transitive dependency, but some other JVM (like IBM's or Azul Systems's >> JVMs) might complain more aggressively. So there may still be a way we can >> improve this without too much effort. >> >> In addition to this, you can also provide a list of packages that contain >> custom plugins in the configuration: >> <configuration status="TRACE" packages="com.a.b,com.x.y"> ... >> >> >> It's possible this could also use improvement, once again because loading >> classes that are never actually used can cause problems like this. >> >> N >> >> I'm not too worried about the performance of this as there is no >> scanning or searching outside of these packages in the jars in the >> classpath. >> >> >> On Sat, Aug 17, 2013 at 11:46 PM, Nick Williams < >> [email protected]> wrote: >> >>> >>> On Aug 17, 2013, at 8:20 AM, Gary Gregory wrote: >>> >>> > Detective Nick is one the case! :) >>> >>> Thank, Gary! I don't give up. I do NOT like not knowing why something is >>> working in an unexpected way. It means I don't know something I should. >>> >>> > Do those other projects use third party line for this or roll their >>> own? >>> >>> Tomcat rolls its own, AFAICT. Spring and Hibernate use third-party >>> libraries for sure. I'm going to look into what's necessary. My >>> understanding is we would get both a startup performance boost /and/ use >>> less memory by not loading every class to scan for annotations. >>> >>> Question: WHAT does Log4j scan to look for plugins? Does it scan every >>> class in the JAR (in which case the performance and memory improvements >>> would be minor), or does it scan every class on the entire class path (in >>> which case the performance and memory improvements would me major)? The >>> larger the likely improvements, the more effort we should invest it making >>> this happen. >>> >>> Nick >>> >>> > On Aug 17, 2013, at 4:17, Nick Williams <[email protected]> >>> wrote: >>> > >>> >> Solved it! >>> >> >>> >> And you're never gonna believe what I learned tonight...(well, maybe >>> you will) >>> >> >>> >> I solved the error by changing this: >>> >> >>> >> public final class MongoDBProvider implements >>> NoSQLProvider<MongoDBConnection> { >>> >> ... >>> >> + try { >>> >> + if (!database.authenticate(username, >>> password.toCharArray())) { >>> >> + LOGGER.error("Failed to authenticate against >>> MongoDB server. Unknown error."); >>> >> + } >>> >> + } catch (MongoException e) { >>> >> + LOGGER.error("Failed to authenticate against >>> MongoDB: " + e.getMessage(), e); >>> >> + } catch (IllegalStateException e) { >>> >> + LOGGER.error("Factory-supplied MongoDB database >>> connection already authenticated with different" + >>> >> + "credentials but lost connection."); >>> >> + } >>> >> ... >>> >> } >>> >> >>> >> To this: >>> >> >>> >> public final class MongoDBProvider implements >>> NoSQLProvider<MongoDBConnection> { >>> >> ... >>> >> + MongoDBConnection.authenticate(database, username, >>> password); >>> >> ... >>> >> } >>> >> >>> >> public final class MongoDBConnection implements >>> NoSQLConnection<BasicDBObject, MongoDBObject> { >>> >> ... >>> >> + static void authenticate(final DB database, final String >>> username, final String password) { >>> >> + try { >>> >> + if (!database.authenticate(username, >>> password.toCharArray())) { >>> >> + LOGGER.error("Failed to authenticate against MongoDB >>> server. Unknown error."); >>> >> + } >>> >> + } catch (final MongoException e) { >>> >> + LOGGER.error("Failed to authenticate against MongoDB: " >>> + e.getMessage(), e); >>> >> + } catch (final IllegalStateException e) { >>> >> + LOGGER.error("Factory-supplied MongoDB database >>> connection already authenticated with different" + >>> >> + "credentials but lost connection."); >>> >> + } >>> >> + } >>> >> ... >>> >> } >>> >> >>> >> Crazy, right!? Here's what I've learned: >>> >> >>> >> The errors were occurring in tests for the Log4j 1.2 API and the >>> SLF4J Bridge. These tests use the core Logger which triggers plugin >>> discovery. In order to scan for annotations, plugin discovery loads the >>> MongoDBProvider, CouchDBProvider, and JPAAppender classes, among many >>> others, all of which have transitive dependencies that are not on the >>> classpath for running the unit tests for Log4j 1.2 API and SLF4J. So how >>> did it ever work in the first place? >>> >> >>> >> As you may already know, when Java loads a class it also >>> automatically loads any classes it extends or implements, any classes that >>> are the types of static members of that class, any static inner classes, >>> and any classes used within the static initializer. It doesn't load any >>> other classes that class uses in any methods or constructors or as instance >>> members--like com.mongodb.DB or javax.persistence.*--until the code that >>> uses them actually executes for the first time. Because of this, we can do >>> something like load the MongoDBProvider class to scan for @Plugin >>> annotations even though the com.mongodb classes it uses are not on the >>> classpath (as long as they aren't static members of or extended by the >>> MongoDBProvider, that is). >>> >> >>> >> However, Java has a special behavior with exceptions. Because we have >>> these lovely things called checked exceptions that methods must declare to >>> be thrown, exceptions are naturally part of a class's interface. Thus, when >>> Java loads a class it must load the exceptions the class's methods might >>> throw so that it can complete the interface in memory >>> (java.lang.Class.getMethod("someMethod").getExceptionTypes()). Likely for >>> performance reasons, it doesn't differentiate between exceptions that are >>> actually declared to be thrown and exceptions that are just used (caught). >>> ANY dependent classes that are exceptions are loaded when the class loads, >>> even if they're just caught exceptions. This is why this all worked until I >>> started using an exception from a transitive dependency within a plugin >>> class (MongoDBProvider). >>> >> >>> >> (Incidentally, it's also why more advanced class-scanning projects >>> like Spring, Hibernate and Tomcat don't load classes using a ClassLoader >>> just to scan for annotations. Instead, they inspect the byte code manually >>> to scan for annotations, preventing such class loading errors during >>> discovery phases and also saving memory resources since Classes aren't >>> usually garbage collected. It might be worthwhile to look into doing >>> something similar in Log4j plugin discovery. I don't know how much effort >>> would be involved.) >>> >> >>> >> I haven't confirmed any of this with JLS documentation because no >>> amount of Google searching for the combination of "class loading" and >>> "exception" brings up anything other than 10,000,000 people asking >>> questions about what's wrong with their classpath. I simply can't find that >>> needle in a planet full of haystacks. But my thorough experimentation has >>> some pretty clear results. This is exactly what's happening. >>> >> >>> >> Nick >>> >> >>> >> On Aug 17, 2013, at 12:44 AM, Ralph Goers wrote: >>> >> >>> >>> I'll reiterate what I wrote. Catch the RuntimeException and then do >>> >>> >>> >>> if (e.class.getName().equals("com.mongodb.MongoException")) { >>> >>> LOGGER.error("..."); >>> >>> } else { >>> >>> throw e; >>> >>> } >>> >>> >>> >>> This should give you the same behavior. >>> >>> >>> >>> Ralph >>> >>> >>> >>> On Aug 16, 2013, at 9:49 PM, Nick Williams wrote: >>> >>> >>> >>>> That approach concerns me. Catching RuntimeException essentially >>> opens it up to thousands of possible exceptions that could be the cause, as >>> opposed to looking for that exact cause. I suppose I don't have a choice, >>> though. This apparently just isn't going to work. >>> >>>> >>> >>>> Definitely agreed that there is too much going on for a simple >>> Exception class. >>> >>>> >>> >>>> :-/ >>> >>>> >>> >>>> Nick >>> >>>> >>> >>>> On Aug 16, 2013, at 11:45 PM, Ralph Goers wrote: >>> >>>> >>> >>>>> After following the chain of stuff that gets brought in via >>> BSONObject I still recommend the approach in my other email of just >>> catching RuntimeException. A bunch of other classes are being referenced, >>> one of which is creating a static Logger from java.util.logging. I have no >>> idea why that might fail but there is just way too much going on for a >>> simple Exception class. >>> >>>>> >>> >>>>> Ralph >>> >>>>> >>> >>>>> On Aug 16, 2013, at 9:26 PM, Nick Williams wrote: >>> >>>>> >>> >>>>>> >>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/DB.java >>> >>>>>> >>> >>>>>> That also shows an import for org.bson.BSONObject, but the tests >>> still run with just DB and no MongoException. org.bson is in the >>> org.mongodb:mongo-java-driver JAR file. So, no, that's not the problem. >>> There's something else... >>> >>>>>> >>> >>>>>> Nick >>> >>>>>> >>> >>>>>> On Aug 16, 2013, at 11:20 PM, Ralph Goers wrote: >>> >>>>>> >>> >>>>>>> >>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/MongoException.javashows >>> an import for org.bson.BSONObject. The pom.xml for mongo-java-driver >>> doesn't contain a transitive dependency for that and mvn dependency:tree on >>> core doesn't show it. >>> >>>>>>> >>> >>>>>>> Ralph >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> On Aug 16, 2013, at 3:48 PM, Nick Williams wrote: >>> >>>>>>> >>> >>>>>>>> Guys, I'm having a hard time with this simple fix that should >>> have taken five minutes. I'm getting test failures due to >>> NoClassDefFoundErrors that shouldn't happen. >>> >>>>>>>> >>> >>>>>>>> Here are the tests in error: >>> >>>>>>>> CategoryTest.setupClass:52 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testTraceWithException:415 ? NoClassDefFound >>> com/mongodb/MongoExcep... >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testLog:459 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testRB1:295 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testRB2:314 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testRB3:334 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testTrace:388 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testAdditivity1:119 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testAdditivity2:144 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testAdditivity3:183 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testIsTraceEnabled:443 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.testExists:355 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggingTest.setupClass:44 ? NoClassDefFound >>> com/mongodb/MongoException >>> >>>>>>>> LoggingTest.cleanupClass:49 NullPointer >>> >>>>>>>> >>> >>>>>>>> Here's the code I added: >>> >>>>>>>> >>> >>>>>>>> try { >>> >>>>>>>> if (!database.authenticate(username, >>> password.toCharArray())) { >>> >>>>>>>> LOGGER.error("Failed to authenticate >>> against MongoDB server. Unknown error."); >>> >>>>>>>> } >>> >>>>>>>> } catch (MongoException e) { >>> >>>>>>>> LOGGER.error("Failed to authenticate against >>> MongoDB: " + e.getMessage(), e); >>> >>>>>>>> } catch (IllegalStateException e) { >>> >>>>>>>> LOGGER.error("Factory-supplied MongoDB >>> database connection already authenticated with different" + >>> >>>>>>>> "credentials but lost connection."); >>> >>>>>>>> } >>> >>>>>>>> >>> >>>>>>>> Problem is, "database" is an instance of com.mongodb.DB, which >>> is in the same JAR as com.mongodb.MongoException. If I remove this code, >>> the tests pass. How is this possible? The DB instance is there with or >>> without this new code, which means the JAR is on the classpath, which means >>> MongoException should be on the classpath. >>> >>>>>>>> >>> >>>>>>>> Very confused... >>> >>>>>>>> >>> >>>>>>>> Nick >>> >>>>>>>> >>> >>>>>>>> On Aug 16, 2013, at 5:13 PM, Gary Gregory wrote: >>> >>>>>>>> >>> >>>>>>>>> Thank you for the update Nick! >>> >>>>>>>>> :) >>> >>>>>>>>> Gary >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> On Fri, Aug 16, 2013 at 5:39 PM, Nick Williams < >>> [email protected]> wrote: >>> >>>>>>>>> Answers inline. >>> >>>>>>>>> >>> >>>>>>>>> On Aug 14, 2013, at 2:10 AM, YuCheng Ting wrote: >>> >>>>>>>>> >>> >>>>>>>>>> Hi all, >>> >>>>>>>>>> >>> >>>>>>>>>> I use beta8 log4j2 and wrote log4j2.xml like example in >>> document ( >>> http://logging.apache.org/log4j/2.x/manual/appenders.html#NoSQLAppender): >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> <appenders> >>> >>>>>>>>>> <NoSql name="databaseAppender"> >>> >>>>>>>>>> <MongoDb databaseName="applicationDb" >>> collectionName="applicationLog" >>> >>>>>>>>>> server="mongo.example.org" >>> >>>>>>>>>> username="loggingUser" password="abc123" /> >>> >>>>>>>>>> </NoSql> >>> >>>>>>>>>> </appenders> >>> >>>>>>>>> >>> >>>>>>>>> Yep. That's correct. >>> >>>>>>>>> >>> >>>>>>>>>> but I get the two exception: >>> >>>>>>>>>> >>> >>>>>>>>>> 1, "can't serialize class org.apache.logging.log4j.Level" >>> exception in (BasicBSONEncoder.java:270), I read the code and add follow >>> code in my project before logging, it gone. >>> >>>>>>>>>> >>> >>>>>>>>>> >>> BSON.addEncodingHook(org.apache.logging.log4j.Level.class, new >>> Transformer() { >>> >>>>>>>>>> @Override >>> >>>>>>>>>> public Object transform(Object o) { >>> >>>>>>>>>> return o.toString(); >>> >>>>>>>>>> } >>> >>>>>>>>>> }); >>> >>>>>>>>> >>> >>>>>>>>> This bug was reported and fixed a few weeks ago. The fix will >>> be in the next version, or you can compile locally. >>> https://issues.apache.org/jira/browse/LOG4J2-330 >>> >>>>>>>>> >>> >>>>>>>>>> 2, “not authorized for insert test.log”, because my MongoDB >>> need auth to write, but the the "username" and "password" attributes in >>> log4j2.xml is nearly useless, after I read source code, found it NOT auth in >>> >>>>>>>>>> >>> >>>>>>>>>> >>> org.apache.logging.log4j.core.appender.db.nosql.mongo.MongoDBProvider.createNoSQLProvider >>> >>>>>>>>>> source code line 181 after check username and password and >>> com.mongodb.DB.authenticate never be called. >>> >>>>>>>>> >>> >>>>>>>>> This is a bug. I'm reporting it and fixing it now. The fix >>> will be in the next version, or you can compile locally (after I get the >>> change committed, of course). >>> >>>>>>>>> >>> >>>>>>>>>> so I change log4j2.xml : >>> >>>>>>>>>> >>> >>>>>>>>>> <NoSql name="mongodb"> >>> >>>>>>>>>> <MongoDb collectionName="log" databaseName="test" >>> >>>>>>>>>> >>> factoryClassName="com.yuchs.test.log4j.MainTest" >>> >>>>>>>>>> factoryMethodName="getMongoClient" /> >>> >>>>>>>>>> </NoSql> >>> >>>>>>>>>> >>> >>>>>>>>>> and create MongoClient and call com.mongodb.DB.authenticate >>> method in com.yuchs.test.log4j.MainTest.getMongoClient. >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> This is my question: >>> >>>>>>>>>> >>> >>>>>>>>>> 1, Why not add BSON.addEncodingHook code into log4j2 project >>> to avoid basic exception ? or another rule of method I don't know ? >>> >>>>>>>>>> >>> >>>>>>>>>> 2, Why not auth DB in log4j2 project if password and username >>> is set in log4j2.xml ? or another rule of method I don't know ? >>> >>>>>>>>>> >>> >>>>>>>>>> Thanks everyone! >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> -- >>> >>>>>>>>> E-Mail: [email protected] | [email protected] >>> >>>>>>>>> Java Persistence with Hibernate, Second Edition >>> >>>>>>>>> JUnit in Action, Second Edition >>> >>>>>>>>> Spring Batch in Action >>> >>>>>>>>> Blog: http://garygregory.wordpress.com >>> >>>>>>>>> Home: http://garygregory.com/ >>> >>>>>>>>> Tweet! http://twitter.com/GaryGregory >>> >> >>> >> >>> >> --------------------------------------------------------------------- >>> >> To unsubscribe, e-mail: [email protected] >>> >> For additional commands, e-mail: [email protected] >>> >> >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe, e-mail: [email protected] >>> > For additional commands, e-mail: [email protected] >>> > >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >> >> >
