Hi Martin, If I understand correctly, your intention is to not using slf4j + logback on the client side? How about the twill containers (both AM and runnables)? Is it ok to use logback or you want twill to be more flexible about that?
I understand the failure on the AM that you mentioned, I am just wondering what's your end goal looks like to shape a better solution for this. Terence Sent from my iPhone > On Feb 9, 2017, at 11:33 AM, Martin Serrano <mar...@attivio.com> wrote: > > Terence, > > I'm familiar with the logback appender and Kafka code. My point is this: > > * the AppMaster depends on logback. > * when the YarnTwillPreparer class calls createTwillJar it is creating the > runtime jar for the AppMaster from the current classpath (or more accurately > from the classloader used by the current thread). > * this means the logback jar will not be within the twill jar unless it is > currently on the classpath of the client. The current dependency code > ignores dependent classes which are not found in the classpath while walking > the dependency graph. This is what leads to the class not found exception > when starting the appmaster. This is why I filed TWILL-215. > * having the logback jar in the current classpath turns on logback within my > twill client code since I use slf4j. > > Does that make sense? > > -Martin > > > >> On 02/09/2017 02:19 PM, Terence Yim wrote: >> Hi Martin, >> >> Twill has a logback Appender implementation for capturing logs emitted via >> slf4j api from runnable and publish them to the embedded Kafka running >> inside the AM process. If you are using log4j as the API for emitting logs, >> what you can do is to use the log4j-over-slf4j bridge to have logs emitted >> via the log4j API get bridged to slf4j. >> >> I suspect why you are seeing the class missing error is most likely because >> you have the slf4j to log4j bridge (the reverse of the one I mentioned >> above, look for a jar with name containing "slf4j-log4j12" in the client >> classpath) that comes earlier in the classpath then the logback jars. >> >> Terence >> >>> On Thu, Feb 9, 2017 at 10:47 AM, Martin Serrano <mar...@attivio.com> wrote: >>> >>> Henry, >>> >>> I see this behavior deploying with YARN 2.7.1, HDP 2.3. But I'm not sure >>> you understood my issue. >>> >>> * The logback jar dependency is only picked up if it is on the classpath >>> when the bundle is created. >>> * With logback in my twill client classpath, the appmaster starts fine. >>> However without logback in my client classpath the appmaster will get a >>> ClassNotFoundException. >>> * We use log4j and with logback in my client classpath, it takes over the >>> slf4j bindings and I lose control of the client logging. >>> >>> So my question was about whether this is expected or if there is a >>> well-known procedure for working around it. It seems there should be a way >>> to tell the twill system to where to find the appmaster dependencies >>> without having them in the classpath of the twill client. >>> >>> Thanks! >>> -Martin >>> >>> >>>> On 02/08/2017 08:09 PM, Henry Saputra wrote: >>>> >>>> But the logback dependency should be included in the jar packaging that >>>> YARN client sends for Twill ApplicationMaster. >>>> >>>> Are you seeing this behavior in deploying Twill app in latest YARN? >>>> >>>> - Henry >>>> >>>> On Wed, Feb 8, 2017 at 12:30 PM, Martin Serrano <mar...@attivio.com> >>>> wrote: >>>> >>>> Hey Devs, >>>>> It seems like the twill project goes through some pain to try to insulate >>>>> itself logging frameworks. I see use of the slf4j API. However, the >>>>> appmaster code has a dependency on logback via the >>>>> org.apache.twill.internal.logging.Loggings class. The appmaster will >>>>> not start up without this dependency present. With the dependency code >>>>> as >>>>> it is now, there is no way to include the logback jar in the generated >>>>> bundle without it being on the current classpath. I've created a ticket >>>>> (TWILL-215) to make a missing dependency trigger an exception at bundle >>>>> generation time rather than appmaster execution time. >>>>> >>>>> When the logback jar is on my classpath, my client code picks up logback >>>>> instead of our current logger (log4j). Is this what is expected? Is >>>>> there >>>>> any known workaround? It seems like there may be a case for specifying >>>>> dependencies of the appmaster that are located outside of the current jvm >>>>> classpath. >>>>> >>>>> Thanks, >>>>> Martin >>>>> >>>>> >>>>> >
