Ah ok, thanks for clarifying your concern, Martin. Twill currently do need the logback in the runnables running the YARN to be able to collect the logs and publish them to the embedded Kafka.
So in your case, you want to use slf4j but backed up by log4j binding instead for client and runnables? - Henry On Fri, Feb 10, 2017 at 9:24 AM, Martin Serrano <mar...@attivio.com> wrote: > Terence, > > Correct, I don't want to use logback on the client side. slf4j is okay > -- we ship with the log4j binding. In the twill containers, the AM doesn't > matter for me, but the runnables does. So yes, flexibility for the > runnables is important. > > Thanks, > Martin > > > On 02/10/2017 01:56 AM, Terence Yim wrote: > >> Hi Martin, >> >> If I understand correctly, your intention is to not using slf4j + logback >> on the client side? How about the twill containers (both AM and runnables)? >> Is it ok to use logback or you want twill to be more flexible about that? >> >> I understand the failure on the AM that you mentioned, I am just >> wondering what's your end goal looks like to shape a better solution for >> this. >> >> Terence >> >> Sent from my iPhone >> >> On Feb 9, 2017, at 11:33 AM, Martin Serrano <mar...@attivio.com> wrote: >>> >>> Terence, >>> >>> I'm familiar with the logback appender and Kafka code. My point is this: >>> >>> * the AppMaster depends on logback. >>> * when the YarnTwillPreparer class calls createTwillJar it is creating >>> the runtime jar for the AppMaster from the current classpath (or more >>> accurately from the classloader used by the current thread). >>> * this means the logback jar will not be within the twill jar unless it >>> is currently on the classpath of the client. The current dependency code >>> ignores dependent classes which are not found in the classpath while >>> walking the dependency graph. This is what leads to the class not found >>> exception when starting the appmaster. This is why I filed TWILL-215. >>> * having the logback jar in the current classpath turns on logback >>> within my twill client code since I use slf4j. >>> >>> Does that make sense? >>> >>> -Martin >>> >>> >>> >>> On 02/09/2017 02:19 PM, Terence Yim wrote: >>>> Hi Martin, >>>> >>>> Twill has a logback Appender implementation for capturing logs emitted >>>> via >>>> slf4j api from runnable and publish them to the embedded Kafka running >>>> inside the AM process. If you are using log4j as the API for emitting >>>> logs, >>>> what you can do is to use the log4j-over-slf4j bridge to have logs >>>> emitted >>>> via the log4j API get bridged to slf4j. >>>> >>>> I suspect why you are seeing the class missing error is most likely >>>> because >>>> you have the slf4j to log4j bridge (the reverse of the one I mentioned >>>> above, look for a jar with name containing "slf4j-log4j12" in the client >>>> classpath) that comes earlier in the classpath then the logback jars. >>>> >>>> Terence >>>> >>>> On Thu, Feb 9, 2017 at 10:47 AM, Martin Serrano <mar...@attivio.com> >>>>> wrote: >>>>> >>>>> Henry, >>>>> >>>>> I see this behavior deploying with YARN 2.7.1, HDP 2.3. But I'm not >>>>> sure >>>>> you understood my issue. >>>>> >>>>> * The logback jar dependency is only picked up if it is on the >>>>> classpath >>>>> when the bundle is created. >>>>> * With logback in my twill client classpath, the appmaster starts fine. >>>>> However without logback in my client classpath the appmaster will get a >>>>> ClassNotFoundException. >>>>> * We use log4j and with logback in my client classpath, it takes over >>>>> the >>>>> slf4j bindings and I lose control of the client logging. >>>>> >>>>> So my question was about whether this is expected or if there is a >>>>> well-known procedure for working around it. It seems there should be >>>>> a way >>>>> to tell the twill system to where to find the appmaster dependencies >>>>> without having them in the classpath of the twill client. >>>>> >>>>> Thanks! >>>>> -Martin >>>>> >>>>> >>>>> On 02/08/2017 08:09 PM, Henry Saputra wrote: >>>>>> >>>>>> But the logback dependency should be included in the jar packaging >>>>>> that >>>>>> YARN client sends for Twill ApplicationMaster. >>>>>> >>>>>> Are you seeing this behavior in deploying Twill app in latest YARN? >>>>>> >>>>>> - Henry >>>>>> >>>>>> On Wed, Feb 8, 2017 at 12:30 PM, Martin Serrano <mar...@attivio.com> >>>>>> wrote: >>>>>> >>>>>> Hey Devs, >>>>>> >>>>>>> It seems like the twill project goes through some pain to try to >>>>>>> insulate >>>>>>> itself logging frameworks. I see use of the slf4j API. However, the >>>>>>> appmaster code has a dependency on logback via the >>>>>>> org.apache.twill.internal.logging.Loggings class. The appmaster >>>>>>> will >>>>>>> not start up without this dependency present. With the dependency >>>>>>> code >>>>>>> as >>>>>>> it is now, there is no way to include the logback jar in the >>>>>>> generated >>>>>>> bundle without it being on the current classpath. I've created a >>>>>>> ticket >>>>>>> (TWILL-215) to make a missing dependency trigger an exception at >>>>>>> bundle >>>>>>> generation time rather than appmaster execution time. >>>>>>> >>>>>>> When the logback jar is on my classpath, my client code picks up >>>>>>> logback >>>>>>> instead of our current logger (log4j). Is this what is expected? Is >>>>>>> there >>>>>>> any known workaround? It seems like there may be a case for >>>>>>> specifying >>>>>>> dependencies of the appmaster that are located outside of the >>>>>>> current jvm >>>>>>> classpath. >>>>>>> >>>>>>> Thanks, >>>>>>> Martin >>>>>>> >>>>>>> >>>>>>> >>>>>>> >
