Terence,
Correct, I don't want to use logback on the client side. slf4j is okay
-- we ship with the log4j binding. In the twill containers, the AM
doesn't matter for me, but the runnables does. So yes, flexibility
for the runnables is important.
Thanks,
Martin
On 02/10/2017 01:56 AM, Terence Yim wrote:
Hi Martin,
If I understand correctly, your intention is to not using slf4j + logback on
the client side? How about the twill containers (both AM and runnables)? Is it
ok to use logback or you want twill to be more flexible about that?
I understand the failure on the AM that you mentioned, I am just wondering
what's your end goal looks like to shape a better solution for this.
Terence
Sent from my iPhone
On Feb 9, 2017, at 11:33 AM, Martin Serrano <mar...@attivio.com> wrote:
Terence,
I'm familiar with the logback appender and Kafka code. My point is this:
* the AppMaster depends on logback.
* when the YarnTwillPreparer class calls createTwillJar it is creating the
runtime jar for the AppMaster from the current classpath (or more accurately
from the classloader used by the current thread).
* this means the logback jar will not be within the twill jar unless it is
currently on the classpath of the client. The current dependency code ignores
dependent classes which are not found in the classpath while walking the
dependency graph. This is what leads to the class not found exception when
starting the appmaster. This is why I filed TWILL-215.
* having the logback jar in the current classpath turns on logback within my
twill client code since I use slf4j.
Does that make sense?
-Martin
On 02/09/2017 02:19 PM, Terence Yim wrote:
Hi Martin,
Twill has a logback Appender implementation for capturing logs emitted via
slf4j api from runnable and publish them to the embedded Kafka running
inside the AM process. If you are using log4j as the API for emitting logs,
what you can do is to use the log4j-over-slf4j bridge to have logs emitted
via the log4j API get bridged to slf4j.
I suspect why you are seeing the class missing error is most likely because
you have the slf4j to log4j bridge (the reverse of the one I mentioned
above, look for a jar with name containing "slf4j-log4j12" in the client
classpath) that comes earlier in the classpath then the logback jars.
Terence
On Thu, Feb 9, 2017 at 10:47 AM, Martin Serrano <mar...@attivio.com> wrote:
Henry,
I see this behavior deploying with YARN 2.7.1, HDP 2.3. But I'm not sure
you understood my issue.
* The logback jar dependency is only picked up if it is on the classpath
when the bundle is created.
* With logback in my twill client classpath, the appmaster starts fine.
However without logback in my client classpath the appmaster will get a
ClassNotFoundException.
* We use log4j and with logback in my client classpath, it takes over the
slf4j bindings and I lose control of the client logging.
So my question was about whether this is expected or if there is a
well-known procedure for working around it. It seems there should be a way
to tell the twill system to where to find the appmaster dependencies
without having them in the classpath of the twill client.
Thanks!
-Martin
On 02/08/2017 08:09 PM, Henry Saputra wrote:
But the logback dependency should be included in the jar packaging that
YARN client sends for Twill ApplicationMaster.
Are you seeing this behavior in deploying Twill app in latest YARN?
- Henry
On Wed, Feb 8, 2017 at 12:30 PM, Martin Serrano <mar...@attivio.com>
wrote:
Hey Devs,
It seems like the twill project goes through some pain to try to insulate
itself logging frameworks. I see use of the slf4j API. However, the
appmaster code has a dependency on logback via the
org.apache.twill.internal.logging.Loggings class. The appmaster will
not start up without this dependency present. With the dependency code
as
it is now, there is no way to include the logback jar in the generated
bundle without it being on the current classpath. I've created a ticket
(TWILL-215) to make a missing dependency trigger an exception at bundle
generation time rather than appmaster execution time.
When the logback jar is on my classpath, my client code picks up logback
instead of our current logger (log4j). Is this what is expected? Is
there
any known workaround? It seems like there may be a case for specifying
dependencies of the appmaster that are located outside of the current jvm
classpath.
Thanks,
Martin
