Apologies, outlook for mac is ridiculous. Copy and paste the original below:
----- I’m running into a strange issue with trying to use a custom Log4j layout for Spark (1.6.1) on YARN (CDH). The layout is: https://github.com/michaeltandy/log4j-json If I use a log4j.properties file (supplied with --files) with: log4j.appender.consolejson=org.apache.log4j.ConsoleAppender log4j.appender.consolejson.target=System.err log4j.appender.consolejson.layout=uk.me.mjt.log4jjson.SimpleJsonLayout And supply the log4j-json.1.0.jar with ‘--jars’ to spark-submit, the driver and executors throw an exception right at the start of the log file: log4j:ERROR Could not instantiate class [uk.me.mjt.log4jjson.SimpleJsonLayout]. java.lang.ClassNotFoundException: uk.me.mjt.log4jjson.SimpleJsonLayout However, a simple spark job that does something like: sc.parallelize(List(1,2,3)).foreach(i => {Class.forName("uk.me.mjt.log4jjson.SimpleJsonLayout")}) Doesn’t throw an error. So the class is being loaded, but just not in time for Log4j to use it. I've tried a few different options trying to get it to work (including it in the YARN application classpath, spark executor classpaths etc) and they all produce the same results. The only thing that seems to work is building a custom spark-assembly with the maven dependency included in core/pom.xml. This way, the layout is included in the spark assembly jar, and I get the JSON log output desired. Is there a classloading issue on Log4j when using --jars? I can't imagine why it works with bundling in spark-assembly, but doesn't work with --jars. From: Ted Yu <yuzhih...@gmail.com> Date: Friday, 22 April 2016 at 14:55 To: Andrew Rowson <andrew.row...@thomsonreuters.com> Cc: "user@spark.apache.org" <user@spark.apache.org> Subject: Re: Custom Log4j layout on YARN = ClassNotFoundException There is not much in the body of email. Can you elaborate what issue you encountered ? Thanks On Fri, Apr 22, 2016 at 2:27 AM, Rowson, Andrew G. (TR Technology & Ops) <andrew.row...@thomsonreuters.com> wrote: ________________________________ This e-mail is for the sole use of the intended recipient and contains information that may be privileged and/or confidential. If you are not an intended recipient, please notify the sender by return e-mail and delete this e-mail and any attachments. Certain required legal entity disclosures can be accessed on our website.<http://site.thomsonreuters.com/site/disclosures/> ---------- Forwarded message ---------- From: "Rowson, Andrew G. (TR Technology & Ops)" <andrew.row...@thomsonreuters.com> To: "user@spark.apache.org" <user@spark.apache.org> Cc: Date: Fri, 22 Apr 2016 10:27:53 +0100 Subject: Custom Log4j layout on YARN = ClassNotFoundException --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
smime.p7s
Description: S/MIME cryptographic signature
