Re: How do I include the newer version of Commons-lang in my jar?
Have you tried setting 'mapreduce.user.classpath.first'? It allows user jars to be put in the classpath before hadoop jars. -Original Message- From: Sky USC sky...@hotmail.com Reply-To: common-user@hadoop.apache.org common-user@hadoop.apache.org Date: Mon, 9 Apr 2012 15:46:52 -0500 To: common-user@hadoop.apache.org common-user@hadoop.apache.org Subject: RE: How do I include the newer version of Commons-lang in my jar? Thanks for the reply. I appreciate your helpfulness. I created Jars by following instructions at http://blog.mafr.de/2010/07/24/maven-hadoop-job/;. So external Jars are stored in lib/ folder within a jar. Am I summarizing this correctly: 1. If hadoop version = 0.20.203 or lower - then, there is not possible for me to use an external jar such as commons-lang from apache in my application. Any external jars packaged within my jar under lib directory are not captured. This appears like a huge limitation to me? 2. If hadoop version 0.20.204 to 1.0.x - then use HADOOP_USER_CLASSPATH_FIRST=true environment variable before launching hadoop jar might help. I tried this for version 0.20.205 but it didnt work. 3. If hadoop version 2.x or formerly 0.23.x - then this can be set via API? Is there a working version of testable jar that has these dependencies that I can try to figure out if its my way of packaging jar or something else?? Thx From: ha...@cloudera.com Date: Mon, 9 Apr 2012 13:50:37 +0530 Subject: Re: How do I include the newer version of Commons-lang in my jar? To: common-user@hadoop.apache.org Answer is a bit messy. Perhaps you can set the environment variable export HADOOP_USER_CLASSPATH_FIRST=true before you do a hadoop jar Š to launch your job. However, although this approach is present in 0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on the tasks as well. I don't see it changing anything but for the driver CP. I've not tested it - please let us know if it works in your environment. In higher versions (2.x or formerly 0.23.x), this is doable from within your job if you set mapreduce.job.user.classpath.first to true inside your job, and ship your replacement jars along. Some versions would also let you set this via JobConf/Job.setUserClassesTakesPrecedence(true/false) API calls. On Mon, Apr 9, 2012 at 11:14 AM, Sky sky...@hotmail.com wrote: Hi. I am new to Hadoop and I am working on project on AWS Elastic MapReduce. The problem I am facing is: * org.apache.commons.lang.time.DateUtils: parseDate() works OK but parseDateStrictly() fails. I think parseDateStrictly might be new in lang 2.5. I thought I included all dependencies. However, for some reason, during runtime, my app is not picking up the newer commons-lang. Would love some help. Thx - sky -- Harsh J
Re: How do I include the newer version of Commons-lang in my jar?
Answer is a bit messy. Perhaps you can set the environment variable export HADOOP_USER_CLASSPATH_FIRST=true before you do a hadoop jar … to launch your job. However, although this approach is present in 0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on the tasks as well. I don't see it changing anything but for the driver CP. I've not tested it - please let us know if it works in your environment. In higher versions (2.x or formerly 0.23.x), this is doable from within your job if you set mapreduce.job.user.classpath.first to true inside your job, and ship your replacement jars along. Some versions would also let you set this via JobConf/Job.setUserClassesTakesPrecedence(true/false) API calls. On Mon, Apr 9, 2012 at 11:14 AM, Sky sky...@hotmail.com wrote: Hi. I am new to Hadoop and I am working on project on AWS Elastic MapReduce. The problem I am facing is: * org.apache.commons.lang.time.DateUtils: parseDate() works OK but parseDateStrictly() fails. I think parseDateStrictly might be new in lang 2.5. I thought I included all dependencies. However, for some reason, during runtime, my app is not picking up the newer commons-lang. Would love some help. Thx - sky -- Harsh J
RE: How do I include the newer version of Commons-lang in my jar?
Thanks for the reply. I appreciate your helpfulness. I created Jars by following instructions at http://blog.mafr.de/2010/07/24/maven-hadoop-job/;. So external Jars are stored in lib/ folder within a jar. Am I summarizing this correctly: 1. If hadoop version = 0.20.203 or lower - then, there is not possible for me to use an external jar such as commons-lang from apache in my application. Any external jars packaged within my jar under lib directory are not captured. This appears like a huge limitation to me? 2. If hadoop version 0.20.204 to 1.0.x - then use HADOOP_USER_CLASSPATH_FIRST=true environment variable before launching hadoop jar might help. I tried this for version 0.20.205 but it didnt work. 3. If hadoop version 2.x or formerly 0.23.x - then this can be set via API? Is there a working version of testable jar that has these dependencies that I can try to figure out if its my way of packaging jar or something else?? Thx From: ha...@cloudera.com Date: Mon, 9 Apr 2012 13:50:37 +0530 Subject: Re: How do I include the newer version of Commons-lang in my jar? To: common-user@hadoop.apache.org Answer is a bit messy. Perhaps you can set the environment variable export HADOOP_USER_CLASSPATH_FIRST=true before you do a hadoop jar … to launch your job. However, although this approach is present in 0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on the tasks as well. I don't see it changing anything but for the driver CP. I've not tested it - please let us know if it works in your environment. In higher versions (2.x or formerly 0.23.x), this is doable from within your job if you set mapreduce.job.user.classpath.first to true inside your job, and ship your replacement jars along. Some versions would also let you set this via JobConf/Job.setUserClassesTakesPrecedence(true/false) API calls. On Mon, Apr 9, 2012 at 11:14 AM, Sky sky...@hotmail.com wrote: Hi. I am new to Hadoop and I am working on project on AWS Elastic MapReduce. The problem I am facing is: * org.apache.commons.lang.time.DateUtils: parseDate() works OK but parseDateStrictly() fails. I think parseDateStrictly might be new in lang 2.5. I thought I included all dependencies. However, for some reason, during runtime, my app is not picking up the newer commons-lang. Would love some help. Thx - sky -- Harsh J
Re: How do I include the newer version of Commons-lang in my jar?
Sky, Yes you are right in your summary. Thanks also for reporting back on the 0.205/1.x issue. One ugly hack that just comes to mind, would be to manipulate the classloader at runtime. I've not tried it out to know for sure if its possible/will work, but just thought I'd note it down. However, your statement then, there is not possible for me to use an external jar such as commons-lang from apache in my application. Any external jars packaged within my jar under lib directory are not captured. needs some refinement - You can plug and send dependent jars to a job for sure - but if that jar is already present installed on the cluster (i.e., hadoop uses it via its $HADOOP_HOME/lib/ paths), then you can't override it unless you work around. On Tue, Apr 10, 2012 at 2:16 AM, Sky USC sky...@hotmail.com wrote: Thanks for the reply. I appreciate your helpfulness. I created Jars by following instructions at http://blog.mafr.de/2010/07/24/maven-hadoop-job/;. So external Jars are stored in lib/ folder within a jar. Am I summarizing this correctly: 1. If hadoop version = 0.20.203 or lower - then, there is not possible for me to use an external jar such as commons-lang from apache in my application. Any external jars packaged within my jar under lib directory are not captured. This appears like a huge limitation to me? 2. If hadoop version 0.20.204 to 1.0.x - then use HADOOP_USER_CLASSPATH_FIRST=true environment variable before launching hadoop jar might help. I tried this for version 0.20.205 but it didnt work. 3. If hadoop version 2.x or formerly 0.23.x - then this can be set via API? Is there a working version of testable jar that has these dependencies that I can try to figure out if its my way of packaging jar or something else?? Thx From: ha...@cloudera.com Date: Mon, 9 Apr 2012 13:50:37 +0530 Subject: Re: How do I include the newer version of Commons-lang in my jar? To: common-user@hadoop.apache.org Answer is a bit messy. Perhaps you can set the environment variable export HADOOP_USER_CLASSPATH_FIRST=true before you do a hadoop jar … to launch your job. However, although this approach is present in 0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on the tasks as well. I don't see it changing anything but for the driver CP. I've not tested it - please let us know if it works in your environment. In higher versions (2.x or formerly 0.23.x), this is doable from within your job if you set mapreduce.job.user.classpath.first to true inside your job, and ship your replacement jars along. Some versions would also let you set this via JobConf/Job.setUserClassesTakesPrecedence(true/false) API calls. On Mon, Apr 9, 2012 at 11:14 AM, Sky sky...@hotmail.com wrote: Hi. I am new to Hadoop and I am working on project on AWS Elastic MapReduce. The problem I am facing is: * org.apache.commons.lang.time.DateUtils: parseDate() works OK but parseDateStrictly() fails. I think parseDateStrictly might be new in lang 2.5. I thought I included all dependencies. However, for some reason, during runtime, my app is not picking up the newer commons-lang. Would love some help. Thx - sky -- Harsh J -- Harsh J
How do I include the newer version of Commons-lang in my jar?
Hi. I am new to Hadoop and I am working on project on AWS Elastic MapReduce. The problem I am facing is: * org.apache.commons.lang.time.DateUtils: parseDate() works OK but parseDateStrictly() fails. I think parseDateStrictly might be new in lang 2.5. I thought I included all dependencies. However, for some reason, during runtime, my app is not picking up the newer commons-lang. Would love some help. Thx - sky
