One more question in the same context, so do i have to create the sharelib path in hdfs prior to create sharelib using oozie-setup.sh or that path will automatically get create when i run that command? for me if i dont have that path in hdfs i am getting bellow error.
./oozie-setup.sh sharelib create -fs hdfs://localhost:8020 SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory] the destination path for sharelib is: hdfs:/user/hadoop/share/lib/lib_20160616173312 Error: Wrong FS: hdfs:/user/hadoop/share/lib/lib_20160616173312, expected: hdfs://localhost:8020 Stack trace for the error was (for debug purposes): -------------------------------------- java.lang.IllegalArgumentException: Wrong FS: hdfs:/user/hadoop/share/lib/lib_20160616173312, expected: hdfs://eqp049wo:8020 But in config i use "hdfs://eqp049wo:8020/user/hadoop/share/lib, and run that command that dir is getting created but i am getting this error, so not sure what is the right way to do this sharelib thingy, any of you know what is the best way to do this? [ShareLib update status] host = http://localhost:11000/oozie status = java.io.FileNotFoundException: File /user/hadoop/share/lib does not exist Ram On Wed, Jun 15, 2016 at 3:03 PM, rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Ok by adding bellow config it worked. > > > <property> > <name>oozie.service.WorkflowAppService.system.libpath</name> > <value>hdfs:///user/hadoop/share/lib/</value> > </property> > > <property> > > <name>oozie.service.HadoopAccessorService.hadoop.configurations</name> > <value>*=/usr/hadoop/etc/hadoop/</value> > <description> > Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is > the HOST:PORT of > the Hadoop service (JobTracker, YARN, HDFS). The wildcard '*' > configuration is > used when there is no exact match for an authority. The > HADOOP_CONF_DIR contains > the relevant Hadoop *-site.xml files. If the path is relative > is looked within > the Oozie configuration directory; though the path can be > absolute (i.e. to point > to Hadoop client conf/ directories in the local filesystem. > </description> > </property> > > On Wed, Jun 15, 2016 at 2:55 PM, rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> But i still have a question, is this sharelib path points to local FS or >> HDFS? >> >> oozie.service.WorkflowAppService.system.libpath <-- what is the correct >> syntax for this property in my case it worked if i point it to local FS >> where i have sharelibs extracted, if i give hdfs:// its not working. any >> idea what is the right way to do it? also when i create sharelib using >> oozie-setup.sh can i give static path? >> >> Ram >> >> On Wed, Jun 15, 2016 at 2:00 PM, rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> I have updated the oozie config, when i ran create sharelib its creating >>> under "the destination path for sharelib is: >>> /uap/oozie/share/lib/lib_20160615205749" how can i make to create under >>> just /uap/oozie/share/lib ? >>> >>> >>> <property> >>> <name>oozie.service.WorkflowAppService.system.libpath</name> >>> <value>/uap/oozie/share/lib/</value> >>> </property> >>> >>> I think its working now :) >>> >>> >>> >>> oozie admin -sharelibupdate -oozie http://localhost:11000/oozie >>> [ShareLib update status] >>> host = http://localhost:11000/oozie >>> status = Successful >>> sharelibDirOld = /uap/oozie/share/lib >>> sharelibDirNew = /uap/oozie/share/lib >>> >>> >>> [hadoop@localhost bin]$ oozie admin -shareliblist -oozie >>> http://localhost:11000/oozie >>> [Available ShareLib] >>> oozie >>> hive >>> distcp >>> hcatalog >>> sqoop >>> mapreduce-streaming >>> spark >>> hive2 >>> pig >>> >>> >>> Ram >>> >>> On Wed, Jun 15, 2016 at 1:51 PM, Peter Cseh <gezap...@cloudera.com> >>> wrote: >>> >>>> Check your configuration, oozie is looking for the sharelib under >>>> /user/hadoop/share/lib instead of /user/oozie/share/lib >>>> >>>> Where is your sharelib installed on hdfs? >>>> >>>> >>>> On Wed, Jun 15, 2016 at 10:43 PM, rammohan ganapavarapu < >>>> rammohanga...@gmail.com> wrote: >>>> >>>> > Yes, i did that after i send my previous email now i am not seeing >>>> that 500 >>>> > error but still dont see any thing in sharelib list. >>>> > >>>> > oozie admin -sharelibupdate -oozie http://localhost:11000/oozie >>>> > [ShareLib update status] >>>> > host = http://localhost:11000/oozie >>>> > status = java.io.FileNotFoundException: File >>>> /user/hadoop/share/lib >>>> > does not exist >>>> > >>>> > On Wed, Jun 15, 2016 at 1:40 PM, Peter Cseh <gezap...@cloudera.com> >>>> wrote: >>>> > >>>> > > Based on that e-mail thread: >>>> > > >>>> > > An update of the library from 2.1 to 2.4 into >>>> > > oozie_webapp_folder/WEB-INF/lib solved my problem >>>> > > >>>> > > >>>> > > Try copiing the jar there and restart oozie afterwards >>>> > > >>>> > > gp >>>> > > >>>> > > >>>> > > >>>> > > On Wed, Jun 15, 2016 at 10:35 PM, rammohan ganapavarapu < >>>> > > rammohanga...@gmail.com> wrote: >>>> > > >>>> > > > Peter, >>>> > > > >>>> > > > What is the processes to upgrade common-io-lib? i already have ( >>>> > manually >>>> > > > copied) that jar under $OOZIE_HOME/lib dir but still having >>>> issues. >>>> > > > >>>> > > > Thanks, >>>> > > > Ram >>>> > > > >>>> > > > On Wed, Jun 15, 2016 at 12:10 PM, Peter Cseh < >>>> gezap...@cloudera.com> >>>> > > > wrote: >>>> > > > >>>> > > > > You could try updating the common-io library to 2.4, that >>>> helped a >>>> > > > similar >>>> > > > > issue before: >>>> > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E >>>> > > > > >>>> > > > > gp >>>> > > > > >>>> > > > > On Wed, Jun 15, 2016 at 8:54 PM, rammohan ganapavarapu < >>>> > > > > rammohanga...@gmail.com> wrote: >>>> > > > > >>>> > > > > > I am getting this for update. >>>> > > > > > >>>> > > > > > oozie admin -sharelibupdate -oozie >>>> http://localhost:11000/oozie >>>> > > > > > Error: HTTP error code: 500 : Internal Server Error >>>> > > > > > >>>> > > > > > and log from oozie.log >>>> > > > > > >>>> > > > > > >>>> > > > > > 2016-06-15 18:54:04,974 WARN AuthenticationFilter:509 - >>>> > > > > SERVER[localhost] >>>> > > > > > AuthenticationToken ignored: >>>> > > > > > >>>> org.apache.hadoop.security.authentication.util.SignerException: >>>> > > Invalid >>>> > > > > > signature >>>> > > > > > >>>> > > > > > On Wed, Jun 15, 2016 at 5:28 AM, Peter Cseh < >>>> gezap...@cloudera.com >>>> > > >>>> > > > > wrote: >>>> > > > > > >>>> > > > > > > Hi Ram, >>>> > > > > > > >>>> > > > > > > Have you told oozie to update the sharelibs with the >>>> command: >>>> > > > > > > >>>> > > > > > > oozie admin -sharelibupdate -oozie >>>> http://localhost:11000/oozie >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > BRs >>>> > > > > > > >>>> > > > > > > gezapeti >>>> > > > > > > >>>> > > > > > > On Wed, Jun 15, 2016 at 1:06 AM, rammohan ganapavarapu < >>>> > > > > > > rammohanga...@gmail.com> wrote: >>>> > > > > > > >>>> > > > > > > > Hi, >>>> > > > > > > > >>>> > > > > > > > I am getting this error in oozie-error.log, i have created >>>> > > sharelib >>>> > > > > > using >>>> > > > > > > > setup script it got successful but when i do shareliblist >>>> i am >>>> > > > > getting >>>> > > > > > > > empty output but i can list it from hdfs FS, any help? >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > 1. ./oozie-setup.sh sharelib create -fs >>>> hdfs://localhost:8020 >>>> > > > > > > > the destination path for sharelib is: >>>> > > > > > > > /user/hadoop/share/lib/lib_20160614225830 >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > 2. oozie admin -shareliblist -oozie >>>> > > http://localhost:11000/oozie >>>> > > > > > > > [Available ShareLib] >>>> > > > > > > > >>>> > > > > > > > 3. hadoop fs -ls /uap/oozie/share/lib/lib_20160614164826/ >>>> > > > > > > > Found 10 items >>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48 >>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/distcp >>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48 >>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/hcatalog >>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48 >>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/hive >>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48 >>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/hive2 >>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48 >>>> > > > > > > > >>>> /uap/oozie/share/lib/lib_20160614164826/mapreduce-streaming >>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48 >>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/oozie >>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48 >>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/pig >>>> > > > > > > > -rw-r--r-- 3 hadoop hadoop 1361 2016-06-14 16:48 >>>> > > > > > > > >>>> /uap/oozie/share/lib/lib_20160614164826/sharelib.properties >>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48 >>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/spark >>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48 >>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/sqoop >>>> > > > > > > > [hadoop@eqp049wo bin]$ >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > org.apache.oozie.service.ServiceException: E0104: Could >>>> not >>>> > fully >>>> > > > > > > > initialize service >>>> [org.apache.oozie.service.ShareLibService], >>>> > > Not >>>> > > > > able >>>> > > > > > > to >>>> > > > > > > > cache sharelib. An Admin needs to install the sharelib >>>> with >>>> > > > > > > oozie-setup.sh >>>> > > > > > > > and issue the 'oozie admin' CLI command to update the >>>> sharelib >>>> > > > > > > > org.apache.oozie.service.ServiceException: E0104: Could >>>> not >>>> > fully >>>> > > > > > > > initialize service >>>> [org.apache.oozie.service.ShareLibService], >>>> > > Not >>>> > > > > able >>>> > > > > > > to >>>> > > > > > > > cache sharelib. An Admin needs to install the sharelib >>>> with >>>> > > > > > > oozie-setup.sh >>>> > > > > > > > and issue the 'oozie admin' CLI command to update the >>>> sharelib >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > >>>> > >>>> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:139) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > >>>> > >>>> org.apache.oozie.service.Services.setServiceInternal(Services.java:386) >>>> > > > > > > > at >>>> > > > > org.apache.oozie.service.Services.setService(Services.java:372) >>>> > > > > > > > at >>>> > > > > > >>>> org.apache.oozie.service.Services.loadServices(Services.java:305) >>>> > > > > > > > at >>>> > org.apache.oozie.service.Services.init(Services.java:213) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5068) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584) >>>> > > > > > > > at >>>> > > > > > > >>>> > > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > >>>> > >>>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > >>>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > >>>> > org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > >>>> > >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> > > > > > > > at >>>> java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> > > > > > > > at java.lang.Thread.run(Thread.java:745) >>>> > > > > > > > Caused by: java.lang.NoClassDefFoundError: >>>> > > > > > org/apache/commons/io/Charsets >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > >>>> org.apache.hadoop.security.Credentials.<clinit>(Credentials.java:222) >>>> > > > > > > > at >>>> > org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.oozie.service.HadoopAccessorService.createJobConf(HadoopAccessorService.java:297) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > >>>> > >>>> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:125) >>>> > > > > > > > ... 18 more >>>> > > > > > > > Caused by: java.lang.ClassNotFoundException: >>>> > > > > > > org.apache.commons.io.Charsets >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1858) >>>> > > > > > > > at >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701) >>>> > > > > > > > ... 22 more >>>> > > > > > > > >>>> > > > > > > > Thanks, >>>> > > > > > > > Ram >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > -- >>>> > > > > > > Peter Cseh >>>> > > > > > > Software Engineer >>>> > > > > > > <http://www.cloudera.com> >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > -- >>>> > > > > Peter Cseh >>>> > > > > Software Engineer >>>> > > > > <http://www.cloudera.com> >>>> > > > > >>>> > > > >>>> > > >>>> > > >>>> > > >>>> > > -- >>>> > > Peter Cseh >>>> > > Software Engineer >>>> > > <http://www.cloudera.com> >>>> > > >>>> > >>>> >>>> >>>> >>>> -- >>>> Peter Cseh >>>> Software Engineer >>>> <http://www.cloudera.com> >>>> >>> >>> >> >