One more question in the same context, so do i have to create the sharelib
path in hdfs prior to create sharelib using oozie-setup.sh or that path
will automatically get create when i run that command? for me if i dont
have that path in hdfs i am getting bellow error.
./oozie-setup.sh sharelib create -fs hdfs://localhost:8020
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
the destination path for sharelib is:
hdfs:/user/hadoop/share/lib/lib_20160616173312
Error: Wrong FS: hdfs:/user/hadoop/share/lib/lib_20160616173312, expected:
hdfs://localhost:8020
Stack trace for the error was (for debug purposes):
--------------------------------------
java.lang.IllegalArgumentException: Wrong FS:
hdfs:/user/hadoop/share/lib/lib_20160616173312, expected:
hdfs://eqp049wo:8020
But in config i use "hdfs://eqp049wo:8020/user/hadoop/share/lib, and run
that command that dir is getting created but i am getting this error, so
not sure what is the right way to do this sharelib thingy, any of you know
what is the best way to do this?
[ShareLib update status]
host = http://localhost:11000/oozie
status = java.io.FileNotFoundException: File /user/hadoop/share/lib
does not exist
Ram
On Wed, Jun 15, 2016 at 3:03 PM, rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Ok by adding bellow config it worked.
>
>
> <property>
> <name>oozie.service.WorkflowAppService.system.libpath</name>
> <value>hdfs:///user/hadoop/share/lib/</value>
> </property>
>
> <property>
>
> <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
> <value>*=/usr/hadoop/etc/hadoop/</value>
> <description>
> Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is
> the HOST:PORT of
> the Hadoop service (JobTracker, YARN, HDFS). The wildcard '*'
> configuration is
> used when there is no exact match for an authority. The
> HADOOP_CONF_DIR contains
> the relevant Hadoop *-site.xml files. If the path is relative
> is looked within
> the Oozie configuration directory; though the path can be
> absolute (i.e. to point
> to Hadoop client conf/ directories in the local filesystem.
> </description>
> </property>
>
> On Wed, Jun 15, 2016 at 2:55 PM, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> But i still have a question, is this sharelib path points to local FS or
>> HDFS?
>>
>> oozie.service.WorkflowAppService.system.libpath <-- what is the correct
>> syntax for this property in my case it worked if i point it to local FS
>> where i have sharelibs extracted, if i give hdfs:// its not working. any
>> idea what is the right way to do it? also when i create sharelib using
>> oozie-setup.sh can i give static path?
>>
>> Ram
>>
>> On Wed, Jun 15, 2016 at 2:00 PM, rammohan ganapavarapu <
>> rammohanga...@gmail.com> wrote:
>>
>>> I have updated the oozie config, when i ran create sharelib its creating
>>> under "the destination path for sharelib is:
>>> /uap/oozie/share/lib/lib_20160615205749" how can i make to create under
>>> just /uap/oozie/share/lib ?
>>>
>>>
>>> <property>
>>> <name>oozie.service.WorkflowAppService.system.libpath</name>
>>> <value>/uap/oozie/share/lib/</value>
>>> </property>
>>>
>>> I think its working now :)
>>>
>>>
>>>
>>> oozie admin -sharelibupdate -oozie http://localhost:11000/oozie
>>> [ShareLib update status]
>>> host = http://localhost:11000/oozie
>>> status = Successful
>>> sharelibDirOld = /uap/oozie/share/lib
>>> sharelibDirNew = /uap/oozie/share/lib
>>>
>>>
>>> [hadoop@localhost bin]$ oozie admin -shareliblist -oozie
>>> http://localhost:11000/oozie
>>> [Available ShareLib]
>>> oozie
>>> hive
>>> distcp
>>> hcatalog
>>> sqoop
>>> mapreduce-streaming
>>> spark
>>> hive2
>>> pig
>>>
>>>
>>> Ram
>>>
>>> On Wed, Jun 15, 2016 at 1:51 PM, Peter Cseh <gezap...@cloudera.com>
>>> wrote:
>>>
>>>> Check your configuration, oozie is looking for the sharelib under
>>>> /user/hadoop/share/lib instead of /user/oozie/share/lib
>>>>
>>>> Where is your sharelib installed on hdfs?
>>>>
>>>>
>>>> On Wed, Jun 15, 2016 at 10:43 PM, rammohan ganapavarapu <
>>>> rammohanga...@gmail.com> wrote:
>>>>
>>>> > Yes, i did that after i send my previous email now i am not seeing
>>>> that 500
>>>> > error but still dont see any thing in sharelib list.
>>>> >
>>>> > oozie admin -sharelibupdate -oozie http://localhost:11000/oozie
>>>> > [ShareLib update status]
>>>> > host = http://localhost:11000/oozie
>>>> > status = java.io.FileNotFoundException: File
>>>> /user/hadoop/share/lib
>>>> > does not exist
>>>> >
>>>> > On Wed, Jun 15, 2016 at 1:40 PM, Peter Cseh <gezap...@cloudera.com>
>>>> wrote:
>>>> >
>>>> > > Based on that e-mail thread:
>>>> > >
>>>> > > An update of the library from 2.1 to 2.4 into
>>>> > > oozie_webapp_folder/WEB-INF/lib solved my problem
>>>> > >
>>>> > >
>>>> > > Try copiing the jar there and restart oozie afterwards
>>>> > >
>>>> > > gp
>>>> > >
>>>> > >
>>>> > >
>>>> > > On Wed, Jun 15, 2016 at 10:35 PM, rammohan ganapavarapu <
>>>> > > rammohanga...@gmail.com> wrote:
>>>> > >
>>>> > > > Peter,
>>>> > > >
>>>> > > > What is the processes to upgrade common-io-lib? i already have (
>>>> > manually
>>>> > > > copied) that jar under $OOZIE_HOME/lib dir but still having
>>>> issues.
>>>> > > >
>>>> > > > Thanks,
>>>> > > > Ram
>>>> > > >
>>>> > > > On Wed, Jun 15, 2016 at 12:10 PM, Peter Cseh <
>>>> gezap...@cloudera.com>
>>>> > > > wrote:
>>>> > > >
>>>> > > > > You could try updating the common-io library to 2.4, that
>>>> helped a
>>>> > > > similar
>>>> > > > > issue before:
>>>> > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E
>>>> > > > >
>>>> > > > > gp
>>>> > > > >
>>>> > > > > On Wed, Jun 15, 2016 at 8:54 PM, rammohan ganapavarapu <
>>>> > > > > rammohanga...@gmail.com> wrote:
>>>> > > > >
>>>> > > > > > I am getting this for update.
>>>> > > > > >
>>>> > > > > > oozie admin -sharelibupdate -oozie
>>>> http://localhost:11000/oozie
>>>> > > > > > Error: HTTP error code: 500 : Internal Server Error
>>>> > > > > >
>>>> > > > > > and log from oozie.log
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > 2016-06-15 18:54:04,974 WARN AuthenticationFilter:509 -
>>>> > > > > SERVER[localhost]
>>>> > > > > > AuthenticationToken ignored:
>>>> > > > > >
>>>> org.apache.hadoop.security.authentication.util.SignerException:
>>>> > > Invalid
>>>> > > > > > signature
>>>> > > > > >
>>>> > > > > > On Wed, Jun 15, 2016 at 5:28 AM, Peter Cseh <
>>>> gezap...@cloudera.com
>>>> > >
>>>> > > > > wrote:
>>>> > > > > >
>>>> > > > > > > Hi Ram,
>>>> > > > > > >
>>>> > > > > > > Have you told oozie to update the sharelibs with the
>>>> command:
>>>> > > > > > >
>>>> > > > > > > oozie admin -sharelibupdate -oozie
>>>> http://localhost:11000/oozie
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > BRs
>>>> > > > > > >
>>>> > > > > > > gezapeti
>>>> > > > > > >
>>>> > > > > > > On Wed, Jun 15, 2016 at 1:06 AM, rammohan ganapavarapu <
>>>> > > > > > > rammohanga...@gmail.com> wrote:
>>>> > > > > > >
>>>> > > > > > > > Hi,
>>>> > > > > > > >
>>>> > > > > > > > I am getting this error in oozie-error.log, i have created
>>>> > > sharelib
>>>> > > > > > using
>>>> > > > > > > > setup script it got successful but when i do shareliblist
>>>> i am
>>>> > > > > getting
>>>> > > > > > > > empty output but i can list it from hdfs FS, any help?
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > > 1. ./oozie-setup.sh sharelib create -fs
>>>> hdfs://localhost:8020
>>>> > > > > > > > the destination path for sharelib is:
>>>> > > > > > > > /user/hadoop/share/lib/lib_20160614225830
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > > 2. oozie admin -shareliblist -oozie
>>>> > > http://localhost:11000/oozie
>>>> > > > > > > > [Available ShareLib]
>>>> > > > > > > >
>>>> > > > > > > > 3. hadoop fs -ls /uap/oozie/share/lib/lib_20160614164826/
>>>> > > > > > > > Found 10 items
>>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48
>>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/distcp
>>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48
>>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/hcatalog
>>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48
>>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/hive
>>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48
>>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/hive2
>>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48
>>>> > > > > > > >
>>>> /uap/oozie/share/lib/lib_20160614164826/mapreduce-streaming
>>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48
>>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/oozie
>>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48
>>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/pig
>>>> > > > > > > > -rw-r--r-- 3 hadoop hadoop 1361 2016-06-14 16:48
>>>> > > > > > > >
>>>> /uap/oozie/share/lib/lib_20160614164826/sharelib.properties
>>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48
>>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/spark
>>>> > > > > > > > drwxr-xr-x - hadoop hadoop 0 2016-06-14 16:48
>>>> > > > > > > > /uap/oozie/share/lib/lib_20160614164826/sqoop
>>>> > > > > > > > [hadoop@eqp049wo bin]$
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > > org.apache.oozie.service.ServiceException: E0104: Could
>>>> not
>>>> > fully
>>>> > > > > > > > initialize service
>>>> [org.apache.oozie.service.ShareLibService],
>>>> > > Not
>>>> > > > > able
>>>> > > > > > > to
>>>> > > > > > > > cache sharelib. An Admin needs to install the sharelib
>>>> with
>>>> > > > > > > oozie-setup.sh
>>>> > > > > > > > and issue the 'oozie admin' CLI command to update the
>>>> sharelib
>>>> > > > > > > > org.apache.oozie.service.ServiceException: E0104: Could
>>>> not
>>>> > fully
>>>> > > > > > > > initialize service
>>>> [org.apache.oozie.service.ShareLibService],
>>>> > > Not
>>>> > > > > able
>>>> > > > > > > to
>>>> > > > > > > > cache sharelib. An Admin needs to install the sharelib
>>>> with
>>>> > > > > > > oozie-setup.sh
>>>> > > > > > > > and issue the 'oozie admin' CLI command to update the
>>>> sharelib
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > >
>>>> >
>>>> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:139)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > >
>>>> >
>>>> org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
>>>> > > > > > > > at
>>>> > > > > org.apache.oozie.service.Services.setService(Services.java:372)
>>>> > > > > > > > at
>>>> > > > > >
>>>> org.apache.oozie.service.Services.loadServices(Services.java:305)
>>>> > > > > > > > at
>>>> > org.apache.oozie.service.Services.init(Services.java:213)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5068)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584)
>>>> > > > > > > > at
>>>> > > > > > >
>>>> > > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > >
>>>> >
>>>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > >
>>>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > >
>>>> > org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > >
>>>> >
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> > > > > > > > at
>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> > > > > > > > at java.lang.Thread.run(Thread.java:745)
>>>> > > > > > > > Caused by: java.lang.NoClassDefFoundError:
>>>> > > > > > org/apache/commons/io/Charsets
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > >
>>>> org.apache.hadoop.security.Credentials.<clinit>(Credentials.java:222)
>>>> > > > > > > > at
>>>> > org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.oozie.service.HadoopAccessorService.createJobConf(HadoopAccessorService.java:297)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > >
>>>> >
>>>> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:125)
>>>> > > > > > > > ... 18 more
>>>> > > > > > > > Caused by: java.lang.ClassNotFoundException:
>>>> > > > > > > org.apache.commons.io.Charsets
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1858)
>>>> > > > > > > > at
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
>>>> > > > > > > > ... 22 more
>>>> > > > > > > >
>>>> > > > > > > > Thanks,
>>>> > > > > > > > Ram
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > --
>>>> > > > > > > Peter Cseh
>>>> > > > > > > Software Engineer
>>>> > > > > > > <http://www.cloudera.com>
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > > --
>>>> > > > > Peter Cseh
>>>> > > > > Software Engineer
>>>> > > > > <http://www.cloudera.com>
>>>> > > > >
>>>> > > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Peter Cseh
>>>> > > Software Engineer
>>>> > > <http://www.cloudera.com>
>>>> > >
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Peter Cseh
>>>> Software Engineer
>>>> <http://www.cloudera.com>
>>>>
>>>
>>>
>>
>
