The install package command copies a local file to HDFS. After that you
will reference the file's location in HDFS. Your application.def should be
set to ".slider/package/jmemcached/jmemcached-1.0.1.zip" (jmemcached is the
--name you provided). You only need to install the package once, not every
time you run an application.
To get Slider to pick up Hadoop configs, add "export
HADOOP_CONF_DIR=/etc/hadoop/conf" (or whatever the correct path is for your
system) to the slider-env.sh file.
On Fri, Apr 21, 2017 at 12:39 PM, Foolish Ewe <foolish...@hotmail.com>
wrote:
> Thanks Billie for the quick and thoughtful reply.
>
>
> You are correct, I'm testing in a single node docker container and have
> slider built from source and installed during the docker build action.
>
>
> Regarding the file systems (in the docker container) I want to be precise
> and make sure that I understand what needs to be in the hdfs and what needs
> to be on the local linux (to the docker container) file system.
>
> I have not had much luck in getting slider to pickup code or configuration
> data from HDFS, is there a setting I need to apply to do this?
>
> Also does slider write any logs? Where would they be?
>
>
> Regarding the protobuf-java-2.5.0.jar, I see it in the local (not HDFS)
> directory: /usr/lib/slider/lib/
>
> # ls /usr/lib/slider/lib/protobuf-java-2.5.0.jar
>
> /usr/lib/slider/lib/protobuf-java-2.5.0.jar
>
> Is this the correct location? Should I install these jars in HDFS, if so
> what is the appropriate hdfs path?
>
>
> Regarding jmemcached-1.0.0.zip, you are correct, I forgot to update the
> jmemcached file name in the script (I updated to 1.0.1 since I built
> memcached for source to create an uber jar using the maven assembly
> plugin), I should have caught that.
>
> The revised test script was updated to reflect this the install-package
> command now has the correct version number in the zip name and now reads:
>
>
> So now the script reads (the commented out lines using hdfs did not work,
> not sure why, even when I change appConfig to use hdfs url paths):
>
> #!/bin/bash
>
> # this does not work when run during docker-build, we need to do this once
> we are logged in
>
> echo "setting up the enviornment variables"
>
> env
>
> export JAVA_HOME="/usr/lib/jvm/java"
>
> export HADOOP_CONF="/etc/hadoop/conf"
>
> env
>
> echo "memcached built, registering memcached with slider and then running
> it"
>
> /usr/lib/slider/bin/slider install-package --package
> /tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
> --name jmemcached --debug
>
> #echo "using hdfs version, posting artifacts to hdfs"
>
> #hadoop fs -put ./incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
> ./incubator-slider/app-packages/memcached/*.json
> /user/root/artifacts/memcached
>
> #/usr/lib/slider/bin/slider install-package --package
> hdfs://quickstart.cloudera/user/root/artifacts/memcached/jmemcached-1.0.1.zip
> --name jmemcached --debug
>
> echo "Listing applications - before starting jmemcached"
>
> /usr/lib/slider/bin/slider list --manager localhost:8032
>
> # startup memcached properly, yarn task name will be mymemcached
>
> /usr/lib/slider/bin/slider create mymemcached --template
> /tmp/mybuild/incubator-slider/app-packages/memcached/appConfig.json
> --resources
> /tmp/mybuild/incubator-slider/app-packages/memcached/resources-default.json
> --manager localhost:8032 --debug
>
> #/usr/lib/slider/bin/slider create mymemcached --template
> hdfs://quickstart.cloudera/user/root/artifacts/memcached/appConfig.json
> --resources
> hdfs://quickstart.cloudera/user/root/artifacts/memcached/resources-default.json
> --manager localhost:8032 --debug
>
> echo "Listing applications - after starting jmemcached"
>
> /usr/lib/slider/bin/slider list --manager localhost:8032
>
> echo "Finished $0"
>
>
> appConfig.json reads:
>
> {
>
> "schema": "http://example.org/specification/v2.0.0";,
>
> "metadata": {
>
> },
>
> "global": {
>
> "agent.conf": "/tmp/mybuild/agent/conf/agent.ini",
>
> "application.def": "/tmp/mybuild/incubator-
> slider/app-packages/memcached/jmemcached-1.0.1.zip",
>
> "java_home": "/usr/lib/jvm/java/",
>
> "site.global.additional_cp": "/tmp/mybuild/incubator-
> slider/slider-assembly/target/lib/",
>
> "site.global.xmx_val": "256m",
>
> "site.global.xms_val": "128m",
>
> "site.global.memory_val": "200M",
>
> "site.global.listen_port": "${MEMCACHED.ALLOCATED_PORT}{
> PER_CONTAINER}"
>
> },
>
> "components": {
>
> "slider-appmaster": {
>
> "jvm.heapsize": "256M"
>
> }
>
> }
>
> }
>
>
> /tmp/mybuild/agent/conf/agent.ini reads:
>
>
> app_task_dir=.
>
> app_log_dir=.
>
> app_tmp_dir=app/tmp
>
>
> log_dir=.
>
> run_dir=infra/run
>
> version_file=infra/version
>
>
> ;log_level=INFO
>
> log_level=DEBUG
>
>
> [python]
>
>
> [command]
>
> max_retries=2
>
> sleep_between_retries=1
>
> auto_restart=5,5
>
>
> [security]
>
>
> [heartbeat]
>
> state_interval=6
>
> log_lines_count=300
>
>
>
> And then running the corrected script I see:
>
> + bash -x ./test_slider.sh
>
> + echo 'setting up the enviornment variables'
>
> setting up the enviornment variables
>
> + env
>
> HOSTNAME=quickstart.cloudera
>
> TERM=xterm
>
> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=
> 01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=
> 37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:
> *.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:
> *.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;
> 31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;
> 31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*
> .lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*
> .tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:
> *.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:
> *.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;
> 31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=
> 01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=
> 01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.
> svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:
> *.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=
> 01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.
> vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.
> rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:
> *.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:
> *.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:
> *.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:
> *.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=
> 01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=
> 01;36:*.spx=01;36:*.xspf=01;36:
>
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>
> PWD=/tmp/mybuild
>
> LANG=en_US.UTF-8
>
> SHLVL=3
>
> HOME=/root
>
> no_proxy=*.local, 169.254/16
>
> LESSOPEN=||/usr/bin/lesspipe.sh %s
>
> container=docker
>
> _=/usr/bin/env
>
> + export JAVA_HOME=/usr/lib/jvm/java
>
> + JAVA_HOME=/usr/lib/jvm/java
>
> + export HADOOP_CONF=/etc/hadoop/conf
>
> + HADOOP_CONF=/etc/hadoop/conf
>
> + env
>
> HOSTNAME=quickstart.cloudera
>
> TERM=xterm
>
> HADOOP_CONF=/etc/hadoop/conf
>
> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=
> 01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=
> 37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:
> *.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:
> *.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;
> 31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;
> 31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*
> .lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*
> .tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:
> *.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:
> *.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;
> 31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=
> 01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=
> 01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.
> svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:
> *.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=
> 01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.
> vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.
> rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:
> *.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:
> *.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:
> *.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:
> *.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=
> 01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=
> 01;36:*.spx=01;36:*.xspf=01;36:
>
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>
> PWD=/tmp/mybuild
>
> JAVA_HOME=/usr/lib/jvm/java
>
> LANG=en_US.UTF-8
>
> SHLVL=3
>
> HOME=/root
>
> no_proxy=*.local, 169.254/16
>
> LESSOPEN=||/usr/bin/lesspipe.sh %s
>
> container=docker
>
> _=/usr/bin/env
>
> + echo 'memcached built, registering memcached with slider and then
> running it'
>
> memcached built, registering memcached with slider and then running it
>
> + /usr/lib/slider/bin/slider install-package --package
> /tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
> --name jmemcached --debug --replacepkg
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> slider-core-0.60.0-incubating.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
>
> SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 2017-04-21 19:22:12,948 [main] INFO client.RMProxy - Connecting to
> ResourceManager at /0.0.0.0:8032
>
> 2017-04-21 19:22:13,093 [main] INFO client.SliderClient - Installing
> package
> file:/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
> at file:/root/.slider/package/jmemcached/jmemcached-1.0.1.zip and
> overwrite is true.
>
> 2017-04-21 19:22:13,109 [main] INFO util.ExitUtil - Exiting with status 0
>
> + echo 'Listing applications - before starting jmemcached'
>
> Listing applications - before starting jmemcached
>
> + /usr/lib/slider/bin/slider list --manager localhost:8032
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> slider-core-0.60.0-incubating.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
>
> SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 2017-04-21 19:22:15,070 [main] INFO client.RMProxy - Connecting to
> ResourceManager at localhost/127.0.0.1:8032
>
> 2017-04-21 19:22:15,222 [main] INFO util.ExitUtil - Exiting with status 0
>
> + /usr/lib/slider/bin/slider create mymemcached --template
> /tmp/mybuild/incubator-slider/app-packages/memcached/appConfig.json
> --resources
> /tmp/mybuild/incubator-slider/app-packages/memcached/resources-default.json
> --manager localhost:8032 --debug
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> slider-core-0.60.0-incubating.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
>
> SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 2017-04-21 19:22:17,183 [main] INFO client.RMProxy - Connecting to
> ResourceManager at localhost/127.0.0.1:8032
>
> 2017-04-21 19:22:17,661 [main] INFO agent.AgentClientProvider -
> Validating app definition /tmp/mybuild/incubator-slider/
> app-packages/memcached/jmemcached-1.0.1.zip
>
> 2017-04-21 19:22:17,662 [main] INFO agent.AgentUtils - Reading metainfo
> at /tmp/mybuild/incubator-slider/app-packages/memcached/
> jmemcached-1.0.1.zip
>
> 2017-04-21 19:22:17,685 [main] INFO tools.SliderUtils - Reading
> metainfo.xml of size 2202
>
> 2017-04-21 19:22:17,909 [main] INFO client.SliderClient - No credentials
> requested
>
> 2017-04-21 19:22:18,108 [main] INFO agent.AgentUtils - Reading metainfo
> at /tmp/mybuild/incubator-slider/app-packages/memcached/
> jmemcached-1.0.1.zip
>
> 2017-04-21 19:22:18,116 [main] INFO tools.SliderUtils - Reading
> metainfo.xml of size 2202
>
> 2017-04-21 19:22:18,177 [main] INFO launch.AbstractLauncher - Log include
> patterns:
>
> 2017-04-21 19:22:18,177 [main] INFO launch.AbstractLauncher - Log exclude
> patterns:
>
> 2017-04-21 19:22:18,938 [main] INFO slideram.SliderAMClientProvider -
> Loading all dependencies for AM.
>
> 2017-04-21 19:22:18,939 [main] INFO tools.SliderUtils - Loading all
> dependencies from /usr/lib/slider/lib
>
> 2017-04-21 19:22:20,687 [main] INFO agent.AgentClientProvider -
> Automatically uploading the agent tarball at file:/root/.slider/cluster/
> mymemcached/tmp/application_1492799085220_0003/agent
>
> 2017-04-21 19:22:20,727 [main] INFO agent.AgentClientProvider -
> Validating app definition /tmp/mybuild/incubator-slider/
> app-packages/memcached/jmemcached-1.0.1.zip
>
> 2017-04-21 19:22:20,727 [main] INFO agent.AgentUtils - Reading metainfo
> at /tmp/mybuild/incubator-slider/app-packages/memcached/
> jmemcached-1.0.1.zip
>
> 2017-04-21 19:22:20,735 [main] INFO tools.SliderUtils - Reading
> metainfo.xml of size 2202
>
> 2017-04-21 19:22:20,807 [main] INFO Configuration.deprecation -
> slider.registry.path is deprecated. Instead, use hadoop.registry.zk.root
>
> 2017-04-21 19:22:20,813 [main] INFO launch.AppMasterLauncher - Submitting
> application to Resource Manager
>
> 2017-04-21 19:22:20,838 [main] INFO impl.YarnClientImpl - Submitted
> application application_1492799085220_0003
>
> 2017-04-21 19:22:20,845 [main] INFO util.ExitUtil - Exiting with status 0
>
> + echo 'Listing applications - after starting jmemcached'
>
> Listing applications - after starting jmemcached
>
> + /usr/lib/slider/bin/slider list --manager localhost:8032
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> slider-core-0.60.0-incubating.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
>
> SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 2017-04-21 19:22:22,857 [main] INFO client.RMProxy - Connecting to
> ResourceManager at localhost/127.0.0.1:8032
>
> mymemcached ACCEPTED application_1492799085220_0003
>
> 2017-04-21 19:22:23,142 [main] INFO util.ExitUtil - Exiting with status 0
>
> + echo 'Finished ./test_slider.sh'
>
> Finished ./test_slider.sh
>
> I am still getting the dreaded failure as reported in the yarn resource
> manager:
> Application application_1492799085220_0003 failed 2 times due to AM
> Container for appattempt_1492799085220_0003_000002 exited with exitCode:
> -1000
> For more detailed output, check application tracking page:
> http://quickstart.cloudera:8088/proxy/application_1492799085220_0003/Then,
> click on links to logs of each attempt.
> Diagnostics: File file:/root/.slider/cluster/mymemcached/tmp/application_
> 1492799085220_0003/am/lib/protobuf-java-2.5.0.jar does not exist
> java.io.FileNotFoundException: File file:/root/.slider/cluster/
> mymemcached/tmp/application_1492799085220_0003/am/lib/protobuf-java-2.5.0.jar
> does not exist
> at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(
> RawLocalFileSystem.java:542)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(
> RawLocalFileSystem.java:755)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(
> RawLocalFileSystem.java:532)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(
> FilterFileSystem.java:425)
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1693)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Failing this attempt. Failing the application.
>
>
>
> With best regards:
>
> Bill
>
> ________________________________
> From: Billie Rinaldi <billie.rina...@gmail.com>
> Sent: Wednesday, April 19, 2017 5:36 PM
> To: dev@slider.incubator.apache.org
> Subject: Re: Trying to get he memcached example to run for a new build
>
> The protobuf-java-2.5.0.jar should be in the Slider lib directory. It will
> get pushed out to the AM lib directory from there.
>
> There are some other issues with the script output. The install package
> command has not succeeded, and says "ERROR main.ServiceLauncher - Unable to
> access supplied pkg file at /tmp/mybuild/incubator-slider/
> app-packages/memcached/jmemcached-1.0.0.zip"
>
> Also the line INFO agent.AgentClientProvider - Validating app definition
> /tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
> indicates that it is reading the app definition from the local filesystem,
> when it should be reading it from HDFS. Maybe since this is a single VM
> setup it is able to continue despite these errors, but if you were running
> on a real cluster you'd have to make sure install package was working and
> fix the app definition path in the appConfig.
>
> On Tue, Apr 18, 2017 at 6:35 PM, Foolish Ewe <foolish...@hotmail.com>
> wrote:
>
> > Hello All:
> >
> >
> > I'm trying to run the memcached example locally but it fails pretty quick
> > and claims "protobuf-java-2.5.0.jar does not exist".
> >
> >
> > Consider the following script (test-slider.sh):
> >
> > #!/bin/bash
> >
> > # this does not work when run during docker-build, we need to do this
> once
> > we are logged in
> >
> > echo "memcached built, registering memcached with slider and then running
> > it"
> >
> > /usr/lib/slider/bin/slider install-package --package
> > /tmp/mybuild/incubator-slider/app-packages/memcached/
> jmemcached-1.0.0.zip
> > --name jmemcached --debug
> >
> > echo "Listing applications - before starting jmemcached"
> >
> > /usr/lib/slider/bin/slider list --manager localhost:8032
> >
> > # startup memcached properly, yarn task name will be mymemcached
> >
> > /usr/lib/slider/bin/slider create mymemcached --template
> > /tmp/mybuild/incubator-slider/app-packages/memcached/appConfig.json
> > --resources /tmp/mybuild/incubator-slider/app-packages/memcached/
> resources-default.json
> > --manager localhost:8032 --debug
> >
> > echo "Listing applications - after starting jmemcached"
> >
> > /usr/lib/slider/bin/slider list --manager localhost:8032
> >
> > echo "Finished $0"
> >
> > I am having difficulty in locating the logs after the failure, any idea
> > where they would be? yarn logs cannot find them, and I'm not having luck
> > with the name node. I did modify the yarn-site.xml read at startup to
> > include the lines:
> >
> > <!-- Begin modifications for debugging slider -->
> >
> > <!-- 60 minutes after a failure to see what is left in the directory-->
> >
> > <property>
> >
> > <name>yarn.nodemanager.delete.debug-delay-sec</name>
> >
> > <value>3600</value>
> >
> > </property>
> >
> >
> > <!--time before the process gets a -9 (Should it be 30 seconds?)-->
> >
> > <property>
> >
> > <name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name>
> >
> > <value>3600000</value>
> >
> > </property>
> >
> > <!-- End modifications for debugging slider -->
> >
> >
> > Regarding the error, I'm seeing, the slider install-package appears to
> > work correctly from what I can tell, but the slider create encounters run
> > time errors when launched, it seems to be looking for
> > protobuf-java-2.5.0.jar. I tried making fat jars for the memcached
> > application and the various slider packages, to no avail. Any ideas how
> to
> > resolve these errors reported in the resource manager:
> > Application application_1492557996590_0002 failed 2 times due to AM
> > Container for appattempt_1492557996590_0002_000002 exited with exitCode:
> > -1000
> > For more detailed output, check application tracking page:
> > http://quickstart.cloudera:8088/proxy/application_
> 1492557996590_0002/Then,
> > click on links to logs of each attempt.
> > Diagnostics: File file:/root/.slider/cluster/
> mymemcached/tmp/application_
> > 1492557996590_0002/am/lib/protobuf-java-2.5.0.jar does not exist
> > java.io.FileNotFoundException: File file:/root/.slider/cluster/
> > mymemcached/tmp/application_1492557996590_0002/am/lib/
> protobuf-java-2.5.0.jar
> > does not exist
> > at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(
> > RawLocalFileSystem.java:542)
> > at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(
> > RawLocalFileSystem.java:755)
> > at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(
> > RawLocalFileSystem.java:532)
> > at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(
> > FilterFileSystem.java:425)
> > at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
> > at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
> > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
> > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at org.apache.hadoop.security.UserGroupInformation.doAs(
> > UserGroupInformation.java:1693)
> > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
> > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Failing this attempt. Failing the application.
> >
> >
> > The test-slider.sh script's output is as follows:
> >
> > # bash -x /tmp/mybuild/test_slider.sh
> >
> > + echo 'memcached built, registering memcached with slider and then
> > running it'
> >
> > memcached built, registering memcached with slider and then running it
> >
> > + /usr/lib/slider/bin/slider install-package --package
> > /tmp/mybuild/incubator-slider/app-packages/memcached/
> jmemcached-1.0.0.zip
> > --name jmemcached --debug
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> >
> > SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> > slider-core-0.60.0-incubating.jar!/org/slf4j/impl/
> > StaticLoggerBinder.class]
> >
> > SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> > slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> SLF4J Error Codes<http://www.slf4j.org/codes.html#multiple_bindings>
> www.slf4j.org
> SLF4J warning or error messages and their meanings No SLF4J providers were
> found. This warning, i.e. not an error, message is reported when no SLF4J
> ...
>
>
>
> > explanation.
> >
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >
> > 2017-04-19 01:10:44,080 [main] INFO client.RMProxy - Connecting to
> > ResourceManager at /0.0.0.0:8032
> >
> > 2017-04-19 01:10:44,243 [main] ERROR main.ServiceLauncher - Unable to
> > access supplied pkg file at /tmp/mybuild/incubator-slider/
> > app-packages/memcached/jmemcached-1.0.0.zip
> >
> > 2017-04-19 01:10:44,247 [main] INFO util.ExitUtil - Exiting with status
> 40
> >
> > + echo 'Listing applications - before starting jmemcached'
> >
> > Listing applications - before starting jmemcached
> >
> > + /usr/lib/slider/bin/slider list --manager localhost:8032
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> >
> > SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> > slider-core-0.60.0-incubating.jar!/org/slf4j/impl/
> > StaticLoggerBinder.class]
> >
> > SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> > slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> SLF4J Error Codes<http://www.slf4j.org/codes.html#multiple_bindings>
> www.slf4j.org
> SLF4J warning or error messages and their meanings No SLF4J providers were
> found. This warning, i.e. not an error, message is reported when no SLF4J
> ...
>
>
>
> > explanation.
> >
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >
> > 2017-04-19 01:10:46,066 [main] INFO client.RMProxy - Connecting to
> > ResourceManager at localhost/127.0.0.1:8032
> >
> > 2017-04-19 01:10:46,217 [main] INFO util.ExitUtil - Exiting with status
> 0
> >
> > + /usr/lib/slider/bin/slider create mymemcached --template
> > /tmp/mybuild/incubator-slider/app-packages/memcached/appConfig.json
> > --resources /tmp/mybuild/incubator-slider/app-packages/memcached/
> resources-default.json
> > --manager localhost:8032 --debug
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> >
> > SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> > slider-core-0.60.0-incubating.jar!/org/slf4j/impl/
> > StaticLoggerBinder.class]
> >
> > SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> > slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> SLF4J Error Codes<http://www.slf4j.org/codes.html#multiple_bindings>
> www.slf4j.org
> SLF4J warning or error messages and their meanings No SLF4J providers were
> found. This warning, i.e. not an error, message is reported when no SLF4J
> ...
>
>
>
> > explanation.
> >
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >
> > 2017-04-19 01:10:48,070 [main] INFO client.RMProxy - Connecting to
> > ResourceManager at localhost/127.0.0.1:8032
> >
> > 2017-04-19 01:10:48,647 [main] INFO agent.AgentClientProvider -
> > Validating app definition /tmp/mybuild/incubator-slider/
> > app-packages/memcached/jmemcached-1.0.1.zip
> >
> > 2017-04-19 01:10:48,648 [main] INFO agent.AgentUtils - Reading metainfo
> > at /tmp/mybuild/incubator-slider/app-packages/memcached/
> > jmemcached-1.0.1.zip
> >
> > 2017-04-19 01:10:48,675 [main] INFO tools.SliderUtils - Reading
> > metainfo.xml of size 2202
> >
> > 2017-04-19 01:10:48,893 [main] INFO client.SliderClient - No credentials
> > requested
> >
> > 2017-04-19 01:10:48,938 [main] INFO agent.AgentUtils - Reading metainfo
> > at /tmp/mybuild/incubator-slider/app-packages/memcached/
> > jmemcached-1.0.1.zip
> >
> > 2017-04-19 01:10:48,939 [main] INFO tools.SliderUtils - Reading
> > metainfo.xml of size 2202
> >
> > 2017-04-19 01:10:48,976 [main] INFO launch.AbstractLauncher - Log
> include
> > patterns:
> >
> > 2017-04-19 01:10:48,976 [main] INFO launch.AbstractLauncher - Log
> exclude
> > patterns:
> >
> > 2017-04-19 01:10:49,861 [main] INFO slideram.SliderAMClientProvider -
> > Loading all dependencies for AM.
> >
> > 2017-04-19 01:10:49,862 [main] INFO tools.SliderUtils - Loading all
> > dependencies from /usr/lib/slider/lib
> >
> > 2017-04-19 01:10:51,324 [main] INFO agent.AgentClientProvider -
> > Automatically uploading the agent tarball at file:/root/.slider/cluster/
> > mymemcached/tmp/application_1492557996590_0003/agent
> >
> > 2017-04-19 01:10:51,361 [main] INFO agent.AgentClientProvider -
> > Validating app definition /tmp/mybuild/incubator-slider/
> > app-packages/memcached/jmemcached-1.0.1.zip
> >
> > 2017-04-19 01:10:51,361 [main] INFO agent.AgentUtils - Reading metainfo
> > at /tmp/mybuild/incubator-slider/app-packages/memcached/
> > jmemcached-1.0.1.zip
> >
> > 2017-04-19 01:10:51,364 [main] INFO tools.SliderUtils - Reading
> > metainfo.xml of size 2202
> >
> > 2017-04-19 01:10:51,429 [main] INFO Configuration.deprecation -
> > slider.registry.path is deprecated. Instead, use hadoop.registry.zk.root
> >
> > 2017-04-19 01:10:51,436 [main] INFO launch.AppMasterLauncher -
> Submitting
> > application to Resource Manager
> >
> > 2017-04-19 01:10:51,480 [main] INFO impl.YarnClientImpl - Submitted
> > application application_1492557996590_0003
> >
> > 2017-04-19 01:10:51,484 [main] INFO util.ExitUtil - Exiting with status
> 0
> >
> > + echo 'Listing applications - after starting jmemcached'
> >
> > Listing applications - after starting jmemcached
> >
> > + /usr/lib/slider/bin/slider list --manager localhost:8032
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> >
> > SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> > slider-core-0.60.0-incubating.jar!/org/slf4j/impl/
> > StaticLoggerBinder.class]
> >
> > SLF4J: Found binding in [jar:file:/usr/lib/slider/lib/
> > slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> SLF4J Error Codes<http://www.slf4j.org/codes.html#multiple_bindings>
> www.slf4j.org
> SLF4J warning or error messages and their meanings No SLF4J providers were
> found. This warning, i.e. not an error, message is reported when no SLF4J
> ...
>
>
>
> > explanation.
> >
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >
> > 2017-04-19 01:10:53,893 [main] INFO client.RMProxy - Connecting to
> > ResourceManager at localhost/127.0.0.1:8032
> >
> > mymemcached ACCEPTED
> application_1492557996590_0003
> >
> > 2017-04-19 01:10:54,186 [main] INFO util.ExitUtil - Exiting with status
> 0
> >
> > + echo 'Finished /tmp/mybuild/test_slider.sh'
> >
> > Finished /tmp/mybuild/test_slider.sh
> >
> >
> > Thanks:
> >
> >
> > Bill
> >
>
