Here comes the error. It seems a configuration file error again. I used
the default hbase-site.xml(only change the hdfs url to mine). Thanks.
14/06/20 12:26:36 INFO appmaster.SliderAppMaster: onContainersCompleted([1]
14/06/20 12:26:36 INFO appmaster.SliderAppMaster: Container Completion for
containerID=container_1403279768841_0003_01_000023, state=COMPLETE,
exitStatus=-100, diagnostics=Container released by application
14/06/20 12:26:36 INFO state.AppState: RoleStatus{name='HBASE_REGIONSERVER',
key=2, desired=1, actual=1, requested=0, releasing=0, failed=0, started=1,
startFailed=0, completed=0, failureMessage=''}
14/06/20 12:26:36 INFO state.AppState: RoleStatus{name='HBASE_MASTER',
key=1, desired=1, actual=1, requested=0, releasing=0, failed=5, started=6,
startFailed=5, completed=0, failureMessage='Failure
container_1403279768841_0003_01_000013 on host localhost'}
14/06/20 12:26:40 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:26:40 INFO agent.AgentProviderService: Installing HBASE_MASTER
on container_1403279768841_0003_01_000018.
14/06/20 12:26:40 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:26:40 INFO agent.AgentProviderService: Component operation.
Status: IN_PROGRESS
14/06/20 12:26:40 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:26:40 INFO agent.AgentProviderService: Component operation.
Status: COMPLETED
14/06/20 12:26:40 INFO agent.AgentProviderService: publishing
PublishedConfiguration{description='LogFolders' entries = 14}
14/06/20 12:26:40 INFO agent.AgentProviderService: Starting HBASE_MASTER
on container_1403279768841_0003_01_000018.
14/06/20 12:26:42 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:26:42 INFO agent.AgentProviderService: Component operation.
Status: FAILED
14/06/20 12:26:42 INFO agent.AgentProviderService: Starting HBASE_MASTER
on container_1403279768841_0003_01_000018.
14/06/20 12:26:44 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:26:44 INFO agent.ComponentCommandOrder: Cannot schedule
HBASE_REGIONSERVER START as dependency HBASE_MASTER is INSTALLED
14/06/20 12:26:44 INFO agent.AgentProviderService: Start of
HBASE_REGIONSERVER on container_1403279768841_0003_01_000003 delayed as
dependencies have not started.
14/06/20 12:26:45 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:26:45 INFO agent.AgentProviderService: Component operation.
Status: FAILED
14/06/20 12:26:45 INFO agent.AgentProviderService: Starting HBASE_MASTER
on container_1403279768841_0003_01_000018.
14/06/20 12:26:47 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:26:47 INFO agent.AgentProviderService: Component operation.
Status: FAILED
14/06/20 12:26:54 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:26:54 INFO agent.ComponentCommandOrder: Cannot schedule
HBASE_REGIONSERVER START as dependency HBASE_MASTER is INSTALLED
14/06/20 12:26:54 INFO agent.AgentProviderService: Start of
HBASE_REGIONSERVER on container_1403279768841_0003_01_000003 delayed as
dependencies have not started.
14/06/20 12:26:57 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:27:04 WARN web.SliderAmIpFilter: Could not find proxy-user
cookie, so user will not be set
14/06/20 12:27:04 INFO agent.ComponentCommandOrder: Cannot schedule
HBASE_REGIONSERVER START as dependency HBASE_MASTER is INSTALLED
14/06/20 12:27:04 INFO agent.AgentProviderService: Start of
HBASE_REGIONSERVER on container_1403279768841_0003_01_000003 delayed as
dependencies have not started.
14/06/20 12:27:08 INFO appmaster.SliderAppMaster: onContainersCompleted([1]
14/06/20 12:27:08 INFO appmaster.SliderAppMaster: Container Completion for
containerID=container_1403279768841_0003_01_000018, state=COMPLETE,
exitStatus=0, diagnostics=
14/06/20 12:27:08 INFO state.AppState: Failed container in role 1
14/06/20 12:27:08 ERROR appmaster.SliderAppMaster: Role instance
RoleInstance{container=ContainerID=container_1403279768841_0003_01_000018
nodeID=localhost:43179 http=localhost:8042 priority=1,
id='container_1403279768841_0003_01_000018', createTime=1403281590545,
startTime=1403281590552, released=false, role='HBASE_MASTER', roleId=1,
host=localhost, hostURL=http://localhost:8042, state=5, exitCode=0,
command='python ./infra/agent/slider-agent/agent/main.py --label
container_1403279768841_0003_01_000018___HBASE_MASTER --host
rzhang-HP-ZBook-15 --port 37025 ; ', diagnostics='', output=null,
environment=[AGENT_WORK_ROOT="$PWD", HADOOP_USER_NAME="root",
AGENT_LOG_ROOT="$LOG_DIRS", MALLOC_ARENA_MAX="4"]} failed
14/06/20 12:27:08 INFO state.AppState: RoleStatus{name='HBASE_REGIONSERVER',
key=2, desired=1, actual=1, requested=0, releasing=0, failed=0, started=1,
startFailed=0, completed=0, failureMessage=''}
14/06/20 12:27:08 INFO state.AppState: RoleStatus{name='HBASE_MASTER',
key=1, desired=1, actual=0, requested=0, releasing=0, failed=6, started=6,
startFailed=6, completed=0, failureMessage='Failure
container_1403279768841_0003_01_000018 on host localhost'}
14/06/20 12:27:08 ERROR appmaster.SliderAppMaster: Cluster teardown
triggered %s
org.apache.slider.core.exceptions.TriggerClusterTeardownException:
Unstable Application Instance : - failed with role HBASE_MASTER failing 6
times (6 in startup); threshold is 5 - last failure: Failure
container_1403279768841_0003_01_000018 on host localhost
at org.apache.slider.server.appmaster.state.AppState.
checkFailureThreshold(AppState.java:1394)
at org.apache.slider.server.appmaster.state.AppState.
reviewOneRole(AppState.java:1429)
at org.apache.slider.server.appmaster.state.AppState.
reviewRequestAndReleaseNodes(AppState.java:1382)
at org.apache.slider.server.appmaster.SliderAppMaster.
reviewRequestAndReleaseNodes(SliderAppMaster.java:1029)
at org.apache.slider.server.appmaster.SliderAppMaster.
onContainersCompleted(SliderAppMaster.java:996)
at org.apache.hadoop.yarn.client.api.async.impl.
AMRMClientAsyncImpl$CallbackHandlerThread.run(
AMRMClientAsyncImpl.java:303)
14/06/20 12:27:09 INFO appmaster.SliderAppMaster: Triggering shutdown of
the AM: org.apache.slider.core.exceptions.TriggerClusterTeardownException:
Unstable Application Instance : - failed with role HBASE_MASTER failing 6
times (6 in startup); threshold is 5 - last failure: Failure
container_1403279768841_0003_01_000018 on host localhost
14/06/20 12:27:09 INFO appmaster.SliderAppMaster: Process has exited with
exit code 0 mapped to 0 -ignoring
14/06/20 12:27:09 INFO state.AppState: Releasing 1 containers
14/06/20 12:27:09 INFO appmaster.SliderAppMaster: Application completed.
Signalling finish to RM
14/06/20 12:27:09 INFO appmaster.SliderAppMaster: Unregistering AM
status=FAILED message=org.apache.slider.core.exceptions.
TriggerClusterTeardownException: Unstable Application Instance : - failed
with role HBASE_MASTER failing 6 times (6 in startup); threshold is 5 -
last failure: Failure container_1403279768841_0003_01_000018 on host
localhost
14/06/20 12:27:09 INFO impl.AMRMClientImpl: Waiting for application to be
successfully unregistered.
14/06/20 12:27:09 INFO appmaster.SliderAppMaster: Exiting AM; final exit
code = 73
14/06/20 12:27:09 INFO util.ExitUtil: Exiting with status 73
14/06/20 12:27:09 INFO mortbay.log: Stopped SelectChannelConnector@0.0.0.
0:0
14/06/20 12:27:10 INFO zookeeper.ZooKeeper: Session: 0x146b6040f190006
closed
14/06/20 12:27:10 INFO zookeeper.ClientCnxn: EventThread shut down
14/06/20 12:27:10 INFO ipc.Server: Stopping server on 42119
14/06/20 12:27:10 INFO ipc.Server: Stopping IPC Server listener on 42119
14/06/20 12:27:10 INFO ipc.Server: Stopping IPC Server Responder
14/06/20 12:27:10 INFO impl.ContainerManagementProtocolProxy: Closing
proxy : localhost:43179
14/06/20 12:27:10 INFO impl.AMRMClientAsyncImpl: Interrupted while waiting
for queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at java.util.concurrent.LinkedBlockingQueue.take(
LinkedBlockingQueue.java:442)
at org.apache.hadoop.yarn.client.api.async.impl.
AMRMClientAsyncImpl$CallbackHandlerThread.run(
AMRMClientAsyncImpl.java:275)
On 06/20/2014 11:55 AM, Ted Yu wrote:
Let us know if you encounter any other error.
Cheers
On Fri, Jun 20, 2014 at 8:18 AM, Rui Zhang <rzh...@vertica.com> wrote:
I am stupid. There is a punctuation error in this configuration file.
Sorry about that.
Thanks. Billie and Ted.
On 06/20/2014 11:13 AM, Rui Zhang wrote:
My slider-client.xml file is like this:
<property>
<name>yarn.application.classpath</name>
<value>/opt/hadoop/etc/hadoop,/opt/hadoop/*,/opt/hadoop/lib/
*,/opt/hadoop/lib/native/*,/opt/hadoop/share/hadoop/
common/*,/opt/hadoop/share/hadoop/common/lib/*,/opt/
hadoop/share/hadoop/hdfs/*./opt/hadoop/share/hadoop/hdfs/
lib/*,/opt/hadoop/share/hadoop/mapreduce/*,/opt/
hadoop/share/hadoop/mapreduce/lib/*,/opt/hadoop/share/
hadoop/yarn/*,/opt/hadoop/share/hadoop/yarn/lib/*</value>
</property>
core-site.xml:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.http.staticuser.user</name>
<value>hdfs</value>
</property>
</configuration>
hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/var/data/hadoop/hdfs/nn</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>file:/var/data/hadoop/hdfs/snn</value>
</property>
<property>
<name>fs.checkpoint.edits.dir</name>
<value>file:/var/data/hadoop/hdfs/snn</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/var/data/hadoop/hdfs/dn</value>
</property>
</configuration>
I am not using Ambari. I installed my yarn in /opt/hadoop.
Thanks.
On 06/19/2014 07:22 PM, Billie Rinaldi wrote:
Now it looks like it can't find the hadoop jars. Make sure the
yarn.application.classpath is set properly in your slider-client.xml
file.
Below is an example that works with an instance installed with Ambari.
<property>
<name>yarn.application.classpath</name>
<value>/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/
lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/
usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/
lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*</value>
</property>
On Thu, Jun 19, 2014 at 3:27 PM, Rui Zhang <rzh...@vertica.com> wrote:
Now there is a different error after I set the SLIDER_CLASSPATH.
14/06/19 18:24:06 WARN util.NativeCodeLoader: Unable to load
native-hadoop
library for your platform... using builtin-java classes where
applicable
14/06/19 18:24:06 INFO appmaster.SliderAppMaster: Login user is root
(auth:SIMPLE)
14/06/19 18:24:06 INFO appmaster.SliderAppMaster: Slider Core-0.30
Built
against commit# ${buildNumber} on Java 1.7.0_55 by rzhang
14/06/19 18:24:06 INFO appmaster.SliderAppMaster: Compiled against
Hadoop
2.4.0
14/06/19 18:24:06 INFO appmaster.SliderAppMaster: Hadoop runtime
version
branch-2.4.0 with source checksum 375b2832a6641759c6eaf6e3e998147 and
build date 2014-03-31T08:29Z
14/06/19 18:24:06 ERROR main.ServiceLauncher: No FileSystem for
scheme:
hdfs
Exception: No FileSystem for scheme: hdfs
14/06/19 18:24:06 ERROR main.ServiceLauncher: Exception: No FileSystem
for
scheme: hdfs
java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(
FileSystem.java:2385)
at org.apache.hadoop.fs.FileSystem.createFileSystem(
FileSystem.java:2392)
at org.apache.hadoop.fs.FileSystem.access$200(
FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(
FileSystem.java:2431)
at org.apache.hadoop.fs.FileSystem$Cache.get(
FileSystem.java:2413)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)
at org.apache.slider.common.tools.CoreFileSystem.<init>(
CoreFileSystem.java:76)
at org.apache.slider.common.tools.SliderFileSystem.<init>(
SliderFileSystem.java:38)
at org.apache.slider.server.appmaster.SliderAppMaster.
getClusterFS(SliderAppMaster.java:802)
at org.apache.slider.server.appmaster.SliderAppMaster.
createAndRunCluster(SliderAppMaster.java:427)
at org.apache.slider.server.appmaster.SliderAppMaster.
runService(SliderAppMaster.java:392)
at org.apache.slider.core.main.
ServiceLauncher.launchService(
ServiceLauncher.java:193)
at org.apache.slider.core.main.ServiceLauncher.
launchServiceRobustly(ServiceLauncher.java:425)
at org.apache.slider.core.main.ServiceLauncher.
launchServiceAndExit(ServiceLauncher.java:356)
at org.apache.slider.core.main.ServiceLauncher.serviceMain(
ServiceLauncher.java:566)
at org.apache.slider.server.appmaster.SliderAppMaster.
main(SliderAppMaster.java:1514)
14/06/19 18:24:06 INFO util.ExitUtil: Exiting with status 32
On 06/19/2014 06:03 PM, Josh Elser wrote:
Hi Rui,
My guess is that Slider isn't properly picking up your Hadoop site
configuration files and is falling back to trying to use the local
filesystem instead of HDFS
You could try copying your Hadoop core-site.xml and hdfs-site.xml
into
$SLIDER_HOME/conf which will get it on Slider's classpath.
Alternatively
(and probably better), you could try to add it to
SLIDER_CLASSPATH_EXTRA:
`export SLIDER_CLASSPATH_EXTRA=$HADOOP_CONF_DIR`
On 6/19/14, 2:49 PM, Rui Zhang wrote:
Hi,
I am new to Slider and try to run the HBase example under your
instruction.
But I have met an error, the logs for the container are:
OpenJDK 64-Bit Server VM warning: You have loaded library
/opt/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled
stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
2014-06-19 17:44:11,462 WARN [main] util.NativeCodeLoader
(NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
library for your platform... using builtin-java classes where
applicable
2014-06-19 17:44:11,520 INFO [main] appmaster.SliderAppMaster
(SliderAppMaster.java:serviceInit(337)) - Login user is root
(auth:SIMPLE)
2014-06-19 17:44:11,522 INFO [main] appmaster.SliderAppMaster
(SliderVersionInfo.java:loadAndPrintVersionInfo(95)) - Slider
Core-0.30
Built against commit# ${buildNumber} on Java 1.7.0_55 by rzhang
2014-06-19 17:44:11,522 INFO [main] appmaster.SliderAppMaster
(SliderVersionInfo.java:loadAndPrintVersionInfo(96)) - Compiled
against
Hadoop 2.4.0
2014-06-19 17:44:11,524 INFO [main] appmaster.SliderAppMaster
(SliderVersionInfo.java:loadAndPrintVersionInfo(98)) - Hadoop
runtime
version branch-2.4.0 with source checksum
375b2832a6641759c6eaf6e3e998147 and build date 2014-03-31T08:29Z
2014-06-19 17:44:11,654 ERROR [main] main.ServiceLauncher
(ServiceLauncher.java:launchServiceRobustly(454)) - Wrong FS:
hdfs://localhost:9000/user/root/.slider/cluster/cl1, expected:
file:///
2014-06-19 17:44:11,655 ERROR [main] main.ServiceLauncher
(ServiceLauncher.java:error(298)) - Exception: Wrong FS:
hdfs://localhost:9000/user/root/.slider/cluster/cl1, expected:
file:///
java.lang.IllegalArgumentException: Wrong FS:
hdfs://localhost:9000/user/root/.slider/cluster/cl1, expected:
file:///
at org.apache.hadoop.fs.FileSystem.checkPath(
FileSystem.java:643)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(
RawLocalFileSystem.java:79)
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(
RawLocalFileSystem.java:506)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(
RawLocalFileSystem.java:724)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(
RawLocalFileSystem.java:501)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(
FilterFileSystem.java:397)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.
java:1398)
at
org.apache.slider.core.persist.ConfPersister.
acquireReadLock(ConfPersister.java:174)
at
org.apache.slider.core.persist.ConfPersister.load(
ConfPersister.java:275)
at
org.apache.slider.core.build.InstanceIO.
loadInstanceDefinitionUnresolved(InstanceIO.java:54)
at
org.apache.slider.server.appmaster.SliderAppMaster.
createAndRunCluster(SliderAppMaster.java:434)
at
org.apache.slider.server.appmaster.SliderAppMaster.
runService(SliderAppMaster.java:392)
at
org.apache.slider.core.main.ServiceLauncher.launchService(
ServiceLauncher.java:193)
at
org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(
ServiceLauncher.java:425)
at
org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(
ServiceLauncher.java:356)
at
org.apache.slider.core.main.ServiceLauncher.serviceMain(
ServiceLauncher.java:566)
at
org.apache.slider.server.appmaster.SliderAppMaster.
main(SliderAppMaster.java:1514)
2014-06-19 17:44:11,657 INFO [main] util.ExitUtil
(ExitUtil.java:terminate(124)) - Exiting with status 32
What is the possible cause for this?
Thanks,
Rui
