Hi,
Unfortunately there's no enough information to post a diagnostic. See this [1] section of the docs for more information
on what's needed.
In particular the Hive logs posted as a gist since the Hive tends to wrap the underlying issue with other exceptions,
'swallowing' the root cause.
Also, I highly recommend using the latest dev version as mentioned here [2].
Cheers,
[1]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help
[2]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/install.html#download-dev
On 4/4/15 1:49 PM, ozlevka wrote:
Hello!
I have problem with writing data to elasticsearch as described in:
http://www.elastic.co/guide/en/elasticsearch/hadoop/2.0/hive.html#_writing_data_to_elasticsearch_2
I use cloudera-quickstart-vm-5.3.0-0-vmware to test how working with
eslasticsearch in hadoop.
The queries from hive to elasticsearch working as well but writing data as
describe in tutorial is every time failed.
Log for this task is:
Diagnostics:
Application application_1428135716222_0003 failed 2 times due to AM Container
for appattempt_1428135716222_0003_000002
exited with exitCode: 1 due to: Exception from container-launch.
Container id: container_1428135716222_0003_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
in yarn-yarn-nodemanager-quickstart.cloudera.log:
2015-04-04 13:14:59,618 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Start
request for container_1428135716222_0003_02_000001 by user cloudera
2015-04-04 13:14:59,618 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger:
USER=clouderaIP=127.0.0.1OPERATION=Start Container
RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1428135716222_0003CONTAINERID=container_1428135716222_0003_02_000001
2015-04-04 13:14:59,618 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Adding
container_1428135716222_0003_02_000001 to application
application_1428135716222_0003
2015-04-04 13:14:59,619 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container
container_1428135716222_0003_02_000001 transitioned from NEW to LOCALIZING
2015-04-04 13:14:59,619 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event
CONTAINER_INIT for appId application_1428135716222_0003
2015-04-04 13:14:59,619 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container
container_1428135716222_0003_02_000001 transitioned from LOCALIZING to LOCALIZED
2015-04-04 13:14:59,636 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container
container_1428135716222_0003_02_000001 transitioned from LOCALIZED to RUNNING
2015-04-04 13:14:59,642 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
launchContainer: [nice,
-n, 0, bash,
/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1428135716222_0003/container_1428135716222_0003_02_000001/default_container_executor.sh]
2015-04-04 13:15:00,625 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code
from
container container_1428135716222_0003_02_000001 is : 1
2015-04-04 13:15:00,625 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception
from
container-launch with container ID: container_1428135716222_0003_02_000001 and
exit code: 1
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
container-launch.
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
container_1428135716222_0003_02_000001
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
ExitCodeException
exitCode=1:
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.util.Shell.run(Shell.java:455)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
java.util.concurrent.FutureTask.run(FutureTask.java:262)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
java.lang.Thread.run(Thread.java:745)
2015-04-04 13:15:00,626 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Container exited with a non-zero exit code 1
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container
container_1428135716222_0003_02_000001 transitioned from RUNNING to
EXITED_WITH_FAILURE
2015-04-04 13:15:00,626 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_1428135716222_0003_02_000001
2015-04-04 13:15:00,644 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path
:
/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1428135716222_0003/container_1428135716222_0003_02_000001
2015-04-04 13:15:00,644 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger:
USER=clouderaOPERATION=Container
Finished - FailedTARGET=ContainerImplRESULT=FAILUREDESCRIPTION=Container failed
with state:
EXITED_WITH_FAILUREAPPID=application_1428135716222_0003CONTAINERID=container_1428135716222_0003_02_000001
2015-04-04 13:15:00,644 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container
container_1428135716222_0003_02_000001 transitioned from EXITED_WITH_FAILURE to
DONE
2015-04-04 13:15:00,644 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Removing container_1428135716222_0003_02_000001 from application
application_1428135716222_0003
2015-04-04 13:15:00,644 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Considering container
container_1428135716222_0003_02_000001 for log-aggregation
2015-04-04 13:15:00,644 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event
CONTAINER_STOP for appId application_1428135716222_0003
2015-04-04 13:15:02,203 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Starting resource-monitoring for container_1428135716222_0003_02_000001
2015-04-04 13:15:02,203 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Stopping resource-monitoring for container_1428135716222_0003_02_000001
2015-04-04 13:15:02,600 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed
completed
containers from NM context: [container_1428135716222_0003_02_000001]
2015-04-04 13:15:02,602 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1428135716222_0003 transitioned from RUNNING to
APPLICATION_RESOURCES_CLEANINGUP
2015-04-04 13:15:02,603 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path
:
/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1428135716222_0003
2015-04-04 13:15:02,603 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event
APPLICATION_STOP for appId application_1428135716222_0003
2015-04-04 13:15:02,605 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1428135716222_0003 transitioned from
APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2015-04-04 13:15:02,606 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Application just
finished : application_1428135716222_0003
2015-04-04 13:15:02,606 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Starting aggregate
log-file for app application_1428135716222_0003 at
/var/log/hadoop-yarn/apps/cloudera/logs/application_1428135716222_0003/quickstart.cloudera_47277.tmp
2015-04-04 13:15:02,641 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Uploading logs for
container container_1428135716222_0003_01_000001. Current good log dirs are
/var/log/hadoop-yarn/containers
2015-04-04 13:15:02,657 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Uploading logs for
container container_1428135716222_0003_02_000001. Current good log dirs are
/var/log/hadoop-yarn/containers
2015-04-04 13:15:02,658 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/var/log/hadoop-yarn/containers/application_1428135716222_0003
2015-04-04 13:15:02,763 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Finished aggregate
log-file for app application_1428135716222_0003
2015-04-04 13:20:26,071 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for
appattempt_1428135716222_0004_000001 (auth:SIMPLE)
Anybody have some idea how to fix it?
Thanks
Lev
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
<mailto:elasticsearch+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ee24a950-2fbf-4961-8486-f372c3dfae03%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/ee24a950-2fbf-4961-8486-f372c3dfae03%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5522ACAB.8030205%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
