If you're seeing it list progress (or attempted progress) as here, this
isn't a locking issue. All locks are obtained before the job is
submitted to Hadoop.
Alan.
Mich Talebzadeh <mailto:m...@peridale.co.uk>
April 7, 2015 at 14:09
Hi,
Today I have noticed the following issue.
A simple insert into a table is sting there throwing the following
hive> insert into table mytest values(1,'test');
Query ID = hduser_20150407215959_bc030fac-258f-4996-b50f-3d2d49371cca
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1428439695331_0002, Tracking URL =
http://rhes564:8088/proxy/application_1428439695331_0002/
Kill Command = /home/hduser/hadoop/hadoop-2.6.0/bin/hadoop job -kill
job_1428439695331_0002
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 0
2015-04-07 21:59:35,068 Stage-1 map = 0%, reduce = 0%
2015-04-07 22:00:35,545 Stage-1 map = 0%, reduce = 0%
2015-04-07 22:01:35,832 Stage-1 map = 0%, reduce = 0%
2015-04-07 22:02:36,058 Stage-1 map = 0%, reduce = 0%
2015-04-07 22:03:36,279 Stage-1 map = 0%, reduce = 0%
2015-04-07 22:04:36,486 Stage-1 map = 0%, reduce = 0%
I have been messing around with concurrency for hive. That did not
work. My metastore is built in Oracle. So I drooped that schema and
recreated from scratch. Got rid of concurrency parameters. First I was
getting "container is running beyond virtual memory limits" for the
task. I changed the following parameters in yarn-site.xml
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
<description>Amount of physical memory, in MB, that can be allocated
for containers.</description>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>
and mapred-site.xml
<property>
<name>mapreduce.map.memory.mb</name>
<value>4096</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>4096</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx3072m</value>
</property>
<property>
<name>mapreduce.recduce.java.opts</name>
<value>-Xmx6144m</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>400</value>
</property>
However, nothing has helped except that virtual memory error has gone.
Any ideas appreciated.
Thanks
Mich Talebzadeh
http://talebzadehmich.wordpress.com
__
_Publications due shortly:_
*Creating in-memory Data Grid for Trading Systems with Oracle TimesTen
and Coherence Cache*
NOTE: The information in this email is proprietary and confidential.
This message is for the designated recipient only, if you are not the
intended recipient, you should destroy it immediately. Any information
in this message shall not be understood as given or endorsed by
Peridale Ltd, its subsidiaries or their employees, unless expressly so
stated. It is the responsibility of the recipient to ensure that this
email is virus free, therefore neither Peridale Ltd, its subsidiaries
nor their employees accept any responsibility.
