Hello Aaron,
I have got the following Log from the server (Sorry for being late)
job_201304231203_0004
attempt_201304231203_0004_m_000501_0
2013-04-23 16:09:14,196 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2013-04-23 16:09:14,438 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/pigContext
<-
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/pigContext
2013-04-23 16:09:14,453 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/dk
<-
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/dk
2013-04-23 16:09:14,456 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/META-INF
<-
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/META-INF
2013-04-23 16:09:14,459 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/org
<-
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/org
2013-04-23 16:09:14,469 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/com
<-
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/com
2013-04-23 16:09:14,471 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/.job.jar.crc
<-
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/.job.jar.crc
2013-04-23 16:09:14,474 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/job.jar
<-
/egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/job.jar
2013-04-23 16:09:17,329 INFO org.apache.hadoop.util.ProcessTree: setsid exited
with exit code 0
2013-04-23 16:09:17,387 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@256ef705
2013-04-23 16:09:17,838 INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader:
Current split being processed ColumnFamilySplit((9197470410121435301, '-1]
@[p00nosql02.00, p00nosql01.00])
2013-04-23 16:09:18,088 INFO org.apache.pig.data.SchemaTupleBackend: Key
[pig.schematuple] was not set... will not generate code.
2013-04-23 16:09:19,784 INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map:
Aliases being processed per job phase (AliasName[line,offset]): M:
data[12,7],null[-1,-1],filtered[14,11],null[-1,-1],c1[23,5],null[-1,-1],updated[111,10]
C: R:
2013-04-23 17:35:11,199 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-04-23 17:35:11,384 INFO org.apache.hadoop.io.nativeio.NativeIO:
Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2013-04-23 17:35:11,385 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
UserName cassandra for UID 500 from the native implementation
2013-04-23 17:35:11,417 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: TimedOutException()
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:384)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:390)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:313)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:103)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.getProgress(PigRecordReader.java:169)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:514)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:539)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12932)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:734)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:718)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:346)
... 17 more
2013-04-23 17:35:11,427 INFO org.apache.hadoop.mapred.Task: Runnning cleanup
for the task
These Two tasks hanged for long time and crashes with timeout exception. Very
interesting part is as follows
2013-04-23 16:09:17,838 INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader:
Current split being processed ColumnFamilySplit((9197470410121435301, '-1]
@[p00nosql02.00, p00nosql01.00])
Why it's split data from two nodes? we have 6 nodes cassandra cluster + hadoop
slaves - every task should get local input split from local cassandra - am i
right?
--
Best regards
Shamim A.
24.04.2013, 10:59, "Shamim" <sre...@yandex.ru>:
> Hello Aron,
> We have build up our new cluster from the scratch with version 1.2 -
> partition murmor3. We are not using vnodes at all.
> Actually log is clean and nothing serious, now investigating logs and post
> soon if found something criminal
>
>>>> Our cluster is evenly partitioned (Murmur3Partitioner) > >
>>>> Murmor3Partitioner is only available in 1.2 and changing partitioners is
>>>> not supported. Did you change from Random Partitioner under 1.1? > > Are
>>>> you using virtual nodes in your 1.2 cluster ? > >>> We have roughly
>>>> 97million rows in our cluster. Why we are getting above behavior? Do you
>>>> have any suggestion or clue to trouble shoot in this issue? > > Can you
>>>> make some of the logs from the tasks available? > > Cheers > > --
>
> --------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand
> > > @aaronmorton > http://www.thelastpickle.com > > On 23/04/2013, at 5:50
> AM, Shamim wrote: > >> We are using Hadoop 1.0.3 and pig 0.11.1 version >>
> >> -- >> Best regards >> Shamim A. >> >> 22.04.2013, 21:48, "Shamim" : >> >>>
> Hello all, >>> recently we have upgrade our cluster (6 nodes) from cassandra
> version 1.1.6 to 1.2.1. Our cluster is evenly partitioned
> (Murmur3Partitioner). We are using pig for parse and compute aggregate data.
> >>> >>> When we submit job through pig, what i consistently see is that,
> while most of the task have 20-25k row assigned each (Map input records),
> only 2 of them (always 2 ) getting more than 2 million rows. This 2 tasks
> always complete 100% and hang for long time. Also most of the time we are
> getting killed task (2%) with TimeoutException. >>> >>> We increased
> rpc_timeout to 60000, also set cassandra.input.split.size=1024 but nothing
> help. >>> >>> We have roughly 97million rows in our cluster. Why we are
> getting above behavior? Do you have any suggestion or clue to trouble shoot
> in this issue? Any help will be highly thankful. Thankx in advance. >>> >>>
> -- >>> Best regards >>> Shamim A. -- Best regards
> Shamim A.
