We haven't tried using Pig. However, we had a problem where our mapreduce job blew up for a subset of data. It appeared that we had a bug in our code that had generated a row as big as 3Gb. It was actually causing long GC pauses and would cause GC thrashing. The hadoop job of course would time out. Our range batch sizes are 32 and we had wide rows enabled. Your scenario seems similar. Try using nodetool tpstats to see if the column family involved with your job has a max row size which is very large. Also inspect your C* logs looking for long GC pause log lines from GCInspector. You can also refer to heap usage trends if you have them in your monitoring tools.
On Thu, Apr 25, 2013 at 7:03 PM, aaron morton <aa...@thelastpickle.com>wrote: > 2013-04-23 16:09:17,838 INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: > Current split being processed ColumnFamilySplit((9197470410121435301, '-1] > @[p00nosql02.00, p00nosql01.00]) > Why it's split data from two nodes? we have 6 nodes cassandra cluster + > hadoop slaves - every task should get local input split from local > cassandra - am i right? > > My understanding is that it may get it locally, but it's not something > that has to happen. Once of the Hadoop guys will have a better idea. > > Try reducing the cassandra.range.batch.size and/or if you are using wide > rows enable cassandra.input.widerows > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 25/04/2013, at 7:55 PM, Shamim <sre...@yandex.ru> wrote: > > Hello Aaron, > I have got the following Log from the server (Sorry for being late) > > job_201304231203_0004 > attempt_201304231203_0004_m_000501_0 > > 2013-04-23 16:09:14,196 INFO org.apache.hadoop.util.NativeCodeLoader: > Loaded the native-hadoop library > 2013-04-23 16:09:14,438 INFO > org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating > symlink: > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/pigContext > <- > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/pigContext > 2013-04-23 16:09:14,453 INFO > org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating > symlink: > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/dk > <- > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/dk > 2013-04-23 16:09:14,456 INFO > org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating > symlink: > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/META-INF > <- > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/META-INF > 2013-04-23 16:09:14,459 INFO > org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating > symlink: > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/org > <- > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/org > 2013-04-23 16:09:14,469 INFO > org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating > symlink: > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/com > <- > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/com > 2013-04-23 16:09:14,471 INFO > org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating > symlink: > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/.job.jar.crc > <- > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/.job.jar.crc > 2013-04-23 16:09:14,474 INFO > org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating > symlink: > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/jars/job.jar > <- > /egov/data/hadoop/mapred/local/taskTracker/cassandra/jobcache/job_201304231203_0004/attempt_201304231203_0004_m_000501_0/work/job.jar > 2013-04-23 16:09:17,329 INFO org.apache.hadoop.util.ProcessTree: setsid > exited with exit code 0 > 2013-04-23 16:09:17,387 INFO org.apache.hadoop.mapred.Task: Using > ResourceCalculatorPlugin : > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@256ef705 > 2013-04-23 16:09:17,838 INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: > Current split being processed ColumnFamilySplit((9197470410121435301, '-1] > @[p00nosql02.00, p00nosql01.00]) > 2013-04-23 16:09:18,088 INFO org.apache.pig.data.SchemaTupleBackend: Key > [pig.schematuple] was not set... will not generate code. > 2013-04-23 16:09:19,784 INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map: > Aliases being processed per job phase (AliasName[line,offset]): M: > data[12,7],null[-1,-1],filtered[14,11],null[-1,-1],c1[23,5],null[-1,-1],updated[111,10] > C: R: > 2013-04-23 17:35:11,199 INFO org.apache.hadoop.mapred.TaskLogsTruncater: > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 > 2013-04-23 17:35:11,384 INFO org.apache.hadoop.io.nativeio.NativeIO: > Initialized cache for UID to User mapping with a cache timeout of 14400 > seconds. > 2013-04-23 17:35:11,385 INFO org.apache.hadoop.io.nativeio.NativeIO: Got > UserName cassandra for UID 500 from the native implementation > 2013-04-23 17:35:11,417 WARN org.apache.hadoop.mapred.Child: Error running > child > java.lang.RuntimeException: TimedOutException() > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:384) > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:390) > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:313) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:103) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.getProgress(PigRecordReader.java:169) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:514) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:539) > at > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: TimedOutException() > at > org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12932) > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) > at > org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:734) > at > org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:718) > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:346) > ... 17 more > 2013-04-23 17:35:11,427 INFO org.apache.hadoop.mapred.Task: Runnning > cleanup for the task > > These Two tasks hanged for long time and crashes with timeout exception. > Very interesting part is as follows > 2013-04-23 16:09:17,838 INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: > Current split being processed ColumnFamilySplit((9197470410121435301, '-1] > @[p00nosql02.00, p00nosql01.00]) > Why it's split data from two nodes? we have 6 nodes cassandra cluster + > hadoop slaves - every task should get local input split from local > cassandra - am i right? > > -- > Best regards > Shamim A. > > 24.04.2013, 10:59, "Shamim" <sre...@yandex.ru>: > > Hello Aron, > We have build up our new cluster from the scratch with version 1.2 - > partition murmor3. We are not using vnodes at all. > Actually log is clean and nothing serious, now investigating logs and post > soon if found something criminal > > Our cluster is evenly partitioned (Murmur3Partitioner) > > > Murmor3Partitioner is only available in 1.2 and changing partitioners is > not supported. Did you change from Random Partitioner under 1.1? > > Are > you using virtual nodes in your 1.2 cluster ? > >>> We have roughly > 97million rows in our cluster. Why we are getting above behavior? Do you > have any suggestion or clue to trouble shoot in this issue? > > Can you > make some of the logs from the tasks available? > > Cheers > > -- > > > --------------- > Aaron Morton > Freelance Cassandra Consultant > New > Zealand > > @aaronmorton > http://www.thelastpickle.com > > On > 23/04/2013, at 5:50 AM, Shamim wrote: > >> We are using Hadoop 1.0.3 and > pig 0.11.1 version >> >> -- >> Best regards >> Shamim A. >> >> 22.04.2013, > 21:48, "Shamim" : >> >>> Hello all, >>> recently we have upgrade our > cluster (6 nodes) from cassandra version 1.1.6 to 1.2.1. Our cluster is > evenly partitioned (Murmur3Partitioner). We are using pig for parse and > compute aggregate data. >>> >>> When we submit job through pig, what i > consistently see is that, while most of the task have 20-25k row assigned > each (Map input records), only 2 of them (always 2 ) getting more than 2 > million rows. This 2 tasks always complete 100% and hang for long time. > Also most of the time we are getting killed task (2%) with > TimeoutException. >>> >>> We increased rpc_timeout to 60000, also set > cassandra.input.split.size=1024 but nothing help. >>> >>> We have roughly > 97million rows in our cluster. Why we are getting above behavior? Do you > have any suggestion or clue to trouble shoot in this issue? Any help will > be highly thankful. Thankx in advance. >>> >>> -- >>> Best regards >>> > Shamim A. -- Best regards > Shamim A. > > >
