let me take a look, but branch-1 and master should be the same on the proc stuff
Matteo On Thu, Nov 5, 2015 at 10:23 AM, Stack <st...@duboce.net> wrote: > You are looking at master process only? > > I've been looking at branch-1 only of late. > > St.Ack > > On Thu, Nov 5, 2015 at 12:06 AM, Andrew Purtell <apurt...@apache.org> > wrote: > > > I just profiled trunk running YCSB workload A. I was looking for > something > > else but found completely unexpected results. > > > > I see 60% of all allocation pressure in TLABs coming from > > ProcedureExecutors, allocating 16 GB of > > java.util.AbstractMap$SimpleImmutableEntry in this code: > > > > Stack Trace TLABs Total TLAB Size(bytes) Pressure(%) > > java.util.concurrent.ConcurrentSkipListMap$Node.createSnapshot() 11,603 > > 18,402,380,120 100 > > java.util.concurrent.ConcurrentSkipListMap.firstEntry() 11,599 > > 18,401,678,776 99.996 > > > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.nextObject() > > 11,599 18,401,678,776 99.996 > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.poll() > > 11,599 18,401,678,776 99.996 > > > > org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll() > 11,599 > > 18,401,678,776 99.996 > > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop() 11,599 > > 18,401,678,776 99.996 > > > > > > The three ProcedureExecutor threads also consume about 60% of all CPU > time, > > each about 20%. Here are the top hot code arcs from one of them: > > > > Stack Trace Sample Count Percentage(%) > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.poll() 1,521 > > 39.191 > > org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll() > > 1,521 39.191 > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop() > 1,521 > > 39.191 > > java.lang.String$CaseInsensitiveComparator.compare(String, String) 614 > > 15.821 > > java.lang.String$CaseInsensitiveComparator.compare(Object, Object) 614 > > 15.821 > > java.lang.String.compareToIgnoreCase(String) 614 15.821 > > org.apache.hadoop.hbase.ServerName.compareTo(ServerName) 614 > > 15.821 > > org.apache.hadoop.hbase.ServerName.compareTo(Object) 614 > 15.821 > > java.util.concurrent.ConcurrentSkipListMap.cpr(Comparator, > > Object, Object) 614 15.821 > > > > java.util.concurrent.ConcurrentSkipListMap.findNear(Object, int, > > Comparator) > > 614 15.821 > > > > java.util.concurrent.ConcurrentSkipListMap.getNear(Object, int) 614 > 15.821 > > > > java.util.concurrent.ConcurrentSkipListMap.higherEntry(Object) 614 15.821 > > > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.nextObject() > 614 > > 15.821 > > > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.poll() 614 > 15.821 > > > > org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll() 614 > > 15.821 > > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop() 614 > 15.821 > > java.util.concurrent.ConcurrentSkipListMap$Node.createSnapshot() 468 > 12.059 > > java.util.concurrent.ConcurrentSkipListMap.getNear(Object, int) 468 > > 12.059 > > java.util.concurrent.ConcurrentSkipListMap.higherEntry(Object) 468 > > 12.059 > > > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.nextObject() > 468 > > 12.059 > > > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.poll() > > 468 12.059 > > > > org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll() 468 > > 12.059 > > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop() 468 > 12.059 > > java.util.concurrent.ConcurrentSkipListMap.cpr(Comparator, Object, > Object) > > 402 10.358 > > java.util.concurrent.ConcurrentSkipListMap.findNear(Object, int, > > Comparator) 280 7.215 > > java.util.concurrent.ConcurrentSkipListMap.getNear(Object, int) 280 > > 7.215 > > java.util.concurrent.ConcurrentSkipListMap.higherEntry(Object) > 280 > > 7.215 > > > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.nextObject() > 280 > > 7.215 > > > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.poll() 280 > 7.215 > > > > org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll() 280 > > 7.215 > > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop() 280 > 7.215 > > java.util.concurrent.ConcurrentSkipListMap.findPredecessor(Object, > > Comparator) 122 3.144 > > java.util.concurrent.ConcurrentSkipListMap.findNear(Object, int, > > Comparator) 122 3.144 > > java.util.concurrent.ConcurrentSkipListMap.getNear(Object, int) > > 122 > > 3.144 > > > java.util.concurrent.ConcurrentSkipListMap.higherEntry(Object) > > 122 3.144 > > > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.nextObject() > 122 > > 3.144 > > > > org.apache.hadoop.hbase.procedure2.ProcedureFairRunQueues.poll() 122 > 3.144 > > > > org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll() 122 > > 3.144 > > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop() 122 3.144 > > > > > > ​Just me? Someone working on ProcedureV2 should have a look at WTH the > > ProcedureExecutors are doing. They seem to be polling the run queue > > constantly and burning up CPU. I'll try running more workloads at the > next > > opportunity and see if I can repeat this or gain more insight. > > > > > > -- > > Best regards, > > > > - Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > > (via Tom White) > > >
