[ https://issues.apache.org/jira/browse/ACCUMULO-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032870#comment-14032870 ]
John Vines commented on ACCUMULO-2905: -------------------------------------- I'm okay backporting, as it's a fairly light performance adjustment. Just need to evaluate if there would be any negative repercussions from this modified behavior. > Seeing short hold times and no MinC's > ------------------------------------- > > Key: ACCUMULO-2905 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2905 > Project: Accumulo > Issue Type: Bug > Components: tserver > Environment: continuous ingest test > Reporter: Eric Newton > Assignee: Eric Newton > Priority: Minor > Attachments: SlowToMinC.png > > > Started continuous ingest on a single node cluster to investigate oscillating > ingest rates. Set the split size to 200M, and fired up 3 ingesters. I > increase the major/minor compactors to 24/12 respectively. > The native map was set to 4G. > When the table got to 256 tablets, ingest was constantly showing hold times. > MinC would run, but memory would constantly fill before new MinCs would > start. Here's some output from the log: > {noformat} > $ tail -F ../../../logs/tserver_host.debug.log | egrep 'Up|Got|held|MinC \[' > 2014-06-13 14:51:56,553 [tablet.DatafileManager] TABLET_HIST: 2;5d104;5cf07 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003kc/F0000qtk.rf > 2014-06-13 14:51:56,695 [tablet.DatafileManager] TABLET_HIST: 2;11cb1;11ab > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eww/F0000qtl.rf > 2014-06-13 14:51:56,861 [tablet.DatafileManager] TABLET_HIST: 2;0e4a3;0e2a3 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ery/F0000qtm.rf > 2014-06-13 14:51:56,927 [tablet.DatafileManager] TABLET_HIST: 2;44907;44504 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008q9/F0000qtn.rf > 2014-06-13 14:51:57,019 [tablet.DatafileManager] TABLET_HIST: 2;4a0fa;49cfd > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00001lg/F0000qto.rf > 2014-06-13 14:51:57,535 [tablet.DatafileManager] TABLET_HIST: 2;4c0ff;4bd03 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000ut/F0000qtq.rf > 2014-06-13 14:51:58,443 [tablet.DatafileManager] TABLET_HIST: 2;770cf;76ed2f > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lv/F0000qts.rf > 2014-06-13 14:51:58,448 [tserver.TabletServerResourceManager] DEBUG: Commits > held for 2.18 secs > 2014-06-13 14:51:58,449 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 > 225 in 1.669s, at=[0 0 0.00 1] ft=1.668s(pt=1.667s lt=0.001s ct=0.000s) > 2014-06-13 14:51:58,790 [tablet.DatafileManager] TABLET_HIST: 2;4ad04b;4a902 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f05/F0000qtt.rf > 2014-06-13 14:51:59,113 [tablet.DatafileManager] TABLET_HIST: 2;23af3;238f4 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei6/F0000qtw.rf > 2014-06-13 14:52:00,545 [tablet.DatafileManager] TABLET_HIST: 2;4d906e;4d508 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008s0/F0000qtx.rf > 2014-06-13 14:52:00,599 [tablet.DatafileManager] TABLET_HIST: 2;1bcb4f;1b8b6 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f9v/F0000qty.rf > 2014-06-13 14:52:00,640 [tablet.DatafileManager] TABLET_HIST: 2;1d8be;1d4c1 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008j7/F0000qtz.rf > 2014-06-13 14:52:00,648 [tablet.DatafileManager] TABLET_HIST: 2;40ce3;408de > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etz/F0000qu0.rf > 2014-06-13 14:52:01,040 [tablet.DatafileManager] TABLET_HIST: 2;2c13;2bf32 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000uc/F0000qu5.rf > 2014-06-13 14:52:01,939 [tablet.DatafileManager] TABLET_HIST: 2;5fedd;5fce3 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fir/F0000qu7.rf > 2014-06-13 14:52:02,063 [tablet.DatafileManager] TABLET_HIST: 2;26d0a;26907 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eum/F0000qu8.rf > 2014-06-13 14:52:02,067 [tserver.TabletServerResourceManager] DEBUG: Commits > held for 3.19 secs > 2014-06-13 14:52:02,069 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 > 288 in 2.617s, at=[0 0 0.00 1] ft=2.616s(pt=2.614s lt=0.001s ct=0.001s) > 2014-06-13 14:52:02,604 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58940 > 279,538 in 9.656s, at=[0 1 0.00 899] ft=9.274s(pt=7.516s lt=1.099s ct=0.659s) > 2014-06-13 14:52:02,700 [tablet.DatafileManager] TABLET_HIST: 2;758e6;756e8 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008py/F0000qub.rf > 2014-06-13 14:52:02,765 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58937 > 279,538 in 6.735s, at=[0 1 0.00 899] ft=6.446s(pt=4.446s lt=1.359s ct=0.641s) > 2014-06-13 14:52:02,821 [tablet.DatafileManager] TABLET_HIST: 2;53b18;53918 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000kk7/F0000quc.rf > 2014-06-13 14:52:03,503 [tablet.DatafileManager] TABLET_HIST: 2;116af;114af > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jnw/F0000qud.rf > 2014-06-13 14:52:03,554 [tablet.DatafileManager] TABLET_HIST: 2;76ed2f;76cd7 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ju3/F0000que.rf > 2014-06-13 14:52:04,197 [tablet.DatafileManager] TABLET_HIST: 2;7da4e;7d851 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000igh/F0000quf.rf > 2014-06-13 14:52:04,603 [tablet.DatafileManager] TABLET_HIST: 2;38d03;388fe > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000evr/F0000quh.rf > 2014-06-13 14:52:05,119 [tablet.DatafileManager] TABLET_HIST: 2;156d1f;154d1e > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jqx/F0000quj.rf > 2014-06-13 14:52:05,577 [tablet.DatafileManager] TABLET_HIST: 2;3fada;3f8db > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pre/F0000qul.rf > 2014-06-13 14:52:05,582 [tserver.TabletServerResourceManager] DEBUG: Commits > held for 3.29 secs > 2014-06-13 14:52:05,583 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 > 252 in 2.512s, at=[0 0 0.00 1] ft=2.511s(pt=2.510s lt=0.001s ct=0.000s) > 2014-06-13 14:52:06,206 [tablet.DatafileManager] TABLET_HIST: 2;4550e;4510b > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ex0/F0000qum.rf > 2014-06-13 14:52:06,360 [tablet.DatafileManager] TABLET_HIST: 2;7fe06;7fc0b > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000hyp/F0000qun.rf > 2014-06-13 14:52:06,642 [tablet.DatafileManager] TABLET_HIST: 2;238f4;236f4 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008i1/F0000quq.rf > 2014-06-13 14:52:08,626 [tablet.DatafileManager] TABLET_HIST: 2;5931a;59119 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fy8/F0000qus.rf > 2014-06-13 14:52:08,682 [tablet.DatafileManager] TABLET_HIST: 2;4a902;4a5 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008kd/F0000quu.rf > 2014-06-13 14:52:08,740 [tablet.DatafileManager] TABLET_HIST: 2;1dcbe;1d8be > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etq/F0000quv.rf > 2014-06-13 14:52:08,748 [tablet.DatafileManager] TABLET_HIST: 2;23ef2;23cf2 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei9/F0000qut.rf > 2014-06-13 14:52:09,215 [tablet.DatafileManager] TABLET_HIST: 2;0fa97;0f899 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fgk/F0000quy.rf > 2014-06-13 14:52:09,623 [tablet.DatafileManager] TABLET_HIST: 2;0101f;00e1df > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003ka/F0000qv0.rf > 2014-06-13 14:52:09,626 [tserver.TabletServerResourceManager] DEBUG: Commits > held for 3.81 secs > 2014-06-13 14:52:09,628 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 > 250 in 3.043s, at=[0 0 0.00 1] ft=3.042s(pt=3.041s lt=0.000s ct=0.001s) > 2014-06-13 14:52:10,256 [tablet.DatafileManager] TABLET_HIST: 2;29d21;2991f > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f6f/F0000qv2.rf > 2014-06-13 14:52:10,388 [tablet.DatafileManager] TABLET_HIST: 2;6f6f3;6f4f4 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pr5/F0000qv4.rf > 2014-06-13 14:52:10,879 [tablet.DatafileManager] TABLET_HIST: 2;16cc5;168c8 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb5/F0000qv5.rf > 2014-06-13 14:52:11,379 [tablet.DatafileManager] TABLET_HIST: 2;4b104;4ad04b > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lg/F0000qv7.rf > 2014-06-13 14:52:11,879 [tablet.DatafileManager] TABLET_HIST: 2;6f8f;6f6f3 > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008mo/F0000qv9.rf > 2014-06-13 14:52:12,395 [tablet.DatafileManager] TABLET_HIST: 2;3f4e1;3f2e3e > MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb1/F0000qva.rf > {noformat} > Perhaps the server should be more aggressive with MinC's to get better ingest > performance? -- This message was sent by Atlassian JIRA (v6.2#6252)