[
https://issues.apache.org/jira/browse/ACCUMULO-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032555#comment-14032555
]
Eric Newton commented on ACCUMULO-2905:
---------------------------------------
Proposed fix in review board (see link). Needs at least a double-check from
[~kturner].
> Seeing short hold times and no MinC's
> -------------------------------------
>
> Key: ACCUMULO-2905
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2905
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
> Environment: continuous ingest test
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Attachments: SlowToMinC.png
>
>
> Started continuous ingest on a single node cluster to investigate oscillating
> ingest rates. Set the split size to 200M, and fired up 3 ingesters. I
> increase the major/minor compactors to 24/12 respectively.
> The native map was set to 4G.
> When the table got to 256 tablets, ingest was constantly showing hold times.
> MinC would run, but memory would constantly fill before new MinCs would
> start. Here's some output from the log:
> {noformat}
> $ tail -F ../../../logs/tserver_host.debug.log | egrep 'Up|Got|held|MinC \['
> 2014-06-13 14:51:56,553 [tablet.DatafileManager] TABLET_HIST: 2;5d104;5cf07
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003kc/F0000qtk.rf
> 2014-06-13 14:51:56,695 [tablet.DatafileManager] TABLET_HIST: 2;11cb1;11ab
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eww/F0000qtl.rf
> 2014-06-13 14:51:56,861 [tablet.DatafileManager] TABLET_HIST: 2;0e4a3;0e2a3
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ery/F0000qtm.rf
> 2014-06-13 14:51:56,927 [tablet.DatafileManager] TABLET_HIST: 2;44907;44504
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008q9/F0000qtn.rf
> 2014-06-13 14:51:57,019 [tablet.DatafileManager] TABLET_HIST: 2;4a0fa;49cfd
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00001lg/F0000qto.rf
> 2014-06-13 14:51:57,535 [tablet.DatafileManager] TABLET_HIST: 2;4c0ff;4bd03
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000ut/F0000qtq.rf
> 2014-06-13 14:51:58,443 [tablet.DatafileManager] TABLET_HIST: 2;770cf;76ed2f
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lv/F0000qts.rf
> 2014-06-13 14:51:58,448 [tserver.TabletServerResourceManager] DEBUG: Commits
> held for 2.18 secs
> 2014-06-13 14:51:58,449 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363
> 225 in 1.669s, at=[0 0 0.00 1] ft=1.668s(pt=1.667s lt=0.001s ct=0.000s)
> 2014-06-13 14:51:58,790 [tablet.DatafileManager] TABLET_HIST: 2;4ad04b;4a902
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f05/F0000qtt.rf
> 2014-06-13 14:51:59,113 [tablet.DatafileManager] TABLET_HIST: 2;23af3;238f4
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei6/F0000qtw.rf
> 2014-06-13 14:52:00,545 [tablet.DatafileManager] TABLET_HIST: 2;4d906e;4d508
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008s0/F0000qtx.rf
> 2014-06-13 14:52:00,599 [tablet.DatafileManager] TABLET_HIST: 2;1bcb4f;1b8b6
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f9v/F0000qty.rf
> 2014-06-13 14:52:00,640 [tablet.DatafileManager] TABLET_HIST: 2;1d8be;1d4c1
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008j7/F0000qtz.rf
> 2014-06-13 14:52:00,648 [tablet.DatafileManager] TABLET_HIST: 2;40ce3;408de
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etz/F0000qu0.rf
> 2014-06-13 14:52:01,040 [tablet.DatafileManager] TABLET_HIST: 2;2c13;2bf32
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000uc/F0000qu5.rf
> 2014-06-13 14:52:01,939 [tablet.DatafileManager] TABLET_HIST: 2;5fedd;5fce3
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fir/F0000qu7.rf
> 2014-06-13 14:52:02,063 [tablet.DatafileManager] TABLET_HIST: 2;26d0a;26907
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eum/F0000qu8.rf
> 2014-06-13 14:52:02,067 [tserver.TabletServerResourceManager] DEBUG: Commits
> held for 3.19 secs
> 2014-06-13 14:52:02,069 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363
> 288 in 2.617s, at=[0 0 0.00 1] ft=2.616s(pt=2.614s lt=0.001s ct=0.001s)
> 2014-06-13 14:52:02,604 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58940
> 279,538 in 9.656s, at=[0 1 0.00 899] ft=9.274s(pt=7.516s lt=1.099s ct=0.659s)
> 2014-06-13 14:52:02,700 [tablet.DatafileManager] TABLET_HIST: 2;758e6;756e8
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008py/F0000qub.rf
> 2014-06-13 14:52:02,765 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58937
> 279,538 in 6.735s, at=[0 1 0.00 899] ft=6.446s(pt=4.446s lt=1.359s ct=0.641s)
> 2014-06-13 14:52:02,821 [tablet.DatafileManager] TABLET_HIST: 2;53b18;53918
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000kk7/F0000quc.rf
> 2014-06-13 14:52:03,503 [tablet.DatafileManager] TABLET_HIST: 2;116af;114af
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jnw/F0000qud.rf
> 2014-06-13 14:52:03,554 [tablet.DatafileManager] TABLET_HIST: 2;76ed2f;76cd7
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ju3/F0000que.rf
> 2014-06-13 14:52:04,197 [tablet.DatafileManager] TABLET_HIST: 2;7da4e;7d851
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000igh/F0000quf.rf
> 2014-06-13 14:52:04,603 [tablet.DatafileManager] TABLET_HIST: 2;38d03;388fe
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000evr/F0000quh.rf
> 2014-06-13 14:52:05,119 [tablet.DatafileManager] TABLET_HIST: 2;156d1f;154d1e
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jqx/F0000quj.rf
> 2014-06-13 14:52:05,577 [tablet.DatafileManager] TABLET_HIST: 2;3fada;3f8db
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pre/F0000qul.rf
> 2014-06-13 14:52:05,582 [tserver.TabletServerResourceManager] DEBUG: Commits
> held for 3.29 secs
> 2014-06-13 14:52:05,583 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363
> 252 in 2.512s, at=[0 0 0.00 1] ft=2.511s(pt=2.510s lt=0.001s ct=0.000s)
> 2014-06-13 14:52:06,206 [tablet.DatafileManager] TABLET_HIST: 2;4550e;4510b
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ex0/F0000qum.rf
> 2014-06-13 14:52:06,360 [tablet.DatafileManager] TABLET_HIST: 2;7fe06;7fc0b
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000hyp/F0000qun.rf
> 2014-06-13 14:52:06,642 [tablet.DatafileManager] TABLET_HIST: 2;238f4;236f4
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008i1/F0000quq.rf
> 2014-06-13 14:52:08,626 [tablet.DatafileManager] TABLET_HIST: 2;5931a;59119
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fy8/F0000qus.rf
> 2014-06-13 14:52:08,682 [tablet.DatafileManager] TABLET_HIST: 2;4a902;4a5
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008kd/F0000quu.rf
> 2014-06-13 14:52:08,740 [tablet.DatafileManager] TABLET_HIST: 2;1dcbe;1d8be
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etq/F0000quv.rf
> 2014-06-13 14:52:08,748 [tablet.DatafileManager] TABLET_HIST: 2;23ef2;23cf2
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei9/F0000qut.rf
> 2014-06-13 14:52:09,215 [tablet.DatafileManager] TABLET_HIST: 2;0fa97;0f899
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fgk/F0000quy.rf
> 2014-06-13 14:52:09,623 [tablet.DatafileManager] TABLET_HIST: 2;0101f;00e1df
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003ka/F0000qv0.rf
> 2014-06-13 14:52:09,626 [tserver.TabletServerResourceManager] DEBUG: Commits
> held for 3.81 secs
> 2014-06-13 14:52:09,628 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363
> 250 in 3.043s, at=[0 0 0.00 1] ft=3.042s(pt=3.041s lt=0.000s ct=0.001s)
> 2014-06-13 14:52:10,256 [tablet.DatafileManager] TABLET_HIST: 2;29d21;2991f
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f6f/F0000qv2.rf
> 2014-06-13 14:52:10,388 [tablet.DatafileManager] TABLET_HIST: 2;6f6f3;6f4f4
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pr5/F0000qv4.rf
> 2014-06-13 14:52:10,879 [tablet.DatafileManager] TABLET_HIST: 2;16cc5;168c8
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb5/F0000qv5.rf
> 2014-06-13 14:52:11,379 [tablet.DatafileManager] TABLET_HIST: 2;4b104;4ad04b
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lg/F0000qv7.rf
> 2014-06-13 14:52:11,879 [tablet.DatafileManager] TABLET_HIST: 2;6f8f;6f6f3
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008mo/F0000qv9.rf
> 2014-06-13 14:52:12,395 [tablet.DatafileManager] TABLET_HIST: 2;3f4e1;3f2e3e
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb1/F0000qva.rf
> {noformat}
> Perhaps the server should be more aggressive with MinC's to get better ingest
> performance?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
