Re: ingest performance oscillations and Xceivers

2013-01-04 Thread Kepner, Jeremy - 0553 - MITLL
Thanks. I didn't realize this. I have created an issue and attached the performance plots to the issue: https://issues.apache.org/jira/browse/ACCUMULO-931 On Jan 3, 2013, at 8:26 PM, Drew Farris wrote: Yep, most apache lists strip attachments. On Thursday, January 3, 2013, Kepner, Jeremy - 05

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread Drew Farris
Yep, most apache lists strip attachments. On Thursday, January 3, 2013, Kepner, Jeremy - 0553 - MITLL wrote: > Here it is again. It got sent the last time. Does the e-mail list strip > out attachments? > > On Jan 3, 2013, at 7:47 PM, Josh Elser wrote: > > > I think you missed your attachment :)

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread Josh Elser
Actually, I'm apparently blind. Just kidding, didn't receive. Perhaps over a certain size? I've seen some before. You could open something on Jira and attach it there. On 01/03/2013 08:10 PM, Josh Elser wrote: I don't think it should/does. I got it this time. On 01/03/2013 07:55 PM, Kepner, J

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread John Vines
I think so, because I got it on the first Terry when I was directly listed Sent from my phone, please pardon the typos and brevity. On Jan 3, 2013 8:06 PM, "Kepner, Jeremy - 0553 - MITLL" wrote: > Here it is again. It got sent the last time. Does the e-mail list strip > out attachments? > > On

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread Josh Elser
I don't think it should/does. I got it this time. On 01/03/2013 07:55 PM, Kepner, Jeremy - 0553 - MITLL wrote: Here it is again. It got sent the last time. Does the e-mail list strip out attachments?

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread Kepner, Jeremy - 0553 - MITLL
Here it is again. It got sent the last time. Does the e-mail list strip out attachments? On Jan 3, 2013, at 7:47 PM, Josh Elser wrote: > I think you missed your attachment :) > > On 01/03/2013 07:31 PM, Kepner, Jeremy - 0553 - MITLL wrote: >> Attached is the log for a set for 3 ingests. The f

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread Josh Elser
I think you missed your attachment :) On 01/03/2013 07:31 PM, Kepner, Jeremy - 0553 - MITLL wrote: Attached is the log for a set for 3 ingests. The first hump is with 1 ingestor, the second hump is with 2 ingestors, and the third hump is with 3 ingestors. The 3 ingestor case starts oscillati

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread Kepner, Jeremy - 0553 - MITLL
Attached is the log for a set for 3 ingests. The first hump is with 1 ingestor, the second hump is with 2 ingestors, and the third hump is with 3 ingestors. The 3 ingestor case starts oscillating about 21:40. I don't see any spikes in any of the fields. It would be nice if there was also a pl

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread John Vines
How many hard drives and what's your max minor/maor compactions set to? These can severely limit your performance due to potential disk thrashing. If you observe the monitor, when ingest starts trailing, do you see it undergoing, or possibly being backed up on, any form of compaction? And lastly, a

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread Jeremy Kepner
No correlation with compactions. No queries. On Thu, Jan 03, 2013 at 11:24:17AM -0500, William Slacum wrote: > Have you also been tracking compactions? Did you have a query load? > > > On Wed, Jan 2, 2013 at 7:25 PM, Kepner, Jeremy - 0553 - MITLL < > kep...@ll.mit.edu> wrote: > > > Hmmm, that'

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread Eric Newton
You may be seeing some impact due to ACCUMULO-893: http://issues.apache.org/jira/browse/ACCUMULO-893 Are you seeing 2-minute hold times popping up? -Eric On Wed, Jan 2, 2013 at 7:25 PM, Kepner, Jeremy - 0553 - MITLL < kep...@ll.mit.edu> wrote: > Hmmm, that's interesting, because in the past I

Re: ingest performance oscillations and Xceivers

2013-01-03 Thread William Slacum
Have you also been tracking compactions? Did you have a query load? On Wed, Jan 2, 2013 at 7:25 PM, Kepner, Jeremy - 0553 - MITLL < kep...@ll.mit.edu> wrote: > Hmmm, that's interesting, because in the past I didn't see this behavior. > It might be worth having someone look into because it seems

Re: ingest performance oscillations and Xceivers

2013-01-02 Thread Kepner, Jeremy - 0553 - MITLL
Hmmm, that's interesting, because in the past I didn't see this behavior. It might be worth having someone look into because it seems to have a 2x impact on sustained ingest. Regards. -Jeremy On Jan 2, 2013, at 2:23 PM, Keith Turner wrote: > On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner wr

Re: ingest performance oscillations and Xceivers

2013-01-02 Thread Jeremy Kepner
So what mechanism causes the number of Xceivers to increase? I am carefully controlling the number of ingestors and the data isn't varying too much. I would expect the number of Xceivers to remain consant. Regards. -Jeremy On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote: > Hey Jerem

Re: ingest performance oscillations and Xceivers

2013-01-02 Thread William Slacum
How many disks do you have? That can be bottle-necking throughput as the number of Xceivers is related to the number of resources (threads, sockets: http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/) used at once to perform operations. On Tue, Jan 1, 2013 at 6:45 PM, Eric Newton wrote:

Re: ingest performance oscillations and Xceivers

2013-01-01 Thread Eric Newton
Hey Jeremy, Can you compare the ingest rate to the number of tablets, too? I've found, that if I have 20-80 tablets per server (on similar hardware) I get the best performance. # of Xceivers == number of writers when ingest is the primary target. Also, is this 1.4 or trunk? -Eric On Tue, Ja

ingest performance oscillations and Xceivers

2013-01-01 Thread Kepner, Jeremy - 1010 - MITLL
Accumulo Colleagues, I am trying to optimize my ingest into a single node Accumulo instance running on a 32 core node with 96 GB of RAM. I am seeing the follow ingest variations as a I change the number of ingest processes (see attached): - Ingestors, Inges