Re: Where to place logs
Varun, this really depends on your log rotation and retention policy. Logs usually pretty big, but if you rotate it once a day(for example) and remove old logs after, say, 1 week, you probably will not need to have huge amount of space for it… You should estimate logs side before making decision. Thank you! Sincerely, Leonid Fedotov On Dec 28, 2012, at 12:51 AM, Varun Sharma wrote: > Thanks for the tip. I was just wondering what people have done in the past > - do people typically reserve a separate disk for logging activity ? > > Thanks > Varun > > On Wed, Dec 26, 2012 at 1:13 PM, Stack wrote: > >> On Mon, Dec 24, 2012 at 9:27 AM, Varun Sharma wrote: >> >>> Hi, >>> >>> I am wondering where people usually place hbase + hadoop logs. I have 4 >>> disks and 1 very tiny disk with barely 500 megs (thats the typical setup >> on >>> amazon ec2). The 4 disks shall be used for hbase data. Since 500M is too >>> small, should I place logs on one of the 4 disks. Could it potentially >>> steal IOP(s) from hbase ? Does anyone have an idea how much of an >> overhead >>> logging really is ? >>> >> >> Looks like you have no choice but to put your logging on a data disk. >> Logging can be expensive, yes. It should be easy enough to measure in >> your setup. You can disable all logging for a short period while the >> cluster is under load; do it on a single node even (you can do this from >> the UI). See how your io changes when logging is off. >> >> St.Ack >>
Re: Where to place logs
Thanks for the tip. I was just wondering what people have done in the past - do people typically reserve a separate disk for logging activity ? Thanks Varun On Wed, Dec 26, 2012 at 1:13 PM, Stack wrote: > On Mon, Dec 24, 2012 at 9:27 AM, Varun Sharma wrote: > > > Hi, > > > > I am wondering where people usually place hbase + hadoop logs. I have 4 > > disks and 1 very tiny disk with barely 500 megs (thats the typical setup > on > > amazon ec2). The 4 disks shall be used for hbase data. Since 500M is too > > small, should I place logs on one of the 4 disks. Could it potentially > > steal IOP(s) from hbase ? Does anyone have an idea how much of an > overhead > > logging really is ? > > > > Looks like you have no choice but to put your logging on a data disk. > Logging can be expensive, yes. It should be easy enough to measure in > your setup. You can disable all logging for a short period while the > cluster is under load; do it on a single node even (you can do this from > the UI). See how your io changes when logging is off. > > St.Ack >
Re: Where to place logs
On Mon, Dec 24, 2012 at 9:27 AM, Varun Sharma wrote: > Hi, > > I am wondering where people usually place hbase + hadoop logs. I have 4 > disks and 1 very tiny disk with barely 500 megs (thats the typical setup on > amazon ec2). The 4 disks shall be used for hbase data. Since 500M is too > small, should I place logs on one of the 4 disks. Could it potentially > steal IOP(s) from hbase ? Does anyone have an idea how much of an overhead > logging really is ? > Looks like you have no choice but to put your logging on a data disk. Logging can be expensive, yes. It should be easy enough to measure in your setup. You can disable all logging for a short period while the cluster is under load; do it on a single node even (you can do this from the UI). See how your io changes when logging is off. St.Ack
Where to place logs
Hi, I am wondering where people usually place hbase + hadoop logs. I have 4 disks and 1 very tiny disk with barely 500 megs (thats the typical setup on amazon ec2). The 4 disks shall be used for hbase data. Since 500M is too small, should I place logs on one of the 4 disks. Could it potentially steal IOP(s) from hbase ? Does anyone have an idea how much of an overhead logging really is ? Varun
