Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-09 Thread Christian Aistleitner
Hi Pine, On Sat, Mar 07, 2015 at 08:15:18PM -0800, Pine W wrote: Chris, may I quote your email on BASH? They take emails too? Regardless ... feel free to quote or forward any of my emails wherever you seem fit. Have fun, Christian -- quelltextlich e.U. \\ Christian

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-09 Thread Joseph Allemandou
Thanks a lot Christian :) I had not meant by any mean last Friday to overload the cluster ... I did it nonetheless. Your page on how to 'keep an eye on it' will really be useful! Cheers Joseph On Sun, Mar 8, 2015 at 8:26 PM, Leila Zia le...@wikimedia.org wrote: This is really useful,

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-09 Thread Andrew Otto
Should have icinga alarms arround these types of issues? Seems like that would be the way to go. Aside from this, I get daily emails about webrequest partition statuses, and I would at least notice the morning after that something is wrong. On Mar 7, 2015, at 21:20, Nuria Ruiz

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-09 Thread Christian Aistleitner
Hi Andrew, On Mon, Mar 09, 2015 at 11:54:56AM -0400, Andrew Otto wrote: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Load Christian, may I move this page into the Cluster/Hadoop/Administration page? I think a separate page is worth it as the target audience is different from

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-08 Thread Leila Zia
This is really useful, Christian. Thanks for explaining and documenting it. Leila On Sat, Mar 7, 2015 at 6:14 AM, Christian Aistleitner christ...@quelltextlich.at wrote: Hi, around running jobs on the Analytics cluster, I've sometime seen people say in IRC: “Let's run this heavy job. I'll

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Federico Leva (Nemo)
Christian Aistleitner, 07/03/2015 15:14: P.S.: The above URL has diagrams! Click the URL! And with colours! So it's like checking heartbeats, cute. :) Nemo ___ Analytics mailing list Analytics@lists.wikimedia.org

[Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Christian Aistleitner
Hi, around running jobs on the Analytics cluster, I've sometime seen people say in IRC: “Let's run this heavy job. I'll keep an eye on it”. But more often than not, this seems to have meant: “Let's just run this heavy job and wait. If QChris joins IRC, let's hope he doesn't ping us about having

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Andrew Otto
Thanks Christian! On Mar 7, 2015, at 09:14, Christian Aistleitner christ...@quelltextlich.at wrote: Hi, around running jobs on the Analytics cluster, I've sometime seen people say in IRC: “Let's run this heavy job. I'll keep an eye on it”. But more often than not, this seems to have

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Nuria Ruiz
Thanks much Christian for the writeup. Should have icinga alarms arround these types of issues? Seems like that would be the way to go. Thanks, Nuria On Sat, Mar 7, 2015 at 4:00 PM, Andrew Otto ao...@wikimedia.org wrote: Thanks Christian! On Mar 7, 2015, at 09:14, Christian Aistleitner

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Pine W
Chris, may I quote your email on BASH? Pine On Mar 7, 2015 6:14 AM, Christian Aistleitner christ...@quelltextlich.at wrote: Hi, around running jobs on the Analytics cluster, I've sometime seen people say in IRC: “Let's run this heavy job. I'll keep an eye on it”. But more often than not,