Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Pine W
Chris, may I quote your email on BASH? Pine On Mar 7, 2015 6:14 AM, "Christian Aistleitner" wrote: > Hi, > > around running jobs on the Analytics cluster, I've sometime seen > people say in IRC: “Let's run this heavy job. I'll keep an eye on it”. > > But more often than not, this seems to have m

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Nuria Ruiz
Thanks much Christian for the writeup. Should have icinga alarms arround these types of issues? Seems like that would be the way to go. Thanks, Nuria On Sat, Mar 7, 2015 at 4:00 PM, Andrew Otto wrote: > Thanks Christian! > > > > On Mar 7, 2015, at 09:14, Christian Aistleitner < > christ...@q

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Andrew Otto
Thanks Christian! > On Mar 7, 2015, at 09:14, Christian Aistleitner > wrote: > > Hi, > > around running jobs on the Analytics cluster, I've sometime seen > people say in IRC: “Let's run this heavy job. I'll keep an eye on it”. > > But more often than not, this seems to have meant: > “Let's j

Re: [Analytics] [Release] explore the (parsed) (common) Wikimedia user agents

2015-03-07 Thread Nuria
Amir, I think you need to re-read Oliver's e-mail as this data is not going to be updated regularly. He is explained why as part of this thread. Thanks, Nuria > On Mar 6, 2015, at 10:40 AM, Amir E. Aharoni > wrote: > > Very neat! This pretty much resolves https://phabricator.wikimedia.or

Re: [Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Federico Leva (Nemo)
Christian Aistleitner, 07/03/2015 15:14: P.S.: The above URL has diagrams! Click the URL! And with colours! So it's like checking heartbeats, cute. :) Nemo ___ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/

[Analytics] [Cluster] Monitoring the impact Hive jobs have on the Analytics cluster

2015-03-07 Thread Christian Aistleitner
Hi, around running jobs on the Analytics cluster, I've sometime seen people say in IRC: “Let's run this heavy job. I'll keep an eye on it”. But more often than not, this seems to have meant: “Let's just run this heavy job and wait. If QChris joins IRC, let's hope he doesn't ping us about having o

[Analytics] [Cluster] If Analytics Cluster seems slow over the weekend, read this email [aka HDFS balancing]

2015-03-07 Thread Christian Aistleitner
Hi, TL;DR: If you think your Hive queries are currently taking longer than usual, please find qchris in IRC, and if he is not responsive, kindly ask someone with root on stat1002 (like Ops) to kill the process java -Dproc_balancer -Xmx1000m [...] ---