[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 Kevin Leduc changed: What|Removed |Added Status|NEW |RESOLVED Resolution|---

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 --- Comment #8 from christ...@quelltextlich.at --- TL;DR: * analytics1003 alarms were harmless. * oxygen alarms point to real packet-loss, which affected all udp2log multicast consumers during two ~4 hour periods. * The issue is expect

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 christ...@quelltextlich.at changed: What|Removed |Added Assignee|dandree...@wikimedia.org|christ...@quelltextlich.at

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 Dan Andreescu changed: What|Removed |Added Whiteboard|u=Kevin c=General/Unknown |u=Kevin c=General/Unknown

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 Dan Andreescu changed: What|Removed |Added Whiteboard|u=Kevin c=General/Unknown |u=Kevin c=General/Unknown

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 --- Comment #7 from Dan Andreescu --- Just writing some notes for myself on how to troubleshoot this. For now, I'm putting it on hold until I get two more things: 1. kafka broker data is available to query again (right now we have 1 week roll

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 Dario Taraborelli changed: What|Removed |Added CC||dtarabore...@wikimedia.org --- Com

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 Bawolff (Brian Wolff) changed: What|Removed |Added CC||bawolff...@gmail.com --- Comme

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 --- Comment #4 from Dan Andreescu --- problems seem fairly contained to esams, for example: http://ganglia.wikimedia.org/latest/?hreg[]=%28amssq%7Ccp%29.%2B&mreg[]=kafka.varnishkafka%5C.kafka_drerr.per_second&z=large>ype=line&title=kafka.varni

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 --- Comment #3 from Dan Andreescu --- according to puppet, analytics1003 is set up with role::analytics::kafkatee::webrequest::mobile, but I'm not too familiar with the infrastructure. My theory so far is that the FIFA World Cup semifinals is

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 --- Comment #2 from Toby Negrin --- Kafka graphs show a spike from 1022: http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&tab=v&vn=kafka&hide-hf=false -- You are receiving this mail because: You are on the CC list for the bug. ___

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 --- Comment #1 from Dan Andreescu --- Will write here anything I find. So it looks like packets into Analytics1003 increased by an unusual margin starting around 20:30 and going through 22:00 at the time of this comment: http://ganglia.wikime

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 Dan Andreescu changed: What|Removed |Added Whiteboard||u=Kevin c=General/Unknown

[Bug 67694] Packetloss issues on oxygen (and analytics1003)

https://bugzilla.wikimedia.org/show_bug.cgi?id=67694 Dan Andreescu changed: What|Removed |Added CC||dandree...@wikimedia.org Ass