fgiunchedi added a comment.
FWIW I agree with testing the different boundaries, especially as you pointed
out the alert is lax in terms of "reactivity"
TASK DETAIL
https://phabricator.wikimedia.org/T348831
EMAIL PREFERENCES
https://phabricator.wikimedia.org/sett
fgiunchedi added a comment.
For most cases I think alerting on "no data" and "values are all null" is
sensible, in other words you expect to have data returned by the query at all
times.
In this case I can't quite figure out why the alert went "no data"
fgiunchedi added a comment.
Thank you for reaching out @Lucas_Werkmeister_WMDE ! Yes indeed known issue,
we (o11y) recommend turning off notifications for datasource errors (full
rationale in https://phabricator.wikimedia.org/T347221#9264101) and the
instructions being at
https
fgiunchedi added a comment.
In T281267#8954763 <https://phabricator.wikimedia.org/T281267#8954763>,
@ArielGlenn wrote:
> @fgiunchedi I notice that in some cases phab tasks are autocreated when
systemd units fail. Is that true for systemd jobs on snapshot hosts? Could we
g
fgiunchedi updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T332953
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi, WMDE-leszek, leila, fkaelin, ItamarWMDE, elukey, KartikMistry,
santhosh
fgiunchedi removed a project: Observability-Metrics.
fgiunchedi added a comment.
Not a problem AFAICS, removing o11y
TASK DETAIL
https://phabricator.wikimedia.org/T294014
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Addshore
fgiunchedi added a comment.
Thank you for following up, I think the culprit is the fact that the S3
<https://phabricator.wikimedia.org/S3> compat API stores chunks of big files in
a separate container (suffixed with `+segments`). See also the audit below I
ran logged into
fgiunchedi added a comment.
In T314835#8178848 <https://phabricator.wikimedia.org/T314835#8178848>,
@dcausse wrote:
> Moving forward we will:
>
> - stop the presto-swift client in favor of an S3
<https://phabricator.wikimedia.org/S3> connector.
> - cl
fgiunchedi closed subtask T314914: Bump memcache connections and swift-proxy
limits as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T314835
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: MatthewVernon, gmodena, elukey
fgiunchedi added a comment.
In T314835#8141914 <https://phabricator.wikimedia.org/T314835#8141914>,
@fgiunchedi wrote:
> Thank you @dcausse for diving deep into this issue and mitigating it! I can
confirm that the space has stopped growing at the same rate (i.e. not gro
fgiunchedi added a comment.
Thank you @dcausse for diving deep into this issue and mitigating it! I can
confirm that the space has stopped growing at the same rate (i.e. not growing
ATM).
I can confirm that I've seen the same failures from swift client doing mass
deletes, not sure why
fgiunchedi created this task.
fgiunchedi added projects: Wikidata-Query-Service, SRE, SRE-swift-storage.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
It looks like wdqs more than tripled its storage space usage in the span of
10 days (from ~6T to ~21T
fgiunchedi removed a subtask: T300723: Migrate Traffic Prometheus alerts from
Icinga to Alertmanager.
TASK DETAIL
https://phabricator.wikimedia.org/T281454
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Jdlrobson, fgiunchedi
fgiunchedi removed a subtask: T294564: Migrate Foundations Prometheus alerts to
AlertManager.
TASK DETAIL
https://phabricator.wikimedia.org/T281454
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Jdlrobson, fgiunchedi, Aklapper
fgiunchedi removed a subtask: T293399: Migrate the majority of the analytics
cluster alerts from Icinga to AlertManager.
TASK DETAIL
https://phabricator.wikimedia.org/T281454
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Jdlrobson
fgiunchedi removed a subtask: T289077: Migrate Search team's prometheus-based
alerts from Icinga to alert-manager.
TASK DETAIL
https://phabricator.wikimedia.org/T281454
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Jdlrobson
fgiunchedi removed a subtask: T285328: Migrate OSM sync alerts from icinga to
AlertManager.
TASK DETAIL
https://phabricator.wikimedia.org/T281454
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Jdlrobson, fgiunchedi, Aklapper
fgiunchedi closed this task as "Resolved".
fgiunchedi claimed this task.
fgiunchedi added a comment.
Resolving this in favor of parent task
TASK DETAIL
https://phabricator.wikimedia.org/T281454
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailp
fgiunchedi edited projects, added SRE Observability (FY2022/2023-Q1); removed
SRE Observability (FY2021/2022-Q4).
TASK DETAIL
https://phabricator.wikimedia.org/T281454
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Jdlrobson
fgiunchedi added a comment.
In T297494#8036682 <https://phabricator.wikimedia.org/T297494#8036682>,
@ItamarWMDE wrote:
>> My understanding is that you are solving for the former problem (i.e. MW)
(?)
>
> In this particular case, yes, the metrics are collected fro
fgiunchedi added subscribers: colewhite, Krinkle.
fgiunchedi added a comment.
In T297494#8022604 <https://phabricator.wikimedia.org/T297494#8022604>,
@ItamarWMDE wrote:
> Reposting from a Slack discussion, it appears as though statsd is still the
preferred way to gather som
fgiunchedi added a comment.
This is complete I believe, we're backing up the `daily` hierarchy now
TASK DETAIL
https://phabricator.wikimedia.org/T297145
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi
fgiunchedi closed this task as "Resolved".
fgiunchedi claimed this task.
fgiunchedi added a comment.
I'm tentatively resolving the task since all short term mitigations are
completed, feel free to reopen if sth is amiss
TASK DETAIL
https://phabricator.wikimedia.org/T294
fgiunchedi added a comment.
In T294355#7563057 <https://phabricator.wikimedia.org/T294355#7563057>,
@Manuel wrote:
> Thank you for the suggestion @fgiunchedi! Do we have an explanation
somewhere of how to do this?
Sure no problem! My understanding is that thes
fgiunchedi added a comment.
@Manuel @Lydia_Pintscher going forward I suggest also investing resources to
switch to Prometheus as the supported metric system. Graphite is deprecated and
in "life support" mode while all producers (essentially mediawiki and related)
are being p
fgiunchedi added a comment.
In T294355#7559074 <https://phabricator.wikimedia.org/T294355#7559074>,
@Lucas_Werkmeister_WMDE wrote:
> In T294355#7531241 <https://phabricator.wikimedia.org/T294355#7531241>,
@fgiunchedi wrote:
>
>> In T294355#7531236 <https:
fgiunchedi added a comment.
In T294355#7531236 <https://phabricator.wikimedia.org/T294355#7531236>,
@Lucas_Werkmeister_WMDE wrote:
> I’m not sure I understand the discussion correctly :) do you still need a
list of paths to back up, or does it look like we can back up every
fgiunchedi added a comment.
In T294355#7528880 <https://phabricator.wikimedia.org/T294355#7528880>,
@jcrespo wrote:
> One more question, to finally decide if setting up weekly full backups or
daily but incremental- do all files mostly change completely, or only a subset
fgiunchedi added a comment.
In T294355#7527157 <https://phabricator.wikimedia.org/T294355#7527157>,
@jcrespo wrote:
> number of files are (within reason) a non-blocker for bacula, as files are
packaged into volumes. It is true that each file is stored as a mysql record,
but th
fgiunchedi added a subscriber: jcrespo.
fgiunchedi added a comment.
In T294355#7527026 <https://phabricator.wikimedia.org/T294355#7527026>,
@Lucas_Werkmeister_WMDE wrote:
> Sounds like a good idea to me, I can’t judge how much would fit in Bacula.
Do you need a list of importan
fgiunchedi added a comment.
I've sent the incident up for review, what do you think re: my proposal of
adding parts of the hierarchy to bacula (if it is feasible in terms of number
of files, e.g. `daily` is ~100k files now)
TASK DETAIL
https://phabricator.wikimedia.org/T294355
EMAIL
fgiunchedi added a comment.
Draft incident report:
https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-10-29_graphite
Please feel free to integrate/change as needed. I'll be OOO until the 18th
and I'll pick this back up
TASK DETAIL
https://phabricator.wikimedia.org
fgiunchedi added a comment.
Audit completed, what I did is count the number of null data points in the
year leading to the graphite2003 reimage (i.e. the first reimage, where the
backfill would have first failed) from 2020/10/14 to 2021/10/11 (first column).
And the number of nulls after
fgiunchedi added a comment.
Status update: I'm running a full audit on all ~4M metric files looking for
similar cases. The backfill from yesterday completed in the mean time and some
metrics were able to be backfilled successfully.
I'll be following up with an incident report about
fgiunchedi added a comment.
Status update: the backfill is still ongoing since I lowered the concurrency.
The good news is that some metrics are already backfilled, e.g. api backend
summary:
https://grafana.wikimedia.org/d/2/api-backend-summary?viewPanel=31=1=161723520
fgiunchedi added a comment.
@Lucas_Werkmeister_WMDE thank you for the report. Yes pretty sure the
graphite bullseye migration is related. We backfilled graphite1004 from
graphite2003 (which in turn was the first host we reimaged, and backfilled it
from graphite1004), I suspect some metric
fgiunchedi added a project: Observability-Metrics.
TASK DETAIL
https://phabricator.wikimedia.org/T294014
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Aklapper, fgiunchedi, Invadibot, maantietaja, lmata, Akuckartz, Nandana,
Lahi
fgiunchedi created this task.
fgiunchedi added a project: Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Similarly to T293329: Invalid wikidata graphite metrics received
<https://phabricator.wikimedia.org/T293329> there are `wikidata.daily` metrics
fgiunchedi added a comment.
@Lucas_Werkmeister_WMDE can confirm the invalid metrics don't show up
anymore, thank you!
I found others for `wikidata.daily` but will file a separate task for that:
carbon-cache@b/listener.log:21/10/2021 03:00:00 :: invalid line received
from client
fgiunchedi created this task.
fgiunchedi added projects: Wikidata, Observability-Metrics.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
I noticed this in carbon logs on graphite1004, looks like some wikidata
processes don't send a metric value
==> carbon-cach
fgiunchedi closed this task as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T281359
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: lmata, fgiunchedi
Cc: fgiunchedi, Aklapper, Suran38, Biggs657, Invadibot, Lalamarie69,
m
fgiunchedi added a comment.
I agree, this is done
TASK DETAIL
https://phabricator.wikimedia.org/T290080
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup, fgiunchedi
Cc: colewhite, Addshore, Ladsgroup, fgiunchedi, Aklapper, the0001
fgiunchedi added a comment.
Thank you @Addshore and @Ladsgroup ! Much easier to go Grafana for now, I've
retitled/repurposed the task and thanks for your help on T240685
<https://phabricator.wikimedia.org/T240685> !
TASK DETAIL
https://phabricator.wikimedia.org/T290080
EMAIL PREFE
fgiunchedi renamed this task from "Collect wikidata/siteinfo in Prometheus" to
"Move wikidata lag checks off Icinga".
fgiunchedi updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T290080
EMAIL PREFERENCES
https://phabricator.wikimedi
fgiunchedi created this task.
fgiunchedi added projects: observability, MediaWiki-General, Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
This is a followup in the context of migrating wikidata alerts to
AlertManager (T287741 <https://phabricator.wikimedia.
fgiunchedi updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T281359
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: lmata, fgiunchedi
Cc: fgiunchedi, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF,
maantietaja, lmata
fgiunchedi added a parent task: T288622: All Prometheus based alerts move from
Icinga to alert manager exclusively.
TASK DETAIL
https://phabricator.wikimedia.org/T281359
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi
fgiunchedi added a parent task: T288622: All Prometheus based alerts move from
Icinga to alert manager exclusively.
TASK DETAIL
https://phabricator.wikimedia.org/T281454
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Jdlrobson
fgiunchedi added a parent task: T281359: Onboard teams with Grafana alerts to
AlertManager.
TASK DETAIL
https://phabricator.wikimedia.org/T287741
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Aklapper, Addshore, Invadibot
fgiunchedi added a subtask: T287741: Convert wikidata-alerts grafana dashboard
to AlertManager.
TASK DETAIL
https://phabricator.wikimedia.org/T281359
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi, Aklapper, Biggs657
fgiunchedi closed subtask T282806: Port traffic/netops grafana alerts to
AlertManager as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T281359
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi, Aklapper, Biggs657
fgiunchedi updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T281359
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF,
maantietaja, lmata
fgiunchedi updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T281359
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF,
maantietaja, lmata
fgiunchedi updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T281359
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi, Aklapper, Invadibot, MPhamWMF, maantietaja, lmata, CBogen,
Akuckartz, Nandana
fgiunchedi added a comment.
In T272128#7212524 <https://phabricator.wikimedia.org/T272128#7212524>,
@Ladsgroup wrote:
> Tomorrow all metrics starting with `wikibase.queryService.ui.app.` should
be migrated to `wikibase.queryService.ui.index.app.` I will deploy it around
fgiunchedi added a project: observability.
TASK DETAIL
https://phabricator.wikimedia.org/T262741
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Lydia_Pintscher, Addshore, abian, Aklapper, Invadibot, maantietaja, lmata,
Akuckartz
fgiunchedi closed this task as "Resolved".
fgiunchedi claimed this task.
fgiunchedi added a comment.
Chatted with @gehel and concluded we're ok to offboard him and @RKemper from
legacy paging as-is!
TASK DETAIL
https://phabricator.wikimedia.org/T274249
EMAIL PREFERENC
fgiunchedi added a comment.
Restricted Application added a project: wdwb-tech-focus.
random-ish update re: checkpoint storage after a chat with @Zbyszko: the
current situation is that we're using thanos-swift cluster for wdqs flink
checkpoints. This is meant to be a temporary allocation
fgiunchedi added a project: observability.
TASK DETAIL
https://phabricator.wikimedia.org/T269204
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: RKemper, dcausse, Aklapper, lmata, CBogen, Akuckartz, Nandana, Namenlos314,
Lahi, Gq86
fgiunchedi added a comment.
In T246004#6567338 <https://phabricator.wikimedia.org/T246004#6567338>,
@Zbyszko wrote:
> Thank you all for swift (pun intended) action!
haha! the account is setup now, I've written the credentials in your home on
`deploy1001`
TASK DETAI
fgiunchedi added a comment.
In T246004#6560303 <https://phabricator.wikimedia.org/T246004#6560303>,
@Ottomata wrote:
> I don't know! @fgiunchedi how does one access the cluster? @elukey can
check the network VLAN ACLs and update accordingly.
The canonical url is https
fgiunchedi added a comment.
Ok, thank you for the information. It doesn't seem we have an isolated test
environment anyways so even though I'm reluctant we'll have to test on the
production swift cluster. A middle ground I suppose would be to create the
accounts on the `thanos` swift
fgiunchedi removed a project: Wikimedia-Logstash.
fgiunchedi added a comment.
Removing wikimedia-logstash since it doesn't seem to be a logstash-specific
issue, please add back if that's not the case!
TASK DETAIL
https://phabricator.wikimedia.org/T265015
EMAIL PREFERENCES
https
fgiunchedi removed a project: Wikimedia-Logstash.
fgiunchedi added a comment.
Removing wikimedia-logstash since it doesn't seem to be a logstash-specific
issue, please add back if that's not the case!
TASK DETAIL
https://phabricator.wikimedia.org/T265035
EMAIL PREFERENCES
https
fgiunchedi removed a project: Wikimedia-Logstash.
fgiunchedi added a comment.
Removing wikimedia-logstash since it doesn't seem to be a logstash-specific
issue, please add back if that's not the case!
TASK DETAIL
https://phabricator.wikimedia.org/T265037
EMAIL PREFERENCES
https
fgiunchedi removed a project: Wikimedia-Logstash.
fgiunchedi added a comment.
Removing `wikimedia-logstash` since it doesn't seem to be a logstash-specific
issue, please add back if that's not the case!
TASK DETAIL
https://phabricator.wikimedia.org/T265022
EMAIL PREFERENCES
https
fgiunchedi added a comment.
In T246004#6520077 <https://phabricator.wikimedia.org/T246004#6520077>,
@Zbyszko wrote:
> @fgiunchedi Currently, Flink pipeline resides on the Analytics Hadoop
cluster. As for the question whether Flink creates it's containers - I think
not, it did
fgiunchedi added a comment.
In T246004#6501106 <https://phabricator.wikimedia.org/T246004#6501106>,
@Zbyszko wrote:
> @fgiunchedi We estimate we'd need around 500GB of storage for the streaming
updater (not accounting for replicas). Our use case is almost always write only
(ch
fgiunchedi added a subscriber: JMeybohm.
fgiunchedi added a comment.
@Zbyszko re: docker and swift. @JMeybohm suggested using
https://github.com/swiftstack/docker-swift (and possibly lowering auth token
TTLs to make sure renewing expired tokens works as expected)
re: monitoring, we have
fgiunchedi edited projects, added Traffic, observability; removed Graphite.
Restricted Application added a project: Operations.
TASK DETAIL
https://phabricator.wikimedia.org/T238540
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc
fgiunchedi added a comment.
I can confirm that a DELETE of
https://grafana.wikimedia.org/api/dashboards/uid/00599 results in a 403,
further I don't see the request reaching grafana1001's apache logs. I'm adding
#traffic <https://phabricator.wikimedia.org/tag/traffic/> since this
fgiunchedi added a project: observability.
TASK DETAIL
https://phabricator.wikimedia.org/T204364
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel, fgiunchedi
Cc: gerritbot, Smalyshev, fgiunchedi, Gehel, Aklapper, Hook696, Daryl-TTMG
fgiunchedi added a project: observability.
TASK DETAIL
https://phabricator.wikimedia.org/T136852
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: bd808, fgiunchedi
Cc: gerritbot, hoo, Addshore, Aklapper, bd808, Zppix, Hook696, Daryl-TTMG
fgiunchedi added a project: observability.
TASK DETAIL
https://phabricator.wikimedia.org/T178530
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: debt, fgiunchedi
Cc: Stashbot, gerritbot, dcausse, Gehel, EBernhardson, Aklapper, Smalyshev,
Hook696
fgiunchedi added a comment.
In T221774#5155621 <https://phabricator.wikimedia.org/T221774#5155621>, @hoo
wrote:
> Possible way to do this:
>
> Create `PrometheusBlazegraphLagService` class which internally fetches the
lag from a given Blazegraph instance li
fgiunchedi updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T187960
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Cmjohnson, fgiunchedi
Cc: akosiaris, Joe, fgiunchedi, hashar, Krinkle, ArielGlenn, jijiki, Addshore
fgiunchedi added a comment.
Any update?TASK DETAILhttps://phabricator.wikimedia.org/T208215EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: fgiunchedi, Aklapper, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune
fgiunchedi created this task.fgiunchedi added a project: Wikidata-Query-Service.Restricted Application added a subscriber: Aklapper.Restricted Application added a project: Wikidata.
TASK DESCRIPTIONNoticed this while investigating something else, metrics exposed by jmx_exporter running on wdqs
fgiunchedi added a comment.
In T195121#4529715, @aborrero wrote:
In T195121#4527757, @fgiunchedi wrote:
Sounds like a nice project! With my swift maintainer hat on, testing a single 200-300 GB chunk of data sounds good to me. Let's coordinate though before uploading the full data set because
fgiunchedi added a comment.
Sounds like a nice project! With my swift maintainer hat on, testing a single 200-300 GB chunk of data sounds good to me. Let's coordinate though before uploading the full data set because swift is pending its annual expansion (T201937) and I'd like to have
fgiunchedi closed subtask T195530: status.wikimedia.org showing all lights green during major outage as "Invalid".
TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, fgiunchediCc
fgiunchedi added a comment.
No planned upgrades ATM, though a newer upstream version might help with understanding (hopefully fixing) T192456: Prometheus metrics missing for some hosts too, so definitely welcome!TASK DETAILhttps://phabricator.wikimedia.org/T192768EMAIL PREFERENCEShttps
fgiunchedi triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T186815EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: Superyetkin, Aklapper, Davinaclare77, Qtn1293, Lahi, Gq86, GoranSMilovanovic, Th3d3v
fgiunchedi closed this task as "Resolved".fgiunchedi claimed this task.fgiunchedi added a comment.
Done, fix deployedTASK DETAILhttps://phabricator.wikimedia.org/T184434EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: fgiunchedi,
fgiunchedi added a project: User-fgiunchedi.fgiunchedi added a comment.
@Addshore yes! I'll try taking a look in the next couple of weeks I thinkTASK DETAILhttps://phabricator.wikimedia.org/T147328EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore
fgiunchedi triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T160685EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: Lydia_Pintscher, Krinkle, Reedy, jeblad, Aklapper, Th3d3v1ls, Hfbn0, QZanden,
fgiunchedi triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T150356EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: Smalyshev, Gehel, Aklapper, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas
fgiunchedi added a comment.
Merged, though following up from IRC: we're keeping grafana labs/prod segregated in their datasources to avoid introducing more production/labs dependencies. It'd be nice if the tool ran somewhere in production. AFAICS it is stateless so that shouldn't be too hard
fgiunchedi triaged this task as "Normal" priority.
TASK DETAIL
https://phabricator.wikimedia.org/T133490
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: Bovlb, Aklapper, Mushroom, Avner, debt, Gehel, D3r1ck01, FloNight, Izn
fgiunchedi added a comment.
drive-by comment: partman will likely to be adjusted too so we don't run into
surprises when reprovisioning
TASK DETAIL
https://phabricator.wikimedia.org/T119579
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RobH
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi closed this task as "Resolved".
fgiunchedi claimed this task.
fgiunchedi added a comment.
{{done}}
TASK DETAIL
https://phabricator.wikimedia.org/T118850
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailp
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi closed this task as "Resolved".
fgiunchedi claimed this task.
fgiunchedi added a comment.
{{done}}
TASK DETAIL
https://phabricator.wikimedia.org/T118836
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailp
fgiunchedi triaged this task as "Normal" priority.
fgiunchedi added a subscriber: fgiunchedi.
TASK DETAIL
https://phabricator.wikimedia.org/T119579
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi, hoo, Akl
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi added a comment.
thanks for expanding on that, here's my (as the person who's been looking after
our graphite stack) opinion:
- graphite isn't really data warehouse, thus I wouldn't recommend it as the
primary storage for the verbatim
fgiunchedi added a comment.
@addshore to clarify, more than functionality I was pointing out guarantees
about the data stored. if the metrics are also being archived to hdfs for
example so it is possible to dump/load into graphite then IMO that's acceptable.
re: analytics graphite instance, I
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi added a comment.
for long term data warehousing or analytics type of workflows using
ourhadoop/analytics infrastructure will be more appropriate I think. graphite
is more focused on operational metrics from applications, services and so
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi assigned this task to GLavagetto.
fgiunchedi added a comment.
moving to @joe
TASK DETAIL
https://phabricator.wikimedia.org/T95679
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GLavagetto
fgiunchedi reassigned this task from GLavagetto to Joe.
fgiunchedi added a subscriber: GLavagetto.
TASK DETAIL
https://phabricator.wikimedia.org/T95679
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Joe, fgiunchedi
Cc: GLavagetto, fgiunchedi
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi added a comment.
looks like this is completed, anything else left @joe ?
TASK DETAIL
https://phabricator.wikimedia.org/T84902
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username
99 matches
Mail list logo
