Smalyshev added a comment.
I also discover for some items the data is not the latest revision: e.g. for Q57529925 we have all servers except wdq5 on 795730255 but wdq5 on 795729753. This seems to be related to bursts of robotic edits on the same entry, which may suggest there's some kind of race
Smalyshev triaged this task as "High" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T209776EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: hashar, Addshore, Aklapper, Smalyshev, Nandana, Lahi, Gq86, Lucas_Werkme
Smalyshev added a comment.
The build is still failing. Is there somebody that should do something, and if so, who and what?TASK DETAILhttps://phabricator.wikimedia.org/T209776EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: hashar, Addshore
Smalyshev added a comment.
Possibly a consequence of T210044: Data corruption when loading RDF data into WDQS.TASK DETAILhttps://phabricator.wikimedia.org/T207826EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Smalyshev, Nandana, Lahi
Smalyshev added subscribers: Floatingpurr, EBjune, Jane023, Tarrow, Gstupp.Smalyshev merged a task: T207675: Some items are in an inconsistent state.
TASK DETAILhttps://phabricator.wikimedia.org/T210044EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
Smalyshev closed this task as a duplicate of T210044: Data corruption when loading RDF data into WDQS.
TASK DETAILhttps://phabricator.wikimedia.org/T207675EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: EBjune, Jane023, Tarrow
Smalyshev closed subtask T206189: Set sensible thread limit to Blazegraph as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T206108EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Gehel, Nandana,
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.
TASK DETAILhttps://phabricator.wikimedia.org/T206189EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: gerritbot, Aklapper, Gehel, Smalyshev, CucyNoiD, Nandana, Ne
Smalyshev triaged this task as "High" priority.Smalyshev added a project: User-Smalyshev.
TASK DETAILhttps://phabricator.wikimedia.org/T210044EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: mhl20, Wikidata-Query-Service,
Smalyshev claimed this task.
TASK DETAILhttps://phabricator.wikimedia.org/T210044EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: mhl20, Wikidata-Query-Service, Oravrattas, Lucas_Werkmeister_WMDE, Stashbot, Alexsdutton, Aklapper, Smalyshev
Smalyshev closed this task as a duplicate of T210044: Data corruption when loading RDF data into WDQS.
TASK DETAILhttps://phabricator.wikimedia.org/T203646EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Stashbot, Lucas_Werkmeister_WMDE
Smalyshev added subscribers: Alexsdutton, Stashbot, Lucas_Werkmeister_WMDE, Oravrattas, Wikidata-Query-Service, mhl20.Smalyshev merged a task: T203646: Wikidata Query Service nodes out of sync.
TASK DETAILhttps://phabricator.wikimedia.org/T210044EMAIL PREFERENCEShttps://phabricator.wikimedia.org
Smalyshev added a comment.
Timestamps for data updates:
wdq10: 2018-11-20T05:49:19Z
wdq6: 2018-11-20T05:49:25Z
wdq26: 2018-11-20T05:49:31Z
wdq21: 2018-11-20T05:49:32Z
wdq22: 2018-11-20T05:49:32Z
wdq3: 2018-11-20T05:49:35Z
wdq7: 2018-11-20T05:49:40Z
wdq9: 2018-11-20T05:49:39Z
wdq8: 2018-11-20T05
Smalyshev created this task.Smalyshev added projects: Wikidata-Query-Service, Discovery-Wikidata-Query-Service-Sprint.Restricted Application added a subscriber: Aklapper.Restricted Application added a project: Wikidata.
TASK DESCRIPTIONSome data are not loaded correctly into WDQS
Smalyshev added a comment.
Tried with incoming stream, the machine can't keep up - by now it's 5 hours behind, and processing about half the necessary requests.
I think the conclusion is mostly clear - this VM as such is not suitable for any performance testing or any workloads that are close
Smalyshev added a comment.
Trying to run Updater on labs (for T206636) where there's no Kafka, I still get these errors all the time. So the problem does not seem to be solved.TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
Smalyshev added a comment.
Updater seems to be able to get about 4-5 updates per second, which is abut 2x slower than production. Summarily, it looks like this setup may be fit for functionality testing, but decidedly unfit for any performance testing, as it is 2-3x slower than production.
Final
Smalyshev created this task.Smalyshev added a project: Wikidata Query UI.Restricted Application added a subscriber: Aklapper.Restricted Application added a project: Wikidata.
TASK DESCRIPTIONWhen building WDQS with newest GUI, I get this:
01:24:39 [INFO] Running "qunit:all" (qunit) tas
Smalyshev added a comment.
Loading finished, overall took 8 days and 9 hours, or 201 hours, or 3x compared to production. Launching updater next to see the updates speed.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
Smalyshev added a comment.
ensuring that the data in the WDQS nodes accurately reflects the data upstream of the service, or at least that the data is consistent between query nodes
I am not sure how you would propose ensuring that. Given the database of almost 7 billion triples
Smalyshev triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T207826EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Smalyshev, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovi
Smalyshev lowered the priority of this task from "Unbreak Now!" to "High".
TASK DETAILhttps://phabricator.wikimedia.org/T209201EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Tarrow, Krenair, Addshore, TerraCodes, Liux
Smalyshev added a comment.
Another thing we might want to consider is there's a queue of requests that are in Jetty queue even before they reach QueryServlet. On high load, a lot of these requests, by the time they reach execution, would be already abandoned by their clients, disconnected, etc. I
Smalyshev added a comment.
Testing with the thread limiting patch on wdqs1010, I see the thread count never go over 2500, and as soon as load is removed, the service starts recovering within minutes, with no lingering effects.TASK DETAILhttps://phabricator.wikimedia.org/T206189EMAIL
Smalyshev added a parent task: T206108: Limit CPU consumption of blazegraph.
TASK DETAILhttps://phabricator.wikimedia.org/T206189EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Gehel, Smalyshev, Nandana, Lahi, Gq86
Smalyshev added a project: Wikidata-Query-Service.Restricted Application added a project: Wikidata.
TASK DETAILhttps://phabricator.wikimedia.org/T206108EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Gehel, Nandana, Lahi, Gq86
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
This is done.TASK DETAILhttps://phabricator.wikimedia.org/T112127EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Lydia_Pint
Smalyshev closed this task as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T206123EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Mathew.onipe, SmalyshevCc: Stashbot, Smalyshev, Mathew.onipe, gerritbot, Aklapper, Gehel, CucyNoi
Smalyshev added a comment.
Looking at performance graphs, in regular operation number of threads stays well under 1000. Of those about 300 come from non-Executor services, so normal count of executorService threads is around 700. I think if we start to refuse service when executorService thread
Smalyshev added a project: Upstream.Smalyshev added a comment.
Filed https://github.com/blazegraph/database/issues/107 with upstream.TASK DETAILhttps://phabricator.wikimedia.org/T197598EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper
Smalyshev added a comment.
One thing to consider here to stop the situation getting too terrible would be to add the wdqs lag to the maxlag for wikidata.org
@Addshore I am not very informed on this one, what's maxlag and how it works?TASK DETAILhttps://phabricator.wikimedia.org/T209201EMAIL
Smalyshev added a comment.
So it sounds like the only place to fix this is within blazegraph itself.?
One of the solutions may be to try and figure out how to do faster updates. Another would be to add servers to production cluster to spread query load. The pattern seems to be very dependant
Smalyshev added a comment.
The difference seems to come because Blazegraph chooses different query plans with and without limit. Without limit:
P7814 (An Untitled Masterwork)
with limit:
P7815 (An Untitled Masterwork)
The difference is JVMSolutionSetHashJoinOp vs
Smalyshev claimed this task.
TASK DETAILhttps://phabricator.wikimedia.org/T197598EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Smalyshev, Lucas_Werkmeister_WMDE, Daniel_Mietchen, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113
Smalyshev added a comment.
@Imarlier nothing special in GC that can be linked to the errors. GC times seem to be low and unexceptional.TASK DETAILhttps://phabricator.wikimedia.org/T207718EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Imarlier, SmalyshevCc
Smalyshev added a comment.
This file is mentioned as appender for com.bigdata.relation.rule.eval.RuleLog but I don't think we even use these logging configs anymore. Not sure though how our logging and log4j logging interacts...TASK DETAILhttps://phabricator.wikimedia.org/T144539EMAIL
Smalyshev added a comment.
Is this still relevant? We do have regular deploys now at 10am PDT Monday.TASK DETAILhttps://phabricator.wikimedia.org/T144932EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Lydia_Pintscher, Smalyshev, Aklapper, hoo
Smalyshev closed this task as "Resolved".Smalyshev added a comment.
Should be ok now.TASK DETAILhttps://phabricator.wikimedia.org/T207643EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, SmalyshevCc: gerritbot, Aklapper, fgiunchedi, Smalyshe
Smalyshev triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T207665EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Mathew.onipe, Smalyshev, Gehel, Nandana, Lahi, Gq86, Lucas_Werkme
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
Should be fixed with new logging system.TASK DETAILhttps://phabricator.wikimedia.org/T150356EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc:
Smalyshev added a project: User-Smalyshev.
TASK DETAILhttps://phabricator.wikimedia.org/T207675EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Jane023, Tarrow, Lucas_Werkmeister_WMDE, Aklapper, Smalyshev, Floatingpurr, Gstupp, Nandana, Lahi, Gq86
Smalyshev added a comment.
@Floatingpurr This is weird indeed, looks like some data is missing. Not sure why it happened, I'll research.TASK DETAILhttps://phabricator.wikimedia.org/T207675EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Jane023
Smalyshev added a comment.
It could be -- how quickly does it retry? Immediately? Or is there a delay?
I don't think there's a delay for NoHttpResponseException. There's 500ms delay for 503 and 429, but for other exceptions I think it retries immediately.
https://phabricator.wikimedia.org
Smalyshev added a comment.
@Imarlier yes the patches have been deployed though we don't use RC API now for production so T207718 is more important than this one.TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
Smalyshev added a comment.
So, an interesting thing: in at least some of these cases, there is a web request that is making it to wikidata, and that is returning a 200.
The request is retried if it fails, are you sure it's not the retry that you are seeing with 200?TASK DETAILhttps
Smalyshev added a comment.
The patch has been deployed, and doesn't look like it prevents the issue:
18:05:29.346 [update 4] WARN org.wikidata.query.rdf.tool.Updater - Retryable error syncing. Retrying.
18:05:29.346 [update 7] WARN org.wikidata.query.rdf.tool.Updater - Retryable error syncing
Smalyshev closed subtask T195203: Federated SPARQL queries https://data.pdok.nl/sparql failing with error 500 as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T197530EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklappe
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
This seems to work now, looks like upgrade to jetty 9.4 fixed it.TASK DETAILhttps://phabricator.wikimedia.org/T195203EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailp
Smalyshev added a project: User-Smalyshev.
TASK DETAILhttps://phabricator.wikimedia.org/T200612EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: seav, Aklapper, jindrichmynarz, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden
Smalyshev added a project: Wikidata-Query-Service.Restricted Application added a project: Wikidata.
TASK DETAILhttps://phabricator.wikimedia.org/T206123EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Mathew.onipe, SmalyshevCc: Stashbot, Smalyshev
Smalyshev added a comment.
@Floatingpurr:
Something similar seems happening for this query:
This query produces nothing for me. Is that expected result?TASK DETAILhttps://phabricator.wikimedia.org/T207675EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
Smalyshev added a comment.
Could it be related to this report
We are not using RC changes now, but Kafka stream, so not likely.
I wonder if running them on a different host might provide a small bit of processing relief for the wdqs hosts?
Updater process does not consume a lot of CPU (most
Smalyshev added a comment.
Looks like this VM is substantially slower - I started data load at Nov 9, now is end of Nov 13, and it's only 75% done, which means it'll take about a week to load all data, which is significantly slower than in production (it was done in 63 hours).
However, the real
Smalyshev triaged this task as "Lowest" priority.Smalyshev added a comment.
Hmm yes this looks like we'd need to duplicate the client-side SVG rendering on server-side, which is a huge piece of work. I'd probably suggest for now using browser automation if one needs to automatically ge
Smalyshev closed this task as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T204364EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, SmalyshevCc: gerritbot, Smalyshev, fgiunchedi, Gehel, Aklapper, Legado_Shulgin, CucyNoi
Smalyshev closed this task as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T200696EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: gerritbot, Aklapper, Lucas_Werkmeister_WMDE, CucyNoiD, Nandana, NebulousIris, Gaboe420
Smalyshev moved this task from Up Next to Current work on the Discovery-Search board.Smalyshev edited projects, added Discovery-Search (Current work); removed Discovery-Search.
TASK DETAILhttps://phabricator.wikimedia.org/T206613WORKBOARDhttps://phabricator.wikimedia.org/project/board/1849/EMAIL
Smalyshev triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T204364EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, SmalyshevCc: gerritbot, Smalyshev, fgiunchedi, Gehel, Aklapper, Legado_Shulgin, CucyNoi
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
I think swapping helped, so closing for now.TASK DETAILhttps://phabricator.wikimedia.org/T200563EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: S
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.
TASK DETAILhttps://phabricator.wikimedia.org/T206961EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Gehel, gerritbot, Aklapper, Smalyshev, CucyNoiD, Nandana, Ne
Smalyshev claimed this task.
TASK DETAILhttps://phabricator.wikimedia.org/T55652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: gerritbot, Smalyshev, Wikidata-bugs, Yair_rand, Lydia_Pintscher, daniel, CucyNoiD, Nandana, NebulousIris, Gaboe420
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
Put a space between 0 and the dot. 0. is parsed as the beginning of a floating point number.TASK DETAILhttps://phabricator.wikimedia.org/T209223EMAIL PREFERENCEShttps://phabricator.wik
Smalyshev added a comment.
Looking at the servers, we have very low update throughput:
01:17:43.485 [main] INFO org.wikidata.query.rdf.tool.Updater - Polled up to 2018-11-09T23:13:34Z at (3.5, 3.9, 1.9) updates per second and (543.7, 598.2, 291.9) milliseconds per second
Which probably means
Smalyshev triaged this task as "Unbreak Now!" priority.Restricted Application added subscribers: Liuxinyu970226, TerraCodes.
TASK DETAILhttps://phabricator.wikimedia.org/T209201EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc:
Smalyshev created this task.Smalyshev added a project: Wikidata-Query-Service.Restricted Application added a subscriber: Aklapper.Restricted Application added a project: Wikidata.
TASK DESCRIPTIONThe situation with update lag keeps deteritoriating (it's 2 hours behind now and is not improving
Smalyshev created this task.Smalyshev added a project: Wikidata-Query-Service.Restricted Application added a subscriber: Aklapper.Restricted Application added a project: Wikidata.
TASK DESCRIPTIONWhen running builds on Maven which include tests, I get a lot of unnecessary logging from Blazegraph
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
Seems to be OK now.TASK DETAILhttps://phabricator.wikimedia.org/T202785EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Mchlrch, Gehel
Smalyshev closed subtask T202785: Federation request to https://ld.stadt-zuerich.ch/query fails as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T197530EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Esc3300, ET4Ev
Smalyshev added a comment.
This seems to be happening only to a small set of values, namely:
4576 b2995751623614f950797f80606ddb60
173 a843a14d6be3111e93a253fd623f18cf
152 7e281616976c7de150357c18e76abfd1
48 b3795d3425e0bbdd474f3138cad4a069
6 a4cd577c35e40a9190ebcdbeb49b0d38
6
Smalyshev added a project: User-Smalyshev.
TASK DETAILhttps://phabricator.wikimedia.org/T197598EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Smalyshev, Lucas_Werkmeister_WMDE, Daniel_Mietchen, Nandana, Lahi, Gq86, Darkminds3113
Smalyshev closed this task as "Resolved".Smalyshev added a comment.
Doesn't seem to be missing anymore, maybe it was update lag?TASK DETAILhttps://phabricator.wikimedia.org/T208042EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc
Smalyshev claimed this task.Smalyshev added projects: User-Smalyshev, Discovery-Wikidata-Query-Service-Sprint.
TASK DETAILhttps://phabricator.wikimedia.org/T200696EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: gerritbot, Aklapper
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
I think we have mostly achieved this.TASK DETAILhttps://phabricator.wikimedia.org/T206121EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc:
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
Looks like it's fine now.TASK DETAILhttps://phabricator.wikimedia.org/T208986EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Stashbot,
Smalyshev added a comment.
-Dorg.eclipse.jetty.annotations.AnnotationParser.LEVEL=OFF seems to suppress the messages, but it'd be nice to get rid of their source.TASK DETAILhttps://phabricator.wikimedia.org/T209020EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
Smalyshev added a comment.
Jetty produces about 7400 such messages for Blazegraph...TASK DETAILhttps://phabricator.wikimedia.org/T209020EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Gehel, Smalyshev, Nandana, Lahi, Gq86
Smalyshev created this task.Smalyshev added a project: Wikidata-Query-Service.Restricted Application added a subscriber: Aklapper.Restricted Application added a project: Wikidata.
TASK DESCRIPTIONAfter upgrade to jetty 9.4, launching Blazegraph produces a myriad of messages like this:
Nov 08 07
Smalyshev triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T206189EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Gehel, Smalyshev, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSM
Smalyshev triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T204045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Lucas_Werkmeister_WMDE, Aklapper, Nandana, Lahi, Gq86, GoranSMilovanovic, QZand
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
Updated.TASK DETAILhttps://phabricator.wikimedia.org/T208925EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Smalyshev, Marsupium
Smalyshev added a comment.
It's not alright, we know about it but hasn't found the cause for it as of yet.TASK DETAILhttps://phabricator.wikimedia.org/T208928EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Smalyshev, Lucas_Werkmeister_WMDE
Smalyshev triaged this task as "High" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T208986EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Stashbot, gerritbot, bd808, faidon, Gehel, Aklapper, Smalyshev, Legado_Shulgin, CucyNoi
Smalyshev added projects: Discovery-Wikidata-Query-Service-Sprint, User-Smalyshev.Smalyshev added a comment.
Bryan advises against setting hard limits on executor, so the options for limiting thread growth are:
Not launching new queries if the thread count too high
Limit number of simultaneous
Smalyshev closed subtask T206880: Investigate runaway Blazegraph threads as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T206189EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Gehel, Smalyshev, Nandana,
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
I think it's pretty clear what is going on, so next thing would be in T206189 to set some limits. Bryan advises against setting hard limits on executor, so the options are:
Not launching n
Smalyshev added projects: Operations, Cloud-Services.
TASK DETAILhttps://phabricator.wikimedia.org/T208986EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Gehel, Aklapper, Smalyshev, Legado_Shulgin, Nandana, thifranc, AndyTan, Zylc, Davinaclare77
Smalyshev created this task.Smalyshev added a project: Wikidata-Query-Service.Restricted Application added a subscriber: Aklapper.Restricted Application added a project: Wikidata.
TASK DESCRIPTIONCI jobs for WDQS now fail with:
18:12:20> Throwable
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
I think we're mostly done with implementing the followup. For reference, incident report is here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20181024-WDQSTASK D
Smalyshev closed subtask T207656: WDQS logging should be rate limited as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T207817EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: greg, Ottomata, Pchelolo, Tarrow, WMDE-leszek
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.Smalyshev added a comment.
I still need a config for enabling TRACE level, but the rest seems to work fine.TASK DETAILhttps://phabricator.wikimedia.org/T207834EMAIL PREFERENCEShttps://phabricator.wikimedia.org/sett
Smalyshev closed subtask T207834: Cleanup Wikidata Query Service logging configuration as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T207817EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: greg, Ottomata, Pchelolo, Ta
Smalyshev added a comment.
Additional wrinkle: https://github.com/eclipse/jetty.project/issues/920 - looks like latest 9.2 runner is buggy, and the fix is only in later versions. At least 9.2.26.v20180806 didn't work. We can use current runner I guess but then build package is broken. We need
Smalyshev added a comment.
I am noticing that there's no use of the TabulistBot at all. So I wonder if the use case is there...TASK DETAILhttps://phabricator.wikimedia.org/T67626EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Smalyshev
Smalyshev added a comment.
I think we should keep it around for now, unless it becomes maintenance burden. Some things can be tested on beta, but experiments that involve direct-patching configs and code are hard to test on beta, and easy on labs. Main thing for lexemes is that we need a bunch
Smalyshev added a subscriber: Gehel.Smalyshev added a comment.
Looks like this hits unexpected bump - wiremock is not compatible with Jetty above 9.2, as it seems: https://github.com/tomakehurst/wiremock/pull/887 and the maintainer is opposed to making it work. See also: https://github.com
Smalyshev added a comment.
Not sure where that error is coming from - SPARQL responses have access-control-allow-origin: *. Maybe it's something in Mediawiki settings?TASK DETAILhttps://phabricator.wikimedia.org/T208329EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
Smalyshev added a comment.
Excluding the above, I think we're now ready to move with this. Patches needed here:
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexeme/+/468183
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/269357
https://gerrit.wikimedia.org/r/c
Smalyshev added a comment.
@Lucas_Werkmeister_WMDE given that we haven't had much progress with T99907 since 3 years ago, and any change there would be incremental, I don't think we should block on it. If we make a sprint on it and have the resolution soon (like, before 2019) I definitely can hold
Smalyshev added a comment.
I'm guessing this is more of a "follow-up" task vs an "active situation" task, yes?
Yes, by now it's definitely a follow-up tracking task.TASK DETAILhttps://phabricator.wikimedia.org/T207817EMAIL PREFERENCEShttps://phabricator.wikimedi
Smalyshev closed this task as "Resolved".Smalyshev claimed this task.
TASK DETAILhttps://phabricator.wikimedia.org/T207947EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Stashbot, ema, gerritbot, Aklapper, Gehel, Legado_Shulgin, CucyNoi
1101 - 1200 of 6330 matches
Mail list logo
