[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

Yes, with some promising discussions recently about ways to support continuous updates and open tasks inviting contribs On Wed, Oct 2, 2024 at 8:27 AM Peter F. Patel-Schneider < pfpschnei...@g

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

I believe that the QLever service does not work this way. Instead the QLever service is based on the Wikidata RDF dumps and its data is reloaded in a batch process each week. The data in the service is thus between a few days and two weeks behind the data in Wikidata. There are several reaso

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

The most promising show of "we are ready" for the Virtuoso Open Source edition would be what QLever has been doing for a while: to provide a public endpoint with the data loaded, and kept up to date using the public edit stream. That would be an undeniably strong argument for "just use this!" On

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

An updated benchmark eval (or self-evals) for the top db candidates seems called for :) we would all love to see it. Ideally with a canonical hardware or vm spec that all can use... I don't know if the QLever self-eval from the spring is the right place to start, but I believe these

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

Hi Denny, On 9/25/24 1:40 AM, Denny Vrandečić wrote: If my memory serves me well, the Open Source version of Virtuoso doesn't have certain scalability features that would be necessary to run a graph as large and dynamic as Wikidata's. Is this information out-of-date? Cheers, Denny Yes, th

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

t: 25 September 2024 07:40 To: Discussion list for the Wikidata project Subject: [Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025 If my memory serves me well, the

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

If my memory serves me well, the Open Source version of Virtuoso doesn't have certain scalability features that would be necessary to run a graph as large and dynamic as Wikidata's. Is this information out-of-date? Cheers, Denny On Tue, Sep 24, 2024 at 10:35 PM Kingsley Idehen via Wikidata < wiki

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

Hi Everyone, On 9/6/24 11:46 AM, Samuel Klein wrote: On Fri, Sep 6, 2024 at 4:52 AM Luca Martinelli [Sannita@WMF] wrote: no "magic solution" exists, each comes with its load of problems and costs Given the reload speed, approach to more continuous updating

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

Looking to alternatives to Blazegraph (Qlever being one of those) is definitely on our list of things to do once we have completed our current project of splitting the graph! On Fri, 6 Sept 2024 at 17:47, Samuel Klein wrote: > On Fri, Sep 6, 2024 at 4:52 AM Luca Martinelli [Sannita@WMF] < > sann

[Wikidata] Re: timing information from the Wikidata Query Service

{ ?s ?p ?o } limit 1000" \ -d "explain=details" -o ~/explainDetails.html ( equivalent GET URL: https://query.wikidata.org/sparql?explain=details&query=select%20*%20where%20%7B%20%3Fs%20%3Fp%20%3Fo%20%7D%20limit%201000 ) -Adam [1] http

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

On Fri, Sep 6, 2024 at 4:52 AM Luca Martinelli [Sannita@WMF] < sann...@wikimedia.org> wrote: > no "magic solution" exists, each comes with its load of problems and > costs Given the reload speed, approach to more continuous updating

[Wikidata] timing information from the Wikidata Query Service

Is there a way to get the Wikidata Query Service to report the total time taken to evaluate a query? This would be better than estimating the time from the elapsed time between a request and the beginning of a response. peter ___ Wikidata mailing

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

On Thu, Sep 5, 2024 at 8:15 PM Physikerwelt wrote: > > Dear Luca, > > the communication was good I think. > > However, I don't understand the decision process. Who is the > responsible person, e.g., the product manager, who eventually decided > to cut WDQS into pieces? > > Moritz The decision was

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

On Thu, Sep 5, 2024 at 6:02 PM Physikerwelt wrote: > > Dear Guillaume, > > thanks for sharing. Could you explain WHO has decided that the graph split > will be done? > > All the best > Moritz Hi Physikerwelt, the problems with the Wikidata Query Service backend are be

[Wikidata] Re: [Wikitech-l] Re: [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

and “scholarly”[3] subgraphs of >> Wikidata. >> >> As you might be aware we are addressing the Wikidata Query Service >> stability and scaling issues. We have been working on several projects to >> address these issues. This announcement is about one of them, the WDQS >>

[Wikidata] Re: [Wikitech-l] [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

ew SPARQL endpoints > available for serving the “main”[2] and “scholarly”[3] subgraphs of > Wikidata. > > As you might be aware we are addressing the Wikidata Query Service > stability and scaling issues. We have been working on several projects to > address these issues. This announ

[Wikidata] [BREAKING CHANGE ANNOUNCEMENT] Wikidata Query Service graph split available in production; scholarly entity queries require migration by March 2025

Hi all! As part of the WDQS Graph Split project,[1] we have new SPARQL endpoints available for serving the “main”[2] and “scholarly”[3] subgraphs of Wikidata. As you might be aware we are addressing the Wikidata Query Service stability and scaling issues. We have been working on several projects

[Wikidata] Wikidata Query Service - Scaling Update - February 2024

various use cases and workflows around Wikidata Query Service. **What is the WDQS Graph Split experiment?** We want to address the growing size of the Wikidata graph by splitting it into 2 subgraphs of roughly half the size of the full graph, which should support the growth of Wikidata for the next

[Wikidata] Wikidata Query Service - Scaling Update - October 2023

scale Wikidata Query Service as mentioned in our annual plan. What are we trying to address? We are convinced that the current highest risk to Wikidata Query Service is the data size and data growth. Wikidata is growing at a rate of roughly 1 billion triples per year and is already one of the l

[Wikidata] Re: Wikidata Query Service overloaded in codfw

Hello Wikidata, We have had some ups and downs with the WDQS service in our CODFW datacenter today, but we believe the service is stabilized. We have opened an incident report for the issue[0] . I'm still filling it out, but I should have more information within the next 24 hours. Please accep

[Wikidata] Wikidata Query Service overloaded in codfw

Hello all! We are currently experiencing issues with the codfw WDQS cluster, with a sharp drop in successful queries [1]. We suspect that the cluster is overloaded by some expensive queries, but we are still tracking them down. This should affect only traffic routed to the codfw cluster [2]. We w

[Wikidata] Just for fun - chatGPT creating a simple wikidata query

The ChatGPT service that's all the rage is OK at some basic wikidata queries: https://twitter.com/erik_paulson/status/1599491203866558464 (first 4 chats) https://twitter.com/erik_paulson/status/1599491207247187968 (next 3) It's a terrible search engine so it doesn't know any q numbers, but I was s

[Wikidata] Re: Awareness: Wikidata Query Service interruption

a great weekend! Best, Brian King SRE, Search Platform Team Wikimedia Foundation IRC: inflatador > On Dec 2, 2022, at 11:06 AM, Brian King wrote: > > Hello Wikidata Query users, > > We are aware of an ongoing issue with WDQS that begun around 1500 UTC today. > To sta

[Wikidata] Awareness: Wikidata Query Service interruption

Hello Wikidata Query users, We are aware of an ongoing issue with WDQS that begun around 1500 UTC today. To stabilize the service, we have introduced drastic measures including banning cloud provider IPs and certain user agents. Unfortunately, this will likely affect legitimate users of the

[Wikidata] Awareness: Brief Wikidata Query Service Outage 2022-11-22

Hello Wikidata Users, Wikidata Query Service experienced a brief outage on 2022 November 22[1]. Please see the link below for further details. If you have any questions about this, please respond here or on the talk page for the article below. Our apologies for any disruption you may have

[Wikidata] Query

Hello, I made this query. https://query.wikidata.org/#SELECT%20DISTINCT%20?item%20WHERE%20%7B%0A%20%20?item%20p:P414%20?statement0.%0A%20%20?statement0%20(ps:P414/(wdt:P279*))%20_:anyValueP414.%0A%7D It shows all items which are having "stock exchange" property used. I need to make a query which

[Wikidata] A new update on the Wikidata Query Service scaling process

Hi all! We have an important update to share with the community regarding the Wikidata Query Service scaling process: we closed our study phase and published a working paper,[1] in which we short-listed and analyzed four potential candidates for replacing Blazegraph. You can find out a summary of

[Wikidata] Re: New Streaming Updater for Wikidata Query Service in production 18 Oct 2021

roduct Manager, Search Wikimedia Foundation <https://wikimediafoundation.org/> On 16September, 2021 at 13:04:01, Mike Pham (mp...@wikimedia.org) wrote: Hello all, Thank you again for your all recent thoughts and feedback with regard to the recent Wikidata: Query Service (WDQS) scaling update Aug

[Wikidata] Re: New Streaming Updater for Wikidata Query Service in production 18 Oct 2021

> > *Mike Pham* (he/him) > Sr Product Manager, Search > Wikimedia Foundation <https://wikimediafoundation.org/> > > On 16September, 2021 at 13:04:01, Mike Pham (mp...@wikimedia.org) wrote: > > Hello all, > > Thank you again for your all recent thoughts an

[Wikidata] Re: New Streaming Updater for Wikidata Query Service in production 18 Oct 2021

regard to the recent Wikidata: Query Service (WDQS) scaling update Aug 2021 <https://www.wikidata.org/wiki/Wikidata:Query_Service_scaling_update_Aug_2021>, and for everyone who has responded to the WDQS user survey <https://www.wikidata.org/wiki/Wikidata:Project_chat/

[Wikidata] New Streaming Updater for Wikidata Query Service in production 18 Oct 2021

Hello all, Thank you again for your all recent thoughts and feedback with regard to the recent Wikidata: Query Service (WDQS) scaling update Aug 2021 <https://www.wikidata.org/wiki/Wikidata:Query_Service_scaling_update_Aug_2021>, and for everyone who has responded to the WDQS user survey

[Wikidata] Technical Issues with Wikidata Query Service

Hi all, At around 1PM UTC today (Sep 3) we started experiencing stability issues with WDQS, localized (at least at the moment) to a single, of two, datacenter. Unfortunately, we haven't been able to pinpoint the issue as of now. We suspect that someone is running a query that affects Blazegraph -

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

etter in the future. >>> >>> >>> >>> >>> >>> On 23/08/2021 21:36, Samuel Klein wrote: >>> > Ah, that's lovely.  Thanks for the update, Kingsley!  >>> Uniprot is a good >>> >

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

s who work with them: Is there someone you'd >> > recommend chatting with at uniprot? >> > "scaling alongside uniprot" or at least engaging them on how to solve >> > shared + comparable issues (they also offer authentication-free SPARQL >> > querying) sounds li

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

atting with at uniprot? >> > "scaling alongside uniprot" or at least engaging them on how to >> solve >> > shared + comparable issues (they also offer authentication-free >> SPARQL >> > querying) sounds like a compelling option. >

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

> mailto:wikidata@lists.wikimedia.org>> > wrote: > > > > On 8/18/21 5:07 PM, Mike Pham wrote: > >> > >> Wikidata community members, > >> > >> > >> Thank you for all of your work helping Wikidata grow and improve >

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

ing option. > > > > S. > > > > On Thu, Aug 19, 2021 at 4:32 PM Kingsley Idehen via Wikidata > > mailto:wikidata@lists.wikimedia.org>> > wrote: > > > > On 8/18/21 5:07 PM, Mike Pham wrote: > >> > >> Wikidata community mem

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

helping Wikidata grow and improve over the years. In the spirit of better communication, we would like to take this opportunity to share some of the current challenges Wikidata Query Service (WDQS) is facing, and some strategies we have for dealing with them. WDQS current

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

> > Thank you for all of your work helping Wikidata grow and improve over the > years. In the spirit of better communication, we would like to take this > opportunity to share some of the current challenges Wikidata Query Service > (WDQS) is facing, and some strategies we have for d

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

On 8/18/21 5:07 PM, Mike Pham wrote: > > Wikidata community members, > > > Thank you for all of your work helping Wikidata grow and improve over > the years. In the spirit of better communication, we would like to > take this opportunity to share some of the current challe

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

Hey everyone, On Thu, Aug 19, 2021 at 9:42 PM Ed Summers wrote: > First, I just wanted to say it is *awesome* to see this level of > transparency and clarity about the state of the service. +1. Really glad that we're hearing about this now, before the problem becomes so seriously that measures ar

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

Marko, First, I just wanted to say it is *awesome* to see this level of transparency and clarity about the state of the service. Maybe this is over simplifying things but is it accurate to say that there are two orthogonal problems here? 1. The underlying technology (BlazeGraph) is end of life a

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

; Wikidata community members, > > > Thank you for all of your work helping Wikidata grow and improve > over the years. In the spirit of better communication, we would like > to take this opportunity to share some of the current challenges > Wikidata Query Service (WDQS) is facing, and s

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

Hi Marco, The problem with LDF is that not only the compute is shifted to the client, but in order to do the compute... the data that is to be computed must sometimes be transferred...and sometimes for even a simple query, the data has to be transferred and can be substantial. It's not always sub

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

d like to take this opportunity to share some of the current challenges Wikidata Query Service (WDQS) is facing, and some strategies we have for dealing with them. WDQS currently risks failing to provide acceptable service quality due to the following reasons:

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

Hi Mike, Is Blazegraph instances running within Java 9+ JVM ? I assume they are configured using G1GC garbage collector? Did you also try enabling the -XX+UseStringDeduplication to see if it can at least help a little and reduce some of the heap overhead on long lived strings? https://github.com/

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

data grow and improve over the > years. In the spirit of better communication, we would like to take this > opportunity to share some of the current challenges Wikidata Query Service > (WDQS) is facing, and some strategies we have for dealing with them. > > WDQS currently risks f

[Wikidata] Re: Wikidata Query Service scaling update Aug 2021

communication, we would like to take this opportunity to share some of the current challenges Wikidata Query Service (WDQS) is facing, and some strategies we have for dealing with them. WDQS currently risks failing to provide acceptable service quality due to the following reasons: 1. Blaz

[Wikidata] Wikidata Query Service scaling update Aug 2021

Wikidata community members, Thank you for all of your work helping Wikidata grow and improve over the years. In the spirit of better communication, we would like to take this opportunity to share some of the current challenges Wikidata Query Service (WDQS) is facing, and some strategies we have

[Wikidata] Wikidata Query Service (WDQS) User Survey 2021

Hi all, In the 2021–2022 fiscal year, the WMF Search team is working on scaling up Wikidata Query Service (WDQS) to handle increasing graph size and queries – we will be following up shortly with more updates regarding this. In order to design a querying service that works for most use cases, it

[Wikidata] Wikidata Query Service status update

Hello all! Here is a summary of what the Search Platform team is doing around WDQS: * The database responsible for unit conversions [7] has been updated on Friday Jan 29. It means that entities served from WDQS and updated since this date will use the new conversion data for normalized quantities

Re: [Wikidata] Does the Wikidata Query Service provide a REST API?

:29:14AM Subject: Re: [Wikidata] Does the Wikidata Query Service provide a REST API? Docs please? Can someone point me to the Wikidata docs on this? I'd imagine it's somewhere buried under subpages from [1]https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/Wikidata_Query_Help [

Re: [Wikidata] Does the Wikidata Query Service provide a REST API?

JavaScript ones) don’t use any special SPARQL libraries. You can also see >>> the “URL” option, which is just the URL you’re looking for. The returned >>> format depends on the Accept request header (or “format” query parameter), >>> so if you add a request header “Accept: t

Re: [Wikidata] Does the Wikidata Query Service provide a REST API?

o if you add a request header “Accept: text/csv”, you’ll get CSV back. >> >> Cheers, >> Lucas >> On 27.01.21 15:49, Yaron Koren wrote: >> >> Hi, >> >> After you run a SPARQL query in the Wikidata Query Service ( >> https://query.wikidata.org), the i

Re: [Wikidata] Does the Wikidata Query Service provide a REST API?

; > Hi, > > After you run a SPARQL query in the Wikidata Query Service ( > https://query.wikidata.org), the interface provides a lot of options - > you can download the results in formats like CSV and JSON, and it even > provides code to let you run that query directly in a variety

Re: [Wikidata] Does the Wikidata Query Service provide a REST API?

format depends on the Accept request header (or “format” query parameter), so if you add a request header “Accept: text/csv”, you’ll get CSV back. Cheers, Lucas On 27.01.21 15:49, Yaron Koren wrote: > Hi, > > After you run a SPARQL query in the Wikidata Query Service > (https://query.

Re: [Wikidata] Does the Wikidata Query Service provide a REST API?

ill return json with the response. Usage is quite simple - SPARQL query is a "query" param in a GET request to "https://query.wikidata.org/sparql"; endpoint. Hope that helps! Regards, Zbyszko On Wed, Jan 27, 2021 at 3:50 PM Yaron Koren wrote: > Hi, > > After you r

[Wikidata] Does the Wikidata Query Service provide a REST API?

Hi, After you run a SPARQL query in the Wikidata Query Service ( https://query.wikidata.org), the interface provides a lot of options - you can download the results in formats like CSV and JSON, and it even provides code to let you run that query directly in a variety of different computer

Re: [Wikidata] Wikidata Query Service status update

> https://www.linkedin.com/in/thadguidry/ > > > On Mon, Oct 12, 2020 at 8:59 AM Guillaume Lederrey < > gleder...@wikimedia.org> wrote: > >> Hello all! >> >> Here are a few updates from Wikidata Query Service: >> >> * We are getting close to full

Re: [Wikidata] Wikidata Query Service status update

along! Thad https://www.linkedin.com/in/thadguidry/ On Mon, Oct 12, 2020 at 8:59 AM Guillaume Lederrey wrote: > Hello all! > > Here are a few updates from Wikidata Query Service: > > * We are getting close to full functional coverage of our Flink based > Streaming Updater [1

[Wikidata] Wikidata Query Service status update

Hello all! Here are a few updates from Wikidata Query Service: * We are getting close to full functional coverage of our Flink based Streaming Updater [1]. We are starting to work on productionizing it and having a deployment strategy. The current goal is deploy on top of Kubernetes. * We&#x

Re: [Wikidata] Status of Wikidata Query Service

Le lun. 10 févr. 2020 à 17:11, Amirouche Boubekki a écrit : > > Hello Guillaume, > > Le ven. 7 févr. 2020 à 14:33, Guillaume Lederrey > a écrit : > > > > Hello all! > > > > First of all, my apologies for the long silence. We need to do better in > > terms of communication. I'll try my best to se

Re: [Wikidata] Status of Wikidata Query Service

Oh, wow, I just tried that out too, and indeed, it used to be possible to link to the L-number very quickly if I remember correctly, but now this is not the case anymore. Weirdly enough, the SPARQL endpoint got updated with the new Lexeme very quickly. So I think these two things are not related.

Re: [Wikidata] Status of Wikidata Query Service

I am sorry to bring more problems to the table, but the indexing of lexemes in the "ordinary" elasticsearch-based search is now also often slow. The Q-items are also indexed slowly, but there you can at least type in the Q-number in the edit field and it will lookup the item. For L-numbers, I

Re: [Wikidata] Status of Wikidata Query Service

On Mon, 10 Feb 2020 at 16:11, Amirouche Boubekki wrote: > Who needs to see the updates live in WDQS as soon as edits > are done in wikidata? https://www.wikidata.org/w/index.php?title=Wikidata:Contact_the_development_team/Query_Service_and_search&diff=prev&oldid=1110094246 -- Andy Mabbett

Re: [Wikidata] Status of Wikidata Query Service

Hi, I have limited knowledge of SPARQL and RDF so I might not have understood fully. On Tue, 11 Feb 2020 at 07:05, Samuel Klein wrote: > A linked data fragments approach w/ a [thick client] seems to me to be one > useful option to explore and benchmark. > > As far as I can see WDQS does offer a

Re: [Wikidata] Status of Wikidata Query Service

Hoi, I find it interesting that some do not need to see the result of what they do. What it tells me is that they deal with collections, big amounts of data that like stamp collections are dumped in Wikidata. What is the point of such? I consider them prime examples of what can be set aside. I do

Re: [Wikidata] Status of Wikidata Query Service

The earth is not flat :) I appreciate all of your thoughts in this thread, Amirouche. A linked data fragments approach w/ a [thick client] seems to me to be one useful option to explore and benchmark. Other possible thoughts: ~ have some core highly-used subgraphs within which queries are lightni

Re: [Wikidata] Status of Wikidata Query Service

Hello Amirouche, Regarding "most cell phone plans" being unlimited, here in the United States there are many phone plans which are not unlimited. I don't know what the proportion of unlimited to limited users are. My understanding is that Twitter charges money for the use of their API under some

Re: [Wikidata] Status of Wikidata Query Service

Le lun. 10 févr. 2020 à 18:23, Marco Neumann a écrit : > > why all the sad faces? > the Semantic Web will be distributed after all The semantic Web is already distributed. > and there is no need to stuff everything into one graph. Everything into one graph, or if you prefer in one place, is th

Re: [Wikidata] Status of Wikidata Query Service

why all the sad faces? the Semantic Web will be distributed after all and there is no need to stuff everything into one graph. it just requires us as a RDF community to spend more time developing ideas around efficient query distribution and focus on relationships and links in wikidata rather than

Re: [Wikidata] Status of Wikidata Query Service

I accept your apology Guillaume, no worries. Regards, Marco On Mon, Feb 10, 2020 at 2:37 PM Guillaume Lederrey wrote: > On Fri, Feb 7, 2020 at 5:18 PM Guillaume Lederrey > wrote: > >> On Fri, Feb 7, 2020 at 2:54 PM Marco Neumann >> wrote: >> >>> thank you Guillaume, when do you expect a publi

Re: [Wikidata] Status of Wikidata Query Service

On Tue, Feb 11, 2020, 12:11 AM Amirouche Boubekki, < amirouche.boube...@gmail.com> wrote: > > Again, we're not sure. Of course, reducing the load (both in terms of > edits on Wikidata and of reads on WDQS) will help. But not using those > services makes them useless. > > What about making the lag

Re: [Wikidata] Status of Wikidata Query Service

Hello Guillaume, Le ven. 7 févr. 2020 à 14:33, Guillaume Lederrey a écrit : > > Hello all! > > First of all, my apologies for the long silence. We need to do better in > terms of communication. I'll try my best to send a monthly update from now > on. Keep me honest, remind me if I fail. > It w

Re: [Wikidata] Status of Wikidata Query Service

Hi, if you need some "Wikibase item diff" function, have a look at the Rust crate I am co-authoring: https://gitlab.com/tobias47n9e/wikibase_rs It comes with diff code: https://gitlab.com/tobias47n9e/wikibase_rs/-/blob/master/src/entity_diff.rs Should not be too hard to build eg a simple diff co

Re: [Wikidata] Status of Wikidata Query Service

Le ven. 7 févr. 2020 à 19:01, Pine W a écrit : > > I don't know if this is helpful, as I'm not very familiar with > Wikidata's infrastructure, but I think that an idea that was discussed > in the Wikimedia Strategy 2030 process is charging real money to > organizations that consume large amounts o

Re: [Wikidata] Status of Wikidata Query Service

Indeed, having WikiData Query Service (WDQS) responsibility under a separate sub-organization as known as the 'Search Platform' is mischief. Le lun. 10 févr. 2020 à 15:55, Gerard Meijssen a écrit : > > Hoi, > In my opinion, Wikidata is a flagship project of the Wikimedia Foun

Re: [Wikidata] Status of Wikidata Query Service

Hoi, In my opinion, Wikidata is a flagship project of the Wikimedia Foundation. With the current underperformance vis a vis demand, it is not only a technical question what to do, it is also an organisational issue. As I have written in several blogposts, the current underperformance affects the

Re: [Wikidata] Status of Wikidata Query Service

On Fri, Feb 7, 2020 at 5:18 PM Guillaume Lederrey wrote: > On Fri, Feb 7, 2020 at 2:54 PM Marco Neumann > wrote: > >> thank you Guillaume, when do you expect a public update on the security >> incident [1]? Is any of our personal and private data (email, password etc) >> affected? >> > > It shou

Re: [Wikidata] Status of Wikidata Query Service

I don't know if this is helpful, as I'm not very familiar with Wikidata's infrastructure, but I think that an idea that was discussed in the Wikimedia Strategy 2030 process is charging real money to organizations that consume large amounts of data from the Wikimedia API. By extension, an idea to co

Re: [Wikidata] Status of Wikidata Query Service

On 07.02.20 14:32, Guillaume Lederrey wrote: Keeping all of Wikidata in a single graph is most probably not going to work long term. We have not found examples of public SPARQL endpoints with > 10 B triples and there is probably a good reason for that. We will probably need to split the graphs

Re: [Wikidata] Status of Wikidata Query Service

On Fri, Feb 7, 2020 at 3:12 PM wrote: > > Better update granularity is probably good and may be a good priority. > > It is (still) unclear for me as a tool writer whether I can do anything. > For instance it is not clear to me whether the parallel SPARQL queries > that comes when a user visit a S

Re: [Wikidata] Status of Wikidata Query Service

On Fri, Feb 7, 2020 at 2:54 PM Marco Neumann wrote: > thank you Guillaume, when do you expect a public update on the security > incident [1]? Is any of our personal and private data (email, password etc) > affected? > It should be made public in the next few days. I'm not going to go into any mo

Re: [Wikidata] Status of Wikidata Query Service

Better update granularity is probably good and may be a good priority. It is (still) unclear for me as a tool writer whether I can do anything. For instance it is not clear to me whether the parallel SPARQL queries that comes when a user visit a Scholia page is important for the load on WDQS

Re: [Wikidata] Status of Wikidata Query Service

thank you Guillaume, when do you expect a public update on the security incident [1]? Is any of our personal and private data (email, password etc) affected? best, Marco [1] https://phabricator.wikimedia.org/T241410 On Fri, Feb 7, 2020 at 1:33 PM Guillaume Lederrey wrote: > Hello all! > > Firs

[Wikidata] Status of Wikidata Query Service

Hello all! First of all, my apologies for the long silence. We need to do better in terms of communication. I'll try my best to send a monthly update from now on. Keep me honest, remind me if I fail. First, we had a security incident at the end of December, which forced us to move from our Kafka

Re: [Wikidata] Wikidata Query Service update lag

>>>>> either splitting the processing over multiple CPU or sharding the data >>>>>>> over >>>>>>> multiple servers. Neither of which Blazegraph supports (at least not in >>>>>>> our >>>>>>> current c

Re: [Wikidata] Wikidata Query Service update lag

on. >>>>>> >>>>>> What you can do (short term): >>>>>> >>>>>> * Keep bots usage well behaved (don't do parallel queries, provide a >>>>>> meaningful user agent, smooth the load over time if possible, ...)

Re: [Wikidata] Wikidata Query Service update lag

>>>>> * Optimize your queries: better queries will use less resources, which >>>>> should help. Time to completion is a good approximation of the resources >>>>> used. I don't really have any more specific advice, SPARQL is not my area >>>>> of expertise. &

Re: [Wikidata] Wikidata Query Service update lag

gt;>> of expertise. >>>> >>>> What you can do (longer term): >>>> >>>> * Help us think out of the box. Can we identify higher level use cases? >>>> Could we implement some of our workflows on a higher level API than SPARQL, >>>&

Re: [Wikidata] Wikidata Query Service update lag

;> Sadly, we don't have the bandwidth right now to engage meaningfully in >>> this conversation. Feel free to send thoughts already, but don't expect any >>> timely response. >>> >>> Yet another thought is the large discrepancy between Virginia and Tex

Re: [Wikidata] Wikidata Query Service update lag

nt in the parallelism of the updater [6]. This has >>> just >>> > been identified. While it will probably also provide some improvement >>> in >>> > throughput, we haven't actually started working on that and we don't >>> > have any numbers

Re: [Wikidata] Query Service break down on FactGrId

As always: thank you Lucas! - everything is back to normal. (I will need a private lesson sooner or later...) Olaf > Lucas Werkmeister hat am 14. November 2019 um > 18:26 geschrieben: > > > I’ve restarted Blazegraph, it looks like it ran out of memory. Not sure why. > > Cheers, > Lucas >

Re: [Wikidata] Query Service break down on FactGrId

I’ve restarted Blazegraph, it looks like it ran out of memory. Not sure why. Cheers, Lucas On 14.11.19 18:19, Olaf Simons wrote: > Dear list, > > I am experiencing unexpected problems with running Queries on FactGrid. (You > find some sample queries here: > https://database.factgrid.de/wiki/Fac

[Wikidata] Query Service break down on FactGrId

Dear list, I am experiencing unexpected problems with running Queries on FactGrid. (You find some sample queries here: https://database.factgrid.de/wiki/FactGrid:Projects) The error code is: Can anyone make more sense of the information and how to handle it? 502 Proxy Error Proxy Error The

Re: [Wikidata] Wikidata Query Service update lag

; difference? Rather than editing or BlazeGraph, could the issue be some >> form of network issue? >> > > As pointed out by Lucas, this is expected. Due to how our GeoDNS works, we > see more traffic on eqiad than on codfw. > > Thanks for the help! > >Guillaume

Re: [Wikidata] Wikidata Query Service update lag

tps://wikitech.wikimedia.org/wiki/Wikidata_query_service/Usage > > > [1] > > https://grafana.wikimedia.org/d/00489/wikidata-query-service?panelId=8&fullscreen&orgId=1&from=now-7d&to=now > > /Finn > > > > On 14/11/2019 10:50, Guillaume Lederrey wrote: > >

Re: [Wikidata] Wikidata Query Service update lag

s far as I > understand the hardware (and software) are the same. So why is there > this large difference? Rather than editing or BlazeGraph, could the > issue be some form of network issue? > > > [1] > https://grafana.wikimedia.org/d/00489/wikidata-query-service?panelId=8&am

Re: [Wikidata] Wikidata Query Service update lag

00489/wikidata-query-service?panelId=8&fullscreen&orgId=1&from=now-7d&to=now /Finn On 14/11/2019 10:50, Guillaume Lederrey wrote: Hello all! As you've probably noticed, the update lag on the public WDQS endpoint [1] is not doing well [2], with lag climbing to > 12h for s

[Wikidata] Wikidata Query Service update lag

[1] https://query.wikidata.org/ [2] https://grafana.wikimedia.org/d/00489/wikidata-query-service?orgId=1&from=1571131796906&to=1573723796906&var-cluster_name=wdqs&panelId=8&fullscreen [3] https://phabricator.wikimedia.org/T238229 [4] https://blazegraph.com/ [5

  1   2   3   4   >