[Wikidata-bugs] [Maniphest] [Commented On] T202785: Federation request to https://ld.stadt-zuerich.ch/query fails
Mchlrch added a comment. I was able to reproduce the issue outside the WDQR codebase [1], using the blazegraph bigdata-client library. Unfortunately, I can only confirm what Stas already suspected and commented earlier: There is no easy fix. What I conclude, is that current WDQR does not support federation to endpoints that also provide HTTP/2 over TLS, because HTTP/2 over TLS requires the use of ALPN. The blazegraph bigdata-client library used in WDQR uses jetty version 9.2.3.v20140905, which predates HTTP/2 standardization from 2015 and therefore lacks ALPN support. WDQR federation however does work with endpoints that also provide HTTP/2 but without TLS - because negotation works without ALPN in that case. Looking forward, it would seem reasonable to aim at including support for HTTP/2 over TLS in WDQR. Jetty apparently supports HTTP/2 starting with version 9.3 [2], so maybe upgrading blazegraph to jetty 9.3 is doable. On the other hand, I noticed that the blazegraph codebase has not seen any activity in the last two years, which might also be a motivation for Wikidata to move on to another datastore. To resolve on the endpoint side would apparently require to completely drop support for HTTP/2 over TLS. Even though theoretically possible, I personally don't consider that an option, because it's just too regressive. Probably just a bad idea, but I mention it anyway: Another option (pretty far fetched and adventurous) might be to have a MITMing proxy inside Wikidata infrastructure, that terminates outbound HTTP/2-TLS on one side and exposes HTTP/1.1-TLS to WDQR. Also I have no clue if there are products, that actually support such behavior. 1: https://github.com/mchlrch/wdqr-ssz-tc/tree/master/wdqr-ssz-tc 2: http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/VERSION.txtTASK DETAILhttps://phabricator.wikimedia.org/T202785EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, MchlrchCc: Mchlrch, Gehel, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T202785: Federation request to https://ld.stadt-zuerich.ch/query fails
Mchlrch added a comment. I understand your concerns that bringing in ALPN would most likely cause problems, if it even would be possible at all. I suggest to explore if we can find a resolution without bringing in new dependencies. I suggest the following procedure: isolate the code that creates the HttpClient in order to make the issue reproducible outside the Query-Service codebase, in order to confirm that we are really searching in the right spot explore if the setup of the HttpClient could be modified in order to make the request work in case of success, apply the necessary changes to the Query-Service codebase I could help with step 1) and 2). I cloned WDQR [1] and had a look at the source, but I'm not even sure, that this is the right repository to look at. If it is, then it looks like the HttpClient is created in either of the following two places: module blazegraph: org.wikidata.query.rdf.blazegraph.ProxiedHttpConnectionFactory, line 40 module tools: org.wikidata.query.rdf.tool.HttpClientUtils, line 77 Do you know which of the two places is the right one to investigate? Or could you give me a pointer to the right source, in case I'm searching in the wrong spot. According to the parent POM, the Query-Service runs with Java 8, right?TASK DETAILhttps://phabricator.wikimedia.org/T202785EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, MchlrchCc: Mchlrch, Gehel, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T202785: Federation request to https://ld.stadt-zuerich.ch/query fails
Mchlrch added a comment. Allow me to reformulate: How about making the jetty-client ask explicitly for HTTP/1.1?TASK DETAILhttps://phabricator.wikimedia.org/T202785EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, MchlrchCc: Mchlrch, Gehel, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T202785: Federation request to https://ld.stadt-zuerich.ch/query fails
Mchlrch added a comment.
The endpoint https://ld.stadt-zuerich.ch/query supports HTTP/1.0, HTTP/1.1 and HTTP/2. I defaults to HTTP/2, if the client does request an explicit version.
@Smalyshev How about explicitly specifying HTTP/1.1 for the jetty-client?
Query:
$ cat foo.rq
PREFIX rdf:
PREFIX rdfs:
SELECT * WHERE {
GRAPH {
?sub ?pred ?obj .
}
}
LIMIT 1
curl for HTTP/1.0:
$ curl --http1.0 -i --data-urlencode qu...@foo.rq https://ld.stadt-zuerich.ch/query
HTTP/1.0 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public, max-age=120
Content-Type: application/sparql-results+xml
Date: Tue, 28 Aug 2018 11:54:40 GMT
Set-Cookie: _f3bc9=http://10.42.34.100:80; Path=/
Vary: Accept-Encoding
Vary: Accept, Accept-Encoding
Via: 1.1 varnish-v4
X-Varnish: 296846
Content-Length: 540
https://ld.stadt-zuerich.ch/.well-known/void
http://www.w3.org/1999/02/22-rdf-syntax-ns#type
http://rdfs.org/ns/void#DatasetDescription
curl for HTTP/1.1:
$ curl --http1.1 -i --data-urlencode qu...@foo.rq https://ld.stadt-zuerich.ch/query
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public, max-age=120
Content-Type: application/sparql-results+xml
Date: Tue, 28 Aug 2018 11:54:18 GMT
Set-Cookie: _f3bc9=http://10.42.34.100:80; Path=/
Vary: Accept-Encoding
Vary: Accept, Accept-Encoding
Via: 1.1 varnish-v4
X-Varnish: 1540376
Content-Length: 540
https://ld.stadt-zuerich.ch/.well-known/void
http://www.w3.org/1999/02/22-rdf-syntax-ns#type
http://rdfs.org/ns/void#DatasetDescription
TASK DETAILhttps://phabricator.wikimedia.org/T202785EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, MchlrchCc: Mchlrch, Gehel, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
