Re: [QGIS-Developer] mirroring qgis.org repos
Hi Richard,
nocache.qgis.org did the trick for us
thank you for your effort
Bernhard
Am 05.06.2018 um 12:22 schrieb Richard Duivenvoorde:
On 05-06-18 09:54, Richard Duivenvoorde wrote:
On 05-06-18 09:31, Jürgen E. Fischer wrote:
Hi Bernhard,
On Tue, 05. Jun 2018 at 08:29:09 +0200, Bernhard Ströbl wrote:
has anything changed recently? We have -again- since several days problems
mirroring ubuntugis locally:
debmirror --dist=xenial --section=main --arch=i386,amd64 --cleanup
--host=qgis.org --root=ubuntugis-ltr --method=https --rsync-extra=none
/mirrorpath
fails with 403 both via http and https
Odd. With wget and a browser I can get
https://qgis.org/ubuntugis-ltr/dists/xenial/Release.
But I can reproduce with debmirror. I works when when I use nocache.qgis.org
instead of qgis.org.
Looks like a cloudflare issue.
Using myself here (adding -v -debug).
Did some more testing, and it is definitly a cloudflare issue.
If I disable caching on qgis.org temporarily it works.
But if I use a silly (googled) perl script which mimics the perl lines
from debmirror:
# Create a user agent object
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $res = $ua->mirror(
'https://anonym...@qgis.org/ubuntugis-ltr/dists/xenial/Release',
'/tmp/Release' );
# Check the outcome of the response
if ($res->is_success) {
print $res->content;
}
else {
print $res->status_line, "\n";
}
it also(!) just works...
Tried to set the ssl_verification to false, other headers, but I could
not make debmirror work.
We cannot remove cache from qgis.org (as we generate too much web
traffic then, and our services really become slower)
You could use https://nocache.qgis.org ?
I prefer to keep ubuntu.qgis.org cached too, unless others have problems
with it too..
Sorry I cannot be of more help.
Regards,
Richard
__ Information from ESET Mail Security, version of virus signature
database 17500 (20180605) __
The message was checked by ESET Mail Security.
http://www.eset.com
__ Information from ESET Mail Security, version of virus signature
database 17501 (20180605) __
The message was checked by ESET Mail Security.
http://www.eset.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
On 05-06-18 09:54, Richard Duivenvoorde wrote:
> On 05-06-18 09:31, Jürgen E. Fischer wrote:
>> Hi Bernhard,
>>
>> On Tue, 05. Jun 2018 at 08:29:09 +0200, Bernhard Ströbl wrote:
>>> has anything changed recently? We have -again- since several days problems
>>> mirroring ubuntugis locally:
>>>
>>> debmirror --dist=xenial --section=main --arch=i386,amd64 --cleanup
>>> --host=qgis.org --root=ubuntugis-ltr --method=https --rsync-extra=none
>>> /mirrorpath
>>> fails with 403 both via http and https
>>
>> Odd. With wget and a browser I can get
>> https://qgis.org/ubuntugis-ltr/dists/xenial/Release.
>>
>> But I can reproduce with debmirror. I works when when I use
>> nocache.qgis.org
>> instead of qgis.org.
>>
>> Looks like a cloudflare issue.
>
> Using myself here (adding -v -debug).
Did some more testing, and it is definitly a cloudflare issue.
If I disable caching on qgis.org temporarily it works.
But if I use a silly (googled) perl script which mimics the perl lines
from debmirror:
# Create a user agent object
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $res = $ua->mirror(
'https://anonym...@qgis.org/ubuntugis-ltr/dists/xenial/Release',
'/tmp/Release' );
# Check the outcome of the response
if ($res->is_success) {
print $res->content;
}
else {
print $res->status_line, "\n";
}
it also(!) just works...
Tried to set the ssl_verification to false, other headers, but I could
not make debmirror work.
We cannot remove cache from qgis.org (as we generate too much web
traffic then, and our services really become slower)
You could use https://nocache.qgis.org ?
I prefer to keep ubuntu.qgis.org cached too, unless others have problems
with it too..
Sorry I cannot be of more help.
Regards,
Richard
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
On 05-06-18 09:31, Jürgen E. Fischer wrote: > Hi Bernhard, > > On Tue, 05. Jun 2018 at 08:29:09 +0200, Bernhard Ströbl wrote: >> has anything changed recently? We have -again- since several days problems >> mirroring ubuntugis locally: >> >> debmirror --dist=xenial --section=main --arch=i386,amd64 --cleanup >> --host=qgis.org --root=ubuntugis-ltr --method=https --rsync-extra=none >> /mirrorpath >> fails with 403 both via http and https > > Odd. With wget and a browser I can get > https://qgis.org/ubuntugis-ltr/dists/xenial/Release. > > But I can reproduce with debmirror. I works when when I use nocache.qgis.org > instead of qgis.org. > > Looks like a cloudflare issue. Using myself here (adding -v -debug). Using nocache.qgis.org (which is NOT going through cloudflare), I get another problem, but that is maybe because nocache.qgis.org is not known in keys/certificates or so? # debmirror -v -debug --dist=xenial --section=main --arch=i386,amd64 --cleanup --host=nocache.qgis.org --root=ubuntugis-ltr --method=https --rsync-extra=none /mirrorpath Warning: --rsync-extra is not configured to mirror the trace files. This configuration is not recommended. Mirroring to /mirrorpath from https://anonym...@nocache.qgis.org/ubuntugis-ltr/ Arches: i386,amd64 Dists: xenial Sections: main Including source. Pdiff mode: use Will clean up before mirroring. Attempting to get lock ... Getting meta files ... https://nocache.qgis.org/ubuntugis-ltr/dists/xenial/Release => [ 0%] Getting: dists/xenial/Release... 200 OK ok https://nocache.qgis.org/ubuntugis-ltr/dists/xenial/InRelease => [ 0%] Getting: dists/xenial/InRelease... 200 OK ok https://nocache.qgis.org/ubuntugis-ltr/dists/xenial/Release.gpg => [ 0%] Getting: dists/xenial/Release.gpg... 200 OK ok [GNUPG:] ERROR add_keyblock_resource 33554433 [GNUPG:] NEWSIG [GNUPG:] ERROR keydb_search 33554445 [GNUPG:] ERRSIG CAEB3DC3BDF7FB45 1 10 00 1526737603 9 - [GNUPG:] NO_PUBKEY CAEB3DC3BDF7FB45 gpgv: unknown type of key resource 'trustedkeys.kbx' gpgv: keyblock resource '/root/.gnupg/trustedkeys.kbx': General error gpgv: Signature made Sat 19 May 2018 03:46:43 PM CEST gpgv:using RSA key CAEB3DC3BDF7FB45 gpgv: Can't check signature: No public key .temp/.tmp/dists/xenial/Release.gpg signature does not verify. [GNUPG:] ERROR add_keyblock_resource 33554433 [GNUPG:] NEWSIG [GNUPG:] ERROR keydb_search 33554445 [GNUPG:] ERRSIG CAEB3DC3BDF7FB45 1 10 01 1526737603 9 - [GNUPG:] NO_PUBKEY CAEB3DC3BDF7FB45 gpgv: unknown type of key resource 'trustedkeys.kbx' gpgv: keyblock resource '/root/.gnupg/trustedkeys.kbx': General error gpgv: Signature made Sat 19 May 2018 03:46:43 PM CEST gpgv:using RSA key CAEB3DC3BDF7FB45 gpgv: Can't check signature: No public key .temp/.tmp/dists/xenial/InRelease signature does not verify. /usr/bin/debmirror: Created directory: dists/ /usr/bin/debmirror: Created directory: dists/xenial/ /usr/bin/debmirror: Created directory: .temp/dists/ /usr/bin/debmirror: Created directory: .temp/dists/xenial/ Errors: .temp/.tmp/dists/xenial/Release.gpg signature does not verify .temp/.tmp/dists/xenial/InRelease signature does not verify Failed to download some Release, Release.gpg or InRelease files! WARNING: releasing 1 pending lock... ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
Hi Bernhard, On Tue, 05. Jun 2018 at 08:29:09 +0200, Bernhard Ströbl wrote: > has anything changed recently? We have -again- since several days problems > mirroring ubuntugis locally: > > debmirror --dist=xenial --section=main --arch=i386,amd64 --cleanup > --host=qgis.org --root=ubuntugis-ltr --method=https --rsync-extra=none > /mirrorpath > fails with 403 both via http and https Odd. With wget and a browser I can get https://qgis.org/ubuntugis-ltr/dists/xenial/Release. But I can reproduce with debmirror. I works when when I use nocache.qgis.org instead of qgis.org. Looks like a cloudflare issue. Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de QGIS release manager (PSC) GermanyIRC: jef on FreeNode signature.asc Description: PGP signature ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
Hi Richard, hi group, has anything changed recently? We have -again- since several days problems mirroring ubuntugis locally: debmirror --dist=xenial --section=main --arch=i386,amd64 --cleanup --host=qgis.org --root=ubuntugis-ltr --method=https --rsync-extra=none /mirrorpath fails with 403 both via http and https Bernhard __ Information from ESET Mail Security, version of virus signature database 17500 (20180605) __ The message was checked by ESET Mail Security. http://www.eset.com ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
Hi Richard, On Wed, 03. Jan 2018 at 10:39:16 +0100, Richard Duivenvoorde wrote: > FYI: because apparently ubuntu.qgis.org is also used to request > version.txt Not anymore since c67e7f4c3. Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de QGIS release manager (PSC) GermanyIRC: jef on FreeNode signature.asc Description: PGP signature ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
On 02-01-18 16:09, Jürgen E. Fischer wrote: > Hi Bernhard, > > On Tue, 02. Jan 2018 at 09:20:07 +0100, Bernhard Ströbl wrote: >> We have -again- a problem mirroring the QGIS repos locally, see [1]. The >> ubuntu subdomain is not accessible anymore and going directly to qgis.org >> results in the same error as two years ago. > > qgis.org is the domain. ubuntu.qgis.org was just one of the catch alls. FYI: because apparently ubuntu.qgis.org is also used to request version.txt I just added ubuntu.qgis.org as alt name in the qgis.org certificate: https://ubuntu.qgis.org/ubuntugis/ Richard ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
Ok, (more) clear :-) Easiest (for me now) is if you could use this url: https://qgis.org/ubuntugis/ The (letsencrypt) certificate there is valid for qgis.org (going via cache) version.qgis.org (NOT going via cache) and download.qgis.org (not even a DNS entry) So IF https://qgis.org/ubuntugis/ gives cloudflare/certificate problems, you could use https://version.qgis.org/ubuntugis/ Note that 144.76.174.102 will be the ip address for the upcoming future. It is not that we will change back to the other one. If all this is not working, I could add ubuntu.qgis.org to the certificate too (and probably debian.qgis.org then too???) Let me know what you preference is. Regards, Richard On 02-01-18 14:58, Bernhard Ströbl wrote: > Hi Richard, > > thank you for your answer. Our goal is the same as two years ago: We > want to mirror the repo locally, i.e. we want to see all the files. > > This command currently works: > debmirror --dist=xenial --section=main --arch=i386,amd64 --cleanup > --host=144.76.174.102 --root=ubuntugis-ltr --method=http --rsync-extra=none > > BUT: > 1) if the IP changes we will have to call again > 2) if http is shut off we cannot access via https because the SSL > certificate does not contain the address 144.76.174.102 (certificate > verification fails) > > regards > Bernhard > > > Am 02.01.2018 um 14:01 schrieb Richard Duivenvoorde: >> >> Hi Bernhard, >> >> Can you provide some more information on what you exactly want to see? >> >> http://ubuntu.qgis.org ? >> >> That one was still pointing to our old server (which since... I think 3 >> weeks does not have a webserver running anymore). >> >> I let it point to our new qgis2 server now (144.76.174.102). >> (note that this one is NOT being cached by cloudflare, so I doubt it is >> 'the same problem' as earlier) >> >> And I see now: >> http://ubuntu.qgis.org/ubuntugis/ >> which is a repo. >> >> Fixed like this? >> >> Sorry for the inconvenience. >> >> Regards, >> >> Richard Duivenvoorde >> >> >> On 02-01-18 09:20, Bernhard Ströbl wrote: >>> >>> Hi, >>> I wish everybody a good start in 2018! >>> >>> We have -again- a problem mirroring the QGIS repos locally, see [1]. The >>> ubuntu subdomain is not accessible anymore and going directly to >>> qgis.org results in the same error as two years ago. >>> >>> best regards >>> >>> Bernhard >>> >>> [1] >>> http://lists.osgeo.org/pipermail/qgis-developer/2016-January/041107.html >>> >>> >>> __ Information from ESET Mail Security, version of virus >>> signature database 16667 (20180102) __ >>> >>> The message was checked by ESET Mail Security. >>> http://www.eset.com >>> >>> >>> ___ >>> QGIS-Developer mailing list >>> QGIS-Developer@lists.osgeo.org >>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer >>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer >> >> >> >> __ Information from ESET Mail Security, version of virus >> signature database 16669 (20180102) __ >> >> The message was checked by ESET Mail Security. >> http://www.eset.com >> >> > > ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
Hi Bernhard, On Tue, 02. Jan 2018 at 09:20:07 +0100, Bernhard Ströbl wrote: > We have -again- a problem mirroring the QGIS repos locally, see [1]. The > ubuntu subdomain is not accessible anymore and going directly to qgis.org > results in the same error as two years ago. qgis.org is the domain. ubuntu.qgis.org was just one of the catch alls. Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de QGIS release manager (PSC) GermanyIRC: jef on FreeNode signature.asc Description: PGP signature ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
Hi Richard, thank you for your answer. Our goal is the same as two years ago: We want to mirror the repo locally, i.e. we want to see all the files. This command currently works: debmirror --dist=xenial --section=main --arch=i386,amd64 --cleanup --host=144.76.174.102 --root=ubuntugis-ltr --method=http --rsync-extra=none BUT: 1) if the IP changes we will have to call again 2) if http is shut off we cannot access via https because the SSL certificate does not contain the address 144.76.174.102 (certificate verification fails) regards Bernhard Am 02.01.2018 um 14:01 schrieb Richard Duivenvoorde: Hi Bernhard, Can you provide some more information on what you exactly want to see? http://ubuntu.qgis.org ? That one was still pointing to our old server (which since... I think 3 weeks does not have a webserver running anymore). I let it point to our new qgis2 server now (144.76.174.102). (note that this one is NOT being cached by cloudflare, so I doubt it is 'the same problem' as earlier) And I see now: http://ubuntu.qgis.org/ubuntugis/ which is a repo. Fixed like this? Sorry for the inconvenience. Regards, Richard Duivenvoorde On 02-01-18 09:20, Bernhard Ströbl wrote: Hi, I wish everybody a good start in 2018! We have -again- a problem mirroring the QGIS repos locally, see [1]. The ubuntu subdomain is not accessible anymore and going directly to qgis.org results in the same error as two years ago. best regards Bernhard [1] http://lists.osgeo.org/pipermail/qgis-developer/2016-January/041107.html __ Information from ESET Mail Security, version of virus signature database 16667 (20180102) __ The message was checked by ESET Mail Security. http://www.eset.com ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer __ Information from ESET Mail Security, version of virus signature database 16669 (20180102) __ The message was checked by ESET Mail Security. http://www.eset.com -- Bernhard Ströbl Anwendungsbetreuer GIS Kommunale Immobilien Jena Am Anger 26 07743 Jena Tel.: 03641 49- 5190 E-Mail: bernhard.stro...@jena.de Internet: www.kij.de Kommunale Immobilien Jena Eigenbetrieb der Stadt Jena Werkleiter: Karl-Hermann Kliewe __ Information from ESET Mail Security, version of virus signature database 16669 (20180102) __ The message was checked by ESET Mail Security. http://www.eset.com ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] mirroring qgis.org repos
Hi Bernhard, Can you provide some more information on what you exactly want to see? http://ubuntu.qgis.org ? That one was still pointing to our old server (which since... I think 3 weeks does not have a webserver running anymore). I let it point to our new qgis2 server now (144.76.174.102). (note that this one is NOT being cached by cloudflare, so I doubt it is 'the same problem' as earlier) And I see now: http://ubuntu.qgis.org/ubuntugis/ which is a repo. Fixed like this? Sorry for the inconvenience. Regards, Richard Duivenvoorde On 02-01-18 09:20, Bernhard Ströbl wrote: > > Hi, > I wish everybody a good start in 2018! > > We have -again- a problem mirroring the QGIS repos locally, see [1]. The > ubuntu subdomain is not accessible anymore and going directly to > qgis.org results in the same error as two years ago. > > best regards > > Bernhard > > [1] > http://lists.osgeo.org/pipermail/qgis-developer/2016-January/041107.html > > > __ Information from ESET Mail Security, version of virus > signature database 16667 (20180102) __ > > The message was checked by ESET Mail Security. > http://www.eset.com > > > ___ > QGIS-Developer mailing list > QGIS-Developer@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
[QGIS-Developer] mirroring qgis.org repos
Hi, I wish everybody a good start in 2018! We have -again- a problem mirroring the QGIS repos locally, see [1]. The ubuntu subdomain is not accessible anymore and going directly to qgis.org results in the same error as two years ago. best regards Bernhard [1] http://lists.osgeo.org/pipermail/qgis-developer/2016-January/041107.html __ Information from ESET Mail Security, version of virus signature database 16667 (20180102) __ The message was checked by ESET Mail Security. http://www.eset.com ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
