[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"
aborrero added a comment.
the VM is in error state:
This is what I get from the openstack API:
{'code': 500, 'created': '2021-12-10T19:19:35Z', 'message': 'OSError',
'details': 'Traceback (most recent call last):\n File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 205, in
decorated_function\nreturn function(self, context, *args, **kwargs)\n File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3743, in
reboot_instance\ndo_reboot_instance(context, instance, block_device_info,
reboot_type)\n File
"/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in
inner\nreturn f(*args, **kwargs)\n File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3742, in
do_reboot_instance\nreboot_type)\n File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3835, in
_reboot_instance\nself._set_instance_obj_error_state(instance)\n File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__\nself.force_reraise()\n File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise\nsix.reraise(self.type_, self.value, self.tb)\n File
"/usr/lib/python3/dist-packages/six.py", line 703, in reraise\nraise
value\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line
3810, in _reboot_instance\nbad_volumes_callback=bad_volumes_callback)\n
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3250,
in reboot\nblock_device_info, accel_info)\n File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3343, in
_hard_reboot\nmdevs=mdevs, accel_info=accel_info)\n File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6501, in
_get_guest_xml\ncontext, mdevs, accel_info)\n File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6146, in
_get_guest_config\nflavor, guest.os_type)\n File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4817, in
_get_guest_storage_config\ninst_type)\n File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4724, in
_get_guest_disk_config\nconf = disk.libvirt_info(disk_info,
self.disk_cachemode,\n File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 556, in
disk_cachemode\nif not
nova.privsep.utils.supports_direct_io(CONF.instances_path):\n File
"/usr/lib/python3/dist-packages/nova/privsep/utils.py", line 74, in
supports_direct_io\n{\'path\': dirpath, \'ex\': e})\n File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__\nself.force_reraise()\n File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise\nsix.reraise(self.type_, self.value, self.tb)\n File
"/usr/lib/python3/dist-packages/six.py", line 703, in reraise\nraise
value\n File "/usr/lib/python3/dist-packages/nova/privsep/utils.py", line 57,
in supports_direct_io\nfd = os.open(testfile, os.O_CREAT | os.O_WRONLY |
os.O_DIRECT)\n File "/usr/lib/python3/dist-packages/eventlet/green/os.py",
line 118, in open\nfd = __original_open__(file, flags, mode)\nOSError:
[Errno 28] No space left on device:
\'/var/lib/nova/instances/.directio.test.5199023175537772324\'\n'}
I don't know what happened. First time I see this error.
TASK DETAIL
https://phabricator.wikimedia.org/T297454
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: dcaro, aborrero
Cc: aborrero, GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD,
Alicia_Fagerving_WMSE, EBernhardson, Aklapper, Jarekt, Invadibot, MPhamWMF,
maantietaja, CBogen, Nintendofan885, Akuckartz, Nandana, JKSTNK, Namenlos314,
Lahi, Gq86, E1presidente, Ramsey-WMF, Cparle, Anoop, SandraF_WMF,
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer,
merbst, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS,
Jonas, Xmlizer, Susannaanas, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base,
matthiasmullie, aude, Tobias1984, Manybubbles, Ricordisamoa, Wesalius,
Lydia_Pintscher, Raymond, Steinsplitter, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
aborrero added a subtask: T273579: cloudvirt-wdqs1001 getting out of space due to huge VM. TASK DETAIL https://phabricator.wikimedia.org/T206636 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel, aborrero Cc: dcausse, EBernhardson, Iamamz3, bd808, Addshore, Andrew, Aklapper, Smalyshev, Gehel, Ramtin0071, MPhamWMF, dcaro, Devnull, GeminiAgaloos, nskaggs, lmata, Muchiri124, CBogen, Akuckartz, Phamhi, Legado_Shulgin, Nandana, Namenlos314, sietec, Davinaclare77, Bstorm, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Allthingsgo, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, JJMC89, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T190495: [Epic] Support a selection of diverse new GLAM pilot projects that involve Structured Data on Wikimedia Commons
aborrero closed subtask T195121: Contribution from the IGN to Structured Data on Commons as "Resolved". TASK DETAILhttps://phabricator.wikimedia.org/T190495EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMF, aborreroCc: Aklapper, Abit, Sadads, SandraF_WMF, Nandana, JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, Silverfish, _jensen, D3r1ck01, Susannaanas, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T195121: Contribution from the IGN to Structured Data on Commons
aborrero closed this task as "Resolved".aborrero claimed this task.aborrero added a comment. Since we found a way to do this operation, marking task as resolved now. Feel free to reopen if required.TASK DETAILhttps://phabricator.wikimedia.org/T195121EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: fgiunchedi, Reedy, bd808, Aklapper, aborrero, SandraF_WMF, Platonides, Rodelar, abian, Nandana, JKSTNK, AndyTan, sietec, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Silverfish, _jensen, D3r1ck01, Susannaanas, srodlund, Luke081515, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T209818: Mount dumps NFS share to instances in the soweego VPS project
aborrero closed this task as "Resolved".aborrero added a comment. root@soweego-1:~# ls -l /public/dumps/ total 16 lrwxrwxrwx 1 root root 59 Dec 12 13:52 incr -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/incr lrwxrwxrwx 1 root root 88 Dec 12 13:52 pagecounts-all-sites -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/other/pagecounts-all-sites lrwxrwxrwx 1 root root 82 Dec 12 13:52 pagecounts-raw -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/other/pagecounts-raw lrwxrwxrwx 1 root root 77 Dec 12 13:52 pageviews -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/other/pageviews lrwxrwxrwx 1 root root 61 Dec 12 13:52 public -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public I think you are all set. Feel free to reopen the ticket if you find anything wrong.TASK DETAILhttps://phabricator.wikimedia.org/T209818EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: gerritbot, bd808, aborrero, Aklapper, Hjfocs, CucyNoiD, Nandana, NebulousIris, JKSTNK, GTirloni, Fheredia, AndyTan, Gaboe420, sietec, Zylc, Versusxo, Majesticalreaper22, Giuliamocci, Bstorm, Adrian1985, Cpaulf30, 1978Gage2001, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Chicocvenancio, Allthingsgo, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tbscho, LawExplorer, Lewizho99, JJMC89, Maathavan, _jensen, D3r1ck01, srodlund, Luke081515, Wikidata-bugs, aude, Gryllida, scfc, Mbch331, Jay8g, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Claimed] T209818: Request to access shared storage on the soweego VPS project
aborrero claimed this task. TASK DETAILhttps://phabricator.wikimedia.org/T209818EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: bd808, aborrero, Aklapper, Hjfocs, Nandana, JKSTNK, GTirloni, Fheredia, AndyTan, sietec, Zylc, Bstorm, 1978Gage2001, Lahi, Gq86, GoranSMilovanovic, Chicocvenancio, Allthingsgo, QZanden, Tbscho, LawExplorer, JJMC89, _jensen, D3r1ck01, srodlund, Luke081515, Wikidata-bugs, aude, Gryllida, scfc, Mbch331, Jay8g, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T209818: Request to access shared storage on the soweego VPS project
aborrero edited projects, added cloud-services-team (Kanban); removed cloud-services-team.aborrero triaged this task as "Normal" priority.aborrero added a comment. Looking into this.TASK DETAILhttps://phabricator.wikimedia.org/T209818EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: bd808, aborrero, Aklapper, Hjfocs, Nandana, JKSTNK, GTirloni, Fheredia, AndyTan, sietec, Zylc, Bstorm, 1978Gage2001, Lahi, Gq86, GoranSMilovanovic, Chicocvenancio, Allthingsgo, QZanden, Tbscho, LawExplorer, JJMC89, _jensen, D3r1ck01, srodlund, Luke081515, Wikidata-bugs, aude, Gryllida, scfc, Mbch331, Jay8g, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T208714: Request creation of soweego VPS project
aborrero added a project: cloud-services-team (Kanban). TASK DETAILhttps://phabricator.wikimedia.org/T208714EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GTirloni, aborreroCc: Hjfocs, MaxFrax96, Aklapper, Nandana, JKSTNK, GTirloni, Fheredia, AndyTan, sietec, Zylc, Bstorm, 1978Gage2001, Lahi, aborrero, Gq86, Bsandipan, GoranSMilovanovic, Chicocvenancio, Allthingsgo, QZanden, Tbscho, LawExplorer, JJMC89, srodlund, Luke081515, Wikidata-bugs, aude, Gryllida, scfc, Mbch331, Jay8g, bd808, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204698: cloudvps: wikidata-page-banner project trusty deprecation
aborrero added a comment. In T204698#4599707, @Jdlrobson wrote: @aborrero if we don't hear from Sumit, i suggest we remove this instance, however I'm not sure what the protocol and grace period is for doing so. Ok, you can read more about our timeline here: https://wikitech.wikimedia.org/wiki/News/Trusty_deprecationTASK DETAILhttps://phabricator.wikimedia.org/T204698EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: aborrero, Jdlrobson, Sumit, Aklapper, JKSTNK, AndyTan, Zylc, 1978Gage2001, Lahi, Gq86, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, LawExplorer, Winter, JJMC89, srodlund, Wikidata-bugs, aude, Gryllida, Lydia_Pintscher, scfc, Mbch331, Jay8g, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T204696: cloudvps: wikidata-dev project trusty deprecation
aborrero created this task.aborrero triaged this task as "Normal" priority.aborrero added a project: Cloud-VPS.Restricted Application added a subscriber: Aklapper. TASK DESCRIPTIONUbuntu Trusty is no longer available in Cloud VPS since Nov 2017 for new instances. However, the EOL of Trusty is approaching in 2019 and we need to move to Debian Stretch before that date. All instances in the wikidata-dev project needs to upgrade as soon as possible. The list of affected VMs is: wikibase-vue.wikidata-dev.eqiad.wmflabs Listed administrator are: More info in openstack browser: https://tools.wmflabs.org/openstack-browser/project/wikidata-devTASK DETAILhttps://phabricator.wikimedia.org/T204696EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: Wikidata, aborrero, Aklapper, AndyTan, Zylc, 1978Gage2001, Chicocvenancio, Tbscho, JJMC89, srodlund, Gryllida, scfc, Jay8g, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T204267: Flood of WDQS requests from wbqc
aborrero added a subscriber: Tpt.aborrero added a comment. We stopped now the corhist tool which belongs to @Tpt, please check the tool. Now that the tool has been terminated, the traffic in wikidata seems back to normal.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, aborreroCc: Tpt, aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
aborrero added a comment. In T204267#4583629, @Pintoch wrote: @aborrero thanks for the ping. I do not recognize the shape of the queries as coming from this tool though. The openrefine-wikidata tool should do relatively few SPARQL queries, whose results are cached in redis. How did you determine that this tool is the source of the problem? Sorry for the noise, after stopping your tool we still see the same traffic. Feel free to restart your tool and again: sorry. Problem is that we can't identify the tool by the server logs because there isn't any meaningful UA, so we are walking a bit blind. Your tool had the wikidata name in it, so that was the first candidate.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, aborreroCc: aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T204267: Flood of WDQS requests from wbqc
aborrero added subscribers: Pintoch, aborrero.aborrero added a comment. Ping @Pintoch , this seems to be your tool.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, aborreroCc: aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T195121: Contribution from the IGN to Structured Data on Commons
aborrero added a comment. I just formally requested a project with a single VM for this. See T203072. Please expect a week-long wait until resources are available.TASK DETAILhttps://phabricator.wikimedia.org/T195121EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: fgiunchedi, Reedy, bd808, Aklapper, aborrero, SandraF_WMF, Platonides, Rodelar, abian, AndyTan, sietec, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T195121: Contribution from the IGN to Structured Data on Commons
aborrero added a comment. In T195121#4527757, @fgiunchedi wrote: Sounds like a nice project! With my swift maintainer hat on, testing a single 200-300 GB chunk of data sounds good to me. Let's coordinate though before uploading the full data set because swift is pending its annual expansion (T201937) and I'd like to have that completed to not push swift disk usage too much with substantial uploads. Perhaps I didn't use correct words. Also, I don't know in deep how data looks like. But I believe files are small, like map tiles and other images. In this case was using data chunk to refer to the downloadable files that IGN offers, which seems to composed of many of these map tiles or other images and metadata.TASK DETAILhttps://phabricator.wikimedia.org/T195121EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: fgiunchedi, Reedy, bd808, Aklapper, aborrero, SandraF_WMF, Platonides, Rodelar, abian, stebsco, AndyTan, sietec, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T195121: Contribution from the IGN to Structured Data on Commons
aborrero added a comment. I just had a videoconf with people @abian and Ruben Ojeda from Wikimedia Spain. Some conclusions: IGN offers a lot of data, in many different formats. @abian or someone else should get an idea on how to post-process these files to a format understandable by Commons. We agreed on trying a 200GB VM for data processing before uploading to commons, and work by small chunks of data. Of these 200GB, 100GB is for the raw download, and 100GB for the post-process output before uploading to common. After a chunk is processed, the storage is cleaned to left space for next chunk. Apparently IGN doesn't have an API or other structured web URL for us to download the data using a script. They use some custom POST parameters, and we would need some information on them before we can script those. If we can't automate the download, there is an option to go to the IGN datacenter, plug a hard disk and fetch all the data without using the network. Once we have this hard disk we could either send it to a WMF datacenter or @abian can upload it from his home to our VM. So, there are 2 different issues here: How to fetch the data from IGN (web API, http POST, hard disk, etc) How to process the data we fetched from IGN In case we discover IGN has an API (or @abian can script the http POST easily) we could even think on having this pipeline build on Toolforge in our Grid Engine (download small chunk -> process -> upload to commons -> start again) .TASK DETAILhttps://phabricator.wikimedia.org/T195121EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: fgiunchedi, Reedy, bd808, Aklapper, aborrero, SandraF_WMF, Platonides, Rodelar, abian, AndyTan, sietec, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T195121: Contribution from the IGN to Structured Data on Commons
aborrero added a subscriber: bd808.aborrero added a comment. In T195121#4424983, @abian wrote: In T195121#4224564, @aborrero wrote: Is it possible to do the processing by type (your first 3 points) instead of all at the same time? So you could reuse small chunks of storage instead of having to allocate a big one We'll definitely do the processing by type to reuse small chunks of storage, yes. ACK How do they offer the datasets? Do they have an API you can query for more data? Do they offer a single big file for downloading with all the data? Depending on this, our approach could be very different. We could build a "pipeline" instead of working with large batches The data are spread through several downloadable files. I think I'll be able to structure these data somehow. Do you have an estimation on how big is each downloadable file? You mentioned a series of raster topographic maps at a scale of 1:25, formats TIFF and ECW (~785 GB),, but I guess this is composed of several smaller files. Let's assume it's 10%, i.e. 79GB. We could allocate a ~200GB VM for you to download from there and do the processing, upload to commons, clean and continue with the next block. Do you have an estimation on how many time all the operations would take? i.e, how much time should we provide the storage facilities? 1 week, 1 month, 6 month, 1 year... No, for now we can't guess how many time the operations would take since we still don't know what's the best way to transfer the files from a hard drive to Commons. The points are: We have to buy a hard drive so that the IGN can store all the files in it. Do we need a SSD to be able to transfer/upload the files within a reasonable time but at the expense of having a limited capacity? Would a HDD be enough? Will this difference in time be actually noticeable? How do we transfer the files in the SSD/HDD to the Cloud VPS? Can we physically send the drive to the location of the server? Or should we transfer the files via the Internet? Considering the bandwith and the size of the files, is this last option possible? @aborrero, maybe you can help us with some of these points? Networking (specially international) will surely be a bottleneck. Depending on the file size (and protocol), data transfer might be very painfull. We may use rsync or some bittorrent protocol to ensure that even with long transfer times we are able to actually move the data from one point to the other. I don't think moving around a physical drive would help us in this case (and I ignore which policy we have to plug a random drive in one of our servers). So, this is my proposal, assuming ~79GB data files: let's allocate a VM with a 200GB extra disk (to be discussed with @bd808 and the rest of the WMCS team) let's give you access to this VM start testing our pipeline download of a single ~79GB data file from IGN, using something like rsync, torrent, etc processing uploading to commons clean VM to leave room for next block TASK DETAILhttps://phabricator.wikimedia.org/T195121EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: bd808, Aklapper, aborrero, SandraF_WMF, Platonides, Rodelar, abian, AndyTan, sietec, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T195121: Contribution from the IGN to Structured Data on Commons
aborrero added a comment. Hey, this is great news, since IGN is indeed a powerful source of data regarding geography, topology and mountains (which I love). From the Wikimedia Cloud Services point of view, one thing we could offer is a Cloud VPS temporal project, where you can create one or two virtual machines to do all this work. We can delete the project when you are done. If you plan to hold all the data at the same time, it seems you need about 3TB just to store what IGN offers. Then I guess you need at least the same for processing the data and the intermediate step before uploading to Wikimedia Commons. This is about 6TB storage, which is not trivial to allocate I think. Some questions: Do you have an estimation on how many time all the operations would take? i.e, how much time should we provide the storage facilities? 1 week, 1 month, 6 month, 1 year... Is it possible to do the processing by type (your first 3 points) instead of all at the same time? So you could reuse small chunks of storage instead of having to allocate a big one How do they offer the datasets? Do they have an API you can query for more data? Do they offer a single big file for downloading with all the data? Depending on this, our approach could be very different. We could build a "pipeline" instead of working with large batches TASK DETAILhttps://phabricator.wikimedia.org/T195121EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: aborrero, Aklapper, SandraF_WMF, Platonides, Rodelar, abian, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, bd808, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T192785: Create a discussion around finding different connected wikibase instances
aborrero closed subtask T192892: Request creation of wikibase-registry VPS project as "Resolved". TASK DETAILhttps://phabricator.wikimedia.org/T192785EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aborreroCc: Addshore, Aklapper, RazShuty, LJ, Lahi, Gq86, Andrawaag, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Daniel_Mietchen, Lydia_Pintscher, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
