[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  Links latest-full.ttl.bz2 -> 20200116/commons-20200116-full.ttl.bz2 and 
latest-full.ttl.gz -> 20200116/commons-20200116-full.ttl.gz have been cleaned 
up. Thanks for the suggestion!

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: DD063520, D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, 
Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, 
jannee_e, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Taiwania_Justo, 
Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, gnosygnu, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[dcausse]

dcausse added a comment.


  should `latest-full.*` be deleted to avoid confusion?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, dcausse
Cc: DD063520, D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, 
Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, 
jannee_e, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Taiwania_Justo, 
Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, gnosygnu, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  It's linked off the 'other datasets' page near the top. But here's the direct 
link: https://dumps.wikimedia.org/other/wikibase/commonswiki/

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: DD063520, D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, 
Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, 
jannee_e, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Taiwania_Justo, 
Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, gnosygnu, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[DD063520]

DD063520 added a comment.


  Cool, but what is the link?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, DD063520
Cc: DD063520, D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, 
Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, 
jannee_e, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Taiwania_Justo, 
Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, gnosygnu, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 610816 **merged** by ArielGlenn:
  [operations/puppet@production] Add commons structured data dumps to the 
webpage!
  
  https://gerrit.wikimedia.org/r/610816

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, jannee_e, Beast1978, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, 
Lewizho99, Maathavan, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, 
Xmlizer, Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 610816 had a related patch set uploaded (by ArielGlenn; owner: 
ArielGlenn):
  [operations/puppet@production] Add commons structured data dumps to the 
webpage!
  
  https://gerrit.wikimedia.org/r/610816

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, jannee_e, Beast1978, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, 
Lewizho99, Maathavan, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, 
Xmlizer, Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 609823 **merged** by ArielGlenn:
  [operations/puppet@production] add a README about the content of the commons 
structured data dumps
  
  https://gerrit.wikimedia.org/r/609823

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, jannee_e, Beast1978, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, 
Lewizho99, Maathavan, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, 
Xmlizer, Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 609823 had a related patch set uploaded (by ArielGlenn; owner: 
ArielGlenn):
  [operations/puppet@production] add a README about the content of the commons 
structured data dumps
  
  https://gerrit.wikimedia.org/r/609823

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, jannee_e, Beast1978, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, 
Lewizho99, Maathavan, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, 
Xmlizer, Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 609114 **merged** by Gehel:
  [operations/puppet@production] snapshots: enable dumps of structured data 
from commons
  
  https://gerrit.wikimedia.org/r/c/operations/puppet/ /609114

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, jannee_e, Beast1978, 
Un1tY, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, Lewizho99, 
Maathavan, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 609114 had a related patch set uploaded (by Gehel; owner: Gehel):
  [operations/puppet@production] snapshots: enable dumps of structured data 
from commons
  
  https://gerrit.wikimedia.org/r/c/operations/puppet/ /609114

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Alter-paule, jannee_e, Beast1978, 
Un1tY, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, Lewizho99, 
Maathavan, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 601162 **merged** by ArielGlenn:
  [operations/puppet@production] sdc dumps: add placeholder entry in dcat setup 
to avoid syntax errors
  
  https://gerrit.wikimedia.org/r/601162

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Blissjay007, Oblanco79, 
Alter-paule, jannee_e, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, 
E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, CucyNoiD, 
Nandana, NebulousIris, Namenlos314, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, 
Lordiis, GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, 
QZanden, EBjune, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, 
rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, 
gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 601162 had a related patch set uploaded (by ArielGlenn; owner: 
ArielGlenn):
  [operations/puppet@production] sdc dumps: add placeholder entry in dcat setup 
to avoid syntax errors
  
  https://gerrit.wikimedia.org/r/601162

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Blissjay007, Oblanco79, 
Alter-paule, jannee_e, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, 
E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, CucyNoiD, 
Nandana, NebulousIris, Namenlos314, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, 
Lordiis, GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, 
QZanden, EBjune, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, 
rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, 
gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 599856 **merged** by Gehel:
  [operations/puppet@production] Revert "enable dumps of structured data from 
commons"
  
  https://gerrit.wikimedia.org/r/599856

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Blissjay007, Oblanco79, 
Alter-paule, jannee_e, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, 
E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, CucyNoiD, 
Nandana, NebulousIris, Namenlos314, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, 
Lordiis, GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, 
QZanden, EBjune, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, 
rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, 
gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 599856 had a related patch set uploaded (by Gehel; owner: Gehel):
  [operations/puppet@production] Revert "enable dumps of structured data from 
commons"
  
  https://gerrit.wikimedia.org/r/599856

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: D063520, CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, 
Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, 
Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, darthmon_wmde, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[dcausse]

dcausse added a comment.


  @WMDE-leszek oops, sorry I replied before reading you comment and was reading 
an old code base... if this is just a config change it can hopefully be merged 
soon. Thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, dcausse
Cc: CBogen, nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, 
jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, darthmon_wmde, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[WMDE-leszek]

WMDE-leszek added a comment.


  @dcausse it is possible to customize the prefix. 
https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/569260 once 
merged will enable this. The chain of patches is a bit stalled. I will remove 
some dust off it and hopefully we'll get this in soon

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, WMDE-leszek
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, CBogen, darthmon_wmde, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[dcausse]

dcausse added a comment.


  Just a note on the current problem:
  the prefixes defined in ttl dumps are identical to the ones used by wikidata 
e.g.:
  
@prefix wdt:  .
  
  This is perfectly valid but might cause some confusions because when using 
commons query service we will likely change these prefixes so that federation 
with wdqs is more obvious.
  I had a look at the code base but haven't found an easy way to override this.
  @Lucas_Werkmeister_WMDE do you know if it'd be possible to change the 
prefixes for the local repository such that for commons 
`http://commons.wikimedia.org/prop/direct/` would not be `wdt`?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, dcausse
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, CBogen, darthmon_wmde, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  @dcausse what's your time frame?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, CBogen, darthmon_wmde, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[dcausse]

dcausse added a comment.


  @ArielGlenn we plan to make a subtle change to the dump (prefixes), this 
won't be technically a breaking change but could cause some confusion if users 
start to assume the presence of some prefixes. Would it be possible to pause 
the publication of the dumps while we change this? Sorry for the late notice.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, dcausse
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, CBogen, darthmon_wmde, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 599052 **merged** by ArielGlenn:
  [operations/puppet@production] fix up cron name of commons structured data 
dumps
  
  https://gerrit.wikimedia.org/r/599052

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Blissjay007, Oblanco79, Alter-paule, 
jannee_e, Beast1978, CBogen, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, 
E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, CucyNoiD, 
Nandana, NebulousIris, Namenlos314, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, 
Lordiis, GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, 
QZanden, EBjune, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, 
rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, 
gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 599052 had a related patch set uploaded (by ArielGlenn; owner: 
ArielGlenn):
  [operations/puppet@production] fix up cron name of commons structured data 
dumps
  
  https://gerrit.wikimedia.org/r/599052

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Blissjay007, Oblanco79, Alter-paule, 
jannee_e, Beast1978, CBogen, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, 
E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, CucyNoiD, 
Nandana, NebulousIris, Namenlos314, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, 
Lordiis, GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, 
QZanden, EBjune, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, 
rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, 
gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 598787 **merged** by ArielGlenn:
  [operations/puppet@production] enable dumps of structured data from commons
  
  https://gerrit.wikimedia.org/r/598787

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Blissjay007, Oblanco79, Alter-paule, 
jannee_e, Beast1978, CBogen, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, 
E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, CucyNoiD, 
Nandana, NebulousIris, Namenlos314, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, 
Lordiis, GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, 
QZanden, EBjune, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, 
rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, 
gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 598787 had a related patch set uploaded (by ArielGlenn; owner: 
ArielGlenn):
  [operations/puppet@production] enable dumps of structured data from commons
  
  https://gerrit.wikimedia.org/r/598787

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, gerritbot
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, CBogen, darthmon_wmde, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  I see that we're no longer blocked.  Does this mean that we're good to go for 
weekly runs?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, CBogen, darthmon_wmde, Nandana, 
Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, 
_jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, 
Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[dcausse]

dcausse added a comment.


  @ArielGlenn no not yet, this is still blocked on T243292 
 which requires some investigation 
to determine which component (dump or the wdqs transformation process) is wrong.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, CBogen, darthmon_wmde, Nandana, 
Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, 
_jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, 
Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  Hi, just checking in: any progress on invetigating the 'extra' dumps content?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, CBogen, darthmon_wmde, Nandana, 
Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, 
_jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, 
Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  @Cparle, No blocks on your side, the ball is now in @dcausse 's court. :-)

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, darthmon_wmde, Nandana, JKSTNK, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
Lunewa, QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, Silverfish, 
_jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Susannaanas, 
Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, 
matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, 
Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[Cparle]

Cparle added a comment.


  @ArielGlenn the structured data team keep coming up as blockers on this 
ticket in scrum-of-scrums, but we're not blocking you, are we?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Cparle
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, 
hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, darthmon_wmde, Nandana, JKSTNK, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
Lunewa, QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, Silverfish, 
_jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Susannaanas, 
Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, 
matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, 
Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  In https://dumps.wikimedia.org/other/wikibase/commonswiki/ there are two ttl 
files, gz and bz2 compressed. Please have a look!
  
  The bash script producing them complained that
  
/usr/local/bin/dumpwikibaserdf.sh: line 224: setDcatConfig: command not 
found
  
  so I'm looking at that.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: EBernhardson, Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, 
Poyekhali, Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, 
Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, 
darthmon_wmde, Nandana, JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, 
Anooprao, SandraF_WMF, GoranSMilovanovic, Lunewa, QZanden, EBjune, Tramullas, 
Acer, merbst, LawExplorer, Silverfish, _jensen, rosalieper, Taiwania_Justo, 
Scott_WUaS, Jonas, Xmlizer, Susannaanas, Ixocactus, Wong128hk, gnosygnu, 
Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, 
Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  I found a ticket that mentions use of ttl files so I'll run
  
/usr/local/bin/dumpwikibaserdf.sh commons full ttl
  
  and keep an eye on it. Running on snapshot1008 in a screen session. Here we 
go!

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: EBernhardson, Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, 
Poyekhali, Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, 
Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, 
darthmon_wmde, Nandana, JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, 
Anooprao, SandraF_WMF, GoranSMilovanovic, Lunewa, QZanden, EBjune, Tramullas, 
Acer, merbst, LawExplorer, Silverfish, _jensen, rosalieper, Taiwania_Justo, 
Scott_WUaS, Jonas, Xmlizer, Susannaanas, Ixocactus, Wong128hk, gnosygnu, 
Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, 
Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[WMDE-leszek]

WMDE-leszek added a comment.


  @EBernhardson or some other person from #search-team 
 would now for certain.
  Quickly looking at 
https://wikitech.wikimedia.org/wiki/Wikidata_query_service#Data_reload_procedure,
 commands there mention ttl dumps though. Note that I don't really know what I 
am talking about, just trying to be useful.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: WMDE-leszek
Cc: EBernhardson, Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, 
Poyekhali, Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, 
Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, 
darthmon_wmde, Nandana, JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, 
Anooprao, SandraF_WMF, GoranSMilovanovic, Lunewa, QZanden, EBjune, Tramullas, 
Acer, merbst, LawExplorer, Silverfish, _jensen, rosalieper, Taiwania_Justo, 
Scott_WUaS, Jonas, Xmlizer, Susannaanas, Ixocactus, Wong128hk, gnosygnu, 
Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, 
Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  I plan to try running
  
/usr/local/bin/dumpwikibaserdf.sh commons full nt
  
  on Thursday morning and see how long it takes with the 8 shards that are 
currently configured.  @Abit is the nt format the one needed for WDQS testing?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, 
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Af420, E1presidente, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 516444 **merged** by ArielGlenn:
  [operations/puppet@production] Set up dumps for mediainfo RDF generation
  
  https://gerrit.wikimedia.org/r/516444

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, 
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Af420, E1presidente, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  Ran
  
php /srv/mediawiki/multiversion/MWScript.php 
extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki commonswiki 
--batch-size 500 --format nt --flavor full-dump --entity-type mediainfo 
--no-cache --dbgroupdefault dump  --ignore-missing --first-page-id 78846320 
--last-page-id 79046320 --shard 0 --sharding-factor 1  
2>/var/lib/dumpsgen/mediainfo-log-small-shard-oom.txt | gzip > 
/mnt/dumpsdata/temp/dumpsgen/mediainfo-dumps-test-nt-one-shard-small-oom.gz
  
  which should cover the page range where we had the oom; it ran to completion 
fine. I guess that there is some small memory leak that must accumulate over 
batches, which is what did us in earlier. As long as we limit runs to some 
reasonable number of pages each time, we should be fine.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, 
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Af420, E1presidente, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  Ran
  
php /srv/mediawiki/multiversion/MWScript.php 
extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki commonswiki 
--batch-size 1000 --format nt --flavor full-dump --entity-type mediainfo 
--no-cache --dbgroupdefault dump  --ignore-missing --first-page-id 1 
--last-page-id 21 --shard 1 --sharding-factor 4  
2>/var/lib/dumpsgen/mediainfo-log-small-shard.txt | gzip > 
/mnt/dumpsdata/temp/dumpsgen/mediainfo-dumps-test-nt-one-shard-of-4-small.gz
  
  and it also ran fine.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, 
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Af420, E1presidente, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  Note to self that a run of
  
php /srv/mediawiki/multiversion/MWScript.php 
extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki commonswiki 
--batch-size 250 --format nt --flavor full-dump --entity-type mediainfo 
--no-cache --dbgroupdefault dump  --ignore-missing --first-page-id 1 
--last-page-id 21 --shard 0 --sharding-factor 1  
2>/var/lib/dumpsgen/mediainfo-log-small.txt | gzip > 
/mnt/dumpsdata/temp/dumpsgen/mediainfo-dumps-test-nt-noshard-small.gz
  
  worked fine. Going to run one with a sharding factor of 4 and a batch size 4 
times larger, to see how that is.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, 
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Af420, E1presidente, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  This morning the job was terminated by the oom killer:
  
[4288057.417443] Out of memory: Kill process 117265 (php) score 868 or 
sacrifice child
[4288057.425084] Killed process 117265 (php) total-vm:58241128kB, 
anon-rss:56901636kB, file-rss:0kB, shmem-rss:0kB
  
  It produced a file of size 380M with 2224612 entitites in it before being 
shot. One of the last entries in it is the page File:Gerrardina_foliosa_1.jpg 
with page id 78 846 520 and mediainfo entity (Depicts) added on Jan 10th, 2020. 
Also the gz output file is not truncated, so perhaps it is complete. @Abit   
Should I move it somewhere for folks to test with?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, 
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Af420, E1presidente, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  A batchsize of 50k turned out to be too large. Same with 5k. I'm now running 
with a batchsize of 500, which will surely be too small, but at least I am 
getting output. I'll check on it tomorrow and see how it's doing.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, 
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Af420, E1presidente, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  Because I've gotten a nice run in beta with the --ignore-missing flag, I'm 
trying a test run on snapshot1008 in a screen session:
  
php /srv/mediawiki/multiversion/MWScript.php 
extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki commonswiki 
--batch-size 5 --format nt --flavor full-dump --entity-type mediainfo 
--no-cache --dbgroupdefault dump  --ignore-missing 
2>>/var/lib/dumpsgen/mediainfo-log.txt | gzip > 
/mnt/dumpsdata/temp/dumpsgen/mediainfo-dumps-test-nt-noshard.gz
  
  If the output looks good, I'll put it somewhere for WQS testing and move 
forward with making these weekly runs with the appropriate number of parallel 
processes.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, 
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Af420, E1presidente, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, 
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[Abit]

Abit added a comment.


  @ArielGlenn What is this blocked or stalled on?  Looks like several of the 
subtasks have been closed, but not all.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Abit
Cc: Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, 
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, 
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, 
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Af420, E1presidente, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, 
Lordiis, GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, 
QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, 
Maathavan, Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, 
Xmlizer, Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, 
Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, 
Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[Smalyshev]

Smalyshev added a comment.


  I tried to manually dump the mediainfo entries over the weekend, it took 376 
minutes for 4 shards (a lot, but less than I expected) and produces 1724656 
items. Does not seem to produce significant load on DB so far - but it gives 
about 20 items/second, which seems to be too slow. If we ever get all files 
having items, that'd take 4 days to process over 8 shards, probably more since 
DB access will get slower, right now they are not to slow because there's only 
2% of files that have items, so not too many DB queries.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, DannyS712, CucyNoiD, 
Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, 
E1presidente, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Morgankevinj, Taiwania_Justo, Jonas, Xmlizer, 
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, 
Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  I'm not thinking about the amount of time it takes, but rather the load on 
the database servers. Reasonable sized batched queries will be better, as I've 
seen already with stub dumps and slot retrieval.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev, ArielGlenn
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, DannyS712, CucyNoiD, 
Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, 
E1presidente, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Morgankevinj, Taiwania_Justo, Jonas, Xmlizer, 
Susannaanas, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, Jdouglas, 
Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, 
Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[Smalyshev]

Smalyshev added a comment.


  I've added fix for one of the issues in T222497 
 already but it doesn't fix 
everything. I think it's still would be interesting to test what happens in 
production - maybe not full dump but just partial, to estimate what we're 
dealing with and how bad is it? Maybe due to the fact we don't have yet too 
many mediainfo records and they're small we could be still fine?
  
  Also, the patch itself doesn't actually turn the cron on, it just puts the 
files there. We'd need to flip the "files only" switch to actually produce the 
working cron.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, DannyS712, CucyNoiD, 
Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, 
E1presidente, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Morgankevinj, Taiwania_Justo, Jonas, Xmlizer, 
Susannaanas, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, Jdouglas, 
Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, 
Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 517670 **merged** by ArielGlenn:
  [operations/puppet@production] refactor wikidata entity dumps into wikibase + 
wikidata specific bits
  
  https://gerrit.wikimedia.org/r/517670

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev, gerritbot
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, DannyS712, CucyNoiD, 
Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, 
E1presidente, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Morgankevinj, Taiwania_Justo, Jonas, Xmlizer, 
Susannaanas, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, Jdouglas, 
Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, 
Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  I sincerely apologize: this weekend the heat baked my brain and I did nothing 
related to computers at all. And Friday evening I was out. I'll set a 
notification to remind me this coming Friday earlier in the day, so that this 
gets done.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev, ArielGlenn
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, DannyS712, CucyNoiD, 
Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, 
E1presidente, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Morgankevinj, Taiwania_Justo, Jonas, Xmlizer, 
Susannaanas, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, Jdouglas, 
Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, 
Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[Smalyshev]

Smalyshev added a comment.


  @ArielGlenn: sounds great!

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, DannyS712, CucyNoiD, 
Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, 
E1presidente, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Morgankevinj, Taiwania_Justo, Jonas, Xmlizer, 
Susannaanas, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, Jdouglas, 
Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, 
Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  The refactor patchset now checks out with all the wikidata dumps including 
json. I'd like to deploy it this weekend, giving plenty of time to make sure 
it's ok, test the structured data patchset, and then be able to deploy that 
separately.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, DannyS712, CucyNoiD, 
Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, 
E1presidente, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, 
Silverfish, _jensen, rosalieper, Morgankevinj, Taiwania_Justo, Jonas, Xmlizer, 
Susannaanas, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs, Jdouglas, 
Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, 
Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 521920 abandoned by Hoo man:
  LoadBalancer::getConnectionIndex: Also fallback to generic group
  
  Reason:
  Better solved through https://gerrit.wikimedia.org/r/c/mediawiki/core/ /523412
  
  https://gerrit.wikimedia.org/r/521920

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, 
NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, 
Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, gnosygnu, 
Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, 
Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 521920 had a related patch set uploaded (by Hoo man; owner: Hoo man):
  [mediawiki/core@master] LoadBalancer::getConnectionIndex: Also fallback to 
generic group
  
  https://gerrit.wikimedia.org/r/521920

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 0010318400, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Cparle, Darkminds3113, 
Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Lunewa, 
Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, merbst, 
LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, rosalieper, 
Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, gnosygnu, Jane023, 
jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, 
Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  Ah I didn't even think about the param being set in the script. I had removed 
it during testing to see if that helped, and... nada.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 0010318400, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Cparle, Darkminds3113, 
Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Lunewa, 
Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, merbst, 
LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, rosalieper, 
Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, gnosygnu, Jane023, 
jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, 
Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[hoo]

hoo added a comment.


  Apparently this is related to the `--dbgroupdefault` (doesn't seem to depend 
on the value):
  
dumpsgen@deployment-snapshot01:~$ /usr/bin/php7.2 
/srv/mediawiki/multiversion/MWScript.php eval.php --wiki wikidatawiki 
--dbgroupdefault dump
> echo get_class(wfGetDB( DB_REPLICA ));
Caught exception Wikimedia\Rdbms\DBConnectionError: Cannot access the 
database: No working replica DB server: Unknown error
…
  
  
  
dumpsgen@deployment-snapshot01:~$ /usr/bin/php7.2 
/srv/mediawiki/multiversion/MWScript.php eval.php --wiki wikidatawiki
> echo get_class(wfGetDB( DB_REPLICA ));
Wikimedia\Rdbms\DatabaseMysqli
  
  This is also going to always hit the Wikibase dumpers as they set the 
dbgroupdefault even if not given via CLI.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 0010318400, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Cparle, Darkminds3113, 
Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Lunewa, 
Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, merbst, 
LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, rosalieper, 
Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, gnosygnu, Jane023, 
jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, 
Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  Running into a new problem testing on beta.
  
dumpsgen@deployment-snapshot01:~$ /usr/bin/php7.2 
/srv/mediawiki/multiversion/MWScript.php 
extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 1 
--sharding-factor 2 --batch-size 2000 --format ttl  --no-cache --dbgroupdefault 
dump --part-id 5-0 --first-page-id 1 --last-page-id 16000   
Dumping entities of type item, property, lexeme, form, sense
Dumping shard 1/2
Wikimedia\Rdbms\DBConnectionError from line 1392 of 
/srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php: 
Cannot access the database: No working replica DB server: Unknown error
#0 
/srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(467):
 Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 
/srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(897):
 Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(false, Array, 'wikidatawiki')
#2 /srv/mediawiki/php-master/includes/site/DBSiteStore.php(78): 
Wikimedia\Rdbms\LoadBalancer->getConnection(-1)
#3 /srv/mediawiki/php-master/includes/site/DBSiteStore.php(65): 
DBSiteStore->loadSites()
#4 /srv/mediawiki/php-master/includes/site/CachingSiteStore.php(111): 
DBSiteStore->getSites()
#5 
/srv/mediawiki/php-master/extensions/Wikibase/repo/maintenance/dumpRdf.php(187):
 CachingSiteStore->getSites()
#6 
/srv/mediawiki/php-master/extensions/Wikibase/repo/maintenance/DumpEntities.php(200):
 Wikibase\DumpRdf->createDumper(Resource id #716)
#7 
/srv/mediawiki/php-master/extensions/Wikibase/repo/maintenance/dumpRdf.php(138):
 Wikibase\DumpEntities->execute()
#8 /srv/mediawiki/php-master/maintenance/doMaintenance.php(99): 
Wikibase\DumpRdf->execute()
#9 
/srv/mediawiki/php-master/extensions/Wikibase/repo/maintenance/dumpRdf.php(202):
 require_once('/srv/mediawiki/...')
#10 /srv/mediawiki/multiversion/MWScript.php(101): 
require_once('/srv/mediawiki/...')
#11 {main}
  
  That's new broken behavior. Same for json dumps:
  
dumpsgen@deployment-snapshot01:~$ /usr/bin/php7.2 
/srv/mediawiki/multiversion/MWScript.php 
extensions/Wikibase/repo/maintenance/dumpJson.php --wiki wikidatawiki --shard 1 
--sharding-factor 2 --batch-size 500 --snippet 2 --entity-type item 
--entity-type property --no-cache  --first-page-id 1
Dumping entities of type item, property
Dumping shard 1/2
Wikimedia\Rdbms\DBConnectionError from line 1392 of 
/srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php: 
Cannot access the database: No working replica DB server: Unknown error
#0 
/srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(467):
 Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 
/srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(897):
 Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(false, Array, 'wikidatawiki')
#2 /srv/mediawiki/php-master/includes/GlobalFunctions.php(2568): 
Wikimedia\Rdbms\LoadBalancer->getConnection(-1, Array, 'wikidatawiki')
#3 
/srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Store/Sql/SqlEntityIdPager.php(104):
 wfGetDB(-1)
#4 
/srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php(257):
 Wikibase\Repo\Store\Sql\SqlEntityIdPager->fetchIds(500)
#5 
/srv/mediawiki/php-master/extensions/Wikibase/repo/maintenance/DumpEntities.php(221):
 
Wikibase\Dumpers\DumpGenerator->generateDump(Object(Wikibase\Repo\Store\Sql\SqlEntityIdPager))
#6 
/srv/mediawiki/php-master/extensions/Wikibase/repo/maintenance/dumpJson.php(100):
 Wikibase\DumpEntities->execute()
#7 /srv/mediawiki/php-master/maintenance/doMaintenance.php(99): 
Wikibase\DumpJson->execute()
#8 
/srv/mediawiki/php-master/extensions/Wikibase/repo/maintenance/dumpJson.php(126):
 require_once('/srv/mediawiki/...')
#9 /srv/mediawiki/multiversion/MWScript.php(101): 
require_once('/srv/mediawiki/...')
#10 {main}
  
  But I can request a replica and apparently get one:
  
dumpsgen@deployment-snapshot01:~$ /usr/bin/php7.2 
/srv/mediawiki/multiversion/MWScript.php maintenance/getReplicaServer.php 
--wiki wikidatawiki
deployment-db06
  
  Must be missing something but I don't see it.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 0010318400, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Cparle, Darkminds3113, 
Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  I plan to deploy the refactoring patchset Sunday in between runs (possibly 
today if the json dump and the others all finish up at a reasonable hour).  I 
ran wikdiata dumps in beta with and without the changeset (with altered values 
for shard, minfilesize and batchsize) and the new dumps look fine.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 0010318400, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Cparle, Darkminds3113, 
Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, 
Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, merbst, 
LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, rosalieper, 
Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, jkroll, 
Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, 
Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[Smalyshev]

Smalyshev added a comment.


  I've added my reviews and updated 
https://gerrit.wikimedia.org/r/c/operations/puppet/+/516444 to base on 
refactoring patch.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 0010318400, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, 
jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, 
Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  The gerrit change is ready for me to test now, probably by playing with it in 
beta a whole bunch.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 0010318400, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, 
jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, 
Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  Should we have a ticket for misc cleanup like 'get rid of the BETA in 
filenames' and 'get rid of the 'legacy directory' stuff for the json dumps?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, MB-one, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, 
NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, 
jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, 
Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[Smalyshev]

Smalyshev added a comment.


  No idea about JSON dumps, but I don't see any reason not to. I don't need 
them, but since it's Wikibase, it makes sense to have them too.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, MB-one, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, 
NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, 
jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, 
Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  While I"m sure the above is quite broken in a variety of ways, this is the 
sort of thing I had in mind, being able to drop in one file with just values 
specific to commons (or whatever wikibase thing might come our way later), and 
change the project name only, getting everything else 'for free'. If we wind up 
wanting json dumps for commons/future projects, it should not be hard to do 
something similar for them. I personally would like to see json dumps happen 
btw; is that on the road map?

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, MB-one, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, 
NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, 
jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, 
Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 517670 had a related patch set uploaded (by ArielGlenn; owner: 
ArielGlenn):
  [operations/puppet@production] refactor wikidata entity dumps into wikibase + 
wikidata specific bits
  
  https://gerrit.wikimedia.org/r/517670

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, MB-one, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 
0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, 
NebulousIris, JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Taiwania_Justo, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, 
jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, 
El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, 
Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 516441 **merged** by jenkins-bot:
  [mediawiki/extensions/Wikibase@master] Add option --ignore-missing to dumper
  
  https://gerrit.wikimedia.org/r/516441

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, MB-one, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, jkroll, 
Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, 
Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[Smalyshev]

Smalyshev added a comment.


  > Can we get away with a 'dumpwikibaserdf.sh' and some well-chosen variables 
for both wikidata and commons (or at worst two much much shorter scripts) 
instead of a whole new dumpcommonsrdf.sh?
  
  I thought about it, and it might be possible, but it would make the script 
even more complex and even less readable, as it would be pretty much everything 
variables on top of variables on top of variables. That's why I didn't go that 
route - too easy to make an error while piling those variables on top of each 
other, and would be too hard to find one... I understand copypasting is usually 
bad, but I wonder if we want to tolerate it for the sake of readability. If you 
think it's a must I can redo it using common script but that would probably 
make the common part harder to understand and verify.
  
  > Mediainfo is the first name I would have chosen too but maybe we can find 
something else that will keep future dump maintainers from swearing at us.
  
  That's a good time for proposals, while we have not committed to anything 
yet. That said, given that it'd be in other/wikibase/commonswiki, I don't think 
it can be confused with anything else.
  
  > Last comment is that I really dislike the links under 
xmldatadumps/public/{wikidata,commonswiki}. That directory should be only for 
xml/sql dumps and we should be moving away from having links to other stuff 
there, not adding new ones. We need a plan...
  
  I just did what wikidata dump is already doing. I do not have any opinion on 
which way is better, but I think it should be symmetric for both RDF dumps - 
either both have it, or none has it. So you are welcome to propose any model 
that you prefer, I do not have preferences here except for consistency.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, MB-one, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, jkroll, 
Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, 
Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[ArielGlenn]

ArielGlenn added a comment.


  A couple thoughts on the above patchset:
  
  Can we get away with a 'dumpwikibaserdf.sh' and some well-chosen variables 
for both wikidata and commons (or at worst two much much shorter scripts) 
instead of a whole new dumpcommonsrdf.sh?
  
  Additionally, we already have media, mediatitles, mediacounts and imageinfo 
being served out of other/. Mediainfo is the first name I would have chosen too 
but maybe we can find something else that will keep future dump maintainers 
from swearing at us.
  
  Last comment is that I really dislike the links under 
xmldatadumps/public/{wikidata,commonswiki}. That directory should be only for 
xml/sql dumps and we should be moving away from having links to other stuff 
there, not adding new ones. We need a plan...
  
  I'm trying to think of these dumps in the context of other projects possibly 
using structured data at some point, and how we can facilitate new dumps in the 
future.

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, 
Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, MB-one, 
Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Daryl-TTMG, RomaAmorRoma, 
E.S.A-Sheild, darthmon_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, 
Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, 
rosalieper, Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, jkroll, 
Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, 
Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, 
Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 516444 had a related patch set uploaded (by Smalyshev; owner: 
Smalyshev):
  [operations/puppet@production] Set up dumps for mediainfo RDF generation
  
  https://gerrit.wikimedia.org/r/516444

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, Lydia_Pintscher, 
Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, MB-one, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, E.S.A-Sheild, darthmon_wmde, joker88john, 
CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, 
Baloch007, E1presidente, Cparle, Darkminds3113, Anooprao, SandraF_WMF, 
Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, 
QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, 
Maathavan, Silverfish, _jensen, rosalieper, Jonas, Xmlizer, Susannaanas, 
Wong128hk, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, 
aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, 
Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T221917: Create RDF dump of structured data on Commons

[gerritbot]

gerritbot added a comment.


  Change 516441 had a related patch set uploaded (by Smalyshev; owner: 
Smalyshev):
  [mediawiki/extensions/Wikibase@master] Add option --ignore-missing to dumper
  
  https://gerrit.wikimedia.org/r/516441

TASK DETAIL
  https://phabricator.wikimedia.org/T221917

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, Lydia_Pintscher, 
Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, MB-one, Salgo60, 
Lucas_Werkmeister_WMDE, Smalyshev, darthmon_wmde, Nandana, JKSTNK, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, SandraF_WMF, 
GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, 
Silverfish, _jensen, rosalieper, Jonas, Xmlizer, Susannaanas, Wong128hk, 
Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, 
Fabrice_Florin, Raymond, Jdforrester-WMF, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs