gerritbot added a comment.
Change 569836 **merged** by Fdans:
[analytics/refinery@master] Add oozie job converting wikidata dumps to parquet
https://gerrit.wikimedia.org/r/569836
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
gerritbot added a comment.
Change 346726 **merged** by jenkins-bot:
[analytics/refinery/source@master] Add spark code for wikidata json dumps
parsing
https://gerrit.wikimedia.org/r/346726
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
gerritbot added a comment.
Change 571253 **merged** by Elukey:
[operations/puppet@production]
profile::analytics::refinery::job::import_wikidata_entities_dumps: cleanup
https://gerrit.wikimedia.org/r/571253
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
gerritbot added a comment.
Change 571253 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production]
profile::analytics::refinery::job::import_wikidata_entities_dumps: cleanup
https://gerrit.wikimedia.org/r/571253
TASK DETAIL
gerritbot added a comment.
Change 571238 **merged** by Elukey:
[operations/puppet@production]
profile::analytics::refinery::job::import_wikidata_entities_dumps: fix typo
https://gerrit.wikimedia.org/r/571238
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
gerritbot added a comment.
Change 571238 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Correct
profile::analytics::refinery::job::import_wikidata_entities_dumps
https://gerrit.wikimedia.org/r/571238
TASK DETAIL
gerritbot added a comment.
Change 571237 abandoned by Joal:
Correct profile::analytics::refinery::job::import_wikidata_entities_dumps
Reason:
New patch at https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /571238/
https://gerrit.wikimedia.org/r/571237
TASK DETAIL
gerritbot added a comment.
Change 571237 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Correct
profile::analytics::refinery::job::import_wikidata_entities_dumps
https://gerrit.wikimedia.org/r/571237
TASK DETAIL
gerritbot added a comment.
Change 567954 **merged** by Elukey:
[operations/puppet@production] Add
profile::analytics::refinery::job::import_wikidata_entites_dumps
https://gerrit.wikimedia.org/r/567954
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
gerritbot added a comment.
Change 346726 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Add spark code for wikidata json dumps
parsing
https://gerrit.wikimedia.org/r/346726
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL
gerritbot added a comment.
Change 569836 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Add oozie job converting wikidata dumps to parquet
https://gerrit.wikimedia.org/r/569836
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL
gerritbot added a comment.
Change 567954 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Add
profile::analytics::refinery::job::import_wikidata_entites_dumps
https://gerrit.wikimedia.org/r/567954
TASK DETAIL
Isaac added a comment.
> @JAllemandou Thank you - as ever!
+1: these wikidata parquet (specifically item_page_link) dumps are super
useful for us!
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
GoranSMilovanovic added a comment.
@JAllemandou Thank you - as ever!
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GoranSMilovanovic
Cc: Groceryheist, MGerlach, WMDE-leszek, abian, leila,
GoranSMilovanovic added a comment.
@JAllemandou Do you think it would be possible to produce a new version of
this data set?
The latest update seems to be: `2019-10-03 09:29
/user/joal/wmf/data/wmf/mediawiki/wikidata_parquet/20190902` - which you have
pointed me at in T209655#5543575
GoranSMilovanovic added a comment.
@JAllemandou Awesome, thank you!
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GoranSMilovanovic
Cc: WMDE-leszek, abian, leila, Ottomata, Nuria,
JAllemandou added a comment.
this is done @GoranSMilovanovic.
Raw data is here
`/user/joal/wmf/data/raw/mediawiki/wikidata/all_jsondumps/20190902` and parquet
data is here `/user/joal/wmf/data/wmf/mediawiki/wikidata_parquet/20190902`
TASK DETAIL
https://phabricator.wikimedia.org/T209655
GoranSMilovanovic added a comment.
@JAllemandou Would it be possible to have another update (beyond the most
recent `20190603`) of the dump in hdfs?
I would like to present some of the analytical systems based on this in the
WikidataCon 2019, and would be **very, very grateful** if a new
JAllemandou added a comment.
@GoranSMilovanovic : You're welcome :) At some point I'll manage to have that
productionize ;)
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: JAllemandou
Cc:
GoranSMilovanovic added a comment.
@JAllemandou Thanks for the recent `20190603` dump copy in HDFS.
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GoranSMilovanovic
Cc: abian, leila,
Nuria added a comment.
@abian : this is still not happening on a recurrent schedule yet.
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Nuria
Cc: abian, leila, Ottomata, Nuria,
abian added a comment.
@JAllemandou, do you think this is now unblocked?
TASK DETAIL
https://phabricator.wikimedia.org/T209655
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: abian
Cc: abian, leila, Ottomata, Nuria, GoranSMilovanovic, Addshore,
Nuria added a comment.
Having missed most of goals this quarter due to our mw woes i think this might need to be moved to next quarter (q4?)TASK DETAILhttps://phabricator.wikimedia.org/T209655EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: Ottomata,
bmansurov added a comment.
Thanks, @JAllemandou! The more recent dumps are very useful.TASK DETAILhttps://phabricator.wikimedia.org/T209655EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: bmansurovCc: GoranSMilovanovic, Addshore, JAllemandou, bmansurov,
24 matches
Mail list logo
