JAllemandou added a comment.
Code is ready: - Import `commons-mediainfo` json dumps to HDFS (https://gerrit.wikimedia.org/r/738874) - Update spark transformation job to work with both wikidata and commons dumps (https://gerrit.wikimedia.org/r/739129) - Update `wikidata_entity` table creation script and oozie job for the new fields added by the patch above (https://gerrit.wikimedia.org/r/c/analytics/refinery/+/740589) - Add `commons_entoty` table creation script (https://gerrit.wikimedia.org/r/c/analytics/refinery/+/740590) - Update spark transformation job to write directly to a hive table instead of to files (https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/747508/) What we need after having merged/deployed the above is: - A new airflow job for the `commons_entity` data genration - A migration of the `wikidata_entity` oozie job to Airflow TASK DETAIL https://phabricator.wikimedia.org/T258834 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JAllemandou Cc: AKhatun_WMF, JAllemandou, cchen, Nuria, Miriam, nettrom_WMF, 786, EChetty, Suran38, Biggs657, toberto, ldelench_wmf, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, Base, aude, Tobias1984, Manybubbles, Mbch331, jeremyb
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
