dr0ptp4kt added a comment.
I kicked off a run using the current version of the patch with the following command and backing table, and its status should be able to be followed here: https://yarn.wikimedia.org/cluster/app/application_1713178047802_16409 So long as I haven't made an error somewhere in here that produces a runtime exception (e.g., pathing), we should be able to see after a couple hours how it's going. spark3-submit --master yarn --driver-cores 2 --conf spark.sql.autoBroadcastJoinThreshold=-1 --conf spark.dynamicAllocation.maxExecutors=128 --conf spark.sql.shuffle.partitions=512 --conf spark.executor.memoryOverhead=4g --executor-cores 4 --executor-memory 12g --driver-memory 16g --name scholarly_article_split_manual__scholarly_article_split_triples__T362060_personal_namespace --conf spark.yarn.maxAppAttempts=1 --class org.wikidata.query.rdf.spark.transform.structureddata.dumps.ScholarlyArticleSplit --deploy-mode cluster /home/dr0ptp4kt/rdf-spark-tools-0.3.138-SNAPSHOT-jar-with-dependencies-T362060.jar --input-table-partition-spec discovery.wikibase_rdf_t337013/date=20231016/wiki=wikidata --output-table-partition-spec dr0ptp4kt.wikibase_rdf_scholarly_split_T362060/snapshot=20231016/wiki=wikidata Here was the manual table creation I did while `use`ing the `dr0ptp4kt` namespace. CREATE TABLE IF NOT EXISTS dr0ptp4kt.wikibase_rdf_scholarly_split_T362060 ( `subject` string, `predicate` string, `object` string, `context` string ) PARTITIONED BY ( `snapshot` string, `wiki` string, `scope` string ) STORED AS PARQUET LOCATION 'hdfs://analytics-hadoop/user/dr0ptp4kt/wikibase_rdf_scholarly_split_T362060/wikidata/rdf_scholarly_split_T362060/' ; TASK DETAIL https://phabricator.wikimedia.org/T362060 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dr0ptp4kt Cc: dr0ptp4kt, dcausse, Aklapper, Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 786, Biggs657, karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, EBjune, KimKelting, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
