EBernhardson has uploaded a new change for review.
https://gerrit.wikimedia.org/r/266556
Change subject: Add field-name and noop-within parameters to transfer_to_es
workflow
......................................................................
Add field-name and noop-within parameters to transfer_to_es workflow
These values will be needed to update the correct properties. Also
changed default executors back to 3, it turns out the bump in
latencies wasn't caused by indexing and was only a momentary blip.
Letting the indexing run overnight showed no appreciable increase
in search latency.
Change-Id: I26195de6acedb020a00c3691d5672c5385738e7f
---
M oozie/transfer_to_es/bundle.properties
M oozie/transfer_to_es/workflow.xml
2 files changed, 5 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/wikimedia/discovery/analytics
refs/changes/56/266556/1
diff --git a/oozie/transfer_to_es/bundle.properties
b/oozie/transfer_to_es/bundle.properties
index 8fe3dc9..8c9286e 100644
--- a/oozie/transfer_to_es/bundle.properties
+++ b/oozie/transfer_to_es/bundle.properties
@@ -58,7 +58,7 @@
# when spark alocates this resource. This partially controls the amount
# of concurrency between the hadoop cluster and the elasticsearch cluster
# during export.
-spark_number_executors = 1
+spark_number_executors = 3
spark_executor_memory = 1G
spark_driver_memory = 1G
diff --git a/oozie/transfer_to_es/workflow.xml
b/oozie/transfer_to_es/workflow.xml
index 43e967f..66ded00 100644
--- a/oozie/transfer_to_es/workflow.xml
+++ b/oozie/transfer_to_es/workflow.xml
@@ -132,6 +132,10 @@
<arg>${transfer_to_es_batch_size}</arg>
<arg>--hostmap</arg>
<arg>${transfer_to_es_hostmap_file}</arg>
+ <arg>--noop-within</arg>
+ <arg>20</arg>
+ <arg>--field-name</arg>
+ <arg>popularity_score</arg>
</spark>
<ok to="end"/>
--
To view, visit https://gerrit.wikimedia.org/r/266556
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I26195de6acedb020a00c3691d5672c5385738e7f
Gerrit-PatchSet: 1
Gerrit-Project: wikimedia/discovery/analytics
Gerrit-Branch: master
Gerrit-Owner: EBernhardson <ebernhard...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
