Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/3830/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Xikui Wang Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Xikui Wang has submitted this change and it was merged.
Change subject: [NO ISSUE][TEST] Add import order to perf test
..
[NO ISSUE][TEST] Add import order to perf test
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
1. Add data file removal after import to save space.
2. Add import order to import small datasets first.
Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68
Reviewed-on: https://asterix-gerrit.ics.uci.edu/1965
Reviewed-by: Till Westmann
Tested-by: Till Westmann
---
M asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
M
asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
M
asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
M asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/run_sparksql.sh
4 files changed, 15 insertions(+), 13 deletions(-)
Approvals:
Till Westmann: Looks good to me, approved; Verified
diff --git
a/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
b/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
index ff22a7d..9835be9 100644
---
a/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
+++
b/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
@@ -28,8 +28,8 @@
include: loadgen.yml
- name: Load all datasets
- shell: "ansible-playbook -i {{ inventory }} {{ runquery }}
--extra-vars=\"query_file={{ dml }} report=false\""
- with_fileglob:
- - "{{ dml_dir }}/*"
+ shell: "ansible-playbook -i {{ inventory }} {{ runquery }}
--extra-vars=\"query_file={{ dml_dir }}/load{{ dataset }}.sqlpp report=false\";
+ ansible ncs -i {{ inventory }} -m shell -a \"rm -f {{ data_dir }}/{{
dataset | lower}}.tbl*\""
+ with_items: "{{ dataset_import_order }}"
loop_control:
- loop_var: dml
+ loop_var: dataset
diff --git
a/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
b/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
index 17a3638..4f3030e 100644
---
a/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
+++
b/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
@@ -23,6 +23,8 @@
# Datasets with a single file.
centralized_datasets: ["Region", "Nation"]
+dataset_import_order: ["Region", "Nation", "Supplier", "Part", "Customer",
"Partsupp", "Orders", "LineItem"]
+
# Temp directory.
temp_dir: "/tmp/asterixdb"
diff --git
a/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
b/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
index ea635f4..5b07eed 100644
---
a/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
+++
b/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
@@ -30,4 +30,4 @@
- include_vars: "{{ playbook_dir }}/../conf/private_ip.yml"
- include_vars: ../../../benchmarks/tpch/gen/settings.yml
- name: Put data on to HDFS
- shell: for i in `ls -S {{ data_dir }}/`; do {{ hadoop_base }}/bin/hdfs
dfs -put -f {{ data_dir }}/$i hdfs://{{ cc_ip }}:9000/{{ data_dir }}; rm {{
data_dir }}/$i; done
\ No newline at end of file
+ shell: for i in `ls -rS {{ data_dir }}/`; do {{ hadoop_base }}/bin/hdfs
dfs -put -f {{ data_dir }}/$i hdfs://{{ cc_ip }}:9000/{{ data_dir }}; rm {{
data_dir }}/$i; done
\ No newline at end of file
diff --git
a/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/run_sparksql.sh
b/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/run_sparksql.sh
index 9b14f67..18d4962 100755
---
a/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/run_sparksql.sh
+++
b/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/run_sparksql.sh
@@ -45,14 +45,14 @@
exit 1
fi
# Load data
-ansible-playbook -i $INVENTORY $SCRIPT_PATH/../../benchmarks/tpch/gen/gen.yml
+ansible-playbook -i $INVENTORY $SCRIPT_PATH/../../benchmarks/tpch/gen/gen.yml
# Configure HDFS
-ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/install_hdfs.yml
-ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/start_hdfs.yml
+ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/install_hdfs.yml
+ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/start_hdfs.yml
# Configure Spark
-ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/install_spark.yml
-ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/start_spark.yml
-ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/load_tpch.yml
+ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/install_spark.yml
+ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/start_spark.yml
+ansible-playbook -i $INVENTORY $SCRIPT_PATH/ansible/load_tpch.yml
#
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: BAD Compatibility Tests Started https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/1689/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Till Westmann has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Verified+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Till Westmann has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/3830/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/1489/ (6/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6305/ (4/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/1429/ (8/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/1965 to look at the new patch set (#3). Change subject: [NO ISSUE][TEST] Add import order to perf test .. [NO ISSUE][TEST] Add import order to perf test - user model changes: no - storage format changes: no - interface changes: no Details: 1. Add data file removal after import to save space. 2. Add import order to import small datasets first. Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 --- M asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml M asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml M asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml M asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/run_sparksql.sh 4 files changed, 15 insertions(+), 13 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/65/1965/3 -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/1086/ (1/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/4824/ (11/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/751/ (10/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/749/ (9/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-rebalance-cancellation/572/ (3/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/1669/ (5/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/746/ (7/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/1366/ (2/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/3829/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/1488/ (7/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6304/ (1/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/1365/ (3/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/745/ (4/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/1668/ (8/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/1428/ (5/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/1085/ (11/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/4823/ (6/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-rebalance-cancellation/571/ (10/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/750/ (2/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Xikui Wang has uploaded a new patch set (#2). Change subject: [NO ISSUE][TEST] Add import order to perf test .. [NO ISSUE][TEST] Add import order to perf test - user model changes: no - storage format changes: no - interface changes: no Details: 1. Add data file removal after import to save space. 2. Add import order to import small datasets first. Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 --- M asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml M asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml M asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml M asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/run_sparksql.sh 4 files changed, 15 insertions(+), 13 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/65/1965/2 -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/748/ (9/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/3828/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Xikui Wang has uploaded a new change for review.
https://asterix-gerrit.ics.uci.edu/1965
Change subject: [NO ISSUE][TEST] Add import order to perf test
..
[NO ISSUE][TEST] Add import order to perf test
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
1. Add data file removal after import to save space.
2. Add import order to import small datasets first.
Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68
---
M asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
M
asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
M
asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
3 files changed, 7 insertions(+), 5 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/65/1965/1
diff --git
a/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
b/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
index ff22a7d..9835be9 100644
---
a/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
+++
b/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/load.yml
@@ -28,8 +28,8 @@
include: loadgen.yml
- name: Load all datasets
- shell: "ansible-playbook -i {{ inventory }} {{ runquery }}
--extra-vars=\"query_file={{ dml }} report=false\""
- with_fileglob:
- - "{{ dml_dir }}/*"
+ shell: "ansible-playbook -i {{ inventory }} {{ runquery }}
--extra-vars=\"query_file={{ dml_dir }}/load{{ dataset }}.sqlpp report=false\";
+ ansible ncs -i {{ inventory }} -m shell -a \"rm -f {{ data_dir }}/{{
dataset | lower}}.tbl*\""
+ with_items: "{{ dataset_import_order }}"
loop_control:
- loop_var: dml
+ loop_var: dataset
diff --git
a/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
b/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
index 17a3638..4f3030e 100644
---
a/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
+++
b/asterixdb/asterix-benchmark/src/main/resources/benchmarks/tpch/load/settings.yml
@@ -23,6 +23,8 @@
# Datasets with a single file.
centralized_datasets: ["Region", "Nation"]
+dataset_import_order: ["Region", "Nation", "Supplier", "Part", "Customer",
"Partsupp", "Orders", "LineItem"]
+
# Temp directory.
temp_dir: "/tmp/asterixdb"
diff --git
a/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
b/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
index ea635f4..5b07eed 100644
---
a/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
+++
b/asterixdb/asterix-benchmark/src/main/resources/others/SparkSQL/ansible/load_tpch.yml
@@ -30,4 +30,4 @@
- include_vars: "{{ playbook_dir }}/../conf/private_ip.yml"
- include_vars: ../../../benchmarks/tpch/gen/settings.yml
- name: Put data on to HDFS
- shell: for i in `ls -S {{ data_dir }}/`; do {{ hadoop_base }}/bin/hdfs
dfs -put -f {{ data_dir }}/$i hdfs://{{ cc_ip }}:9000/{{ data_dir }}; rm {{
data_dir }}/$i; done
\ No newline at end of file
+ shell: for i in `ls -rS {{ data_dir }}/`; do {{ hadoop_base }}/bin/hdfs
dfs -put -f {{ data_dir }}/$i hdfs://{{ cc_ip }}:9000/{{ data_dir }}; rm {{
data_dir }}/$i; done
\ No newline at end of file
--
To view, visit https://asterix-gerrit.ics.uci.edu/1965
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6303/ (11/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/1364/ (1/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/744/ (10/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/4822/ (2/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/1084/ (6/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/1667/ (4/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-rebalance-cancellation/570/ (8/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/747/ (5/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/1427/ (7/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/1487/ (3/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][TEST] Add import order to perf test
Jenkins has posted comments on this change. Change subject: [NO ISSUE][TEST] Add import order to perf test .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/749/ (9/11) -- To view, visit https://asterix-gerrit.ics.uci.edu/1965 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6ce4e55d4fbd5e082e9f01c526f258891574cb68 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
