Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16155
to look at the new patch set (#2).
Change subject: IMPALA-9887: Add support for sharding end-to-end tests
......................................................................
IMPALA-9887: Add support for sharding end-to-end tests
ASAN maintains stacks for each allocation and free of memory. Impala
sometimes allocates/frees memory from codegen'd code, so this means
that the number of distinct stacks is unbounded. ASAN is storing
these stacks in a hash table with a fixed number of buckets (one million).
As the stacks accumulate, allocations and frees get slower and slower,
because the lookup in this hashtable gets slower. This causes test
execution time to degrade over time. Since backend tests and custom cluster
tests don't have long running daemons, only the end to end tests are
affected.
This adds support for breaking end-to-end test execution into shards,
restarting Impala between each shard. The number of shards is configurable
via the EE_TEST_SHARDS environment variable. By default, EE_TEST_SHARDS=1
and no sharding is used.
Without sharding, an ASAN core job takes about 16-17 hours. With 6 shards,
it takes about 9 hours. It is recommended to always use sharding with ASAN.
Testing:
- Ran core job
- Ran ASAN with EE_TEST_SHARDS=6
Change-Id: I0bdbd79940df2bc7b951efdf0f044e6b40a3fda9
---
M bin/run-all-tests.sh
M tests/run-tests.py
2 files changed, 64 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/16155/2
--
To view, visit http://gerrit.cloudera.org:8080/16155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0bdbd79940df2bc7b951efdf0f044e6b40a3fda9
Gerrit-Change-Number: 16155
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
