[
https://issues.apache.org/jira/browse/HIVE-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046730#comment-17046730
]
Zoltan Haindrich commented on HIVE-22942:
-----------------------------------------
How it works right now:
* we run a [job on the ASF jenkins|
instancehttps://builds.apache.org/job/PreCommit-HIVE-Build/] which logs into
some cloud instance to launch the ptest execution
* the ptest uses a predefined number of executors(16?)
* the tests are batched by a custom logic into ~200 batches
* every executor runs 2 batches at a time
* there are some specially tailored features; like timeout at batch level and a
way to run something in "isolation"
Right now I think the following would be the most promising:
* drop in something else for make use of the [parallel-test-executor plugin for
jenkins|https://plugins.jenkins.io/parallel-test-executor/]
* it basically works by scanning the last result and it makes around equally
sized test groups - and runs that...however it is unable to work if there are
testcases which run for more time than the bucket size.... this could be
probably explored by shoveling in some logic to split the larger cases into
~30m parts
* creating a job which utilizes the plugin is quite straight forward; so adding
all the executors as slaves to a jenkins will be able to utilize the same
compute power
> Replace PTest with an alternative
> ---------------------------------
>
> Key: HIVE-22942
> URL: https://issues.apache.org/jira/browse/HIVE-22942
> Project: Hive
> Issue Type: Improvement
> Reporter: Zoltan Haindrich
> Assignee: Zoltan Haindrich
> Priority: Major
>
> I never opened a jira about this...but it might actually help collect ideas
> and actually start going somewhere sooner than later :D
> Right now we maintain the ptest2 project inside Hive to be able to run Hive
> tests in a distributed fashion...the backstab of this solution is that we are
> putting much effort into maintaining a distributed test execution framework...
> I think it would be better if we could find an off the shelf solution for the
> task and migrate to that instead of putting more efforts into the ptest
> framework
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
