[
https://issues.apache.org/jira/browse/HADOOP-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759393#action_12759393
]
Todd Lipcon commented on HADOOP-6248:
-------------------------------------
Hey Alex,
Here are a couple ideas I think would make Circus a more useful tool beyond
being a simple wrapper for scripts:
- *Automatic log analysis* - if it can collect the logs from the daemons
involved in a test and turn out a report of any WARN or ERROR level messages,
that would be very useful. With this I think each test would need some kind of
whitelist ability for expected errors (eg a test of write pipeline failure
would expect warnings that the connection had been lost). This is starting to
tread on territory of other contrib projects, so I think the differentiator
would have to be that it's clearly aimed towards testing and scaling to
historical data is not within scope.
- *Automatic performance metrics* - the simplest of this would be keeping track
of the wall clock time for the tests. This would help identify performance
regressions on a macro scale. Additionally, one advantage of not using Java is
that you can easily wrap the daemons themselves with the "time" command to keep
track of CPU time, sys time, and wall clock time separately and reliably. In a
similar vein, you could add hooks for running all of the daemons or tasks
inside a profiler.
- *Automatic configuration parameter testing* - users (and developers) often go
through a workflow of Configure -> Deploy -> Test -> Reconfigure -> Deploy ->
etc. Being able to very easily write a script that can run a given circus test
with bracketed configurations and dump the results in a standard (eg TSV)
format would be incredibly useful.
That said, I agree that it's best to solicit feedback early and include this as
a contrib project even in its current state. Though it currently does little
beyond what a skilled user can do with bash, it holds a lot of promise for
future extension. Getting users in the habit of developing their system tests
within this framework now will benefit everyone down the road when some of the
above features have been implemented.
> Circus: Proposal and Preliminary Code for a Hadoop System Testing Framework
> ---------------------------------------------------------------------------
>
> Key: HADOOP-6248
> URL: https://issues.apache.org/jira/browse/HADOOP-6248
> Project: Hadoop Common
> Issue Type: New Feature
> Components: test
> Environment: Python, bash
> Reporter: Alex Loddengaard
> Attachments: HADOOP-6248.diff, HADOOP-6248_v2.diff,
> HADOOP-6248_v3.diff
>
>
> This issue contains a proposal and preliminary source code for Circus, a
> Hadoop system testing framework. At a high level, Circus will help Hadoop
> users and QA engineers to run system tests on a configurable Hadoop cluster,
> or distribution of Hadoop. See the comment below for the proposal itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
