Automation framework evaluation: Difference between revisions
No edit summary |
No edit summary |
||
Line 57: | Line 57: | ||
Xperior is an open source framework developed on perl by Xyratex for executing lustre tests from current acceptance small shell-based testing framework. | Xperior is an open source framework developed on perl by Xyratex for executing lustre tests from current acceptance small shell-based testing framework. | ||
|| | || | ||
* have some knowledge about current luste tests structure | |||
* compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout) | |||
* could be extended for new executors | * could be extended for new executors | ||
* test result is per-test yaml file and html report could be generated | * test result is per-test yaml file and html report could be generated | ||
* could collect some logs from systems under tests on per-test basis, and this could be simple extended | * could collect some logs from systems under tests on per-test basis, and this could be simple extended | ||
|| | || |
Revision as of 13:20, 6 January 2013
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
Some requirements/desires for the automation framework:
- Aware of multiple clusters
- Able to create virtual clusters as VMs
- Able to automatically start testing based on various triggers, e.g. git commit hooks
- Maintains a prioritizable job queue
- Collect test output status in a database
- Visually represent pass/failure in a clear, concise manner
- Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results
Framework(with link) | Advantages | Disadvantages | |
---|---|---|---|
autotest
It is designed primarily to test the Linux kernel, though it is useful for many other functions such asqualifying new hardware. It used and developed by a number of organizations, including Google, IBM, Red Hat, andmany others. Developed on Python |
in server mode:
|
| |
STAF
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. |
... |
... | |
robotframework | Example | Example | |
Xperior
Xperior is an open source framework developed on perl by Xyratex for executing lustre tests from current acceptance small shell-based testing framework. |
|
|