Automation framework evaluation: Difference between revisions
No edit summary |
No edit summary |
||
(2 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure. | In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure. | ||
Some requirements/desires for the automation framework: | Some requirements/desires for the automation framework: | ||
* Aware of multiple clusters | * Aware of multiple clusters | ||
Line 28: | Line 28: | ||
| [https://github.com/autotest/autotest autotest] | | [https://github.com/autotest/autotest autotest] | ||
It is designed primarily to test the Linux kernel, though it is useful for many other | It is designed primarily to test the Linux kernel, though it is useful for many other | ||
functions such | functions such as qualifying new hardware. It used and developed by a number of | ||
organizations, including Google, IBM, Red Hat, | organizations, including Google, IBM, Red Hat, and many others. | ||
Developed on Python | Developed on Python | ||
[[File:autotest_sample_report.tar.bz2]] | |||
|| | || | ||
# Test have meta information but framework don't use it. | # Test have meta information but framework don't use it. | ||
Line 93: | Line 94: | ||
| [https://github.com/Xyratex/xperior Xperior] | | [https://github.com/Xyratex/xperior Xperior] | ||
Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12. | Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12. | ||
[[File:report_sample.zip]] | |||
|| | || | ||
# test could be described, about 10 fields now are used. | # test could be described, about 10 fields now are used. |
Latest revision as of 12:01, 24 January 2013
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
Some requirements/desires for the automation framework:
- Aware of multiple clusters
- Able to create virtual clusters as VMs
- Able to automatically start testing based on various triggers, e.g. git commit hooks
- Maintains a prioritizable job queue
- Collect test output status in a database
- Visually represent pass/failure in a clear, concise manner
- Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results
Framework(with link) | Feature list
|
More advantages | More disadvantages | |
---|---|---|---|---|
autotest
It is designed primarily to test the Linux kernel, though it is useful for many other functions such as qualifying new hardware. It used and developed by a number of organizations, including Google, IBM, Red Hat, and many others. Developed on Python File:Autotest sample report.tar.bz2 |
|
in server mode:
|
| |
STAF
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. Strictly says, it is not a framework but set of services and libraries for making own test |
|
|
| |
robotframework
Robot Framework is a python-based an open source generic test framework. |
|
|
||
Xperior
Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More info was presented at LAD '12. |
|
|
|