Automation framework evaluation: Difference between revisions
m (Nrutman moved page Existing test framework evaluation to Automation framework evaluation) |
No edit summary |
||
Line 1: | Line 1: | ||
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure. | |||
Some requirements/desires for the automation framework: | |||
* Aware of multiple clusters | |||
* Able to create virtual clusters as VMs | |||
* Able to automatically start testing based on various triggers, e.g. git commit hooks | |||
* Maintains a prioritizable job queue | |||
* Collect test output status in a database | |||
* Visually represent pass/failure in a clear, concise manner | |||
* Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results | |||
{| class="wikitable" | {| class="wikitable" | ||
Line 5: | Line 19: | ||
|- | |- | ||
| [https://github.com/autotest/autotest autotest] | | [https://github.com/autotest/autotest autotest] | ||
It is designed primarily | It is designed primarily to test the Linux kernel, though it is useful for many other | ||
functions such asqualifying new hardware. It used and developed by a number of | functions such asqualifying new hardware. It used and developed by a number of | ||
organizations, including Google, IBM, Red Hat, andmany others. | organizations, including Google, IBM, Red Hat, andmany others. | ||
Line 25: | Line 39: | ||
|- | |- | ||
| [http://staf.sourceforge.net/ STAF] | | [http://staf.sourceforge.net/ STAF] | ||
The Software Testing Automation Framework (STAF) is an open source, multi-platform,multi-language framework, lead by IBM | The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM | ||
Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. | Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. | ||
|| | || | ||
* multi-platform | * multi-platform: works on Windows, Linux, AIX | ||
... | ... | ||
|| | || | ||
* | * isn't ready for linux kernel-specific functionality | ||
... | ... | ||
|- | |- | ||
| [http://code.google.com/p/robotframework/ robotframework] || Example || Example | | [http://code.google.com/p/robotframework/ robotframework] || Example || Example | ||
|} | |} |
Revision as of 15:06, 19 December 2012
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
Some requirements/desires for the automation framework:
- Aware of multiple clusters
- Able to create virtual clusters as VMs
- Able to automatically start testing based on various triggers, e.g. git commit hooks
- Maintains a prioritizable job queue
- Collect test output status in a database
- Visually represent pass/failure in a clear, concise manner
- Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results
Framework(with link) | Advantages | Disadvantages |
---|---|---|
autotest
It is designed primarily to test the Linux kernel, though it is useful for many other functions such asqualifying new hardware. It used and developed by a number of organizations, including Google, IBM, Red Hat, andmany others. Developed on Python |
in server mode:
|
|
STAF
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. |
... |
... |
robotframework | Example | Example |