Automation framework evaluation: Difference between revisions
No edit summary |
No edit summary |
||
Line 16: | Line 16: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! Framework(with link) !! | ! Framework(with link) !! | ||
Feature list | |||
* Do tests describe intent clearly? | |||
* Is system information gathered? | |||
* Do results readable by man and computer? | |||
!! More advantages !! More disadvantages | |||
|- | |- | ||
| [https://github.com/autotest/autotest autotest] | | [https://github.com/autotest/autotest autotest] | ||
Line 23: | Line 29: | ||
organizations, including Google, IBM, Red Hat, andmany others. | organizations, including Google, IBM, Red Hat, andmany others. | ||
Developed on Python | Developed on Python | ||
|| | |||
|| | || | ||
* ready for work linux kernel, supports some logging services and crashdump | * ready for work linux kernel, supports some logging services and crashdump | ||
Line 43: | Line 49: | ||
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM | The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM | ||
Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. | Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. | ||
|| | |||
|| | || | ||
* multi-platform: works on Windows, Linux, AIX | * multi-platform: works on Windows, Linux, AIX | ||
Line 55: | Line 62: | ||
| [http://code.google.com/p/robotframework/ robotframework] | | [http://code.google.com/p/robotframework/ robotframework] | ||
Robot Framework is a python-based an open source generic test framework. | Robot Framework is a python-based an open source generic test framework. | ||
|| | |||
|| | |||
* General purpose testing framework. | * General purpose testing framework. | ||
* Supports multiple testing methodologies, such as data-driven test cases | * Supports multiple testing methodologies, such as data-driven test cases | ||
Line 73: | Line 81: | ||
| [https://github.com/Xyratex/xperior Xperior] | | [https://github.com/Xyratex/xperior Xperior] | ||
Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12. | Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12. | ||
|| | |||
|| | |||
* has some knowledge about current luste tests structure | * has some knowledge about current luste tests structure | ||
* compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout) | * compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout) |
Revision as of 08:27, 22 January 2013
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
Some requirements/desires for the automation framework:
- Aware of multiple clusters
- Able to create virtual clusters as VMs
- Able to automatically start testing based on various triggers, e.g. git commit hooks
- Maintains a prioritizable job queue
- Collect test output status in a database
- Visually represent pass/failure in a clear, concise manner
- Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results
Framework(with link) |
Feature list
|
! More advantages | More disadvantages | |
---|---|---|---|---|
autotest
It is designed primarily to test the Linux kernel, though it is useful for many other functions such asqualifying new hardware. It used and developed by a number of organizations, including Google, IBM, Red Hat, andmany others. Developed on Python |
in server mode:
|
| ||
STAF
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. |
|
| ||
robotframework
Robot Framework is a python-based an open source generic test framework. |
|
|
||
Xperior
Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More info was presented at LAD '12. |
|
|