Automation framework evaluation: Difference between revisions
Jump to navigation
Jump to search
(Add link to Xperior presentation at LAD'12.) |
No edit summary |
||
| Line 28: | Line 28: | ||
* under active developemnt | * under active developemnt | ||
* could work in "client" mode with minimum setup | * could work in "client" mode with minimum setup | ||
* | * has good internal unit test coverage | ||
in server mode: | in server mode: | ||
* ready to work on sets of nodes and selecting nodes based on marks/lables | * ready to work on sets of nodes and selecting nodes based on marks/lables | ||
* | * has jobs queue supports | ||
* | * has a basic web UI and test result database | ||
|| | || | ||
* very limited test selection, include list, exclude list, tagging support | * very limited test selection, include list, exclude list, tagging support | ||
* | * has pretty simple web UI | ||
* test structure pretty complex | * test structure pretty complex | ||
* | * in server mode doesn't configuration support multinode execution where hosts plays different roles, need configuration improving | ||
|- | |- | ||
| [http://staf.sourceforge.net/ STAF] | | [http://staf.sourceforge.net/ STAF] | ||
| Line 46: | Line 46: | ||
* multi-platform: works on Windows, Linux, AIX | * multi-platform: works on Windows, Linux, AIX | ||
* test could be written on many languages | * test could be written on many languages | ||
* | * has interface for for remote work with server | ||
* | * has security levels | ||
|| | || | ||
* isn't ready for linux kernel-specific functionality and kernel crashes (maybe special proxies or services are exist) | * isn't ready for linux kernel-specific functionality and kernel crashes (maybe special proxies or services are exist) | ||
* | * doesn't mean work in clusters, need improving configuration | ||
* | * doesn't have test management level | ||
|- | |- | ||
| [http://code.google.com/p/robotframework/ robotframework] || Example || Example | | [http://code.google.com/p/robotframework/ robotframework] || Example || Example | ||
| Line 60: | Line 60: | ||
Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12. | Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12. | ||
|| | || | ||
* | * has some knowledge about current luste tests structure | ||
* compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout) | * compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout) | ||
* could be extended for new executors | * could be extended for new executors | ||
* test result is per-test yaml file and html report could be generated | * test result is per-test yaml file and html report could be generated | ||
* could collect some logs from systems under tests on per-test basis, and this could be simple extended | * could collect some logs from systems under tests on per-test basis, and this could be simple extended | ||
* | * has good internal unit test coverage | ||
|| | || | ||
* doesn't directly maintain nodes, lustre mounts, cluster status | * doesn't directly maintain nodes, lustre mounts, cluster status | ||
|- | |- | ||
|} | |} | ||
Revision as of 12:00, 18 January 2013
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
Some requirements/desires for the automation framework:
- Aware of multiple clusters
- Able to create virtual clusters as VMs
- Able to automatically start testing based on various triggers, e.g. git commit hooks
- Maintains a prioritizable job queue
- Collect test output status in a database
- Visually represent pass/failure in a clear, concise manner
- Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results
| Framework(with link) | Advantages | Disadvantages | |
|---|---|---|---|
| autotest
It is designed primarily to test the Linux kernel, though it is useful for many other functions such asqualifying new hardware. It used and developed by a number of organizations, including Google, IBM, Red Hat, andmany others. Developed on Python |
in server mode:
|
| |
| STAF
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. |
|
| |
| robotframework | Example | Example | |
| Xperior
Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More info was presented at LAD '12. |
|
|