Difference between revisions of "Automation framework evaluation"
Line 53: | Line 53: | ||
* doesn't have test management level | * doesn't have test management level | ||
|- | |- | ||
− | | [http://code.google.com/p/robotframework/ robotframework] || | + | | [http://code.google.com/p/robotframework/ robotframework] |
+ | Robot Framework is a python-based an open source generic test framework. | ||
+ | || | ||
+ | * General purpose testing framework. | ||
+ | * Supports multiple testing methodologies, such as data-driven test cases | ||
+ | * provides a command line interface and XML based output files for integration into build infrastructure | ||
+ | * Has built-in support for variables, which is useful for testing in different environments | ||
+ | * provides tagging to categorize and select test cases to be run. | ||
+ | * provides test-case and test-suite level setup and teardowns | ||
+ | * provides an advanced keyword driven framework | ||
+ | * keyworks are a synonym of a "function" doing "something" | ||
+ | * Provides the ability to create reusable/custom higher-level keywords from the existing keywords | ||
+ | * Easily extendable by Python, C# & Java | ||
+ | || | ||
+ | * Doesn't run in parallel, but can run multiple instances to emulate it. | ||
| | | | ||
Revision as of 14:58, 18 January 2013
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
Some requirements/desires for the automation framework:
- Aware of multiple clusters
- Able to create virtual clusters as VMs
- Able to automatically start testing based on various triggers, e.g. git commit hooks
- Maintains a prioritizable job queue
- Collect test output status in a database
- Visually represent pass/failure in a clear, concise manner
- Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results
Framework(with link) | Advantages | Disadvantages | |
---|---|---|---|
autotest
It is designed primarily to test the Linux kernel, though it is useful for many other functions such asqualifying new hardware. It used and developed by a number of organizations, including Google, IBM, Red Hat, andmany others. Developed on Python |
in server mode:
|
| |
STAF
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. |
|
| |
robotframework
Robot Framework is a python-based an open source generic test framework. |
|
|
|
Xperior
Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More info was presented at LAD '12. |
|
|