Automation framework evaluation: Difference between revisions

From OpenSFS Wiki
Jump to navigation Jump to search
No edit summary
Line 1: Line 1:
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
Some requirements/desires for the automation framework:
* Aware of multiple clusters
* Able to create virtual clusters as VMs
* Able to automatically start testing based on various triggers, e.g. git commit hooks
* Maintains a prioritizable job queue
* Collect test output status in a database
* Visually represent pass/failure in a clear, concise manner
* Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results


{| class="wikitable"
{| class="wikitable"
Line 5: Line 19:
|-
|-
| [https://github.com/autotest/autotest autotest]  
| [https://github.com/autotest/autotest autotest]  
It is designed primarily totest the Linux kernel, though it is useful for many other  
It is designed primarily to test the Linux kernel, though it is useful for many other  
functions such asqualifying new hardware. It used and developed by a number of  
functions such asqualifying new hardware. It used and developed by a number of  
organizations, including Google, IBM, Red Hat, andmany others.
organizations, including Google, IBM, Red Hat, andmany others.
Line 25: Line 39:
|-
|-
| [http://staf.sourceforge.net/ STAF]  
| [http://staf.sourceforge.net/ STAF]  
The Software Testing Automation Framework (STAF) is an open source, multi-platform,multi-language framework, lead by IBM
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM
Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt.  
Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt.  
||  
||  
* multi-platform, works on Windows, Linux, AIX
* multi-platform: works on Windows, Linux, AIX
...
...
  ||  
  ||  
* doesn't ready for linux kernel specific functionality
* isn't ready for linux kernel-specific functionality
...
...
|-
|-
| [http://code.google.com/p/robotframework/ robotframework] || Example || Example
| [http://code.google.com/p/robotframework/ robotframework] || Example || Example
|}
|}

Revision as of 15:06, 19 December 2012

In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.

Some requirements/desires for the automation framework:

  • Aware of multiple clusters
  • Able to create virtual clusters as VMs
  • Able to automatically start testing based on various triggers, e.g. git commit hooks
  • Maintains a prioritizable job queue
  • Collect test output status in a database
  • Visually represent pass/failure in a clear, concise manner
  • Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results



Framework(with link) Advantages Disadvantages
autotest

It is designed primarily to test the Linux kernel, though it is useful for many other functions such asqualifying new hardware. It used and developed by a number of organizations, including Google, IBM, Red Hat, andmany others. Developed on Python

  • ready for work linux kernel, supports some logging services and crashdump
  • under active developemnt
  • could work in "client" mode with minimum setup

in server mode:

  • ready to work on sets of nodes and selecting nodes based on marks/lables
  • have jobs queue supports
  • very limited test selection, include list, exclude list, tagging support
  • have pretty simple web UI
  • test structure pretty complex
STAF

The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt.

  • multi-platform: works on Windows, Linux, AIX

...

  • isn't ready for linux kernel-specific functionality

...

robotframework Example Example