Automation framework evaluation: Difference between revisions

From OpenSFS Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 46: Line 46:
* multi-platform: works on Windows, Linux, AIX
* multi-platform: works on Windows, Linux, AIX
* test could be written on many languages
* test could be written on many languages
* has interface for for remote work with server
* has UI interface for for remote work with server
* has security levels
* has security levels
  ||  
  ||  
Line 67: Line 67:
* has good internal unit test coverage
* has good internal unit test coverage
  ||  
  ||  
* doesn't directly maintain nodes, lustre mounts, cluster status
* doesn't maintain directly nodes, lustre mounts, cluster status
* no security
|-
|-
|}
|}

Revision as of 12:10, 18 January 2013

In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.

Some requirements/desires for the automation framework:

  • Aware of multiple clusters
  • Able to create virtual clusters as VMs
  • Able to automatically start testing based on various triggers, e.g. git commit hooks
  • Maintains a prioritizable job queue
  • Collect test output status in a database
  • Visually represent pass/failure in a clear, concise manner
  • Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results



Framework(with link) Advantages Disadvantages
autotest

It is designed primarily to test the Linux kernel, though it is useful for many other functions such asqualifying new hardware. It used and developed by a number of organizations, including Google, IBM, Red Hat, andmany others. Developed on Python

  • ready for work linux kernel, supports some logging services and crashdump
  • under active developemnt
  • could work in "client" mode with minimum setup
  • has good internal unit test coverage

in server mode:

  • ready to work on sets of nodes and selecting nodes based on marks/lables
  • has jobs queue supports
  • has a basic web UI and test result database
  • very limited test selection, include list, exclude list, tagging support
  • has pretty simple web UI
  • test structure pretty complex
  • in server mode doesn't support configuration for multinode execution where hosts plays different roles, need configuration improving
STAF

The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt.

  • multi-platform: works on Windows, Linux, AIX
  • test could be written on many languages
  • has UI interface for for remote work with server
  • has security levels
  • doesn't have linux kernel-specific and kernel crashes functionality(maybe special proxies or services are exist)
  • doesn't support configuration for multinode execution where hosts plays different roles, need configuration improving
  • doesn't have test management level
robotframework Example Example
Xperior

Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More info was presented at LAD '12.

  • has some knowledge about current luste tests structure
  • compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout)
  • could be extended for new executors
  • test result is per-test yaml file and html report could be generated
  • could collect some logs from systems under tests on per-test basis, and this could be simple extended
  • has good internal unit test coverage
  • doesn't maintain directly nodes, lustre mounts, cluster status
  • no security