Automation framework evaluation: Difference between revisions

From OpenSFS Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(24 intermediate revisions by 4 users not shown)
Line 1: Line 1:
In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
Some requirements/desires for the automation framework:
* Aware of multiple clusters
* Able to create virtual clusters as VMs
* Able to automatically start testing based on various triggers, e.g. git commit hooks
* Maintains a prioritizable job queue
* Collect test output status in a database
* Visually represent pass/failure in a clear, concise manner
* Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results


{| class="wikitable"
{| class="wikitable"
|-
|-
! Framework(with link) !! Advantages !! Disadvantages
! Framework(with link)  
! Feature list
# Do tests describe intent clearly?
# Is system information gathered?
# Do results readable by man and computer?
# Is run and develop process fast?
# Does it able to be run on cluster in parallel?
! More advantages
! More disadvantages
|-
|-
| [https://github.com/autotest/autotest autotest]  
| [https://github.com/autotest/autotest autotest]  
It is designed primarily totest the Linux kernel, though it is useful for many other functions such asqualifying new hardware. It's an open-source project under the GPL and is usedand developed by a number of organizations, including Google, IBM, Red Hat, andmany others.
It is designed primarily to test the Linux kernel, though it is useful for many other  
functions such as qualifying new hardware. It used and developed by a number of  
organizations, including Google, IBM, Red Hat, and many others.
Developed on Python
[[File:autotest_sample_report.tar.bz2]]
||
# Test have meta information but framework don't use it.
# yes, several types of sys info collected, include profiling and crahes
# yes, own text easy-to-parse text report, html report and possible tap. have own sql-based reporting base and limited UI for it.
# yes
# with some limited programming  - yes
||  
||  
* ready for work linux kernel, supports some logging services and crashdump
* ready for work linux kernel, supports some logging services and crashdump
* under active developemnt
* under active developemnt
* could work in "client" mode with minimum setup
* could work in "client" mode with minimum setup
* has good internal unit test coverage
in server mode:
in server mode:


* ready to work set of nodes and selecting nodes based on mark/lables
* ready to work on sets of nodes and selecting nodes based on marks/lables
* have jobs queue supports
* has jobs queue supports
* has a basic web UI and test result database
||  
||  
* very limited test selection, include list, exclude list, tagging support
* very limited test selection, include list, exclude list, tagging support
* have pretty simple web UI
* has pretty simple web UI
* test structure pretty complex
* test structure pretty complex
* in server mode doesn't support configuration for multinode execution where hosts plays different roles, need configuration improving
|-
| [http://staf.sourceforge.net/ STAF]
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM
Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. Strictly says, it is not a framework but set of services and libraries for making own test
||
# No test management provided
# few options about host are collected
#-
#-
#with some limited programming  - yes
||
* multi-platform: works on Windows, Linux, AIX
* test could be written on many languages
* has UI interface for for remote work with server
* has security levels
||
* doesn't have linux kernel-specific and kernel crashes functionality(maybe special proxies or services are exist)
* doesn't support configuration for multinode execution where hosts plays different roles, need configuration improving
* doesn't have test management level
|-
| [http://code.google.com/p/robotframework/ robotframework]
Robot Framework is a python-based an open source generic test framework.
||
||
* General purpose testing framework.
* Supports multiple testing methodologies, such as data-driven test cases
* provides a command line interface and XML based output files for integration into build infrastructure
* Has built-in support for variables, which is useful for testing in different environments
* provides tagging to categorize and select test cases to be run.
* provides test-case and test-suite level setup and teardowns
* provides an advanced keyword driven framework
* keyworks are a synonym of a "function" doing "something"
* Provides the ability to create reusable/custom higher-level keywords from the existing keywords
* Easily extendable by Python, C# & Java
||
* Doesn't run in parallel, but can run multiple instances to emulate it.
|


|-
|-
| [http://staf.sourceforge.net/ STAF] || Example || Example
| [https://github.com/Xyratex/xperior Xperior]
Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework.  More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.
 
[[File:report_sample.zip]]
||
# test could be described, about 10 fields now are used.
# only node status before run and lustre-diagnostic after test failure. Also there is few extension for reformat lustre after every tests, coverage collection, console log collection via netconsole, more could be easily added
# yes, yaml format of results and few convertors to junit, html and tap output documetns
# yes
# with external tool like jenkins - yes.
||
* have separated lustre configuration
* has some knowledge about current luste tests structure
* compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout)
* could be extended for new executors
* test result is per-test yaml file and html report could be generated
* could collect some logs from systems under tests on per-test basis, and this could be simple extended. Also there are script for uploading to mongo database.
* has good internal unit test coverage
* provides tagging to categorize and select test cases to be run. include/exclude lists
* provides test-case and test-suite level setup, properties are inherited from suite to tests
||  
* doesn't maintain directly nodes, lustre mounts, cluster status
* no security
|-
|-
| [http://code.google.com/p/robotframework/ robotframework] || Example || Example
|}
|}

Latest revision as of 12:01, 24 January 2013

In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.

Some requirements/desires for the automation framework:

  • Aware of multiple clusters
  • Able to create virtual clusters as VMs
  • Able to automatically start testing based on various triggers, e.g. git commit hooks
  • Maintains a prioritizable job queue
  • Collect test output status in a database
  • Visually represent pass/failure in a clear, concise manner
  • Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results



Framework(with link) Feature list
  1. Do tests describe intent clearly?
  2. Is system information gathered?
  3. Do results readable by man and computer?
  4. Is run and develop process fast?
  5. Does it able to be run on cluster in parallel?
More advantages More disadvantages
autotest

It is designed primarily to test the Linux kernel, though it is useful for many other functions such as qualifying new hardware. It used and developed by a number of organizations, including Google, IBM, Red Hat, and many others. Developed on Python File:Autotest sample report.tar.bz2

  1. Test have meta information but framework don't use it.
  2. yes, several types of sys info collected, include profiling and crahes
  3. yes, own text easy-to-parse text report, html report and possible tap. have own sql-based reporting base and limited UI for it.
  4. yes
  5. with some limited programming - yes
  • ready for work linux kernel, supports some logging services and crashdump
  • under active developemnt
  • could work in "client" mode with minimum setup
  • has good internal unit test coverage

in server mode:

  • ready to work on sets of nodes and selecting nodes based on marks/lables
  • has jobs queue supports
  • has a basic web UI and test result database
  • very limited test selection, include list, exclude list, tagging support
  • has pretty simple web UI
  • test structure pretty complex
  • in server mode doesn't support configuration for multinode execution where hosts plays different roles, need configuration improving
STAF

The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. Strictly says, it is not a framework but set of services and libraries for making own test

  1. No test management provided
  2. few options about host are collected
  3. -
  4. -
  5. with some limited programming - yes
  • multi-platform: works on Windows, Linux, AIX
  • test could be written on many languages
  • has UI interface for for remote work with server
  • has security levels
  • doesn't have linux kernel-specific and kernel crashes functionality(maybe special proxies or services are exist)
  • doesn't support configuration for multinode execution where hosts plays different roles, need configuration improving
  • doesn't have test management level
robotframework

Robot Framework is a python-based an open source generic test framework.

  • General purpose testing framework.
  • Supports multiple testing methodologies, such as data-driven test cases
  • provides a command line interface and XML based output files for integration into build infrastructure
  • Has built-in support for variables, which is useful for testing in different environments
  • provides tagging to categorize and select test cases to be run.
  • provides test-case and test-suite level setup and teardowns
  • provides an advanced keyword driven framework
  • keyworks are a synonym of a "function" doing "something"
  • Provides the ability to create reusable/custom higher-level keywords from the existing keywords
  • Easily extendable by Python, C# & Java
  • Doesn't run in parallel, but can run multiple instances to emulate it.
Xperior

Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More info was presented at LAD '12.

File:Report sample.zip

  1. test could be described, about 10 fields now are used.
  2. only node status before run and lustre-diagnostic after test failure. Also there is few extension for reformat lustre after every tests, coverage collection, console log collection via netconsole, more could be easily added
  3. yes, yaml format of results and few convertors to junit, html and tap output documetns
  4. yes
  5. with external tool like jenkins - yes.
  • have separated lustre configuration
  • has some knowledge about current luste tests structure
  • compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout)
  • could be extended for new executors
  • test result is per-test yaml file and html report could be generated
  • could collect some logs from systems under tests on per-test basis, and this could be simple extended. Also there are script for uploading to mongo database.
  • has good internal unit test coverage
  • provides tagging to categorize and select test cases to be run. include/exclude lists
  • provides test-case and test-suite level setup, properties are inherited from suite to tests
  • doesn't maintain directly nodes, lustre mounts, cluster status
  • no security