http://wiki.opensfs.org/api.php?action=feedcontributions&user=Jeffg&feedformat=atomOpenSFS Wiki - User contributions [en]2024-03-29T05:55:09ZUser contributionsMediaWiki 1.39.3http://wiki.opensfs.org/index.php?title=Automation_framework_evaluation&diff=201Automation framework evaluation2013-01-18T22:58:17Z<p>Jeffg: </p>
<hr />
<div>In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.<br />
<br />
Some requirements/desires for the automation framework:<br />
* Aware of multiple clusters<br />
* Able to create virtual clusters as VMs<br />
* Able to automatically start testing based on various triggers, e.g. git commit hooks<br />
* Maintains a prioritizable job queue<br />
* Collect test output status in a database<br />
* Visually represent pass/failure in a clear, concise manner<br />
* Should facilitate easy interpretation of test "trends"; i.e. statistical-based test results<br />
<br />
<br />
<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Framework(with link) !! Advantages !! Disadvantages<br />
|-<br />
| [https://github.com/autotest/autotest autotest] <br />
It is designed primarily to test the Linux kernel, though it is useful for many other <br />
functions such asqualifying new hardware. It used and developed by a number of <br />
organizations, including Google, IBM, Red Hat, andmany others.<br />
Developed on Python<br />
<br />
|| <br />
* ready for work linux kernel, supports some logging services and crashdump<br />
* under active developemnt<br />
* could work in "client" mode with minimum setup<br />
* has good internal unit test coverage<br />
in server mode:<br />
<br />
* ready to work on sets of nodes and selecting nodes based on marks/lables<br />
* has jobs queue supports<br />
* has a basic web UI and test result database<br />
|| <br />
* very limited test selection, include list, exclude list, tagging support<br />
* has pretty simple web UI<br />
* test structure pretty complex<br />
* in server mode doesn't support configuration for multinode execution where hosts plays different roles, need configuration improving<br />
|-<br />
| [http://staf.sourceforge.net/ STAF] <br />
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM<br />
Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. <br />
|| <br />
* multi-platform: works on Windows, Linux, AIX<br />
* test could be written on many languages<br />
* has UI interface for for remote work with server<br />
* has security levels<br />
|| <br />
* doesn't have linux kernel-specific and kernel crashes functionality(maybe special proxies or services are exist)<br />
* doesn't support configuration for multinode execution where hosts plays different roles, need configuration improving<br />
* doesn't have test management level<br />
|-<br />
| [http://code.google.com/p/robotframework/ robotframework] <br />
Robot Framework is a python-based an open source generic test framework.<br />
|| <br />
* General purpose testing framework. <br />
* Supports multiple testing methodologies, such as data-driven test cases<br />
* provides a command line interface and XML based output files for integration into build infrastructure<br />
* Has built-in support for variables, which is useful for testing in different environments<br />
* provides tagging to categorize and select test cases to be run.<br />
* provides test-case and test-suite level setup and teardowns<br />
* provides an advanced keyword driven framework<br />
* keyworks are a synonym of a "function" doing "something"<br />
* Provides the ability to create reusable/custom higher-level keywords from the existing keywords<br />
* Easily extendable by Python, C# & Java<br />
|| <br />
* Doesn't run in parallel, but can run multiple instances to emulate it.<br />
|<br />
<br />
|-<br />
| [https://github.com/Xyratex/xperior Xperior] <br />
Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.<br />
|| <br />
* has some knowledge about current luste tests structure<br />
* compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout) <br />
* could be extended for new executors<br />
* test result is per-test yaml file and html report could be generated<br />
* could collect some logs from systems under tests on per-test basis, and this could be simple extended<br />
* has good internal unit test coverage<br />
|| <br />
* doesn't maintain directly nodes, lustre mounts, cluster status<br />
* no security<br />
|-<br />
|}</div>Jeffg