Automation framework evaluation - Revision history

Roman grigoryev at 19:01, 24 January 2013

2013-01-24T19:01:30Z

← Older revision		Revision as of 12:01, 24 January 2013
Line 28:		Line 28:
	\| [https://github.com/autotest/autotest autotest]		\| [https://github.com/autotest/autotest autotest]
	It is designed primarily to test the Linux kernel, though it is useful for many other		It is designed primarily to test the Linux kernel, though it is useful for many other
	functions such ~~asqualifying~~ new hardware. It used and developed by a number of		functions such as qualifying new hardware. It used and developed by a number of
	organizations, including Google, IBM, Red Hat, ~~andmany~~ others.		organizations, including Google, IBM, Red Hat, and many others.
	Developed on Python		Developed on Python
			[[File:autotest_sample_report.tar.bz2]]
	\|\|		\|\|
	# Test have meta information but framework don't use it.		# Test have meta information but framework don't use it.

Roman grigoryev at 18:31, 22 January 2013

2013-01-22T18:31:08Z

← Older revision		Revision as of 11:31, 22 January 2013
Line 93:		Line 93:
	\| [https://github.com/Xyratex/xperior Xperior]		\| [https://github.com/Xyratex/xperior Xperior]
	Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.		Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.

	[[File:report_sample.zip]]		[[File:report_sample.zip]]
	\|\|		\|\|

Roman grigoryev at 18:28, 22 January 2013

2013-01-22T18:28:41Z

← Older revision		Revision as of 11:28, 22 January 2013
Line 1:		Line 1:
	In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.		In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
	~~[[File:Example.jpg]]~~
	Some requirements/desires for the automation framework:		Some requirements/desires for the automation framework:
	* Aware of multiple clusters		* Aware of multiple clusters
Line 93:		Line 93:
	\| [https://github.com/Xyratex/xperior Xperior]		\| [https://github.com/Xyratex/xperior Xperior]
	Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.		Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.
			[[File:report_sample.zip]]
	\|\|		\|\|
	# test could be described, about 10 fields now are used.		# test could be described, about 10 fields now are used.

Roman grigoryev at 18:20, 22 January 2013

2013-01-22T18:20:17Z

← Older revision		Revision as of 11:20, 22 January 2013
Line 1:		Line 1:
	In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.		In order to automatically service and track test requests, as well as deploy testing resources, an upper-level test automation framework would sit "on top of" the Lustre test infrastructure.
			[[File:Example.jpg]]
	Some requirements/desires for the automation framework:		Some requirements/desires for the automation framework:
	* Aware of multiple clusters		* Aware of multiple clusters

Roman grigoryev at 17:24, 22 January 2013

2013-01-22T17:24:04Z

← Older revision		Revision as of 10:24, 22 January 2013
Line 34:		Line 34:
	# Test have meta information but framework don't use it.		# Test have meta information but framework don't use it.
	# yes, several types of sys info collected, include profiling and crahes		# yes, several types of sys info collected, include profiling and crahes
	# yes		# yes, own text easy-to-parse text report, html report and possible tap. have own sql-based reporting base and limited UI for it.
	# yes		# yes
	# with some limited programming - yes		# with some limited programming - yes
Line 55:		Line 55:
	\| [http://staf.sourceforge.net/ STAF]		\| [http://staf.sourceforge.net/ STAF]
	The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM		The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM
	Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt.		Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt. Strictly says, it is not a framework but set of services and libraries for making own test
	\|\|		\|\|
	# No test management provided		# No test management provided
Line 94:		Line 94:
	Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.		Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.
	\|\|		\|\|
			# test could be described, about 10 fields now are used.
			# only node status before run and lustre-diagnostic after test failure. Also there is few extension for reformat lustre after every tests, coverage collection, console log collection via netconsole, more could be easily added
			# yes, yaml format of results and few convertors to junit, html and tap output documetns
			# yes
			# with external tool like jenkins - yes.
	\|\|		\|\|
			* have separated lustre configuration
	* has some knowledge about current luste tests structure		* has some knowledge about current luste tests structure
	* compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout)		* compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout)
	* could be extended for new executors		* could be extended for new executors
	* test result is per-test yaml file and html report could be generated		* test result is per-test yaml file and html report could be generated
	* could collect some logs from systems under tests on per-test basis, and this could be simple extended		* could collect some logs from systems under tests on per-test basis, and this could be simple extended. Also there are script for uploading to mongo database.
	* has good internal unit test coverage		* has good internal unit test coverage
	\|\|		* provides tagging to categorize and select test cases to be run. include/exclude lists
			* provides test-case and test-suite level setup, properties are inherited from suite to tests
			\|\|
	* doesn't maintain directly nodes, lustre mounts, cluster status		* doesn't maintain directly nodes, lustre mounts, cluster status
	* no security		* no security
	\|-		\|-
	\|}		\|}

Roman grigoryev at 16:03, 22 January 2013

2013-01-22T16:03:37Z

← Older revision		Revision as of 09:03, 22 January 2013
Line 36:		Line 36:
	# yes		# yes
	# yes		# yes
	# with some programming - yes		# with some limited programming - yes
	\|\|		\|\|
	* ready for work linux kernel, supports some logging services and crashdump		* ready for work linux kernel, supports some logging services and crashdump
Line 57:		Line 57:
	Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt.		Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt.
	\|\|		\|\|
			# No test management provided
			# few options about host are collected
			#-
			#-
			#with some limited programming - yes
	\|\|		\|\|
	* multi-platform: works on Windows, Linux, AIX		* multi-platform: works on Windows, Linux, AIX

Roman grigoryev at 15:55, 22 January 2013

2013-01-22T15:55:33Z

← Older revision		Revision as of 08:55, 22 January 2013
Line 16:		Line 16:
	{\| class="wikitable"		{\| class="wikitable"
	\|-		\|-
	! Framework(with link) !!		! Framework(with link)
	Feature list		! Feature list
	# Do tests describe intent clearly?		# Do tests describe intent clearly?
	# Is system information gathered?		# Is system information gathered?
	# Do results readable by man and computer?		# Do results readable by man and computer?
	# Is run and develop process fast?		# Is run and develop process fast?
	# Does it able to be run on cluster in parallel?		# Does it able to be run on cluster in parallel?
			! More advantages
	!! More advantages !! More disadvantages		! More disadvantages
	\|-		\|-
	\| [https://github.com/autotest/autotest autotest]		\| [https://github.com/autotest/autotest autotest]
Line 32:		Line 32:
	Developed on Python		Developed on Python
	\|\|		\|\|
			# Test have meta information but framework don't use it.
			# yes, several types of sys info collected, include profiling and crahes
			# yes
			# yes
			# with some programming - yes
	\|\|		\|\|
	* ready for work linux kernel, supports some logging services and crashdump		* ready for work linux kernel, supports some logging services and crashdump

Roman grigoryev at 15:30, 22 January 2013

2013-01-22T15:30:43Z

← Older revision		Revision as of 08:30, 22 January 2013
Line 18:		Line 18:
	! Framework(with link) !!		! Framework(with link) !!
	Feature list		Feature list
	* Do tests describe intent clearly?		# Do tests describe intent clearly?
	* Is system information gathered?		# Is system information gathered?
	* Do results readable by man and computer?		# Do results readable by man and computer?
			# Is run and develop process fast?
			# Does it able to be run on cluster in parallel?

	!! More advantages !! More disadvantages		!! More advantages !! More disadvantages

Roman grigoryev at 15:27, 22 January 2013

2013-01-22T15:27:15Z

← Older revision		Revision as of 08:27, 22 January 2013
Line 16:		Line 16:
	{\| class="wikitable"		{\| class="wikitable"
	\|-		\|-
	! Framework(with link) !! ~~Advantages~~ !! ~~Disadvantages~~		! Framework(with link) !!
			Feature list
			* Do tests describe intent clearly?
			* Is system information gathered?
			* Do results readable by man and computer?

			!! More advantages !! More disadvantages
	\|-		\|-
	\| [https://github.com/autotest/autotest autotest]		\| [https://github.com/autotest/autotest autotest]
Line 23:		Line 29:
	organizations, including Google, IBM, Red Hat, andmany others.		organizations, including Google, IBM, Red Hat, andmany others.
	Developed on Python		Developed on Python
			\|\|
	\|\|		\|\|
	* ready for work linux kernel, supports some logging services and crashdump		* ready for work linux kernel, supports some logging services and crashdump
Line 43:		Line 49:
	The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM		The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework, lead by IBM
	Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt.		Core developed on C. You can interact with STAF from many languages (Java, C, C++, Python,Perl, Tcl, Rexx) and from the command line/shell prompt.
			\|\|
	\|\|		\|\|
	* multi-platform: works on Windows, Linux, AIX		* multi-platform: works on Windows, Linux, AIX
Line 55:		Line 62:
	\| [http://code.google.com/p/robotframework/ robotframework]		\| [http://code.google.com/p/robotframework/ robotframework]
	Robot Framework is a python-based an open source generic test framework.		Robot Framework is a python-based an open source generic test framework.
	\|\|		\|\|
			\|\|
	* General purpose testing framework.		* General purpose testing framework.
	* Supports multiple testing methodologies, such as data-driven test cases		* Supports multiple testing methodologies, such as data-driven test cases
Line 73:		Line 81:
	\| [https://github.com/Xyratex/xperior Xperior]		\| [https://github.com/Xyratex/xperior Xperior]
	Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.		Xperior is an open source framework developed on Perl by Xyratex for executing lustre tests from current shell-based testing framework. More [http://www.eofs.eu/fileadmin/lad2012/22_Rutman-Gearing_Xyratex-Intel_Lustre_Testing_LAD_2012.pdf info] was presented at LAD '12.
	\|\|		\|\|
			\|\|
	* has some knowledge about current luste tests structure		* has some knowledge about current luste tests structure
	* compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout)		* compatible the most of acc-small lustre test set, allow to configure some test parameters (e.g. per-test execution timeout)

Jeffg at 22:58, 18 January 2013

2013-01-18T22:58:17Z

@@ Line 53: / Line 53: @@
 * doesn't have test management level
 |-
-| [http://code.google.com/p/robotframework/ robotframework] || Example || Example
+| [http://code.google.com/p/robotframework/ robotframework]
+|