Test framework requirements - Revision history

Spitzcor: add Google doc link to Nathan's original proposal

2013-01-17T18:12:55Z

add Google doc link to Nathan's original proposal

← Older revision		Revision as of 11:12, 17 January 2013
Line 4:		Line 4:
	Please add your requirements to the table below including child pages where you suggestion is a document or sizeable entry. We want to capture all the valuable input of people from ideas, to language requirements, high level architectural ideas, references to other test methods/language/environments that might be useful...		Please add your requirements to the table below including child pages where you suggestion is a document or sizeable entry. We want to capture all the valuable input of people from ideas, to language requirements, high level architectural ideas, references to other test methods/language/environments that might be useful...

	Nathan's original proposal: [[File:Test Framework 2012.pdf]]		Nathan's original proposal: [[File:Test Framework 2012.pdf]] (Comment-able in Google docs at: http://goo.gl/mCtwU [Nathan can grant edit rights])

	{\| class="wikitable" border="1"		{\| class="wikitable" border="1"

Nrutman at 23:21, 19 December 2012

2012-12-19T23:21:33Z

← Older revision		Revision as of 16:21, 19 December 2012
Line 230:		Line 230:
	\|The framework should have a clear, enforceable API for allowing interaction with the filesystem. Areas that should be covered in the API should include the following:		\|The framework should have a clear, enforceable API for allowing interaction with the filesystem. Areas that should be covered in the API should include the following:
	* Queries for system configuration		* Queries for system configuration
	* Explicit role-based operations		* Explicit role-based operations - the current tests are mostly written as if intended to be run from a client. However, this is not compatible with some testing systems, where the role of the test executor and the role of the client may not coincide - for example some Cray nodes are not amenable to running scripts this way. Operations that must take place on specific nodes should all use a common API format to specify the node or role. This is similar to the “do_node” function in the current test-framework.sh, but it must be extended to explicit instead of implicit client roles.
	* Precondition checks		* Precondition checks - current conditions of the configuration or filesystem state may exclude tests from executing correctly or usefully. These checks should be performed before trying to start a test, so that the output condition of a test is not “skip” (as is the case today); instead the test should not begin.
	* Permissions mask		* Permissions mask
	* Rich file creation methods		* Rich file creation methods
			** no explicit file locations / names. API calls should return a name/file handle rather than be given one.
			** no assumptions about the underlying Linux distribution/OS or availability of files/dirs for copying. (E.g. tests today assume /etc is populated and used a source of “random files”.)
			** a list of created files is maintained, and all files are removed automatically by the framework after the test returns.
	* Parameter setting		* Parameter setting
			**check for parameter setting permissions
			**maintain a list of parameter changes
			**restore the changed parameters automatically upon test completion
	\|-		\|-
	\| Initials		\| Initials

Nrutman at 23:15, 19 December 2012

2012-12-19T23:15:23Z

@@ Line 225: / Line 225: @@
 |Constraint-based testing
 |To more fully characterize response in unusual circumstances, it would be useful for the framework to be able to control and vary some aspects of the environment.  For example, if the framework were able to specify the amount of memory available to the kernel (e.g. when using a virtual machine), then OOM conditions and behavior could be easily tested.  Similarly, limiting the number of MDS threads to (num_clients-1), running with a very high network latency, etc. would provide testing of code paths that are not normally tested.
 |-
 | Initials

Nrutman at 22:14, 19 December 2012

2012-12-19T22:14:51Z

← Older revision		Revision as of 15:14, 19 December 2012
Line 8:		Line 8:
	{\| class="wikitable" border="1"		{\| class="wikitable" border="1"
	\|-		\|-
			! Initials
	! Requirement		! Requirement
	! Discussion		! Discussion
Line 222:		Line 223:
	\|-		\|-
	\|NR		\|NR
	\|		\|Constraint-based testing
	Constraint-based testing\|To more fully characterize response in unusual circumstances, it would be useful for the framework to be able to control and vary some aspects of the environment. For example, if the framework were able to specify the amount of memory available to the kernel (e.g. when using a virtual machine), then OOM conditions and behavior could be easily tested. Similarly, limiting the number of MDS threads to (num_clients-1), running with a very high network latency, etc. would provide testing of code paths that are not normally tested.		\|To more fully characterize response in unusual circumstances, it would be useful for the framework to be able to control and vary some aspects of the environment. For example, if the framework were able to specify the amount of memory available to the kernel (e.g. when using a virtual machine), then OOM conditions and behavior could be easily tested. Similarly, limiting the number of MDS threads to (num_clients-1), running with a very high network latency, etc. would provide testing of code paths that are not normally tested.
	\|-		\|-
	\| Initials		\| Initials

Nrutman at 22:14, 19 December 2012

2012-12-19T22:14:05Z

← Older revision		Revision as of 15:14, 19 December 2012
Line 163:		Line 163:
	\|-		\|-
	\| JG		\| JG
	\| A survey of existing ~~open source~~ test frameworks ~~should be done.~~		\| A survey of existing test frameworks
	\| It's possible that there's a framework that provides 80% of the requirements, making the time to implement the solution much shorter. See frameworks comparsion : [[Exists_test_framework_evaluation]]		\| It's possible that there's a framework that provides 80% of the requirements, making the time to implement the solution much shorter. See frameworks comparsion : [[Exists_test_framework_evaluation]]

			NR: I've tried to clarify a distinction between the Lustre Test Framework, described here, and a test automation framework, described there. I've renamed that page to [[automation framework evaluation]] and added some description.
	\|-		\|-
	\| JG		\| JG
Line 218:		Line 220:
	* environment changes		* environment changes
	This should be called automatically when the test ends (for both pass and fail cases).		This should be called automatically when the test ends (for both pass and fail cases).
			\|-
			\|NR
			\|
			Constraint-based testing\|To more fully characterize response in unusual circumstances, it would be useful for the framework to be able to control and vary some aspects of the environment. For example, if the framework were able to specify the amount of memory available to the kernel (e.g. when using a virtual machine), then OOM conditions and behavior could be easily tested. Similarly, limiting the number of MDS threads to (num_clients-1), running with a very high network latency, etc. would provide testing of code paths that are not normally tested.
	\|-		\|-
	\| Initials		\| Initials

Nrutman at 01:57, 19 December 2012

2012-12-19T01:57:33Z

← Older revision		Revision as of 18:57, 18 December 2012
Line 210:		Line 210:
	Precondition checks		Precondition checks
	\|Current conditions of the configuration or filesystem state may exclude tests from executing correctly or usefully. These checks should be performed before trying to start a test, so that the output condition of a test is not “skip” (as is the case today), but rather a note in the test log saying the test was skipped due to failure of precondition X. The preconditions should be specified via a special section before each test, perhaps along with the test metadata.		\|Current conditions of the configuration or filesystem state may exclude tests from executing correctly or usefully. These checks should be performed before trying to start a test, so that the output condition of a test is not “skip” (as is the case today), but rather a note in the test log saying the test was skipped due to failure of precondition X. The preconditions should be specified via a special section before each test, perhaps along with the test metadata.
			\|-
			\|NR
			\|Automatic cleanup
			\|The framework should enforce an API that allows for automatic cleanup of any test artifacts:
			* created files
			* parameter settings
			* environment changes
			This should be called automatically when the test ends (for both pass and fail cases).
	\|-		\|-
	\| Initials		\| Initials

Nrutman at 01:53, 19 December 2012

2012-12-19T01:53:11Z

← Older revision		Revision as of 18:53, 18 December 2012
Line 16:		Line 16:

	The framework environment must cause scalable tests to be the natural way to write tests. The shortest and simplest test should be scalable, with singular non-scalable being more time consuming.		The framework environment must cause scalable tests to be the natural way to write tests. The shortest and simplest test should be scalable, with singular non-scalable being more time consuming.
			\|-
			\|NR
			\|Parallel test execution
			\|Related to the above, tests and the framework must both be able to run tests in parallel on multiple clients. Sadly, many tests today currently assume they are the only test running (e.g. the obd_fail_loc mechanism is not sophisticated enough to handle parallel tests).
	\|-		\|-
	\| CG		\| CG
Line 207:		Line 211:
	\|Current conditions of the configuration or filesystem state may exclude tests from executing correctly or usefully. These checks should be performed before trying to start a test, so that the output condition of a test is not “skip” (as is the case today), but rather a note in the test log saying the test was skipped due to failure of precondition X. The preconditions should be specified via a special section before each test, perhaps along with the test metadata.		\|Current conditions of the configuration or filesystem state may exclude tests from executing correctly or usefully. These checks should be performed before trying to start a test, so that the output condition of a test is not “skip” (as is the case today), but rather a note in the test log saying the test was skipped due to failure of precondition X. The preconditions should be specified via a special section before each test, perhaps along with the test metadata.
	\|-		\|-

	\| Initials		\| Initials
	\| Requirement		\| Requirement

Nrutman at 01:47, 19 December 2012

2012-12-19T01:47:13Z

← Older revision		Revision as of 18:47, 18 December 2012
Line 202:		Line 202:
	The framework itself should be built with consistency and correctness unit tests, to verify the correct functionality of the test execution, measurement, and reporting functions. These should be executed independently from any Lustre features; ie. we can run test-framework-check without any Lustre installed.		The framework itself should be built with consistency and correctness unit tests, to verify the correct functionality of the test execution, measurement, and reporting functions. These should be executed independently from any Lustre features; ie. we can run test-framework-check without any Lustre installed.
	\|-		\|-
			\|NR
			\|
			Precondition checks
			\|Current conditions of the configuration or filesystem state may exclude tests from executing correctly or usefully. These checks should be performed before trying to start a test, so that the output condition of a test is not “skip” (as is the case today), but rather a note in the test log saying the test was skipped due to failure of precondition X. The preconditions should be specified via a special section before each test, perhaps along with the test metadata.
			\|-

	\| Initials		\| Initials
	\| Requirement		\| Requirement

Nrutman at 01:46, 19 December 2012

2012-12-19T01:46:00Z

← Older revision		Revision as of 18:46, 18 December 2012
Line 196:		Line 196:
	* timing information		* timing information
	* error codes with test line numbers for failures/exits within a test		* error codes with test line numbers for failures/exits within a test
			\|-
			\|NR
			\|Framework Self Test
			\|
			The framework itself should be built with consistency and correctness unit tests, to verify the correct functionality of the test execution, measurement, and reporting functions. These should be executed independently from any Lustre features; ie. we can run test-framework-check without any Lustre installed.
	\|-		\|-
	\| Initials		\| Initials

Nrutman at 01:42, 19 December 2012

2012-12-19T01:42:30Z

@@ Line 178: / Line 178: @@
 | Inter-version testing
 | Currently, testing components are distributed among a number of nodes.  If the Lustre versions don't match (typical between servers and clients), then testing components may not match.  The test framework code should be developed independently and installed independently from any particular Lustre version.  The framework should of course be aware of which versions of Lustre it is using and filter tests appropriately.
 |-
 | Initials