Test framework requirements

This page is intended as the collection point for people to record their requirements of a new test-framework environment. The new test framework environment is intended to support Lustre today as well as into the future with exascale systems and so the entries on this page should encapsulate all that will be needed to test Lustre for the foreseeable exascale future. The current plan is that we should design the best possible new test framework environment and then assess if a transition from the current framework is possible, but we should not hinder the new framework environment in order to make the transition possible. Please add your requirements to the table below including child pages where you suggestion is a document or sizeable entry. We want to capture all the valuable input of people from ideas, to language requirements, high level architectural ideas, references to other test methods/language/environments that might be useful......

Email	What and Why	Requirement/Idea/Thought
[email protected]	Make test scalable Historically test framework tests tend to address single clients against single oss and single ost, extra effort is required to create scaling tests - this behaviour is the inverse of what's required. A simple test should scale to 100,000+ clients, against 1000+ servers with no effort on behalf of the test writer.	The framework environment must cause scalable tests to be the natural way to write tests. The shortest and simplest test should be scalable, with singular non-scalable being more time consuming.
[email protected]	Be able to create safe live test system test sets.	Test need to be organised on a one test per file basis. This will lead easier to read code but more importantly enable a set of tests to be created that are guaranteed to be safe for a live system. The problem with running tests by filter is that people make mistakes if it's possible to assemble an install of 'safe' tests then we are removing the live ammunition from the situation.
[email protected]	Remove all test relationships A significant issue with fully exploiting the test-framework is that tests have interdependences with one test being required before another test if that later is to succeed. A knock on effect is that setup / teardown of test requirements is often done by the first or last test in a sequence.	The requirement is that each test can be successfully run without a requirement for any other test to execute. This has the following knock on effects; • Setup / Teardown be separated entirely from the tests themselves and then the setup called only once before the first test requiring the provided functionality and then the teardown called only once after the last test requiring the provided functionality. • A general mechanism should be created that allows a safe mechanism for sections containing a number of separate tests to be bounded by a setup/teardown pair. The ‘trick’ here is that the setup/teardown must only occur if at least one of the requiring tests is actually run. REQ-1: Identify all dependent tests and create a change request ticket for each test or set of tests. This work may have already been carried out by Xyratex and so the requirement is to create a change request ticket for those found. REQ-2: Create a general mechanism that enables setup/teardown code to be separated from the tests and called in an efficient way. Submit this mechanism for review before implementing on each set of dependent tests. REQ-3: Implement changes for each set of changes described by the ticket in req-1, where applicable apply the mechanism created in req-2 to deal with each setup/teardown requirement. Submit each change for review and landing, during the landing process ensure that extra testing is carried out to validate the modified tests are operating correctly.
[email protected]	Add Permissions/Run-Filter Capability The ability to run tests based on a class of requirement would allow targeted testing to take place more readily that today’s manual selection process using ONLY or EXCEPT. This functionality as well as allow directed testing for functionality should allow filtering by requirements such as file-system-safe, failover-safe…	The ability to run tests based on a class of requirement would allow targeted testing to take place more readily that today’s manual selection process using ONLY or EXCEPT. To make this possible the run_test function must be updated to apply a AND/OR style mask to be applied. This mask could be bitwise, keyword or key/value pair match. It should not be a trigger for functionality, i.e. if bit x set then format the file-system before test, it should be a simple way of preventing a test from running. Suitable and likely masks/filters are: • File system safe. This would indicate that the test can be run on a production file system. • Failover-unsafe. By default all tests should be failover safe, this flag allows a test to be specified explicitly as unsafe.• Metadata (see spec) key:value present • Metadata (see spec) key:value not present The method used should be extensible so that additional flags can be added with minimal overhead.In the above example the first two are examples of some attributes that might well be hardcode with the test, the last two attributes that would be a managed resource that appends additional information to the test. It must be the case that the filter is applied not by the called by the test but by the test itself, or framework calling the test, this is so that to override for example file system safe the actual test must be changed. As each test will need to be modified a bulk tool for changing all tests to the default state is probably worth developing. REQ-1: Design, implement and have reviewed the mechanism for ensuring that the mask/filter is applied correctly for each test in a failsafe manner that allows for example the use of the test framework on a live file system. REQ-2: Develop a bulk change tool that allows all tests to be set by default to the failsafe mode. This may require no changes or may require a standard piece of code to be inserted into each test procedure. REQ-3: Implement changes for each set of changes described by the ticket in req-1, where applicable apply the mechanism created in req-2 to deal with each setup/teardown requirement. REQ-4: Update the obvious candidates for the major masks such as file system safe, fail over unsafe, it may be required for the majority to be updated ad-hoc moving forwards.
[email protected]	What & Why	How