Contract SFS-DEV-001 early draft: Difference between revisions
mNo edit summary |
No edit summary |
||
(6 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
SFS-DEV-001 includes three areas for development. These areas are listed below with highlighted results. | SFS-DEV-001 includes three areas for development. These areas are listed below with highlighted results. | ||
== Project Documents == | |||
This contract defines development in three broad areas described in the [[Media:SFS-DEV-001_scope-of-work.pdf|Statement of Work]]. | |||
* Project 1: Single Server Metadata Performance Improvements | |||
** Subproject 1.1: SMP Node Affinity | |||
** Subproject 1.2: Parallel Directory Operations | |||
* Project 2. Distributed Namespace: Remote Directories and Striped Directories | |||
** Subproject 2.1: Remote Directories | |||
** Subproject 2.2: Striped Directories | |||
* Project 3. Lustre File System Checker | |||
** Subproject 3.1: Inode Iterator & OI Scrub | |||
** Subproject 3.2: MDT-OST Consistency | |||
** Subproject 3.3: MDT-MDT Consistency | |||
** Subproject 3.4: Performance | |||
Each subproject is developed using a 6 point milestone plan. Completed documents are available from the [[Contract SFS-DEV-001|project]] page. | |||
== Lustre FSCK: OI Scrub == | == Lustre FSCK: OI Scrub == | ||
Line 10: | Line 26: | ||
=== Recent results === | === Recent results === | ||
OI Scrub project provides ground work for a complete Lustre FSCK tool. Crucial to success of this tool is performance. OI Scrub has been demonstrated checking over 100K files/second on a typical MDS hardware node. In addition, OI Scrub provides a on-line mode that has been shown to add negligible system load with limited scan rates: | |||
[[File: | [[File:on-line-impact-oiscrub.png|frameless|300px|OI Scrub provides on-line scanning for corruption. A successful online tool should not impact system performance. This figure show the fall-off in performance from a system without OI Scrub running in the background (0 on the x-axis) through to a system with OI Scrub running as fast as possible (-1 on the x-axis.) The results from this graph indicate that negligible performance impact will be observed while running OI Scrub with Kfiles per second limited between zero and five.]] | ||
== Single Server Metadata Performance: Parallel Directory Operations == | |||
=== Key Message === | |||
* Only a single lock is available for a directory on Lustre. | |||
* Clients wanting access to a file in the same directory must do so in serial. | |||
* This is fixed in | |||
=== Recent Results | |||
[http://wiki.whamcloud.com/display/opensfs/OpenSFS+Lustre+Development Whamcloud site] for more information. | [http://wiki.whamcloud.com/display/opensfs/OpenSFS+Lustre+Development Whamcloud site] for more information. |
Latest revision as of 15:24, 12 April 2013
SFS-DEV-001 includes three areas for development. These areas are listed below with highlighted results.
Project Documents
This contract defines development in three broad areas described in the Statement of Work.
- Project 1: Single Server Metadata Performance Improvements
- Subproject 1.1: SMP Node Affinity
- Subproject 1.2: Parallel Directory Operations
- Project 2. Distributed Namespace: Remote Directories and Striped Directories
- Subproject 2.1: Remote Directories
- Subproject 2.2: Striped Directories
- Project 3. Lustre File System Checker
- Subproject 3.1: Inode Iterator & OI Scrub
- Subproject 3.2: MDT-OST Consistency
- Subproject 3.3: MDT-MDT Consistency
- Subproject 3.4: Performance
Each subproject is developed using a 6 point milestone plan. Completed documents are available from the project page.
Lustre FSCK: OI Scrub
Key Message
- Enable file-level backup and restoration of an MDT.
- Core infrastructure for online consistency checking.
Details of the internal of this work is available at on the Whamcloud community wiki.
Recent results
OI Scrub project provides ground work for a complete Lustre FSCK tool. Crucial to success of this tool is performance. OI Scrub has been demonstrated checking over 100K files/second on a typical MDS hardware node. In addition, OI Scrub provides a on-line mode that has been shown to add negligible system load with limited scan rates:
Single Server Metadata Performance: Parallel Directory Operations
Key Message
- Only a single lock is available for a directory on Lustre.
- Clients wanting access to a file in the same directory must do so in serial.
- This is fixed in
=== Recent Results
Whamcloud site for more information.