Contract SFS-DEV-001 early draft: Difference between revisions

From OpenSFS Wiki
Jump to navigation Jump to search
mNo edit summary
No edit summary
 
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
SFS-DEV-001 includes three areas for development. These areas are listed below with highlighted results.
SFS-DEV-001 includes three areas for development. These areas are listed below with highlighted results.
== Project Documents ==
This contract defines development in three broad areas described in the [[Media:SFS-DEV-001_scope-of-work.pdf|Statement of Work]].
* Project 1: Single Server Metadata Performance Improvements
** Subproject 1.1: SMP Node Affinity
** Subproject 1.2: Parallel Directory Operations
* Project 2. Distributed Namespace: Remote Directories and Striped Directories
** Subproject 2.1: Remote Directories
** Subproject 2.2: Striped Directories
* Project 3. Lustre File System Checker
** Subproject 3.1: Inode Iterator & OI Scrub
** Subproject 3.2: MDT-OST Consistency
** Subproject 3.3: MDT-MDT Consistency
** Subproject 3.4: Performance
Each subproject is developed using a 6 point milestone plan.  Completed documents are available from the [[Contract SFS-DEV-001|project]] page.


== Lustre FSCK: OI Scrub ==
== Lustre FSCK: OI Scrub ==

Latest revision as of 15:24, 12 April 2013

SFS-DEV-001 includes three areas for development. These areas are listed below with highlighted results.

Project Documents

This contract defines development in three broad areas described in the Statement of Work.

  • Project 1: Single Server Metadata Performance Improvements
    • Subproject 1.1: SMP Node Affinity
    • Subproject 1.2: Parallel Directory Operations
  • Project 2. Distributed Namespace: Remote Directories and Striped Directories
    • Subproject 2.1: Remote Directories
    • Subproject 2.2: Striped Directories
  • Project 3. Lustre File System Checker
    • Subproject 3.1: Inode Iterator & OI Scrub
    • Subproject 3.2: MDT-OST Consistency
    • Subproject 3.3: MDT-MDT Consistency
    • Subproject 3.4: Performance

Each subproject is developed using a 6 point milestone plan. Completed documents are available from the project page.

Lustre FSCK: OI Scrub

Key Message

  • Enable file-level backup and restoration of an MDT.
  • Core infrastructure for online consistency checking.

Details of the internal of this work is available at on the Whamcloud community wiki.

Recent results

OI Scrub project provides ground work for a complete Lustre FSCK tool. Crucial to success of this tool is performance. OI Scrub has been demonstrated checking over 100K files/second on a typical MDS hardware node. In addition, OI Scrub provides a on-line mode that has been shown to add negligible system load with limited scan rates:

OI Scrub provides on-line scanning for corruption. A successful online tool should not impact system performance. This figure show the fall-off in performance from a system without OI Scrub running in the background (0 on the x-axis) through to a system with OI Scrub running as fast as possible (-1 on the x-axis.) The results from this graph indicate that negligible performance impact will be observed while running OI Scrub with Kfiles per second limited between zero and five.

Single Server Metadata Performance: Parallel Directory Operations

Key Message

  • Only a single lock is available for a directory on Lustre.
  • Clients wanting access to a file in the same directory must do so in serial.
  • This is fixed in

=== Recent Results

Whamcloud site for more information.

Retrieved from "http://wiki.opensfs.org/index.php?title=Contract_SFS-DEV-001_early_draft&oldid=385"