Contract SFS-DEV-001: Difference between revisions

From OpenSFS Wiki
Jump to navigation Jump to search
No edit summary
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{note|This contract was completed with the delivery of Lustre Release 2.8 during March 2016}}
= Project Definition =
= Project Definition =


Line 111: Line 113:
| align="left"| Striped Directories
| align="left"| Striped Directories
| [[Media:DNE_StripedDirectories_ScopeStatement.pdf|2013-05-13]]
| [[Media:DNE_StripedDirectories_ScopeStatement.pdf|2013-05-13]]
[[DNE StripedDirectories ScopeStatement wiki version]]
| [[Media:DNE_StripedDirectories_SolutionArchitecture.pdf|2013-06-13]]
| [[Media:DNE_StripedDirectories_SolutionArchitecture.pdf|2013-06-13]]
[[DNE StripedDirectories SolutionArchitecture wiki version]]
| [[Media:DNE_StripedDirectories_HighLevelDesign.pdf|2013-07-24]]
| [[Media:DNE_StripedDirectories_HighLevelDesign.pdf|2013-07-24]]
[[DNE StripedDirectories HighLevelDesign wiki version]]
| [[Media:DNE_StripedDirectories_Implementation.pdf|2014-12-23]]
| [[Media:DNE_StripedDirectories_Implementation.pdf|2014-12-23]]
| in progress
| [[Media:DNE_StripedDirectories_Demonstration.pdf|2015-08-21]]
|
| [[Media:DNE_StripedDirectories_FinalReport.pdf|2016-03-25]]
| 2.8?
| 2.8
|- valign="top"
|- valign="top"
!rowspan="5" align="left"| Lustre File System <br/> Check (LFSCK)
!rowspan="5" align="left"| Lustre File System <br/> Check (LFSCK)
Line 122: Line 127:
| [[Media:LFSCK_OIScrub-InodeIterator_ScopeStatement.pdf|2011-10-05]]
| [[Media:LFSCK_OIScrub-InodeIterator_ScopeStatement.pdf|2011-10-05]]
| [[Media:LFSCK_OIScrub-InodeIterator_SolutionArchitecture.pdf|2011-11-03]]
| [[Media:LFSCK_OIScrub-InodeIterator_SolutionArchitecture.pdf|2011-11-03]]
[[LFSKC OIScrub Solution Architecture wiki version]]
| n/a
| n/a
| 2012-04-06
| 2012-04-06
Line 131: Line 137:
| [[Media:LFSCK_FID-in-dirent_LinkEA_ScopeStatement.pdf|2012-10-31]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_ScopeStatement.pdf|2012-10-31]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_SolutionArchitecture.pdf|2012-11-20]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_SolutionArchitecture.pdf|2012-11-20]]
[[LFSKC FID-in-dirent Solution Architecture wiki version]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_HighLevelDesign.pdf|2012-12-12]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_HighLevelDesign.pdf|2012-12-12]]
[[LFSKC FID-in-dirent High Level Design wiki version]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_Implementation.pdf|2013-01-16]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_Implementation.pdf|2013-01-16]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_DemonstrationMilestone.pdf|2013-05-01]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_DemonstrationMilestone.pdf|2013-05-01]]
Line 140: Line 148:
| [[Media:LFSCK_MDT-OSTConsistency_ScopeStatement.pdf|2013-05-01]]
| [[Media:LFSCK_MDT-OSTConsistency_ScopeStatement.pdf|2013-05-01]]
| [[Media:LFSCK_MDT-OSTConsistency_SolutionArchitecture.pdf|2013-05-29]]
| [[Media:LFSCK_MDT-OSTConsistency_SolutionArchitecture.pdf|2013-05-29]]
[[LFSKC MDT-OST Consistency Solution Architecture wiki version]]
| [[Media:LFSCK_MDT-OSTConsistency_HighLevelDesign.pdf|2013-07-24]]
| [[Media:LFSCK_MDT-OSTConsistency_HighLevelDesign.pdf|2013-07-24]]
[[LFSKC MDT-OST Consistency High Level Design wiki version]]
| [[Media:LFSCK_MDT-OSTConsistency_Implementation.pdf|2013-12-20]]
| [[Media:LFSCK_MDT-OSTConsistency_Implementation.pdf|2013-12-20]]
| [[Media:LFSCK_MDT-OSTConsistency_Demonstration.pdf|2014-02-26]]
| [[Media:LFSCK_MDT-OSTConsistency_Demonstration.pdf|2014-02-26]]
Line 149: Line 159:
| [[Media:LFSCK_MDT-MDTConsistency_ScopeStatement.pdf|2014-04-22]]
| [[Media:LFSCK_MDT-MDTConsistency_ScopeStatement.pdf|2014-04-22]]
| [[Media:LFSCK_MDT-MDTConsistency_SolutionArchitecture.pdf|2014-05-29]]
| [[Media:LFSCK_MDT-MDTConsistency_SolutionArchitecture.pdf|2014-05-29]]
[[LFSKC MDT-MDT Consistency Solution Architecture wiki version]]
| [[Media:LFSCK_MDT-MDTConsistency_HighLevelDesign.pdf|2014-07-29]]
| [[Media:LFSCK_MDT-MDTConsistency_HighLevelDesign.pdf|2014-07-29]]
[[LFSKC MDT-MDT High Level Design wiki version]]
| [[Media:LFSCK_MDT-MDTConsistency_Implementation.pdf|2014-09-18]]
| [[Media:LFSCK_MDT-MDTConsistency_Implementation.pdf|2014-09-18]]
| [[Media:LFSCK_MDT-MDTConsistency_Demonstration.pdf|2015-03-24]]
| [[Media:LFSCK_MDT-MDTConsistency_Demonstration.pdf|2015-03-24]]
|  
| [[Media:LFSCK_MDT-MDTConsistency_FinalReport.pdf|2015-03-24]]
| 2.7?
| 2.7
|- valign="top"
|- valign="top"
| align="left"| Performance
| align="left"| Performance
| [[Media:LFSCK_Performance_ScopeStatement.pdf|2014-10-12]]
| [[Media:LFSCK_Performance_ScopeStatement.pdf|2014-10-12]]
| [[Media:LFSCK_Performance_SolutionArchitecture.pdf|2014-12-04]]
| [[Media:LFSCK_Performance_SolutionArchitecture.pdf|2014-12-04]]
[[LFSKC Performance Solution Architecture wiki version]]
| n/a
| n/a
| [[Media:LFSCK_Performance_Implementation.pdf|2014-12-23]]
| [[Media:LFSCK_Performance_Implementation.pdf|2014-12-23]]
| in progress
| [[Media:LFSCK_Performance_Demonstration.pdf|2015-05-18]]
|
| [[Media:LFSCK_Performance_FinalReport.pdf|2016-03-25]]
|
| 2.8
|}
|}

Latest revision as of 09:36, 30 March 2016

NOTE: This contract was completed with the delivery of Lustre Release 2.8 during March 2016

Project Definition

This contract, awarded to Whamcloud (now Intel HPDD) in 2011, defines Lustre development for three projects and eight subprojects. The full program description is available in the Statement of Work.

  • Project 1: Single Server Metadata Performance Improvements
    • Subproject 1.1: SMP Node Affinity
    • Subproject 1.2: Parallel Directory Operations
  • Project 2. Distributed Namespace: Remote Directories and Striped Directories
    • Subproject 2.1: Remote Directories
    • Subproject 2.2: Striped Directories
  • Project 3. Lustre File System Checker
    • Subproject 3.1: Inode Iterator & OI Scrub
    • Subproject 3.2: MDT-OST Consistency
    • Subproject 3.3: MDT-MDT Consistency
    • Subproject 3.4: Performance

OpenSFS assigned Dave Dillow as the Technical Representative for this project. In January 2014, Jason Hill was assigned as the Technical Representative for this project. Jason leads a Project Approval Committee (PAC) that meets regularly with Intel and assists reviewing Intel's completion of project milestones.

Key People

OpenSFS

  • Jason Hill (ORNL) - Technical Representative, PAC Chair

Project Approval Committee

  • Cory Spitz, Cray Inc.
  • Patrick Farell, Cray Inc.
  • Nathan Rutman, Seagate
  • ??

Intel

  • Andreas Dilger, Doug Oucharek, Richard Henwood

Project Milestones

Each subproject follows a detailed milestone plan composed of the following phases.

Scope Statement is a brief (2-3 pages) summary of the Contractor's understanding of the problem statement and resulting project scope.
Solution Architecture is a document that outlines requirements, use cases, a solution framework, and appropriate test cases.
High-Level Design (HLD) is a description of the recommended solution that addresses the subproject requirements and describes how the solution will work, including basic protocol structures (as applicable).
Implementation occurs after completion of the HLD and includes development and unit testing of new code. OpenSFS tracks progress during this phase with weekly progress meetings with the Contractor.
Demonstration of the subproject is defined during the architecture phase and occurs after the Contractor completes the appropriate functionality of the subproject with execution of test cases designed to prove the acceptance criteria defined during the Solution Architecture.
Delivery occurs when the Contractor integrates the project development branch for the subproject into the canonical Lustre source tree for inclusion in the next available community Lustre feature release.

Completed milestone documents for each subproject are available in the project status section below.

Project Status

The table below lists the milestone acceptance dates for the subprojects defined in OpenSFS' development contract with Intel HPDD (SFS-DEV-001).
Documentation of each milestone is hyperlinked to its completion date. We will update the table as Intel completes new milestones.


Project Subproject Scope Statement Solution Architecture High-level Design Implementation Demonstration Delivery Landing
MDS Performance PDIR Ops 2011-09-28

MDS PDirOps ScopeStatement wiki version

2011-10-12

MDS PDirOps SolutionArchitecture wiki version

2011-11-09

MDS PDirOps HighLevelDesign wiki version

2012-01-06

MDS PDirOps Implementation wiki version

2012-03-19

MDS PDirOps Demonstration wiki version

2012-04-30 2.2
SMP Node Affinity 2012-03-09

MDS SMP Node Affinity ScopeStatement wiki version

2012-04-04

MDS SMP Node Affinity SolutionArchitecture wiki version

2012-05-02

MDS SMP Node Affinity HighLevelDesign wiki version

2012-09-13

MDS SMP Node Affinity Implementation wiki version

2012-10-07

MDS SMP Node Affinity Demonstration wiki version

2012-12-12

MDS SMP Node Affinity FinalReport wiki version

2.3
Distributed Name
Space (DNE)
Remote Directories 2011-10-19

DNE RemoteDirectories ScopeStatement wiki version

2011-12-23

DNE RemoteDirectories SolutionArchitecture wiki version

2012-02-06

DNE RemoteDirectories HighLevelDesign wiki version

DNE RemoteDirectories Implementation wiki version

2013-05-15

DNE RemoteDirectories Demonstration wiki version

2013-05-31

DNE RemoteDirectories Final wiki version

2.4
Striped Directories 2013-05-13

DNE StripedDirectories ScopeStatement wiki version

2013-06-13

DNE StripedDirectories SolutionArchitecture wiki version

2013-07-24

DNE StripedDirectories HighLevelDesign wiki version

2014-12-23 2015-08-21 2016-03-25 2.8
Lustre File System
Check (LFSCK)
OI Scrub 2011-10-05 2011-11-03

LFSKC OIScrub Solution Architecture wiki version

n/a 2012-04-06 2012-06-06 2012-12-07 2.3
Fid-in-dirent & linkEA consistency 2012-10-31 2012-11-20

LFSKC FID-in-dirent Solution Architecture wiki version

2012-12-12

LFSKC FID-in-dirent High Level Design wiki version

2013-01-16 2013-05-01 2013-05-31 2.4
MDT-OST Consistency 2013-05-01 2013-05-29

LFSKC MDT-OST Consistency Solution Architecture wiki version

2013-07-24

LFSKC MDT-OST Consistency High Level Design wiki version

2013-12-20 2014-02-26 2014-08-07 2.6
MDT-MDT Consistency 2014-04-22 2014-05-29

LFSKC MDT-MDT Consistency Solution Architecture wiki version

2014-07-29

LFSKC MDT-MDT High Level Design wiki version

2014-09-18 2015-03-24 2015-03-24 2.7
Performance 2014-10-12 2014-12-04

LFSKC Performance Solution Architecture wiki version

n/a 2014-12-23 2015-05-18 2016-03-25 2.8