Contract SFS-DEV-001: Difference between revisions

From OpenSFS Wiki
Jump to navigation Jump to search
No edit summary
 
(32 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{note|This contract was completed with the delivery of Lustre Release 2.8 during March 2016}}
= Project Definition =
= Project Definition =


Line 14: Line 16:
** Subproject 3.3: MDT-MDT Consistency
** Subproject 3.3: MDT-MDT Consistency
** Subproject 3.4: Performance
** Subproject 3.4: Performance
OpenSFS assigned Dave Dillow as the Technical Representative for this project.  Dave leads a Project Approval Committee (PAC) that meets regularly with Intel and assists reviewing Intel's completion of project milestones.  
OpenSFS assigned Dave Dillow as the Technical Representative for this project.  In January 2014, Jason Hill was assigned as the Technical Representative for this project. Jason leads a Project Approval Committee (PAC) that meets regularly with Intel and assists reviewing Intel's completion of project milestones.
 
= Key People =
 
== OpenSFS ==
 
* Jason Hill (ORNL) - Technical Representative, PAC Chair
 
=== Project Approval Committee ===
 
* Cory Spitz, Cray Inc.
* Patrick Farell, Cray Inc.
* Nathan Rutman, Seagate
* ??
 
== Intel ==
 
* Andreas Dilger, Doug Oucharek, Richard Henwood
 
= Project Milestones =
= Project Milestones =
Each subproject follows a detailed milestone plan composed of the following phases.  
Each subproject follows a detailed milestone plan composed of the following phases.  
Line 45: Line 64:
!rowspan="2" align="left"| MDS Performance
!rowspan="2" align="left"| MDS Performance
| align="left"| PDIR Ops
| align="left"| PDIR Ops
| [[Media:MDS_PDirOps_ScopeStatement.pdf|2011-09-28]]
| [[Media:MDS_PDirOps_ScopeStatement.pdf|2011-09-28]]
[[MDS PDirOps ScopeStatement wiki version]]
| [[Media:MDS_PDirOps_SolutionArchitecture.pdf|2011-10-12]]
| [[Media:MDS_PDirOps_SolutionArchitecture.pdf|2011-10-12]]
[[MDS PDirOps SolutionArchitecture wiki version]]
| [[Media:MDS_PDirOps_HighLevelDesign.pdf|2011-11-09]]
| [[Media:MDS_PDirOps_HighLevelDesign.pdf|2011-11-09]]
[[MDS PDirOps HighLevelDesign wiki version]]
| [[Media:MDS_PDirOps_Implementation.pdf|2012-01-06]]
| [[Media:MDS_PDirOps_Implementation.pdf|2012-01-06]]
[[MDS PDirOps Implementation wiki version]]
| [[Media:MDS_PDirOps_DemonstrationMilestone.pdf|2012-03-19]]
| [[Media:MDS_PDirOps_DemonstrationMilestone.pdf|2012-03-19]]
[[MDS PDirOps Demonstration wiki version]]
| 2012-04-30
| 2012-04-30
| 2.2
| 2.2
Line 55: Line 79:
| align="left"| SMP Node Affinity
| align="left"| SMP Node Affinity
| [[Media:MDS_SMPNodeAffinity_ScopeStatement.pdf|2012-03-09]]
| [[Media:MDS_SMPNodeAffinity_ScopeStatement.pdf|2012-03-09]]
[[MDS SMP Node Affinity ScopeStatement wiki version]]
| [[Media:MDS_SMPNodeAffinity_SolutionArchitecture.pdf|2012-04-04]]
| [[Media:MDS_SMPNodeAffinity_SolutionArchitecture.pdf|2012-04-04]]
[[MDS SMP Node Affinity SolutionArchitecture wiki version]]
| [[Media:MDS_SMPNodeAffinity_HighLevelDesign.pdf|2012-05-02]]
| [[Media:MDS_SMPNodeAffinity_HighLevelDesign.pdf|2012-05-02]]
[[MDS SMP Node Affinity HighLevelDesign wiki version]]
| [[Media:MDS_SMPNodeAffinity_Implementation.pdf|2012-09-13]]
| [[Media:MDS_SMPNodeAffinity_Implementation.pdf|2012-09-13]]
[[MDS SMP Node Affinity Implementation wiki version]]
| [[Media:MDS_SMPNodeAffinity_DemonstrationMilestone.pdf|2012-10-07]]
| [[Media:MDS_SMPNodeAffinity_DemonstrationMilestone.pdf|2012-10-07]]
[[MDS SMP Node Affinity Demonstration wiki version]]
| [[Media:MDS_SMPNodeAffinity_Demonstration_FinalReport.pdf|2012-12-12]]
| [[Media:MDS_SMPNodeAffinity_Demonstration_FinalReport.pdf|2012-12-12]]
[[MDS SMP Node Affinity FinalReport wiki version]]
| 2.3
| 2.3
|- valign="top"
|- valign="top"
Line 65: Line 95:
| align="left"| Remote Directories
| align="left"| Remote Directories
| [[Media:DNE_RemoteDirectories_ScopeStatement.pdf|2011-10-19]]
| [[Media:DNE_RemoteDirectories_ScopeStatement.pdf|2011-10-19]]
[[DNE RemoteDirectories ScopeStatement wiki version]]
| [[Media:DNE_RemoteDirectories_SolutionArchitecture.pdf|2011-12-23]]
| [[Media:DNE_RemoteDirectories_SolutionArchitecture.pdf|2011-12-23]]
[[DNE RemoteDirectories SolutionArchitecture wiki version]]
| [[Media:DNE_RemoteDirectories_HighLevelDesign.pdf|2012-02-06]]
| [[Media:DNE_RemoteDirectories_HighLevelDesign.pdf|2012-02-06]]
[[DNE RemoteDirectories HighLevelDesign wiki version]]
|  
|  
* [[Media:DNE_RemoteDirectories_Implementation1.pdf|2012-06-08]]  
* [[Media:DNE_RemoteDirectories_Implementation1.pdf|2012-06-08]]  
* [[Media:DNE_RemoteDirectories_Implementation2.pdf|2012-11-21]]
* [[Media:DNE_RemoteDirectories_Implementation2.pdf|2012-11-21]]
* [[Media:DNE_RemoteDirectories_Implementation3.pdf|2012-12-21]]
* [[Media:DNE_RemoteDirectories_Implementation3.pdf|2012-12-21]]
[[DNE RemoteDirectories Implementation wiki version]]
| [[Media:DNE_RemoteDirectories_DemonstrationMilestone.pdf|2013-05-15]]
| [[Media:DNE_RemoteDirectories_DemonstrationMilestone.pdf|2013-05-15]]
[[DNE RemoteDirectories Demonstration wiki version]]
| [[Media:DNE_RemoteDirectories_FinalReport.pdf|2013-05-31]]
| [[Media:DNE_RemoteDirectories_FinalReport.pdf|2013-05-31]]
[[DNE RemoteDirectories Final wiki version]]
| 2.4
| 2.4
|- valign="top"
|- valign="top"
| align="left"| Striped Directories
| align="left"| Striped Directories
| [[Media:DNE_StripedDirectories_ScopeStatement.pdf|2013-05-13]]
| [[Media:DNE_StripedDirectories_ScopeStatement.pdf|2013-05-13]]
[[DNE StripedDirectories ScopeStatement wiki version]]
| [[Media:DNE_StripedDirectories_SolutionArchitecture.pdf|2013-06-13]]
| [[Media:DNE_StripedDirectories_SolutionArchitecture.pdf|2013-06-13]]
[[DNE StripedDirectories SolutionArchitecture wiki version]]
| [[Media:DNE_StripedDirectories_HighLevelDesign.pdf|2013-07-24]]
| [[Media:DNE_StripedDirectories_HighLevelDesign.pdf|2013-07-24]]
|  
[[DNE StripedDirectories HighLevelDesign wiki version]]
|
| [[Media:DNE_StripedDirectories_Implementation.pdf|2014-12-23]]
|
| [[Media:DNE_StripedDirectories_Demonstration.pdf|2015-08-21]]
| 2.6?
| [[Media:DNE_StripedDirectories_FinalReport.pdf|2016-03-25]]
| 2.8
|- valign="top"
|- valign="top"
!rowspan="5" align="left"| Lustre File System <br/> Check (LFSCK)
!rowspan="5" align="left"| Lustre File System <br/> Check (LFSCK)
Line 88: Line 127:
| [[Media:LFSCK_OIScrub-InodeIterator_ScopeStatement.pdf|2011-10-05]]
| [[Media:LFSCK_OIScrub-InodeIterator_ScopeStatement.pdf|2011-10-05]]
| [[Media:LFSCK_OIScrub-InodeIterator_SolutionArchitecture.pdf|2011-11-03]]
| [[Media:LFSCK_OIScrub-InodeIterator_SolutionArchitecture.pdf|2011-11-03]]
[[LFSKC OIScrub Solution Architecture wiki version]]
| n/a
| n/a
| 2012-04-06
| 2012-04-06
Line 94: Line 134:
| 2.3
| 2.3
|- valign="top"
|- valign="top"
| align="left"| Fid-in-dirent & lineEA consistency
| align="left"| Fid-in-dirent & linkEA consistency
| [[Media:LFSCK_FID-in-dirent_LinkEA_ScopeStatement.pdf|2012-10-31]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_ScopeStatement.pdf|2012-10-31]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_SolutionArchitecture.pdf|2012-11-20]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_SolutionArchitecture.pdf|2012-11-20]]
[[LFSKC FID-in-dirent Solution Architecture wiki version]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_HighLevelDesign.pdf|2012-12-12]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_HighLevelDesign.pdf|2012-12-12]]
[[LFSKC FID-in-dirent High Level Design wiki version]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_Implementation.pdf|2013-01-16]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_Implementation.pdf|2013-01-16]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_DemonstrationMilestone.pdf|2013-05-01]]
| [[Media:LFSCK_FID-in-dirent_LinkEA_DemonstrationMilestone.pdf|2013-05-01]]
Line 106: Line 148:
| [[Media:LFSCK_MDT-OSTConsistency_ScopeStatement.pdf|2013-05-01]]
| [[Media:LFSCK_MDT-OSTConsistency_ScopeStatement.pdf|2013-05-01]]
| [[Media:LFSCK_MDT-OSTConsistency_SolutionArchitecture.pdf|2013-05-29]]
| [[Media:LFSCK_MDT-OSTConsistency_SolutionArchitecture.pdf|2013-05-29]]
[[LFSKC MDT-OST Consistency Solution Architecture wiki version]]
| [[Media:LFSCK_MDT-OSTConsistency_HighLevelDesign.pdf|2013-07-24]]
| [[Media:LFSCK_MDT-OSTConsistency_HighLevelDesign.pdf|2013-07-24]]
[[LFSKC MDT-OST Consistency High Level Design wiki version]]
| [[Media:LFSCK_MDT-OSTConsistency_Implementation.pdf|2013-12-20]]
| [[Media:LFSCK_MDT-OSTConsistency_Implementation.pdf|2013-12-20]]
| [[Media:LFSCK_MDT-OSTConsistency_Demonstration.pdf|2014-02-26]]
| [[Media:LFSCK_MDT-OSTConsistency_Demonstration.pdf|2014-02-26]]
|
| [[Media:LFSCK_MDT-OSTConsistency_FinalReport.pdf|2014-08-07]]
| 2.6?
| 2.6
|- valign="top"
|- valign="top"
| align="left"| MDT-MDT Consistency
| align="left"| MDT-MDT Consistency
|  
| [[Media:LFSCK_MDT-MDTConsistency_ScopeStatement.pdf|2014-04-22]]
|
| [[Media:LFSCK_MDT-MDTConsistency_SolutionArchitecture.pdf|2014-05-29]]
|
[[LFSKC MDT-MDT Consistency Solution Architecture wiki version]]
|
| [[Media:LFSCK_MDT-MDTConsistency_HighLevelDesign.pdf|2014-07-29]]
|
[[LFSKC MDT-MDT High Level Design wiki version]]
|
| [[Media:LFSCK_MDT-MDTConsistency_Implementation.pdf|2014-09-18]]
|
| [[Media:LFSCK_MDT-MDTConsistency_Demonstration.pdf|2015-03-24]]
| [[Media:LFSCK_MDT-MDTConsistency_FinalReport.pdf|2015-03-24]]
| 2.7
|- valign="top"
|- valign="top"
| align="left"| Performance
| align="left"| Performance
|
| [[Media:LFSCK_Performance_ScopeStatement.pdf|2014-10-12]]
|
| [[Media:LFSCK_Performance_SolutionArchitecture.pdf|2014-12-04]]
|
[[LFSKC Performance Solution Architecture wiki version]]
|
| n/a
|
| [[Media:LFSCK_Performance_Implementation.pdf|2014-12-23]]
|
| [[Media:LFSCK_Performance_Demonstration.pdf|2015-05-18]]
|
| [[Media:LFSCK_Performance_FinalReport.pdf|2016-03-25]]
| 2.8
|}
|}

Latest revision as of 09:36, 30 March 2016

NOTE: This contract was completed with the delivery of Lustre Release 2.8 during March 2016

Project Definition

This contract, awarded to Whamcloud (now Intel HPDD) in 2011, defines Lustre development for three projects and eight subprojects. The full program description is available in the Statement of Work.

  • Project 1: Single Server Metadata Performance Improvements
    • Subproject 1.1: SMP Node Affinity
    • Subproject 1.2: Parallel Directory Operations
  • Project 2. Distributed Namespace: Remote Directories and Striped Directories
    • Subproject 2.1: Remote Directories
    • Subproject 2.2: Striped Directories
  • Project 3. Lustre File System Checker
    • Subproject 3.1: Inode Iterator & OI Scrub
    • Subproject 3.2: MDT-OST Consistency
    • Subproject 3.3: MDT-MDT Consistency
    • Subproject 3.4: Performance

OpenSFS assigned Dave Dillow as the Technical Representative for this project. In January 2014, Jason Hill was assigned as the Technical Representative for this project. Jason leads a Project Approval Committee (PAC) that meets regularly with Intel and assists reviewing Intel's completion of project milestones.

Key People

OpenSFS

  • Jason Hill (ORNL) - Technical Representative, PAC Chair

Project Approval Committee

  • Cory Spitz, Cray Inc.
  • Patrick Farell, Cray Inc.
  • Nathan Rutman, Seagate
  • ??

Intel

  • Andreas Dilger, Doug Oucharek, Richard Henwood

Project Milestones

Each subproject follows a detailed milestone plan composed of the following phases.

Scope Statement is a brief (2-3 pages) summary of the Contractor's understanding of the problem statement and resulting project scope.
Solution Architecture is a document that outlines requirements, use cases, a solution framework, and appropriate test cases.
High-Level Design (HLD) is a description of the recommended solution that addresses the subproject requirements and describes how the solution will work, including basic protocol structures (as applicable).
Implementation occurs after completion of the HLD and includes development and unit testing of new code. OpenSFS tracks progress during this phase with weekly progress meetings with the Contractor.
Demonstration of the subproject is defined during the architecture phase and occurs after the Contractor completes the appropriate functionality of the subproject with execution of test cases designed to prove the acceptance criteria defined during the Solution Architecture.
Delivery occurs when the Contractor integrates the project development branch for the subproject into the canonical Lustre source tree for inclusion in the next available community Lustre feature release.

Completed milestone documents for each subproject are available in the project status section below.

Project Status

The table below lists the milestone acceptance dates for the subprojects defined in OpenSFS' development contract with Intel HPDD (SFS-DEV-001).
Documentation of each milestone is hyperlinked to its completion date. We will update the table as Intel completes new milestones.


Project Subproject Scope Statement Solution Architecture High-level Design Implementation Demonstration Delivery Landing
MDS Performance PDIR Ops 2011-09-28

MDS PDirOps ScopeStatement wiki version

2011-10-12

MDS PDirOps SolutionArchitecture wiki version

2011-11-09

MDS PDirOps HighLevelDesign wiki version

2012-01-06

MDS PDirOps Implementation wiki version

2012-03-19

MDS PDirOps Demonstration wiki version

2012-04-30 2.2
SMP Node Affinity 2012-03-09

MDS SMP Node Affinity ScopeStatement wiki version

2012-04-04

MDS SMP Node Affinity SolutionArchitecture wiki version

2012-05-02

MDS SMP Node Affinity HighLevelDesign wiki version

2012-09-13

MDS SMP Node Affinity Implementation wiki version

2012-10-07

MDS SMP Node Affinity Demonstration wiki version

2012-12-12

MDS SMP Node Affinity FinalReport wiki version

2.3
Distributed Name
Space (DNE)
Remote Directories 2011-10-19

DNE RemoteDirectories ScopeStatement wiki version

2011-12-23

DNE RemoteDirectories SolutionArchitecture wiki version

2012-02-06

DNE RemoteDirectories HighLevelDesign wiki version

DNE RemoteDirectories Implementation wiki version

2013-05-15

DNE RemoteDirectories Demonstration wiki version

2013-05-31

DNE RemoteDirectories Final wiki version

2.4
Striped Directories 2013-05-13

DNE StripedDirectories ScopeStatement wiki version

2013-06-13

DNE StripedDirectories SolutionArchitecture wiki version

2013-07-24

DNE StripedDirectories HighLevelDesign wiki version

2014-12-23 2015-08-21 2016-03-25 2.8
Lustre File System
Check (LFSCK)
OI Scrub 2011-10-05 2011-11-03

LFSKC OIScrub Solution Architecture wiki version

n/a 2012-04-06 2012-06-06 2012-12-07 2.3
Fid-in-dirent & linkEA consistency 2012-10-31 2012-11-20

LFSKC FID-in-dirent Solution Architecture wiki version

2012-12-12

LFSKC FID-in-dirent High Level Design wiki version

2013-01-16 2013-05-01 2013-05-31 2.4
MDT-OST Consistency 2013-05-01 2013-05-29

LFSKC MDT-OST Consistency Solution Architecture wiki version

2013-07-24

LFSKC MDT-OST Consistency High Level Design wiki version

2013-12-20 2014-02-26 2014-08-07 2.6
MDT-MDT Consistency 2014-04-22 2014-05-29

LFSKC MDT-MDT Consistency Solution Architecture wiki version

2014-07-29

LFSKC MDT-MDT High Level Design wiki version

2014-09-18 2015-03-24 2015-03-24 2.7
Performance 2014-10-12 2014-12-04

LFSKC Performance Solution Architecture wiki version

n/a 2014-12-23 2015-05-18 2016-03-25 2.8