Difference between revisions of "Contract SFS-DEV-004"

From OpenSFS Wiki
Jump to navigation Jump to search
Line 182: Line 182:
 
! Planning !! Review !! Landed
 
! Planning !! Review !! Landed
 
|-
 
|-
|  || [http://review.whamcloud.com/#/c/13426/ 13426 LU-5814 obd: remove unused LSM parameters] || [http://review.whamcloud.com/12442 12442] LU-5814 lov: remove LL_IOC_RECREATE_{FID,OBJ}
+
|  || [http://review.whamcloud.com/13722 13722 LU-5814 lov: remove LSM from struct lustre_md] || [http://review.whamcloud.com/12442 12442] LU-5814 lov: remove LL_IOC_RECREATE_{FID,OBJ}
 
|-
 
|-
 
|  || [http://review.whamcloud.com/13644 13644 LU-5814 lov: add lov_object_xattr_get()] || [http://review.whamcloud.com/12446 12446] LU-5814 echo: remove userspace LSM handling
 
|  || [http://review.whamcloud.com/13644 13644 LU-5814 lov: add lov_object_xattr_get()] || [http://review.whamcloud.com/12446 12446] LU-5814 echo: remove userspace LSM handling
 
|-
 
|-
|  || [http://review.whamcloud.com/13680 13680 LU-5814 lov: add cl_object_layout_get()] || [http://review.whamcloud.com/12445 12445] LU-5814 lov: remove unused {get,set}_info handlers
+
|  || [http://review.whamcloud.com/13696 13696 LU-5814 lov: move LSM to LOV layer] || [http://review.whamcloud.com/12445 12445] LU-5814 lov: remove unused {get,set}_info handlers
 
|-
 
|-
|  || [http://review.whamcloud.com/13690 13690 LU-5814 llite: replace lli_has_smd with lli_layout_type] || [http://review.whamcloud.com/12447 12447] LU-5418 echo: replace lov_stripe_md with lov_oinfo
+
|  || [http://review.whamcloud.com/13695 13695 LU-5814 lov: use obd_get_info() to get def/max LOV EA sizes] || [http://review.whamcloud.com/12447 12447] LU-5418 echo: replace lov_stripe_md with lov_oinfo
 
|-
 
|-
|  || [http://review.whamcloud.com/13694 13694 LU-5814 llite: add cl_object_maxbytes()] || [http://review.whamcloud.com/12618 12618] LU-5814 llite: remove ll_objects_destroy()  
+
|  || || [http://review.whamcloud.com/12618 12618] LU-5814 llite: remove ll_objects_destroy()  
 
|-
 
|-
|  || [http://review.whamcloud.com/13695 13695 LU-5814 lov: use obd_get_info() to get def/max LOV EA sizes] || [http://review.whamcloud.com/12581 12581] LU-5814 lov: flatten struct lov_stripe_md]  
+
|  || || [http://review.whamcloud.com/12581 12581] LU-5814 lov: flatten struct lov_stripe_md]  
 
|-
 
|-
|  || [http://review.whamcloud.com/13696 13696 LU-5814 lov: move LSM to LOV layer] ||
+
|| || [http://review.whamcloud.com/#/c/13426/ 13426 LU-5814 obd: remove unused LSM parameters]  
 
|-
 
|-
|  || [http://review.whamcloud.com/13722 13722 LU-5814 lov: remove LSM from struct lustre_md] ||
+
|| || [http://review.whamcloud.com/13680 13680 LU-5814 lov: add cl_object_layout_get()]
 +
|-
 +
|  || || [http://review.whamcloud.com/13690 13690 LU-5814 llite: replace lli_has_smd with lli_layout_type]
 +
|-
 +
|  || [http://review.whamcloud.com/13694 13694 LU-5814 llite: add cl_object_maxbytes()]
 +
|-
 +
|  ||
 +
|-
 +
||  
 
|}
 
|}
  

Revision as of 10:08, 7 May 2015


Overview

The goal of the CLIO Simplification Implementation contract is the implementation in the Lustre source code of the CLIO Simplification Design that resulted from Project 2 of Contract SFS-DEV-003.

For the contract statement of work, see SFS-DEV-004_SOW.pdf

Key People

OpenSFS

  • Sarp Oral - OpenSFS Contract Administrator
  • Christopher Morrone - OpenSFS Technical Representative

Project Approval Committee (PAC)

  • Christopher Morrone - PAC Chair
  • Colin Faber
  • Patrick Farrell
  • Jason Hill
  • James Simmons
  • Cory Spitz

Intel

  • Richard Henwood - Project Manager
  • Andreas Dilger - Consulting Architect
  • Jinshan Xiong - Lead Engineer

Important Dates

The official start date of work is agreed to be October 13, 2014.

The contract lists milestone target dates in weeks relative to the start date. With the start date agreed, here we can just list actual dates to keep things easy to understand.

Milestone task Target Completion Actual Completion
Implementation Jan 26th 2015
Test and fix Apr 6th 2015
Demonstration May 4th 2015
Landing Jun 1st 2015

Meeting Minutes

Working in Progress or Completed

cl_lock re-factoring (simplified and cache-less) DONE

LU-3259 cl_lock re-factoring The cl_lock is necessary because it communicates the DLM lock for a specific IO. The current implementation is highly complex. This work will write a simplified cl_lock. The new lock will be cache-less and replace the current implementation.

Planning Review Landed
10858 LU-3259 clio: cl_lock simplification

Removal of liblustre DONE

LU-2675 removal of liblustre

Planning Review Landed
10657 LU-2675 build: remove liblustre and libsysio
11772 LU-2675 mgc: remove libmgc.c
Completed as part of Removal of Dead Code project 10172 LU-2675 llite: remove liblustre includes
Completed as part of Removal of Dead Code project 10195 LU-2675 lmv: remove liblustre includes
Completed as part of Removal of Dead Code project 10196 LU-2675 lov: remove liblustre includes
11423 LU-2675 build: remove Darwin "support"
11385 LU-2675 build: remove WinNT "support"

function calls implementation and cleanup obsolete OBD methods STARTED

LU-5823 Replace some obsolete obd operations with CLIO ioctl interface OBD API operations for read, write, setattr, getattr, etc. became obsolete after MDT, OFD and client reconstructing were completed. This work removes these redundant operations. OBD API operations for read, write, setattr, getattr, etc because obsolete after MDT, OFD and client restructuring were completed. Redundant code remains in CLIO and interfaces that are not referenced by any module will be targeted for removal.

Planning Review Landed
13514 LU-5823 llite: Remove access of stripe in ll_setattr_raw 12452 LU-5823 clio: add coo_getstripe interface
12494 LU-5823 clio: add cl_object_find_cbdata()
13422 LU-5823 clio: use CIT_SETATTR for FSFILT_IOC_SETFLAGS
12535 LU-5823 clio: add cl_object_fiemap()
12638 LU-5823 clio: add coo_obd_info_get and coo_data_version
12748 LU-5823 clio: remove IOC_LOV_GETINFO
12639 LU-5823 clio: get rid of lov_stripe_md reference
13426 LU-5814 obd: remove unused LSM parameters
Function Status
o_precreate GONE
o_create Used by Echoclient
o_create_async GONE
o_destroy Used by Echoclient
o_setattr Used by Echoclient
o_setattr_async GONE
o_getattr Used by Echoclient
o_getattr_async GONE
o_brw GONE
o_merge_lvb GONE
o_adjust_kms GONE
o_punch GONE
o_sync GONE
o_migrate GONE
o_copy GONE
o_preprw Used by Echoclient and OST server-side code
o_commitrw Used by Echoclient and OST server-side code
o_enqueue GONE
o_cancel GONE
o_change_cbdata GONE
o_find_cbdata GONE
o_change_cbdata GONE
o_extent_calc GONE

Remove lov_stripe_md (LSM) direct access beyond LOV layer STARTED

LU-5814 encapsulate lov_stripe_md (LSM) to LOV layer The current CLIO implementation has a good interface to file layout operations. Legacy code still exists that does not use this interface. The code that does not use the file layout interface will be reviewed and targeted for removal or re-design to use the file layout interface.

Planning Review Landed
13722 LU-5814 lov: remove LSM from struct lustre_md 12442 LU-5814 lov: remove LL_IOC_RECREATE_{FID,OBJ}
13644 LU-5814 lov: add lov_object_xattr_get() 12446 LU-5814 echo: remove userspace LSM handling
13696 LU-5814 lov: move LSM to LOV layer 12445 LU-5814 lov: remove unused {get,set}_info handlers
13695 LU-5814 lov: use obd_get_info() to get def/max LOV EA sizes 12447 LU-5418 echo: replace lov_stripe_md with lov_oinfo
12618 LU-5814 llite: remove ll_objects_destroy()
12581 LU-5814 lov: flatten struct lov_stripe_md]
13426 LU-5814 obd: remove unused LSM parameters
13680 LU-5814 lov: add cl_object_layout_get()
13690 LU-5814 llite: replace lli_has_smd with lli_layout_type
13694 LU-5814 llite: add cl_object_maxbytes()

NOTE: struct obd_info:oi_md cannot be removed now because of inter-dependencies between clean-up patches.

Remove non-linux interfaces STARTED

Two parts

Remove some cfs_ prefixed functions. DONE

Planning Review Landed
NON-INTEL: 6956 LU-1346 libcfs: cleanup libcfs primitive (linux-prim.h)
NON-INTEL: 11797 LU-3963 libcfs: remove last of cfs list wrappers
NON-INTEL: 13070 LU-3963 libcfs: Use kernel's strncasecmp and remove cfs_get_blocked_sigs

NOTE:

  • cfs_snprintf() does have uses, not equivilent to snprintf(), will remain.
  • cfs_hlist* are needed for Linux kernel compatibility and will remain.

Remove ccc_ layer STARTED

LU-5971 removal of ccc_ layerWith the removal of liblustre, the ccc_ layer is redundant and complex. The remaining useful functions will be merged into vfs vm posix layer and the ccc_ layer will be removed.

Planning Review Landed
13318 LU-5971 llite: reorganize {llite,vvp}_internal.h 12592 LU-5971 llite: merge lclient.h into llite/vvp_internal.h
13717 LU-5971 llite: remove ccc_global_{init,fini} 13075 LU-5971 llite: rename ccc_device to vvp_device
13714 LU-5971 llite: rename struct ccc_grouplock to ll_grouplock 13077 LU-5971 llite: rename ccc_object to vvp_object
13715 LU-5971 llite: rename vvp_thread_info to ll_thread_info 13086 LU-5971 llite: rename ccc_page to vvp_page
13716 LU-5971 rename ccc_thread info to vvp_threadinfo 13088 LU-5971 llite: rename ccc_lock to vvp_lock
13351 LU-5971 llite: merge ccc_io and vvp_io
13347 LU-5971 llite: remove struct ll_ra_read
13363 LU-5971 llite: use vui prefix for struct vvp_io members
13376 LU-5971 llite: move vvp_io functions to vvp_io.c
13377 LU-5971 llite: rename ccc_req to vvp_req

Regressions

Planning Review Landed
Move definition of LDLM_GID_ANY to lustre_dlm.h
LU-6046 audit comments in cl_object.h

Test and Fix Phase

For this phase, we will complete the following:

  1. Contractor demonstrates the code passing the complement of tests in Contractor's Autotest environment with the code applied to the Lustre Master tree.
  1. Contract demonstrates the code runs successfully at scale (typically completing a 48 hour SWL run on the Hyperion platform at Lawrence Livermore National Laboratory).
  1. Contractor executes performance regression testing identifying and addressing performance regressions related to the development of the revised code. This performance testing will be run on a system with at least 100 clients and will compare results of IOR, mdtest on builds before and after the implementation of the CLIO Simplification HLD. Degradation of more than 5% will be taken as a failure, but small drops will be accepted as within normal variation.

Change requests

004-001 CLIO ioctl's should be functions

CHANGE REQUEST: 004-001 CLIO ioctl's should be functions.

BACKGROUND: Ioctl calls were included in the CLIO Simplification design to replace some obsolete ODB operations. Alternatively, individual functions can replace the ODB operations instead.

CHANGE: Do not implement ioctl calls. Implement functions.

ACTIONS REQUIRED:

  • Ensure none of the current patches are land.
  • Update the design document with the new design.
  • Update the ticket LU-5823 with new activity.
  • Execute work to complete LU-5823.


STATUS: Approved on 4th Dec 2014

004-002 Omit CLIO Demonstration milestone and associated milestone payment

CHANGE REQUEST: 004-002 Omit CLIO Demonstration milestone and associated milestone payment

BACKGROUND: The Demonstration milestone has been rendered redundant by a precisely specified 'Test and Fix' milestone that will execute before Demonstration. In the current plan, 'Test and Fix' (10 weeks) includes specific tests to run (including 48hr SWL, and performance characterization). There was agreement that a useful Demonstration Milestone is exactly defined by 'Test and Fix' Milestone. The plan requires specification of Demonstration during the Implementation phase and no additional work beyond 'Test and Fix' has been identified.

CHANGE: Omit Demonstration Milestone from the plan of record.

ACTIONS REQUIRED:

  • Move directly from 'Test and Fix' milestone to 'Landing' milestone.
  • Communicate new plan to stakeholders.

STATUS: Submitted for review 2nd March 2015