Lustre Community Development in Progress

From OpenSFS
Revision as of 13:36, 11 February 2015 by Paf (talk | contribs) (List of features under development)
Jump to: navigation, search

Lustre Development Scope

For the Lustre development cycle, we are tracking all outstanding work for the release right in Jira. We do so by adding a version string (e.g. "Lustre 2.6.0") to the "Fix Version/s" field of any ticket that we have scoped to be part of that release. Jira uses the "Fix Version/s" field in a number of ways, including its built-in Road Map, Change Log, and of course Issues pages.

Thanks to use of this field, the Lustre project page in Jira will better reflect the state of development for the next Lustre release. Here are some quick links:

The OpenSFS LWG provides the initial list of tickets scoped for a release. General release info (scope, schedule, etc.) is tracked on per-version LWG pages:

Keep in mind that the list of work is not static. As bugs are discovered, the list will grow. When new Jira tickets are opened for Lustre bug on the master branch, Intel, as part of the work for the Community Tree Development Contract, will make the initial determination of whether the ticket should be added to the scope for a release.

If a party feels like a ticket should be added to the Lustre version scope, but Intel did not make that initial determination, a request to have the version (e.g. "Lustre 2.6.0") added to the "Fix Version/s" field should be made in the Jira ticket itself.

Should Intel and the interested party disagree, the issue should be raised on Lustre Working Group mailing list so that the issue can be discussed by the larger community and final determination made.

List of features under development

Features that are currently being developed for future Lustre releases in the Lustre community. These will be eligible for inclusion in future Lustre releases as per the defined processes. Features are not listed in priority or landing order. Projects not currently under development, or ideas for features, should be listed on the Project Ideas page.

Please note that the targeted release information is provided by the developing organization for informational purposes only and is in no way a roadmap commitment until it appears on the Lustre Roadmap.

Feature Feature Summary Developer Tracker Targeted Release Landed Release
Patchless Server Remove Lustre kernel patches to allow Lustre servers to be more easily ported to new kernels, and to be built against vendor kernels without changing the vendor kernel RPMs. Intel LU-20 2.7 .
End-to-End Data Integrity with T10 Add a path to pass T10 Protection Information from Lustre clients through to server disks to reduce CPU overhead of multiple checksum calculations. Xyratex LU-2488 LU-2584 2.9 .
LFSCK Phase 3 - DNE consistency Enhance LFSCK to work with DNE filesystems, including remote directory entries, and OST orphan handling for multiple MDTs. Intel LU-2307 2.7 .
DNE Phase IIb Asynchronous Commit of cross-MDT updates for improved performance. Remote rename and remote hard link functionality. Intel LU-3534 2.7 .
Layout Enhancement Add support for multiple layouts on a single file, for File Level Replication, Data on MDT, extent-based layouts, etc. Intel LU-3480 2.8 .
File Level Replication Phase 1 Solution Architecture and HDL, Exclusive Open, RAID 1 Layout, Layout Modification Method, Read Only RAID 1 Intel LU-3254 2.9 .
File Level Replication Phase 2 Immediate asynchronous write from client. Intel LU-3254 2.10 .
Data on MDT Allow small files to be stored directly on the MDT for reduced RPC count and improved performance. Intel . 2.9 .
Dynamic LNET Config Introduces a user space script which will read routes from a config file and add those routes to LNET dynamically with the lctl utility. This allows the support of very large routing tables Intel LU-2950 2.7 .
Quota for Projects Allow specifying a "project" or "subtree" identifier for files for accounting to a project, separate from UID/GID. DDN LU-4017 2.8 .
Large Bulk IO Increase the OST bulk IO maximum size to 16MB or larger for more efficient disk IO submission. Intel/DDN . 2.9 .
Kerberos Revival Fix up existing Kerberos code so it is tested working again. Xyratex . 2.8 .
UID/GID mapping Map UID/GID for remote client nodes to local UID/GID on the MDS and OSS. Allows a single Lustre filesystem to be shared across clients with administrative domains. Indiana University LU-3527 2.7 .
Shared Key Crypto Allow node authentication and/or RPC encryption using symmetric shared key crypto with GSSAPI. Avoids complexity in configuring Kerberos across multiple domains. Indiana University LU-3289 2.9 .
3.6/FC18 Server Patches . ORNL/NRL/SuSE/EMC LU-1812 2.5 2.5
Extended Attribute Batching Allow clients to fetch multiple xattrs in a single RPC for efficiency Xyratex LU-549 2.5 2.5
HSM Hierarchical Storage Manager to allow Lustre to interface with external tape libraries and automatically and transparently store files from Lustre to an external archive and the reverse. CEA LU-3608 2.5 2.5
Client in upstream kernel Include Lustre client in vanilla kernel release. EMC . 2.4 Linux 3.11
Reduce ldlm_poold Execution Time Reduce the overhead of the ldlm_poold thread that calculates which locks to expire on the client. Intel LU-2924 2.5 2.5
LNET Router Priorities Adds a parameter to route module parameters which sets a priority number for that route. Routes with the smallest priority number will be favored over ones with larger numbers Intel LU-2934 2.5 2.5
Endianness Fixes Fixes endian and byte-alignment issues in the Lustre codebase. Platforms under consideration for this project are x86 vs Sparc Intel LU-3221 2.5/2.7 2.5/2.7
errno Translation Tables Add addition of errno translations between architectures with different errno definitions Intel LU-2743 2.5 2.5
Unified parameter syntax Make conf_param and set_param syntax the same; add complete coverage Xyratex LU-3155 2.5 2.5
client-server shared modules cleanup Clean up Lustre client to facilitate pushing it into upstream Linux kernel. Need to split some server code dependencies from client to avoidmaking extensive server code changes EMC LU-1214 LU-1330 2.5 2.5/2.6
libcfs cleanup clean up libcfs to use Linux kernel APIs EMC LU-1346 2.5 2.5/2.6
LFSCK Phase 2 - MDT-OST consistency Traverse all inodes in an MDT and verify their corresponding objects on the OSTs exist. Intel LU-957 2.5 2.6
DNE Phase IIa Migration Tool, OUT Handler, OSP Object Cache, Simplify Commit Intel LU-3531 2.6 2.6
Lock ahead Allow user space to request LDLM extent locks in advance of need. Intended to optimize shared file IO. Cray [  ??

Again, please note that the targeted release information is provided by the developing organization for informational purposes only and is in no way a roadmap commitment until it appears on the Lustre Roadmap.

Lustre Community Roadmap