View source for MDS SMP Node Affinity ScopeStatement wiki version

                 
== Introduction ==
 
The following scope statement applies to the SMP Node Affinity Scope Statement project within the SFS-DEV-001 contract/SOW dates 08/01/2011.
 
== Problem Statement ==
 
Followers of CPU design are observing steady increases in core count. As core count increases thread resource contention (particularly locking) threatens to throttle Lustre server performance.
 
A multi-core CPUs are now common. Core count is expected to continue climbing and as it does a Lustre server will need to increase thread count to exploit the CPU capacity. On Lustre today additional threads will contend a single wait-queue, share one request queue, and share only a couple of global locks. Hence, without additional work, Lustre servers are expected to perform badly on multi-core CPUs for reasons including:
 
* Overhead of lock contention.
* Overhead of process switching between cores.
* Scheduler attempting to balance threads across cores.
 
These factors can be addressed by splitting the computing cores into configurable Compute Partitions. Lustre RPC service threads will then be bound to a specific Compute Partition. This design:
 
* Reduces the overhead of threads switching cores by keeping the thread running on the same core as the cache memory.
* Avoids needless contention on the inter-CPU memory subsystem.
* Keeps RPC request processing local to the resources that they affect.
* Allows the protocol stack to scale effectively as the number of cores increases.
 
== Project Goals ==
 
SMP node affinity is concerned with improving vertical scalability of a Lustre server by addressing software insufficiency on multi-core machines. This will enable Lustre to fully exploit increasingly powerful server hardware as it becomes available.

SMP node affinity will implement the following features:

* General libcfs APIs to provide a framework to support Compute Partitions
* Fine-grained SMP locking for Lustre
* SMP node affinity threading mode for Lustre
 
Benchmarks for metadata performance will be made on a cluster with a reasonable number of clients and a large (12+) number of cores.
 
The new features will demonstrate that performance of a single metadata server with a large (12+) cores is improved for file operations: create/remove/stat.

 
Code will land on WC-Lustre master branch.

 
== In-Scope ==
 
* High level design document for SMP node affinity of Lustre.
* Improve small message rate of LNET on multiple (12+) core server.
* Improve small RPC rate of ptlrpc on multiple (12+) core server.
* Improve general metadata performance on a multiple (12+) core server for narrow strip-count file.
* OST stack performance will not exhibit regression.
* SMP node affinity code will take place against WC-Lustre 2.x baseline. Code will cover:
** General SMP improvements for Lustre and LNET.
** libcfs APIs to support CPU partition and NUMA allocators.
** SMP node affinity threading mode for Lustre and LNET.
*** LND threads
*** ptlrpc service threads
*** ptlrpc reply handling threads
* Update manual to include SMP node affinity tuning parameters.
 
== Out of Scope ==
 
* Wide strip count files involve additional factors that will not be helpful for demonstrating performance improvements.
 
== Project Constraints ==
 
* Liang is the only engineer with the expertise to lead this work.
* Test cluster with suitable multiple (12+) core nodes available to WC engineers regardless of nationality.
 
== Key Deliverables ==
 
* Signed-off Milestone document for the project phases:
** Solution Architecture
** High-level Design
* Test plan
* Source code against WC-Lustre 2.x that implements the feature requirements and runs on the test cluster.
* Code landed on Master WC-Lustre 2.x as Issue [[LU-56]].
 
=== Glossary ===
 
* CPU partition<br> CPU partition is subset of processing resource of system, a CPU partition could be any number of CPU cores in system: it can be a single core, or any specified number of cores, or stand for all cores in a NUMA node, or represent all CPUs of a system.  The number of CPU partitions can be set by libcfs APIs.<br> A fat server will be divided into several compute partitions, each compute partition contains: cores in CPU-partition, memory pool, message queue, threads pool, it's a little like virtual machine, although it's much simpler since the the Lustre RPC processing threads will still have access to all Lustre global state if needed.