Benchmarking Working Group

Benchmarking Working Group Wiki Pages
Information about the BWG is found at the at opensfs.org website

LUG 2013 OpenSFS BWG Update Presentation

File System I/O hero run best practices
Lead: Ben Evans

Members: Mark Nelson, Ilene Carpenter, Rick Roloff, Nathan Rutman, Liam Forbes

Hero Run Best Practices

I/O Workload Characterization
Lead: Pietro Cicotti

Members: Ilene Carpenter, Rick Mohr, Mike Booth, Ben Evans

Application I/O kernel extraction
Lead: Ilene Carpenter

Members: Jeff Layton, Pietro Ciccotti, Bobbie Lind

File System Monitoring
Lead: Liam Forbes Members: Alan Wild, Andrew Uselton, Ben Evans, Cheng Shao, Jeff Garlough, Jeff Layton, Mark Nelson, Nic Henke

Metadata Performance Evaluation
Lead: Sorin Fabish

Members: Branislav Radovanovic, Rick Roloff, Cheng Shao, Wang Yibin, Keith Mannthey, Bobbie Lind, Greg Farnum

[MPEE_BWG]

The task of the BWG Metadata Performance Evaluation Effort (MPEE) group is to:
 * 1) Build/select tools that will allow evaluation of File System Metadata performance and scalability
 * 2) The tools will help detect pockets of Metadata low performance in cases when users complain of extreme slowness  of MD operations
 * 3) Benchmark tools will support: POSIX, MPI, and Transactional operations (for CEPH and DAOS)
 * 4) Address the very high end HPC as well as small and medium installations benchmark needs
 * 5) Tools applicable to Lustre and: CEPH, GPFS…

Current MPEE proposed list of benchmarks:
 * 1) mdtest – widely used in HPC
 * 2) fstest - used by pvfs/OrangeFS community
 * 3) Postmark and MPI version - old NetApp benchmark
 * 4) Netmist and MPI version – used by SPECsfs
 * 5) Synthetic tools – used by LANL, ORNL
 * 6) MDS-Survey - Intel’s metadata workload simulator.
 * 7) Any known open source metadata tools used in HPC
 * 8) Add new Lustre statistics specific to MD operations.

MPEE Usecases
 * mdtest: test file MD operations on MDS: open, create, lookups, readdir; used in academia and as a comparison tool of FS MD.
 * fstest: small I/O’s and small files as well as lookups, targeting both MDS and OSS operations and MD HA for multiple MDS’s.
 * Postmark: old NetApp benchmark – I built an MPI version; it is used to measure MD operations and file size scalability and files per directory scalability.
 * Netmist: used to model any workload from statistics including all MD operations and file operations. Can model Workload objects for I/O performance mixes and combination of I/O and MD. Suitable for initial evaluation of storage as well as for performance troubleshooting.

MPEE Proposed Roadmap
 * Collect benchmark tools candidates from OpenSFS
 * Evaluate all the tools and the workloads that can benchmarked
 * Recommend a small set of MD benchmark tools to cover the majority of MD workloads
 * Collect stats from users of MD benchmarks
 * Build scripts to allow ease of use of the recommended tools
 * Write documentation for troubleshooting MD performance problems using the toolset
 * Create a special website for MD tools

MPEE Asks from OpenSFS
 * Share any open source synthetic benchmarks code
 * Share a list of MD benchmark tools they currently use to allow select the most suitable and used candidates
 * Share MD operations tested to allow build Netmist workload objects
 * Share the MD workloads that create pain points to Lustre FS
 * Share cases of poor MD performance workloads and applications

Return to [Benchmarking_Working_Group] page.