LWG Minutes 2014-12-17

From OpenSFS Wiki
Jump to navigation Jump to search


  • lustre.org status update
  • Agenda for Lustre Developer Project Meeting
  • Community development discussion


  • Peter Jones (Intel)
  • Justin Miller (IU)
  • Richard Henwood (Intel)
  • Cory Spitz (Cray)
  • Ruth Klundt (Sandia)
  • Chris Morrone (LLNL)
  • Colin Faber (Seagate)
  • Ann Koehler (Cray)
  • Andreas Dilger (Intel)
  • James Simmons (ORNL)


lustre.org status update

Chris gave an update on lustre.org:

lustre.org is now officially the property of EOFS and OpenSFS jointly. The contract details have been shared with key people from EOFS, OpenSFS, and Seagate. One of the agreements between EOFS and OpenSFS mandates the creation of a Lustre.org Working Group.

The first meeting of the Lustre.org Working Group occurred just before the LWG meeting. A wiki page for that working group and minutes of the meeting are posted at Lustre.org Working Group.

There was some discussion about revitalizing the lustre.org mailing lists. That is something that everyone agrees will be very valuable to the community, and we need to make that short term goal.

Some suggestions that were made:

The Lustre.org Working Group Technical Designee can be moderator for lustre.org mailing lists For instance, upstream Linux kernel developers share code through email, and [email protected] will be the appropriate place for those messages. Those people are not likely to be list subscribers though, and list non-subscribers need to have their posts moderated by a human (to avoid massive amounts of spam on the list).

We could advertise the January developer meeting on lustre.org.

Some talk about what to do with old content: Andreas suggested generatinga flat list of all pages under lustre.org and having people pick out urls to work on. Make a task list that people can pull from.

There was some discussion about the basic form and mechanics that the new lustre.org will use.

Chris suggested that the front web page and a few other top level pages would probably be better if they were hosted in a model CMS (content management system). For instance OpenSFS uses WordPress, and we could keep going that. The CMS pages would look nice, and serve as a front door to guide various classes of people (users, administrators, developers, etc.) quickly and easily to the content that they would want.

Most of the content would then live in a wiki, which has a good track record of allowing many different content contributors to work on content in parallel. Wikis are also relatively low on the learning curve. Wikis do not necessarily have a very pretty or inviting front page though, which is where the CMS would come in.

Most people seemed to like that idea.

Peter thought it would be nice if didn't need to go through an official web guy (or two) just to post information about events. Chris replied that the OpenSFS web guy has been very responsive, so that might not be much of a problem. But he promised to take that under consideration. We might give more people posting rights in the CMS if going through the official web maintainers proves too much overhead. We might have the events all be in the wiki. Options will be considered.

Agenda for Lustre Developer Project Meeting

We need agenda for the Lustre Developer Project Meeting in January.

Chris relayed that Board is very interested in having at least some content that is aimed at new developers.

Andreas suggested that information for new developers comes in two major forms:

  • Descriptions of the tools and practices (gerrit, jira, jenkins, maloo, patch submission, code style, etc.)
  • Talks describing how the code works for some system in Lustre

He wondered which we had in mind. Chris said that some of each would be fantastic.

Other ideas for the agenda:

Cray wants to talk about HSM improvemnts (someone from Cray, Patrick/Frank, could lead a session) Testing infrastructure, how to set up locally (James?) Talks about testing, test coverage Justin - focused attention about sorting info about lustre.org Peter - focus on the info there that is for developers Cory relayed more Cray topics of interest:

  • Lustre HSM
  • liblustreapi and LGPL
  • ldlm changes for better shared file IO

Ruth - strategy to profile server behavior? memory usage, monitoring Andreas suggested that Jinshan might be a good person to talk about memory profiling techniques too much of a stretch to talk about user manual? Best practices for code reviews (chris)

Community development discussion

Chris relayed the Board's desire for short term well defined projects that can be accomplished by the Lustre development community.

Chris shared his high level list of things that we need to be working on to make the Lustre code more approachable:

  • Document Lustre's protocol
  • Refactor code to make it more understandable and maintainable
  • Comment the code
  • Improve the Lustre web pages (including developer guides)
  • Improve the Lustre build system
  • Improve the Lustre Operations manual
  • Host lustre developer education events

Chris asked for input on the list. Is this the right list? Things that should be added? Things that should be removed.

Cory seconded Chris' list, and just wanted to know how do we get started.

Andreas suggested that the upstream Lustre client in the upstream Linux kernel is a pretty important project.

EMC started the effort and pledged to see it through, but unfortunately after the client got into staging that idea fell flat. Client is now more-or-less in limbo. Not getting nearly enough attention to get the Lustre client out of the staging area and into the Lustre kernel proper.

Upstream kernel is about 2.4-and-one-half.

Chris was worried that a long term project with more-or-less no end is not what the Board is looking for. Suggested that we might be able to break the Upstream Client project up into smaller more palatable sub-projects.

Andreas suggested a couple of stages:

  • Clean up upstream client to get it out of staging
  • Update upstream client from more recent lustre

(Note that the current client that is in Linux's staging area is old, essentially from Lustre 2.4-and-one-half)

Peter noted that his LAD talk had list of some known work needed for upstream client. That could be mined for things that could be called "projects".

Cory wanted to identify some areas in need of refactoring. Some ideas were:

James - libcfs, intermingled user- and kernel-space code. Very messy. Needs refactoring. Chris - statahead might still be in bad shape, despite recent improvements. Chris remembered noting about 10 things that could be improved in one of the jira tickets. That could be dug up. (Ruth?) ptlrpcd - half state machine, half flags, very difficult to follow.

Project to document subsystems. Not just code comments, but higher level documentation that describes subsystem interactions and live in the lustre/doc directory (e.g. clio.txt, osd-api.txt)

Next meeting cancelled

The LWG meeting on January 31 is cancelled due to the holidays.

Action Items