http://wiki.opensfs.org/index.php?action=history&feed=atom&title=BWG_FSM_IO_Process_ModelBWG FSM IO Process Model - Revision history2026-07-23T09:50:02ZRevision history for this page on the wikiMediaWiki 1.43.3http://wiki.opensfs.org/index.php?title=BWG_FSM_IO_Process_Model&diff=1332&oldid=prevUselton at 19:44, 16 December 20142014-12-16T19:44:07Z<p></p>
<table style="background-color: #fff; color: #202122;" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 12:44, 16 December 2014</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l11">Line 11:</td>
<td colspan="2" class="diff-lineno">Line 11:</td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>I've done these experiments enough that I have some expectations, but my main interest is in finding problems and issues. That means I want to look closely enough that I will find ways the system doesn't fit my preconceived notions. Needless to say, the black-box approach is much more time consuming. </div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>I've done these experiments enough that I have some expectations, but my main interest is in finding problems and issues. That means I want to look closely enough that I will find ways the system doesn't fit my preconceived notions. Needless to say, the black-box approach is much more time consuming. </div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div> </div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div> </div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>The mental image of file system performance follows the behavior of an application running on a set compute nodes. The application has periods when it is not conducting I/O (it is computing or waiting at a barrier or whatever) and then periods when it does I/O. During the actual processing of a write() system call, for example, we can guess that the node will work on that I/O as fast as it can, but the I/O may consist of a long sequence of write() calls, that have other activity mixed in. Now the node will probably just copy the write() to a memory buffer and that will go fast. Then the OS will drain those buffers to disk at a somewhat slower rate. So long as the buffers never entirely fill the I/O looks like it is very fast. As soon as the OS has to block the application while buffers drain then the I/O looks like it is slowing down to disk (or controller or network) speed. Finally, there can be multiple layers of such buffer cache behavior, so performance can be intimately dependent of the amount size, frequency and timing of the many write() calls on the many nodes, as well as the details of the system construction. Repeat the foregoing with read I/Os in place of wirte I/Os except that the buffer cache can't hide disk and network read latency while it can read form previously cached writes and it can read ahead.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>The mental image of file system performance follows the behavior of an application running on a set compute nodes. The application has periods when it is not conducting I/O (it is computing or waiting at a barrier or whatever) and then periods when it does I/O. During the actual processing of a write() system call, for example, we can guess that the node will work on that I/O as fast as it can, but the I/O may consist of a long sequence of write() calls, that have other activity mixed in. Now the node will probably just copy the write() to a memory buffer and that will go fast. Then the OS will drain those buffers to disk at a somewhat slower rate. So long as the buffers never entirely fill the I/O looks like it is very fast. As soon as the OS has to block the application while buffers drain then the I/O looks like it is slowing down to disk (or controller or network) speed. Finally, there can be multiple layers of such buffer cache behavior, so performance can be intimately dependent of the amount<ins style="font-weight: bold; text-decoration: none;">, </ins>size, frequency<ins style="font-weight: bold; text-decoration: none;">, </ins>and timing of the many write() calls on the many nodes, as well as the details of the system construction. Repeat the foregoing with read I/Os in place of wirte I/Os except that the buffer cache can't hide disk and network read latency while it can read form previously cached writes and it can read ahead.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div> </div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div> </div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Thus one's mental model looks like a very vague and highly incorrect queuing network who's properties you want to discover. If you have some reason to think that your mental model is very close to correct then you may only do a few experiments to prove that. Of course, that means you won't be able to see if and where it's wrong in many cases :)</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Thus one's mental model looks like a very vague and highly incorrect queuing network who's properties you want to discover. If you have some reason to think that your mental model is very close to correct then you may only do a few experiments to prove that. Of course, that means you won't be able to see if and where it's wrong in many cases :)</div></td></tr>
<!-- diff cache key wiki_opensfs_org-wiki_:diff:1.41:old-1331:rev-1332:php=table -->
</table>Useltonhttp://wiki.opensfs.org/index.php?title=BWG_FSM_IO_Process_Model&diff=1331&oldid=prevUselton at 19:41, 16 December 20142014-12-16T19:41:29Z<p></p>
<table style="background-color: #fff; color: #202122;" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 12:41, 16 December 2014</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l2">Line 2:</td>
<td colspan="2" class="diff-lineno">Line 2:</td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Return to [[BWG_File_System_Monitoring|File System Monitoring]] page.</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Return to [[BWG_File_System_Monitoring|File System Monitoring]] page.</div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">(Paraphrase of discussion with Bevin Brett, December 2014)</ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">There are two ways people approach understanding the performance of a new (to them) file system. Someone who is an expert in the domain often starts with a pretty detailed mental model of the system and will do a few experiments to calibrate that model. You see that a lot, and it can be good for a shoot-from-the-hip situation. </ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"> </ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">The other way of approaching a new system is more black-box. In that case you run a series of experiments, and only as you get the data on performance throughout the space of possible experiments do you then develop a model to explain what you see. If you aren't a super expert in the particular hardware and software then this is pretty much what you have to do. </ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"> </ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">I've done these experiments enough that I have some expectations, but my main interest is in finding problems and issues. That means I want to look closely enough that I will find ways the system doesn't fit my preconceived notions. Needless to say, the black-box approach is much more time consuming. </ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"> </ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">The mental image of file system performance follows the behavior of an application running on a set compute nodes. The application has periods when it is not conducting I/O (it is computing or waiting at a barrier or whatever) and then periods when it does I/O. During the actual processing of a write() system call, for example, we can guess that the node will work on that I/O as fast as it can, but the I/O may consist of a long sequence of write() calls, that have other activity mixed in. Now the node will probably just copy the write() to a memory buffer and that will go fast. Then the OS will drain those buffers to disk at a somewhat slower rate. So long as the buffers never entirely fill the I/O looks like it is very fast. As soon as the OS has to block the application while buffers drain then the I/O looks like it is slowing down to disk (or controller or network) speed. Finally, there can be multiple layers of such buffer cache behavior, so performance can be intimately dependent of the amount size, frequency and timing of the many write() calls on the many nodes, as well as the details of the system construction. Repeat the foregoing with read I/Os in place of wirte I/Os except that the buffer cache can't hide disk and network read latency while it can read form previously cached writes and it can read ahead.</ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"> </ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">Thus one's mental model looks like a very vague and highly incorrect queuing network who's properties you want to discover. If you have some reason to think that your mental model is very close to correct then you may only do a few experiments to prove that. Of course, that means you won't be able to see if and where it's wrong in many cases :)</ins></div></td></tr>
<!-- diff cache key wiki_opensfs_org-wiki_:diff:1.41:old-552:rev-1331:php=table -->
</table>Useltonhttp://wiki.opensfs.org/index.php?title=BWG_FSM_IO_Process_Model&diff=552&oldid=prevLforbes: Created page with " Return to File System Monitoring page."2013-07-26T19:04:34Z<p>Created page with " Return to <a href="/BWG_File_System_Monitoring" title="BWG File System Monitoring">File System Monitoring</a> page."</p>
<p><b>New page</b></p><div><br />
<br />
Return to [[BWG_File_System_Monitoring|File System Monitoring]] page.</div>Lforbes