diff options
| author | Neil Brown <neil@brown.name> | 2015-10-26 15:35:54 +0900 |
|---|---|---|
| committer | Jonathan Corbet <corbet@lwn.net> | 2015-11-02 18:18:25 -0700 |
| commit | 3ce96239d482a7d2dfdc1f332152c580b219fef1 (patch) | |
| tree | 7888c1eeab361d49c5d1ad8d5c0e26789829cbed | |
| parent | 05be961772ef52c7bdfd237e61c3da0631cdb192 (diff) | |
| download | linux-3ce96239d482a7d2dfdc1f332152c580b219fef1.tar.gz linux-3ce96239d482a7d2dfdc1f332152c580b219fef1.tar.bz2 linux-3ce96239d482a7d2dfdc1f332152c580b219fef1.zip | |
Documentation: add new description of path-name lookup.
This document is based on three recent lwn.net articles.
Some of the introductory material and linkage between articles
has been removed, and some time-based descriptions have been
revised.
Also all links to code have been removed as the code is very close by.
Contains corrections and improvements from Randy Dunlap <rdunlap@infradead.org>.
Signed-off-by: NeilBrown <neil@brown.name>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
| -rw-r--r-- | Documentation/filesystems/path-lookup.md | 1297 | ||||
| -rw-r--r-- | Documentation/filesystems/path-lookup.txt | 2 |
2 files changed, 1298 insertions, 1 deletions
diff --git a/Documentation/filesystems/path-lookup.md b/Documentation/filesystems/path-lookup.md new file mode 100644 index 000000000000..1b39e084a2b2 --- /dev/null +++ b/Documentation/filesystems/path-lookup.md @@ -0,0 +1,1297 @@ +<head> +<style> p { max-width:50em} ol, ul {max-width: 40em}</style> +</head> + +Pathname lookup in Linux. +========================= + +This write-up is based on three articles published at lwn.net: + +- <https://lwn.net/Articles/649115/> Pathname lookup in Linux +- <https://lwn.net/Articles/649729/> RCU-walk: faster pathname lookup in Linux +- <https://lwn.net/Articles/650786/> A walk among the symlinks + +Written by Neil Brown with help from Al Viro and Jon Corbet. + +Introduction +------------ + +The most obvious aspect of pathname lookup, which very little +exploration is needed to discover, is that it is complex. There are +many rules, special cases, and implementation alternatives that all +combine to confuse the unwary reader. Computer science has long been +acquainted with such complexity and has tools to help manage it. One +tool that we will make extensive use of is "divide and conquer". For +the early parts of the analysis we will divide off symlinks - leaving +them until the final part. Well before we get to symlinks we have +another major division based on the VFS's approach to locking which +will allow us to review "REF-walk" and "RCU-walk" separately. But we +are getting ahead of ourselves. There are some important low level +distinctions we need to clarify first. + +There are two sorts of ... +-------------------------- + +[`openat()`]: http://man7.org/linux/man-pages/man2/openat.2.html + +Pathnames (sometimes "file names"), used to identify objects in the +filesystem, will be familiar to most readers. They contain two sorts +of elements: "slashes" that are sequences of one or more "`/`" +characters, and "components" that are sequences of one or more +non-"`/`" characters. These form two kinds of paths. Those that +start with slashes are "absolute" and start from the filesystem root. +The others are "relative" and start from the current directory, or +from some other location specified by a file descriptor given to a +"xxx`at`" system call such as "[`openat()`]". + +[`execveat()`]: http://man7.org/linux/man-pages/man2/execveat.2.html + +It is tempting to describe the second kind as starting with a +component, but that isn't always accurate: a pathname can lack both +slashes and components, it can be empty, in other words. This is +generally forbidden in POSIX, but some of those "xxx`at`" system calls +in Linux permit it when the `AT_EMPTY_PATH` flag is given. For +example, if you have an open file descriptor on an executable file you +can execute it by calling [`execveat()`] passing the file descriptor, +an empty path, and the `AT_EMPTY_PATH` flag. + +These paths can be divided into two sections: the final component and +everything else. The "everything else" is the easy bit. In all cases +it must identify a directory that already exists, otherwise an error +such as `ENOENT` or `ENOTDIR` will be reported. + +The final component is not so simple. Not only do different system +calls interpret it quite differently (e.g. some create it, some do +not), but it might not even exist: neither the empty pathname nor the +pathname that is just slashes have a final component. If it does +exist, it could be "`.`" or "`..`" which are handled quite differently +from other components. + +[POSIX]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_12 + +If a pathname ends with a slash, such as "`/tmp/foo/`" it might be +tempting to consider that to have an empty final component. In many +ways that would lead to correct results, but not always. In +particular, `mkdir()` and `rmdir()` each create or remove a directory named +by the final component, and they are required to work with pathnames +ending in "`/`". According to [POSIX] + +> A pathname that contains at least one non- <slash> character and +> that ends with one or more trailing <slash> characters shall not +> be resolved successfully unless the last pathname component before +> the trailing <slash> characters names an existing directory or a +> directory entry that is to be created for a directory immediately +> after the pathname is resolved. + +The Linux pathname walking code (mostly in `fs/namei.c`) deals with +all of these issues: breaking the path into components, handling the +"everything else" quite separately from the final component, and +checking that the trailing slash is not used where it isn't +permitted. It also addresses the important issue of concurrent +access. + +While one process is looking up a pathname, another might be making +changes that affect that lookup. One fairly extreme case is that if +"a/b" were renamed to "a/c/b" while another process were looking up +"a/b/..", that process might successfully resolve on "a/c". +Most races are much more subtle, and a big part of the task of +pathname lookup is to prevent them from having damaging effects. Many +of the possible races are seen most clearly in the context of the +"dcache" and an understanding of that is central to understanding +pathname lookup. + +More than just a cache. +----------------------- + +The "dcache" caches information about names in each filesystem to +make them quickly available for lookup. Each entry (known as a +"dentry") contains three significant fields: a component name, a +pointer to a parent dentry, and a pointer to the "inode" which +contains further information about the object in that parent with +the given name. The inode pointer can be `NULL` indicating that the +name doesn't exist in the parent. While there can be linkage in the +dentry of a directory to the dentries of the children, that linkage is +not used for pathname lookup, and so will not be considered here. + +The dcache has a number of uses apart from accelerating lookup. One +that will be particularly relevant is that it is closely integrated +with the mount table that records which filesystem is mounted where. +What the mount table actually stores is which dentry is mounted on top +of which other dentry. + +When considering the dcache, we have another of our "two types" +distinctions: there are two types of filesystems. + +Some filesystems ensure that the information in the dcache is always +completely accurate (though not necessarily complete). This can allow +the VFS to determine if a particular file does or doesn't exist +without checking with the filesystem, and means that the VFS can +protect the filesystem against certain races and other problems. +These are typically "local" filesystems such as ext3, XFS, and Btrfs. + +Other filesystems don't provide that guarantee because they cannot. +These are typically filesystems that are shared across a network, +whether remote filesystems like NFS and 9P, or cluster filesystems +like ocfs2 or cephfs. These filesystems allow the VFS to revalidate +cached information, and must provide their own protection against +awkward races. The VFS can detect these filesystems by the +`DCACHE_OP_REVALIDATE` flag being set in the dentry. + +REF-walk: simple concurrency management with refcounts and spinlocks +-------------------------------------------------------------------- + +With all of those divisions carefully classified, we can now start +looking at the actual process of walking along a path. In particular +we will start with the handling of the "everything else" part of a +pathname, and focus on the "REF-walk" approach to concurrency +management. This code is found in the `link_path_walk()` function, if +you ignore all the places that only run when "`LOOKUP_RCU`" +(indicating the use of RCU-walk) is set. + +[Meet the Lockers]: https://lwn.net/Articles/453685/ + +REF-walk is fairly heavy-handed with locks and reference counts. Not +as heavy-handed as in the old "big kernel lock" days, but certainly not +afraid of taking a lock when one is needed. It uses a variety of +different concurrency controls. A background understanding of the +various primitives is assumed, or can be gleaned from elsewhere such +as in [Meet the Lockers]. + +The locking mechanisms used by REF-walk include: + +### dentry->d_lockref ### + +This uses the lockref primitive to provide both a spinlock and a +reference count. The special-sauce of this primitive is that the +conceptual sequence "lock; inc_ref; unlock;" can often be performed +with a single atomic memory operation. + +Holding a reference on a dentry ensures that the dentry won't suddenly +be freed and used for something else, so the values in various fields +will behave as expected. It also protects the `->d_inode` reference +to the inode to some extent. + +The association between a dentry and its inode is fairly permanent. +For example, when a file is renamed, the dentry and inode move +together to the new location. When a file is created the dentry will +initially be negative (i.e. `d_inode` is `NULL`), and will be assigned +to the new inode as part of the act of creation. + +When a file is deleted, this can be reflected in the cache either by +setting `d_inode` to `NULL`, or by removing it from the hash table +(described shortly) used to look up the name in the parent directory. +If the dentry is still in use the second option is used as it is +perfectly legal to keep using an open file after it has been deleted +and having the dentry around helps. If the dentry is not otherwise in +use (i.e. if the refcount in `d_lockref` is one), only then will +`d_inode` be set to `NULL`. Doing it this way is more efficient for a +very common case. + +So as long as a counted reference is held to a dentry, a non-`NULL` `->d_inode` +value will never be changed. + +### dentry->d_lock ### + +`d_lock` is a synonym for the spinlock that is part of `d_lockref` above. +For our purposes, holding this lock protects against the dentry being +renamed or unlinked. In particular, its parent (`d_parent`), and its +name (`d_name`) cannot be changed, and it cannot be removed from the +dentry hash table. + +When looking for a name in a directory, REF-walk takes `d_lock` on +each candidate dentry that it finds in the hash table and then checks +that the parent and name are correct. So it doesn't lock the parent +while searching in the cache; it only locks children. + +When looking for the parent for a given name (to handle "`..`"), +REF-walk can take `d_lock` to get a stable reference to `d_parent`, +but it first tries a more lightweight approach. As seen in +`dget_parent()`, if a reference can be claimed on the parent, and if +subsequently `d_parent` can be seen to have not changed, then there is +no need to actually take the lock on the child. + +### rename_lock ### + +Looking up a given name in a given directory involves computing a hash +from the two values (the name and the dentry of the directory), +accessing that slot in a hash table, and searching the linked list +that is found there. + +When a dentry is renamed, the name and the parent dentry can both +change so the hash will almost certainly change too. This would move the +dentry to a different chain in the hash table. If a filename search +happened to be looking at a dentry that was moved in this way, +it might end up continuing the search down the wrong chain, +and so miss out on part of the correct chain. + +The name-lookup process (`d_lookup()`) does _not_ try to prevent this +from happening, but only to detect when it happens. +`rename_lock` is a seqlock that is updated whenever any dentry is +renamed. If `d_lookup` finds that a rename happened while it +unsuccessfully scanned a chain in the hash table, it simply tries +again. + +### inode->i_mutex ### + +`i_mutex` is a mutex that serializes all changes to a particular +directory. This ensures that, for example, an `unlink()` and a `rename()` +cannot both happen at the same time. It also keeps the directory +stable while the filesystem is asked to look up a name that is not +currently in the dcache. + +This has a complementary role to that of `d_lock`: `i_mutex` on a +directory protects all of the names in that directory, while `d_lock` +on a name protects just one name in a directory. Most changes to the +dcache hold `i_mutex` on the relevant directory inode and briefly take +`d_lock` on one or more the dentries while the change happens. One +exception is when idle dentries are removed from the dcache due to +memory pressure. This uses `d_lock`, but `i_mutex` plays no role. + +The mutex affects pathname lookup in two distinct ways. Firstly it +serializes lookup of a name in a directory. `walk_component()` uses +`lookup_fast()` first which, in turn, checks to see if the name is in the cache, +using only `d_lock` locking. If the name isn't found, then `walk_component()` +falls back to `lookup_slow()` which takes `i_mutex`, checks again that +the name isn't in the cache, and then calls in to the filesystem to get a +definitive answer. A new dentry will be added to the cache regardless of +the result. + +Secondly, when pathname lookup reaches the final component, it will +sometimes need to take `i_mutex` before performing the last lookup so +that the required exclusion can be achieved. How path lookup chooses +to take, or not take, `i_mutex` is one of the +issues addressed in a subsequent section. + +### mnt->mnt_count ### + +`mnt_count` is a per-CPU reference counter on "`mount`" structures. +Per-CPU here means that incrementing the count is cheap as it only +uses CPU-local memory, but checking if the count is zero is expensive as +it needs to check with every CPU. Taking a `mnt_count` reference +prevents the mount structure from disappearing as the result of regular +unmount operations, but does not prevent a "lazy" unmount. So holding +`mnt_count` doesn't ensure that the mount remains in the namespace and, +in particular, doesn't stabilize the link to the mounted-on dentry. It +does, however, ensure that the `mount` data structure remains coherent, +and it provides a reference to the root dentry of the mounted +filesystem. So a reference through `->mnt_count` provides a stable +reference to the mounted dentry, but not the mounted-on dentry. + +### mount_lock ### + +`mount_lock` is a global seqlock, a bit like `rename_lock`. It can be used to +check if any change has been made to any mount points. + +While walking down the tree (away from the root) this lock is used when +crossing a mount point to check that the crossing was safe. That is, +the value in the seqlock is read, then the code finds the mount that +is mounted on the current directory, if there is one, and increments +the `mnt_count`. Finally the value in `mount_lock` is checked against +the old value. If there is no change, then the crossing was safe. If there +was a change, the `mnt_count` is decremented and the whole process is +retried. + +When walking up the tree (towards the root) by following a ".." link, +a little more care is needed. In this case the seqlock (which +contains both a counter and a spinlock) is fully locked to prevent +any changes to any mount points while stepping up. This locking is +needed to stabilize the link to the mounted-on dentry, which the +refcount on the mount itself doesn't ensure. + +### RCU ### + +Finally the global (but extremely lightweight) RCU read lock is held +from time to time to ensure certain data structures don't get freed +unexpectedly. + +In particular it is held while scanning chains in the dcache hash +table, and the mount point hash table. + +Bringing it together with `struct nameidata` +-------------------------------------------- + +[First edition Unix]: http://minnie.tuhs.org/cgi-bin/utree.pl?file=V1/u2.s + +Throughout the process of walking a path, the current status is stored +in a `struct nameidata`, "namei" being the traditional name - dating +all the way back to [First Edition Unix] - of the function that +converts a "name" to an "inode". `struct nameidata` contains (among +other fields): + +### `struct path path` ### + +A `path` contains a `struct vfsmount` (which is +embedded in a `struct mount`) and a `struct dentry`. Together these +record the current status of the walk. They start out referring to the +starting point (the current working directory, the root directory, or some other +directory identified by a file descriptor), and are updated on each +step. A reference through `d_lockref` and `mnt_count` is always +held. + +### `struct qstr last` ### + +This is a string together with a length (i.e. _not_ `nul` terminated) +that is the "next" component in the pathname. + +### `int last_type` ### + +This is one of `LAST_NORM`, `LAST_ROOT`, `LAST_DOT`, `LAST_DOTDOT`, or +`LAST_BIND`. The `last` field is only valid if the type is +`LAST_NORM`. `LAST_BIND` is used when following a symlink and no +components of the symlink have been processed yet. Others should be +fairly self-explanatory. + +### `struct path root` ### + +This is used to hold a reference to the effective root of the +filesystem. Often that reference won't be needed, so this field is +only assigned the first time it is used, or when a non-standard root +is requested. Keeping a reference in the `nameidata` ensures that +only one root is in effect for the entire path walk, even if it races +with a `chroot()` system call. + +The root is needed when either of two conditions holds: (1) either the +pathname or a symbolic link starts with a "'/'", or (2) a "`..`" +component is being handled, since "`..`" from the root must always stay +at the root. The value used is usually the current root directory of +the calling process. An alternate root can be provided as when +`sysctl()` calls `file_open_root()`, and when NFSv4 or Btrfs call +`mount_subtree()`. In each case a pathname is being looked up in a very +specific part of the filesystem, and the lookup must not be allowed to +escape that subtree. It works a bit like a local `chroot()`. + +Ignoring the handling of symbolic links, we can now describe the +"`link_path_walk()`" function, which handles the lookup of everything +except the final component as: + +> Given a path (`name`) and a nameidata structure (`nd`), check that the +> current directory has execute permission and then advance `name` +> over one component while updating `last_type` and `last`. If that +> was the final component, then return, otherwise call +> `walk_component()` and repeat from the top. + +`walk_component()` is even easier. If the component is `LAST_DOTS`, +it calls `handle_dots()` which does the necessary locking as already +described. If it finds a `LAST_NORM` component it first calls +"`lookup_fast()`" which only looks in the dcache, but will ask the +filesystem to revalidate the result if it is that sort of filesystem. +If that doesn't get a good result, it calls "`lookup_slow()`" which +takes the `i_mutex`, rechecks the cache, and then asks the filesystem +to find a definitive answer. Each of these will call +`follow_managed()` (as described below) to handle any mount points. + +In the absence of symbolic links, `walk_component()` creates a new +`struct path` containing a counted reference to the new dentry and a +reference to the new `vfsmount` which is only counted if it is +different from the previous `vfsmount`. It then calls +`path_to_nameidata()` to install the new `struct path` in the +`struct nameidata` and drop the unneeded references. + +This "hand-over-hand" sequencing of getting a reference to the new +dentry before dropping the reference to the previous dentry may +seem obvious, but is worth pointing out so that we will recognize its +analogue in the "RCU-walk" version. + +Handling the final component. +----------------------------- + +`link_path_walk()` only walks as far as setting `nd->last` and +`nd->last_type` to refer to the final component of the path. It does +not call `walk_component()` that last time. Handling that final +component remains for the caller to sort out. Those callers are +`path_lookupat()`, `path_parentat()`, `path_mountpoint()` and +`path_openat()` each of which handles the differing requirements of +different system calls. + +`path_parentat()` is clearly the simplest - it just wraps a little bit +of housekeeping around `link_path_walk()` and returns the parent +directory and final component to the caller. The caller will be either +aiming to create a name (via `filename_create()`) or remove or rename +a name (in which case `user_path_parent()` is used). They will use +`i_mutex` to exclude other changes while they validate and then +perform their operation. + +`path_lookupat()` is nearly as simple - it is used when an existing +object is wanted such as by `stat()` or `chmod()`. It essentially just +calls `walk_component()` on the final component through a call to +`lookup_last()`. `path_lookupat()` returns just the final dentry. + +`path_mountpoint()` handles the special case of unmounting which must +not try to revalidate the mounted filesystem. It effectively +contains, through a call to `mountpoint_last()`, an alternate +implementation of `lookup_slow()` which skips that step. This is +important when unmounting a filesystem that is inaccessible, such as +one provided by a dead NFS server. + +Finally `path_openat()` is used for the `open()` system call; it +contains, in support functions starting with "`do_last()`", all the +complexity needed to handle the different subtleties of O_CREAT (with +or without O_EXCL), final "`/`" characters, and trailing symbolic +links. We will revisit this in the final part of this series, which +focuses on those symbolic links. "`do_last()`" will sometimes, but +not always, take `i_mutex`, depending on what it finds. + +Each of these, or the functions which call them, need to be alert to +the possibility that the final component is not `LAST_NORM`. If the +goal of the lookup is to create something, then any value for +`last_type` other than `LAST_NORM` will result in an error. For +example if `path_parentat()` reports `LAST_DOTDOT`, then the caller +won't try to create that name. They also check for trailing slashes +by testing `last.name[last.len]`. If there is any character beyond +the final component, it must be a trailing slash. + +Revalidation and automounts +--------------------------- + +Apart from symbolic links, there are only two parts of the "REF-walk" +process not yet covered. One is the handling of stale cache entries +and the other is automounts. + +On filesystems that require it, the lookup routines will call the +`->d_revalidate()` dentry method to ensure that the cached information +is current. This will often confirm validity or update a few details +from a server. In some cases it may find that there has been change +further up the path and that something that was thought to be valid +previously isn't really. When this happens the lookup of the whole +path is aborted and retried with the "`LOOKUP_REVAL`" flag set. This +forces revalidation to be more thorough. We will see more details of +this retry process in the next article. + +Automount points are locations in the filesystem where an attempt to +lookup a name can trigger changes to how that lookup should be +handled, in particular by mounting a filesystem there. These are +covered in greater detail in autofs4.txt in the Linux documentation +tree, but a few notes specifically related to path lookup are in order +here. + +The Linux VFS has a concept of "managed" dentries which is reflected +in function names such as "`follow_managed()`". There are three +potentially interesting things about these dentries corresponding +to three different flags that might be set in `dentry->d_flags`: + +### `DCACHE_MANAGE_TRANSIT` ### + +If this flag has been set, then the filesystem has requested that the +`d_manage()` dentry operation be called before handling any possible +mount point. This can perform two particular services: + +It can block to avoid races. If an automount point is being +unmounted, the `d_manage()` function will usually wait for that +process to complete before letting the new lookup proceed and possibly +trigger a new automount. + +It can selectively allow only some processes to transit through a +mount point. When a server process is managing automounts, it may +need to access a directory without triggering normal automount +processing. That server process can identify itself to the `autofs` +filesystem, which will then give it a special pass through +`d_manage()` by returning `-EISDIR`. + +### `DCACHE_MOUNTED` ### + +This flag is set on every dentry that is mounted on. As Linux +supports multiple filesystem namespaces, it is possible that the +dentry may not be mounted on in *this* namespace, just in some +other. So this flag is seen as a hint, not a promise. + +If this flag is set, and `d_manage()` didn't return `-EISDIR`, +`lookup_mnt()` is called to examine the mount hash table (honoring the +`mount_lock` described earlier) and possibly return a new `vfsmount` +and a new `dentry` (both with counted references). + +### `DCACHE_NEED_AUTOMOUNT` ### + +If `d_manage()` allowed us to get this far, and `lookup_mnt()` didn't +find a mount point, then this flag causes the `d_automount()` dentry +operation to be called. + +The `d_automount()` operation can be arbitrarily complex and may +communicate with server processes etc. but it should ultimately either +report that there was an error, that there was nothing to mount, or +should provide an updated `struct path` with new `dentry` and `vfsmount`. + +In the latter case, `finish_automount()` will be called to safely +install the new mount point into the mount table. + +There is no new locking of import here and it is important that no +locks (only counted references) are held over this processing due to +the very real possibility of extended delays. +This will become more important next time when we examine RCU-walk +which is particularly sensitive to delays. + +RCU-walk - faster pathname lookup in Linux +========================================== + +RCU-walk is another algorithm for performing pathname lookup in Linux. +It is in many ways similar to REF-walk and the two share quite a bit +of code. The significant difference in RCU-walk is how it allows for +the possibility of concurrent access. + +We noted that REF-walk is complex because there are numerous details +and special cases. RCU-walk reduces this complexity by simply +refusing to handle a number of cases -- it instead falls back to +REF-walk. The difficulty with RCU-walk comes from a different +direction: unfamiliarity. The locking rules when depending on RCU are +quite different from traditional locking, so we will spend a little extra +time when we come to those. + +Clear demarcation of roles +-------------------------- + +The easiest way to manage concurrency is to forcibly stop any other +thread from changing the data structures that a given thread is +looking at. In cases where no other thread would even think of +changing the data and lots of different threads want to read at the +same time, this can be very costly. Even when using locks that permit +multiple concurrent readers, the simple act of updating the count of +the number of current readers can impose an unwanted cost. So the +goal when reading a shared data structure that no other process is +changing is to avoid writing anything to memory at all. Take no +locks, increment no counts, leave no footprints. + +The REF-walk mechanism already described certainly doesn't follow this +principle, but then it is really designed to work when there may well +be other threads modifying the data. RCU-walk, in contrast, is +designed for the common situation where there are lots of frequent +readers and only occasional writers. This may not be common in all +parts of the filesystem tree, but in many parts it will be. For the +other parts it is important that RCU-walk can quickly fall back to +using REF-walk. + +Pathname lookup always starts in RCU-walk mode but only remains there +as long as what it is looking for is in the cache and is stable. It +dances lightly down the cached filesystem image, leaving no footprints +and carefully watching where it is, to be sure it doesn't trip. If it +notices that something has changed or is changing, or if something +isn't in the cache, then it tries to stop gracefully and switch to +REF-walk. + +This stopping requires getting a counted reference on the current +`vfsmount` and `dentry`, and ensuring that these are still valid - +that a path walk with REF-walk would have found the same entries. +This is an invariant that RCU-walk must guarantee. It can only make +decisions, such as selecting the next step, that are decisions which +REF-walk could also have made if it were walking down the tree at the +same time. If the graceful stop succeeds, the rest of the path is +processed with the reliable, if slightly sluggish, REF-walk. If +RCU-walk finds it cannot stop gracefully, it simply gives up and +restarts from the top with REF-walk. + +This pattern of "try RCU-walk, if that fails try REF-walk" can be +clearly seen in functions like `filename_lookup()`, +`filename_parentat()`, `filename_mountpoint()`, +`do_filp_open()`, and `do_file_open_root()`. These five +correspond roughly to the four `path_`* functions we met earlier, +each of which calls `link_path_walk()`. The `path_*` functions are +called using different mode flags until a mode is found which works. +They are first called with `LOOKUP_RCU` set to request "RCU-walk". If +that fails with the error `ECHILD` they are called again with no +special flag to request "REF-walk". If either of those report the +error `ESTALE` a final attempt is made with `LOOKUP_REVAL` set (and no +`LOOKUP_RCU`) to ensure that entries found in the cache are forcibly +revalidated - normally entries are only revalidated if the filesystem +determines that they are too old to trust. + +The `LOOKUP_RCU` attempt may drop that flag internally and switch to +REF-walk, but will never then try to switch back to RCU-walk. Places +that trip up RCU-walk are much more likely to be near the leaves and +so it is very unlikely that there will be much, if any, benefit from +switching back. + +RCU and seqlocks: fast and light +-------------------------------- + +RCU is, unsurprisingly, critical to RCU-walk mode. The +`rcu_read_lock()` is held for the entire time that RCU-walk is walking +down a path. The particular guarantee it provides is that the key +data structures - dentries, inodes, super_blocks, and mounts - will +not be freed while the lock is held. They might be unlinked or +invalidated in one way or another, but the memory will not be +repurposed so values in various fields will still be meaningful. This +is the only guarantee that RCU provides; everything else is done using +seqlocks. + +As we saw above, REF-walk holds a counted reference to the current +dentry and the current vfsmount, and does not release those references +before taking references to the "next" dentry or vfsmount. It also +sometimes takes the `d_lock` spinlock. These references and locks are +taken to prevent certain changes from happening. RCU-walk must not +take those references or locks and so cannot prevent such changes. +Instead, it checks to see if a change has been made, and aborts or +retries if it has. + +To preserve the invariant mentioned above (that RCU-walk may only make +decisions that REF-walk could have made), it must make the checks at +or near the same places that REF-walk holds the references. So, when +REF-walk increments a reference count or takes a spinlock, RCU-walk +samples the status of a seqlock using `read_seqcount_begin()` or a +similar function. When REF-walk decrements the count or drops the +lock, RCU-walk checks if the sampled status is still valid using +`read_seqcount_retry()` or similar. + +However, there is a little bit more to seqlocks than that. If +RCU-walk accesses two different fields in a seqlock-protected +structure, or accesses the same field twice, there is no a priori +guarantee of any consistency between those accesses. When consistency +is needed - which it usually is - RCU-walk must take a copy and then +use `read_seqcount_retry()` to validate that copy. + +`read_seqcount_retry()` not only checks the sequence number, but also +imposes a memory barrier so that no memory-read instruction from +*before* the call can be delayed until *after* the call, either by the +CPU or by the compiler. A simple example of this can be seen in +`slow_dentry_cmp()` which, for filesystems which do not use simple +byte-wise name equality, calls into the filesystem to compare a name +against a dentry. The length and name pointer are copied into local +variables, then `read_seqcount_retry()` is called to confirm the two +are consistent, and only then is `->d_compare()` called. When +standard filename comparison is used, `dentry_cmp()` is called +instead. Notably it does _not_ use `read_seqcount_retry()`, but +instead has a large comment explaining why the consistency guarantee +isn't necessary. A subsequent `read_seqcount_retry()` will be +sufficient to catch any problem that could occur at this point. + +With that little refresher on seqlocks out of the way we can look at +the bigger picture of how RCU-walk uses seqlocks. + +### `mount_lock` and `nd->m_seq` ### + +We already met the `mount_lock` seqlock when REF-walk used it to +ensure that crossing a mount point is performed safely. RCU-walk uses +it for that too, but for quite a bit more. + +Instead of taking a counted reference to each `vfsmount` as it +descends the tree, RCU-walk samples the state of `mount_lock` at the +start of the walk and stores this initial sequence number in the +`struct nameidata` in the `m_seq` field. This one lock and one +sequence number are used to validate all accesses to all `vfsmounts`, +and all mount point crossings. As changes to the mount table are +relatively rare, it is reasonable to fall back on REF-walk any time +that any "mount" or "unmount" happens. + +`m_seq` is checked (using `read_seqretry()`) at the end of an RCU-walk +sequence, whether switching to REF-walk for the rest of the path or +when the end of the path is reached. It is also checked when stepping +down over a mount point (in `__follow_mount_rcu()`) or up (in +`follow_dotdot_rcu()`). If it is ever found to have changed, the +whole RCU-walk sequence is aborted and the path is processed again by +REF-walk. + +If RCU-walk finds that `mount_lock` hasn't changed then it can be sure +that, had REF-walk taken counted references on each vfsmount, the +results would have been the same. This ensures the invariant holds, +at least for vfsmount structures. + +### `dentry->d_seq` and `nd->seq`. ### + +In place of taking a count or lock on `d_reflock`, RCU-walk samples +the per-dentry `d_seq` seqlock, and stores the sequence number in the +`seq` field of the nameidata structure, so `nd->seq` should always be +the current sequence number of `nd->dentry`. This number needs to be +revalidated after copying, and before using, the name, parent, or +inode of the dentry. + +The handling of the name we have already looked at, and the parent is +only accessed in `follow_dotdot_rcu()` which fairly trivially follows +the required pattern, though it does so for three different cases. + +When not at a mount point, `d_parent` is followed and its `d_seq` is +collected. When we are at a mount point, we instead follow the +`mnt->mnt_mountpoint` link to get a new dentry and collect its +`d_seq`. Then, after finally finding a `d_parent` to follow, we must +check if we have landed on a mount point and, if so, must find that +mount point and follow the `mnt->mnt_root` link. This would imply a +somewhat unusual, but certainly possible, circumstance where the +starting point of the path lookup was in part of the filesystem that +was mounted on, and so not visible from the root. + +The inode pointer, stored in `->d_inode`, is a little more +interesting. The inode will always need to be accessed at least +twice, once to determine if it is NULL and once to verify access +permissions. Symlink handling requires a validated inode pointer too. +Rather than revalidating on each access, a copy is made on the first +access and it is stored in the `inode` field of `nameidata` from where +it can be safely accessed without further validation. + +`lookup_fast()` is the only lookup routine that is used in RCU-mode, +`lookup_slow()` being too slow and requiring locks. It is in +`lookup_fast()` that we find the important "hand over hand" tracking +of the current dentry. + +The current `dentry` and current `seq` number are passed to +`__d_lookup_rcu()` which, on success, returns a new `dentry` and a +new `seq` number. `lookup_fast()` then copies the inode pointer and +revalidates the new `seq` number. It then validates the old `dentry` +with the old `seq` number one last time and only then continues. This +process of getting the `seq` number of the new dentry and then +checking the `seq` number of the old exactly mirrors the process of +getting a counted reference to the new dentry before dropping that for +the old dentry which we saw in REF-walk. + +### No `inode->i_mutex` or even `rename_lock` ### + +A mutex is a fairly heavyweight lock that can only be taken when it is +permissible to sleep. As `rcu_read_lock()` forbids sleeping, +`inode->i_mutex` plays no role in RCU-walk. If some other thread does +take `i_mutex` and modifies the directory in a way that RCU-walk needs +to notice, the result will be either that RCU-walk fails to find the +dentry that it is looking for, or it will find a dentry which +`read_seqretry()` won't validate. In either case it will drop down to +REF-walk mode which can take whatever locks are needed. + +Though `rename_lock` could be used by RCU-walk as it doesn't require +any sleeping, RCU-walk doesn't bother. REF-walk uses `rename_lock` to +protect against the possibility of hash chains in the dcache changing +while they are being searched. This can result in failing to find +something that actually is there. When RCU-walk fails to find +something in the dentry cache, whether it is really there or not, it +already drops down to REF-walk and tries again with appropriate +locking. This neatly handles all cases, so adding extra checks on +rename_lock would bring no significant value. + +`unlazy walk()` and `complete_walk()` +------------------------------------- + +That "dropping down to REF-walk" typically involves a call to +`unlazy_walk()`, so named because "RCU-walk" is also sometimes +referred to as "lazy walk". `unlazy_walk()` is called when +following the path down to the current vfsmount/dentry pair seems to +have proceeded successfully, but the next step is problematic. This +can happen if the next name cannot be found in the dcache, if +permission checking or name revalidation couldn't be achieved while +the `rcu_read_lock()` is held (which forbids sleeping), if an +automount point is found, or in a couple of cases involving symlinks. +It is also called from `complete_walk()` when the lookup has reached +the final component, or the very end of the path, depending on which +particular flavor of lookup is used. + +Other reasons for dropping out of RCU-walk that do not trigger a call +to `unlazy_walk()` are when some inconsistency is found that cannot be +handled immediately, such as `mount_lock` or one of the `d_seq` +seqlocks repo |