| Written by: Neil Brown |
| Please see MAINTAINERS file for where to send questions. |
| |
| Overlay Filesystem |
| ================== |
| |
| This document describes a prototype for a new approach to providing |
| overlay-filesystem functionality in Linux (sometimes referred to as |
| union-filesystems). An overlay-filesystem tries to present a |
| filesystem which is the result over overlaying one filesystem on top |
| of the other. |
| |
| The result will inevitably fail to look exactly like a normal |
| filesystem for various technical reasons. The expectation is that |
| many use cases will be able to ignore these differences. |
| |
| This approach is 'hybrid' because the objects that appear in the |
| filesystem do not all appear to belong to that filesystem. In many |
| cases an object accessed in the union will be indistinguishable |
| from accessing the corresponding object from the original filesystem. |
| This is most obvious from the 'st_dev' field returned by stat(2). |
| |
| While directories will report an st_dev from the overlay-filesystem, |
| all non-directory objects will report an st_dev from the lower or |
| upper filesystem that is providing the object. Similarly st_ino will |
| only be unique when combined with st_dev, and both of these can change |
| over the lifetime of a non-directory object. Many applications and |
| tools ignore these values and will not be affected. |
| |
| Upper and Lower |
| --------------- |
| |
| An overlay filesystem combines two filesystems - an 'upper' filesystem |
| and a 'lower' filesystem. When a name exists in both filesystems, the |
| object in the 'upper' filesystem is visible while the object in the |
| 'lower' filesystem is either hidden or, in the case of directories, |
| merged with the 'upper' object. |
| |
| It would be more correct to refer to an upper and lower 'directory |
| tree' rather than 'filesystem' as it is quite possible for both |
| directory trees to be in the same filesystem and there is no |
| requirement that the root of a filesystem be given for either upper or |
| lower. |
| |
| The lower filesystem can be any filesystem supported by Linux and does |
| not need to be writable. The lower filesystem can even be another |
| overlayfs. The upper filesystem will normally be writable and if it |
| is it must support the creation of trusted.* extended attributes, and |
| must provide valid d_type in readdir responses, so NFS is not suitable. |
| |
| A read-only overlay of two read-only filesystems may use any |
| filesystem type. |
| |
| Directories |
| ----------- |
| |
| Overlaying mainly involves directories. If a given name appears in both |
| upper and lower filesystems and refers to a non-directory in either, |
| then the lower object is hidden - the name refers only to the upper |
| object. |
| |
| Where both upper and lower objects are directories, a merged directory |
| is formed. |
| |
| At mount time, the two directories given as mount options "lowerdir" and |
| "upperdir" are combined into a merged directory: |
| |
| mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\ |
| workdir=/work /merged |
| |
| The "workdir" needs to be an empty directory on the same filesystem |
| as upperdir. |
| |
| Then whenever a lookup is requested in such a merged directory, the |
| lookup is performed in each actual directory and the combined result |
| is cached in the dentry belonging to the overlay filesystem. If both |
| actual lookups find directories, both are stored and a merged |
| directory is created, otherwise only one is stored: the upper if it |
| exists, else the lower. |
| |
| Only the lists of names from directories are merged. Other content |
| such as metadata and extended attributes are reported for the upper |
| directory only. These attributes of the lower directory are hidden. |
| |
| credentials |
| ----------- |
| |
| By default, all access to the upper, lower and work directories is the |
| recorded mounter's MAC and DAC credentials. The incoming accesses are |
| checked against the caller's credentials. |
| |
| In the case where caller MAC or DAC credentials do not overlap, a |
| use case available in older versions of the driver, the |
| override_creds mount flag can be turned off and help when the use |
| pattern has caller with legitimate credentials where the mounter |
| does not. Several unintended side effects will occur though. The |
| caller without certain key capabilities or lower privilege will not |
| always be able to delete files or directories, create nodes, or |
| search some restricted directories. The ability to search and read |
| a directory entry is spotty as a result of the cache mechanism not |
| retesting the credentials because of the assumption, a privileged |
| caller can fill cache, then a lower privilege can read the directory |
| cache. The uneven security model where cache, upperdir and workdir |
| are opened at privilege, but accessed without creating a form of |
| privilege escalation, should only be used with strict understanding |
| of the side effects and of the security policies. |
| |
| whiteouts and opaque directories |
| -------------------------------- |
| |
| In order to support rm and rmdir without changing the lower |
| filesystem, an overlay filesystem needs to record in the upper filesystem |
| that files have been removed. This is done using whiteouts and opaque |
| directories (non-directories are always opaque). |
| |
| A whiteout is created as a character device with 0/0 device number. |
| When a whiteout is found in the upper level of a merged directory, any |
| matching name in the lower level is ignored, and the whiteout itself |
| is also hidden. |
| |
| A directory is made opaque by setting the xattr "trusted.overlay.opaque" |
| to "y". Where the upper filesystem contains an opaque directory, any |
| directory in the lower filesystem with the same name is ignored. |
| |
| readdir |
| ------- |
| |
| When a 'readdir' request is made on a merged directory, the upper and |
| lower directories are each read and the name lists merged in the |
| obvious way (upper is read first, then lower - entries that already |
| exist are not re-added). This merged name list is cached in the |
| 'struct file' and so remains as long as the file is kept open. If the |
| directory is opened and read by two processes at the same time, they |
| will each have separate caches. A seekdir to the start of the |
| directory (offset 0) followed by a readdir will cause the cache to be |
| discarded and rebuilt. |
| |
| This means that changes to the merged directory do not appear while a |
| directory is being read. This is unlikely to be noticed by many |
| programs. |
| |
| seek offsets are assigned sequentially when the directories are read. |
| Thus if |
| - read part of a directory |
| - remember an offset, and close the directory |
| - re-open the directory some time later |
| - seek to the remembered offset |
| |
| there may be little correlation between the old and new locations in |
| the list of filenames, particularly if anything has changed in the |
| directory. |
| |
| Readdir on directories that are not merged is simply handled by the |
| underlying directory (upper or lower). |
| |
| |
| Non-directories |
| --------------- |
| |
| Objects that are not directories (files, symlinks, device-special |
| files etc.) are presented either from the upper or lower filesystem as |
| appropriate. When a file in the lower filesystem is accessed in a way |
| the requires write-access, such as opening for write access, changing |
| some metadata etc., the file is first copied from the lower filesystem |
| to the upper filesystem (copy_up). Note that creating a hard-link |
| also requires copy_up, though of course creation of a symlink does |
| not. |
| |
| The copy_up may turn out to be unnecessary, for example if the file is |
| opened for read-write but the data is not modified. |
| |
| The copy_up process first makes sure that the containing directory |
| exists in the upper filesystem - creating it and any parents as |
| necessary. It then creates the object with the same metadata (owner, |
| mode, mtime, symlink-target etc.) and then if the object is a file, the |
| data is copied from the lower to the upper filesystem. Finally any |
| extended attributes are copied up. |
| |
| Once the copy_up is complete, the overlay filesystem simply |
| provides direct access to the newly created file in the upper |
| filesystem - future operations on the file are barely noticed by the |
| overlay filesystem (though an operation on the name of the file such as |
| rename or unlink will of course be noticed and handled). |
| |
| |
| Multiple lower layers |
| --------------------- |
| |
| Multiple lower layers can now be given using the the colon (":") as a |
| separator character between the directory names. For example: |
| |
| mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged |
| |
| As the example shows, "upperdir=" and "workdir=" may be omitted. In |
| that case the overlay will be read-only. |
| |
| The specified lower directories will be stacked beginning from the |
| rightmost one and going left. In the above example lower1 will be the |
| top, lower2 the middle and lower3 the bottom layer. |
| |
| |
| Non-standard behavior |
| --------------------- |
| |
| The copy_up operation essentially creates a new, identical file and |
| moves it over to the old name. The new file may be on a different |
| filesystem, so both st_dev and st_ino of the file may change. |
| |
| Any open files referring to this inode will access the old data and |
| metadata. Similarly any file locks obtained before copy_up will not |
| apply to the copied up file. |
| |
| On a file opened with O_RDONLY fchmod(2), fchown(2), futimesat(2) and |
| fsetxattr(2) will fail with EROFS. |
| |
| If a file with multiple hard links is copied up, then this will |
| "break" the link. Changes will not be propagated to other names |
| referring to the same inode. |
| |
| Symlinks in /proc/PID/ and /proc/PID/fd which point to a non-directory |
| object in overlayfs will not contain valid absolute paths, only |
| relative paths leading up to the filesystem's root. This will be |
| fixed in the future. |
| |
| Some operations are not atomic, for example a crash during copy_up or |
| rename will leave the filesystem in an inconsistent state. This will |
| be addressed in the future. |
| |
| Changes to underlying filesystems |
| --------------------------------- |
| |
| Offline changes, when the overlay is not mounted, are allowed to either |
| the upper or the lower trees. |
| |
| Changes to the underlying filesystems while part of a mounted overlay |
| filesystem are not allowed. If the underlying filesystem is changed, |
| the behavior of the overlay is undefined, though it will not result in |
| a crash or deadlock. |
| |
| Testsuite |
| --------- |
| |
| There's testsuite developed by David Howells at: |
| |
| git://git.infradead.org/users/dhowells/unionmount-testsuite.git |
| |
| Run as root: |
| |
| # cd unionmount-testsuite |
| # ./run --ov |