blob: 02ab997a1ed276d6e32928ec48cd2e82164d807a [file] [log] [blame]
Balbir Singh00f0b822008-03-04 14:28:39 -08001Memory Resource Controller
2
Jörg Sommer67de0162011-06-15 13:00:47 -07003NOTE: The Memory Resource Controller has generically been referred to as the
4 memory controller in this document. Do not confuse memory controller
5 used here with the memory controller that is used in hardware.
Balbir Singh1b6df3a2008-02-07 00:13:46 -08006
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -07007(For editors)
8In this document:
9 When we mention a cgroup (cgroupfs's directory) with memory controller,
10 we call it "memory cgroup". When you see git-log and source code, you'll
11 see patch's title and function names tend to use "memcg".
12 In this document, we avoid using it.
Balbir Singh1b6df3a2008-02-07 00:13:46 -080013
Balbir Singh1b6df3a2008-02-07 00:13:46 -080014Benefits and Purpose of the memory controller
15
16The memory controller isolates the memory behaviour of a group of tasks
17from the rest of the system. The article on LWN [12] mentions some probable
18uses of the memory controller. The memory controller can be used to
19
20a. Isolate an application or a group of applications
Michael Kerrisk1939c552012-10-08 16:33:09 -070021 Memory-hungry applications can be isolated and limited to a smaller
Balbir Singh1b6df3a2008-02-07 00:13:46 -080022 amount of memory.
Michael Kerrisk1939c552012-10-08 16:33:09 -070023b. Create a cgroup with a limited amount of memory; this can be used
Balbir Singh1b6df3a2008-02-07 00:13:46 -080024 as a good alternative to booting with mem=XXXX.
25c. Virtualization solutions can control the amount of memory they want
26 to assign to a virtual machine instance.
27d. A CD/DVD burner could control the amount of memory used by the
28 rest of the system to ensure that burning does not fail due to lack
29 of available memory.
Michael Kerrisk1939c552012-10-08 16:33:09 -070030e. There are several other use cases; find one or use the controller just
Balbir Singh1b6df3a2008-02-07 00:13:46 -080031 for fun (to learn and hack on the VM subsystem).
32
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070033Current Status: linux-2.6.34-mmotm(development version of 2010/April)
34
35Features:
36 - accounting anonymous pages, file caches, swap caches usage and limiting them.
Ying Han6252efc2012-04-12 12:49:10 -070037 - pages are linked to per-memcg LRU exclusively, and there is no global LRU.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070038 - optionally, memory+swap usage can be accounted and limited.
39 - hierarchical accounting
40 - soft limit
Michael Kerrisk1939c552012-10-08 16:33:09 -070041 - moving (recharging) account at moving a task is selectable.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070042 - usage threshold notifier
Anton Vorontsov70ddf632013-04-29 15:08:31 -070043 - memory pressure notifier
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070044 - oom-killer disable knob and oom-notifier
45 - Root cgroup has no limit controls.
46
Michael Kerrisk1939c552012-10-08 16:33:09 -070047 Kernel memory support is a work in progress, and the current version provides
Glauber Costa65c64ce2011-12-22 01:02:27 +000048 basically functionality. (See Section 2.7)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070049
50Brief summary of control files.
51
52 tasks # attach a task(thread) and show list of threads
53 cgroup.procs # show list of processes
54 cgroup.event_control # an interface for event_fd()
Daisuke Nishimuraa111c962011-04-27 15:26:48 -070055 memory.usage_in_bytes # show current res_counter usage for memory
56 (See 5.5 for details)
57 memory.memsw.usage_in_bytes # show current res_counter usage for memory+Swap
58 (See 5.5 for details)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070059 memory.limit_in_bytes # set/show limit of memory usage
60 memory.memsw.limit_in_bytes # set/show limit of memory+Swap usage
61 memory.failcnt # show the number of memory usage hits limits
62 memory.memsw.failcnt # show the number of memory+Swap hits limits
63 memory.max_usage_in_bytes # show max memory usage recorded
Zhu Yanhaid66c1ce2012-01-12 17:18:24 -080064 memory.memsw.max_usage_in_bytes # show max memory+Swap usage recorded
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070065 memory.soft_limit_in_bytes # set/show soft limit of memory usage
66 memory.stat # show various statistics
67 memory.use_hierarchy # set/show hierarchical account enabled
68 memory.force_empty # trigger forced move charge to parent
Anton Vorontsov70ddf632013-04-29 15:08:31 -070069 memory.pressure_level # set memory pressure notifications
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070070 memory.swappiness # set/show swappiness parameter of vmscan
71 (See sysctl's vm.swappiness)
72 memory.move_charge_at_immigrate # set/show controls of moving charges
73 memory.oom_control # set/show oom controls.
Ying Han50c35e52011-06-15 15:08:16 -070074 memory.numa_stat # show the number of memory usage per numa node
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070075
Glauber Costad5bdae72012-12-18 14:22:22 -080076 memory.kmem.limit_in_bytes # set/show hard limit for kernel memory
77 memory.kmem.usage_in_bytes # show current kernel memory allocation
78 memory.kmem.failcnt # show the number of kernel memory usage hits limits
79 memory.kmem.max_usage_in_bytes # show max kernel memory usage recorded
80
Glauber Costa3aaabe22011-12-11 21:47:06 +000081 memory.kmem.tcp.limit_in_bytes # set/show hard limit for tcp buf memory
Glauber Costa5a6dd342011-12-11 21:47:07 +000082 memory.kmem.tcp.usage_in_bytes # show current tcp buf memory allocation
Wanpeng Li05a73ed2012-07-31 16:43:21 -070083 memory.kmem.tcp.failcnt # show the number of tcp buf memory usage hits limits
84 memory.kmem.tcp.max_usage_in_bytes # show max tcp buf memory usage recorded
Glauber Costae5671df2011-12-11 21:47:01 +000085
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800861. History
87
88The memory controller has a long history. A request for comments for the memory
89controller was posted by Balbir Singh [1]. At the time the RFC was posted
90there were several implementations for memory control. The goal of the
91RFC was to build consensus and agreement for the minimal features required
92for memory control. The first RSS controller was posted by Balbir Singh[2]
93in Feb 2007. Pavel Emelianov [3][4][5] has since posted three versions of the
94RSS controller. At OLS, at the resource management BoF, everyone suggested
95that we handle both page cache and RSS together. Another request was raised
96to allow user space handling of OOM. The current memory controller is
97at version 6; it combines both mapped (RSS) and unmapped Page
98Cache Control [11].
99
1002. Memory Control
101
102Memory is a unique resource in the sense that it is present in a limited
103amount. If a task requires a lot of CPU processing, the task can spread
104its processing over a period of hours, days, months or years, but with
105memory, the same physical memory needs to be reused to accomplish the task.
106
107The memory controller implementation has been divided into phases. These
108are:
109
1101. Memory controller
1112. mlock(2) controller
1123. Kernel user memory accounting and slab control
1134. user mappings length controller
114
115The memory controller is the first controller developed.
116
1172.1. Design
118
119The core of the design is a counter called the res_counter. The res_counter
120tracks the current memory usage and limit of the group of processes associated
121with the controller. Each cgroup has a memory controller specific data
122structure (mem_cgroup) associated with it.
123
1242.2. Accounting
125
126 +--------------------+
127 | mem_cgroup |
128 | (res_counter) |
129 +--------------------+
130 / ^ \
131 / | \
132 +---------------+ | +---------------+
133 | mm_struct | |.... | mm_struct |
134 | | | | |
135 +---------------+ | +---------------+
136 |
137 + --------------+
138 |
139 +---------------+ +------+--------+
140 | page +----------> page_cgroup|
141 | | | |
142 +---------------+ +---------------+
143
144 (Figure 1: Hierarchy of Accounting)
145
146
147Figure 1 shows the important aspects of the controller
148
1491. Accounting happens per cgroup
1502. Each mm_struct knows about which cgroup it belongs to
1513. Each page has a pointer to the page_cgroup, which in turn knows the
152 cgroup it belongs to
153
Jeff Liu348b4652012-12-11 16:01:28 -0800154The accounting is done as follows: mem_cgroup_charge_common() is invoked to
155set up the necessary data structures and check if the cgroup that is being
156charged is over its limit. If it is, then reclaim is invoked on the cgroup.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800157More details can be found in the reclaim section of this document.
158If everything goes well, a page meta-data-structure called page_cgroup is
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700159updated. page_cgroup has its own LRU on cgroup.
160(*) page_cgroup structure is allocated at boot/memory-hotplug time.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800161
1622.2.1 Accounting details
163
KAMEZAWA Hiroyuki5b4e6552008-10-18 20:28:10 -0700164All mapped anon pages (RSS) and cache pages (Page Cache) are accounted.
Ying Han6252efc2012-04-12 12:49:10 -0700165Some pages which are never reclaimable and will not be on the LRU
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700166are not accounted. We just account pages under usual VM management.
KAMEZAWA Hiroyuki5b4e6552008-10-18 20:28:10 -0700167
168RSS pages are accounted at page_fault unless they've already been accounted
169for earlier. A file page will be accounted for as Page Cache when it's
170inserted into inode (radix-tree). While it's mapped into the page tables of
171processes, duplicate accounting is carefully avoided.
172
Michael Kerrisk1939c552012-10-08 16:33:09 -0700173An RSS page is unaccounted when it's fully unmapped. A PageCache page is
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700174unaccounted when it's removed from radix-tree. Even if RSS pages are fully
175unmapped (by kswapd), they may exist as SwapCache in the system until they
Michael Kerrisk1939c552012-10-08 16:33:09 -0700176are really freed. Such SwapCaches are also accounted.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700177A swapped-in page is not accounted until it's mapped.
178
Michael Kerrisk1939c552012-10-08 16:33:09 -0700179Note: The kernel does swapin-readahead and reads multiple swaps at once.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700180This means swapped-in pages may contain pages for other tasks than a task
181causing page fault. So, we avoid accounting at swap-in I/O.
KAMEZAWA Hiroyuki5b4e6552008-10-18 20:28:10 -0700182
183At page migration, accounting information is kept.
184
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700185Note: we just account pages-on-LRU because our purpose is to control amount
186of used pages; not-on-LRU pages tend to be out-of-control from VM view.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800187
1882.3 Shared Page Accounting
189
190Shared pages are accounted on the basis of the first touch approach. The
191cgroup that first touches a page is accounted for the page. The principle
192behind this approach is that a cgroup that aggressively uses a shared
193page will eventually get charged for it (once it is uncharged from
194the cgroup that brought it in -- this will happen on memory pressure).
195
KAMEZAWA Hiroyuki4b913552012-05-29 15:06:51 -0700196But see section 8.2: when moving a task to another cgroup, its pages may
197be recharged to the new cgroup, if move_charge_at_immigrate has been chosen.
198
Paul Bolledf7c6b92013-03-25 23:59:16 +0100199Exception: If CONFIG_MEMCG_SWAP is not used.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800200When you do swapoff and make swapped-out pages of shmem(tmpfs) to
KAMEZAWA Hiroyukid13d1442009-01-07 18:07:56 -0800201be backed into memory in force, charges for pages are accounted against the
202caller of swapoff rather than the users of shmem.
203
Andrew Mortonc255a452012-07-31 16:43:02 -07002042.4 Swap Extension (CONFIG_MEMCG_SWAP)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700205
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800206Swap Extension allows you to record charge for swap. A swapped-in page is
207charged back to original page allocator if possible.
208
209When swap is accounted, following files are added.
210 - memory.memsw.usage_in_bytes.
211 - memory.memsw.limit_in_bytes.
212
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700213memsw means memory+swap. Usage of memory+swap is limited by
214memsw.limit_in_bytes.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800215
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700216Example: Assume a system with 4G of swap. A task which allocates 6G of memory
217(by mistake) under 2G memory limitation will use all swap.
218In this case, setting memsw.limit_in_bytes=3G will prevent bad use of swap.
Michael Kerrisk1939c552012-10-08 16:33:09 -0700219By using the memsw limit, you can avoid system OOM which can be caused by swap
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700220shortage.
221
222* why 'memory+swap' rather than swap.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800223The global LRU(kswapd) can swap out arbitrary pages. Swap-out means
224to move account from memory to swap...there is no change in usage of
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700225memory+swap. In other words, when we want to limit the usage of swap without
226affecting global LRU, memory+swap limit is better than just limiting swap from
Michael Kerrisk1939c552012-10-08 16:33:09 -0700227an OS point of view.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800228
KAMEZAWA Hiroyuki22a668d2009-06-17 16:27:19 -0700229* What happens when a cgroup hits memory.memsw.limit_in_bytes
Jörg Sommer67de0162011-06-15 13:00:47 -0700230When a cgroup hits memory.memsw.limit_in_bytes, it's useless to do swap-out
KAMEZAWA Hiroyuki22a668d2009-06-17 16:27:19 -0700231in this cgroup. Then, swap-out will not be done by cgroup routine and file
232caches are dropped. But as mentioned above, global LRU can do swapout memory
233from it for sanity of the system's memory management state. You can't forbid
234it by cgroup.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800235
2362.5 Reclaim
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800237
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700238Each cgroup maintains a per cgroup LRU which has the same structure as
239global VM. When a cgroup goes over its limit, we first try
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800240to reclaim memory from the cgroup so as to make space for the new
241pages that the cgroup has touched. If the reclaim is unsuccessful,
242an OOM routine is invoked to select and kill the bulkiest task in the
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700243cgroup. (See 10. OOM Control below.)
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800244
245The reclaim algorithm has not been modified for cgroups, except that
Michael Kerrisk1939c552012-10-08 16:33:09 -0700246pages that are selected for reclaiming come from the per-cgroup LRU
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800247list.
248
Balbir Singh4b3bde42009-09-23 15:56:32 -0700249NOTE: Reclaim does not work for the root cgroup, since we cannot set any
250limits on the root cgroup.
251
KAMEZAWA Hiroyukidaaf1e62010-03-10 15:22:32 -0800252Note2: When panic_on_oom is set to "2", the whole system will panic.
253
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700254When oom event notifier is registered, event will be delivered.
255(See oom_control section)
256
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -07002572.6 Locking
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800258
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700259 lock_page_cgroup()/unlock_page_cgroup() should not be called under
260 mapping->tree_lock.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800261
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700262 Other lock order is following:
263 PG_locked.
264 mm->page_table_lock
265 zone->lru_lock
266 lock_page_cgroup.
267 In many cases, just lock_page_cgroup() is called.
268 per-zone-per-cgroup LRU (cgroup's private LRU) is just guarded by
269 zone->lru_lock, it has no lock of its own.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800270
Andrew Mortonc255a452012-07-31 16:43:02 -07002712.7 Kernel Memory Extension (CONFIG_MEMCG_KMEM)
Glauber Costae5671df2011-12-11 21:47:01 +0000272
Vladimir Davydov2ee06462014-06-04 16:07:28 -0700273WARNING: Current implementation lacks reclaim support. That means allocation
274 attempts will fail when close to the limit even if there are plenty of
275 kmem available for reclaim. That makes this option unusable in real
276 life so DO NOT SELECT IT unless for development purposes.
277
Glauber Costae5671df2011-12-11 21:47:01 +0000278With the Kernel memory extension, the Memory Controller is able to limit
279the amount of kernel memory used by the system. Kernel memory is fundamentally
280different than user memory, since it can't be swapped out, which makes it
281possible to DoS the system by consuming too much of this precious resource.
282
Glauber Costad5bdae72012-12-18 14:22:22 -0800283Kernel memory won't be accounted at all until limit on a group is set. This
284allows for existing setups to continue working without disruption. The limit
285cannot be set if the cgroup have children, or if there are already tasks in the
286cgroup. Attempting to set the limit under those conditions will return -EBUSY.
287When use_hierarchy == 1 and a group is accounted, its children will
288automatically be accounted regardless of their limit value.
289
290After a group is first limited, it will be kept being accounted until it
291is removed. The memory limitation itself, can of course be removed by writing
292-1 to memory.kmem.limit_in_bytes. In this case, kmem will be accounted, but not
293limited.
294
Glauber Costae5671df2011-12-11 21:47:01 +0000295Kernel memory limits are not imposed for the root cgroup. Usage for the root
Glauber Costad5bdae72012-12-18 14:22:22 -0800296cgroup may or may not be accounted. The memory used is accumulated into
297memory.kmem.usage_in_bytes, or in a separate counter when it makes sense.
298(currently only for tcp).
299The main "kmem" counter is fed into the main counter, so kmem charges will
300also be visible from the user counter.
Glauber Costae5671df2011-12-11 21:47:01 +0000301
Glauber Costae5671df2011-12-11 21:47:01 +0000302Currently no soft limit is implemented for kernel memory. It is future work
303to trigger slab reclaim when those limits are reached.
304
3052.7.1 Current Kernel Memory resources accounted
306
Glauber Costad5bdae72012-12-18 14:22:22 -0800307* stack pages: every process consumes some stack pages. By accounting into
308kernel memory, we prevent new processes from being created when the kernel
309memory usage is too high.
310
Glauber Costa92e79342012-12-18 14:23:08 -0800311* slab pages: pages allocated by the SLAB or SLUB allocator are tracked. A copy
Anatol Pomozovf884ab12013-05-08 16:56:16 -0700312of each kmem_cache is created every time the cache is touched by the first time
Glauber Costa92e79342012-12-18 14:23:08 -0800313from inside the memcg. The creation is done lazily, so some objects can still be
314skipped while the cache is being created. All objects in a slab page should
315belong to the same memcg. This only fails to hold when a task is migrated to a
316different memcg during the page allocation by the cache.
317
Glauber Costae1aab162011-12-11 21:47:03 +0000318* sockets memory pressure: some sockets protocols have memory pressure
319thresholds. The Memory Controller allows them to be controlled individually
320per cgroup, instead of globally.
Glauber Costae5671df2011-12-11 21:47:01 +0000321
Glauber Costad1a4c0b2011-12-11 21:47:04 +0000322* tcp memory pressure: sockets memory pressure for the tcp protocol.
323
Glauber Costad5bdae72012-12-18 14:22:22 -08003242.7.3 Common use cases
325
326Because the "kmem" counter is fed to the main user counter, kernel memory can
327never be limited completely independently of user memory. Say "U" is the user
328limit, and "K" the kernel limit. There are three possible ways limits can be
329set:
330
331 U != 0, K = unlimited:
332 This is the standard memcg limitation mechanism already present before kmem
333 accounting. Kernel memory is completely ignored.
334
335 U != 0, K < U:
336 Kernel memory is a subset of the user memory. This setup is useful in
337 deployments where the total amount of memory per-cgroup is overcommited.
338 Overcommiting kernel memory limits is definitely not recommended, since the
339 box can still run out of non-reclaimable memory.
340 In this case, the admin could set up K so that the sum of all groups is
341 never greater than the total memory, and freely set U at the cost of his
342 QoS.
343
344 U != 0, K >= U:
345 Since kmem charges will also be fed to the user counter and reclaim will be
346 triggered for the cgroup for both kinds of memory. This setup gives the
347 admin a unified view of memory, and it is also useful for people who just
348 want to track kernel memory usage.
349
Balbir Singh1b6df3a2008-02-07 00:13:46 -08003503. User Interface
351
3520. Configuration
353
354a. Enable CONFIG_CGROUPS
355b. Enable CONFIG_RESOURCE_COUNTERS
Andrew Mortonc255a452012-07-31 16:43:02 -0700356c. Enable CONFIG_MEMCG
357d. Enable CONFIG_MEMCG_SWAP (to use swap extension)
Glauber Costad5bdae72012-12-18 14:22:22 -0800358d. Enable CONFIG_MEMCG_KMEM (to use kmem extension)
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800359
Jörg Sommerf6e07d32011-06-15 12:59:45 -07003601. Prepare the cgroups (see cgroups.txt, Why are cgroups needed?)
361# mount -t tmpfs none /sys/fs/cgroup
362# mkdir /sys/fs/cgroup/memory
363# mount -t cgroup none /sys/fs/cgroup/memory -o memory
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800364
3652. Make the new group and move bash into it
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700366# mkdir /sys/fs/cgroup/memory/0
367# echo $$ > /sys/fs/cgroup/memory/0/tasks
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800368
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700369Since now we're in the 0 cgroup, we can alter the memory limit:
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700370# echo 4M > /sys/fs/cgroup/memory/0/memory.limit_in_bytes
Balbir Singh0eea1032008-02-07 00:13:57 -0800371
372NOTE: We can use a suffix (k, K, m, M, g or G) to indicate values in kilo,
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700373mega or gigabytes. (Here, Kilo, Mega, Giga are Kibibytes, Mebibytes, Gibibytes.)
374
Daisuke Nishimurac5b947b2009-06-17 16:27:20 -0700375NOTE: We can write "-1" to reset the *.limit_in_bytes(unlimited).
Balbir Singh4b3bde42009-09-23 15:56:32 -0700376NOTE: We cannot set limits on the root cgroup any more.
Balbir Singh0eea1032008-02-07 00:13:57 -0800377
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700378# cat /sys/fs/cgroup/memory/0/memory.limit_in_bytes
Li Zefan2324c5d2008-02-23 15:24:12 -08003794194304
Balbir Singh0eea1032008-02-07 00:13:57 -0800380
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800381We can check the usage:
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700382# cat /sys/fs/cgroup/memory/0/memory.usage_in_bytes
Li Zefan2324c5d2008-02-23 15:24:12 -08003831216512
Balbir Singh0eea1032008-02-07 00:13:57 -0800384
Michael Kerrisk1939c552012-10-08 16:33:09 -0700385A successful write to this file does not guarantee a successful setting of
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700386this limit to the value written into the file. This can be due to a
Balbir Singh0eea1032008-02-07 00:13:57 -0800387number of factors, such as rounding up to page boundaries or the total
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700388availability of memory on the system. The user is required to re-read
Balbir Singh0eea1032008-02-07 00:13:57 -0800389this file after a write to guarantee the value committed by the kernel.
390
Balbir Singhfb789222008-03-04 14:28:24 -0800391# echo 1 > memory.limit_in_bytes
Balbir Singh0eea1032008-02-07 00:13:57 -0800392# cat memory.limit_in_bytes
Li Zefan2324c5d2008-02-23 15:24:12 -08003934096
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800394
395The memory.failcnt field gives the number of times that the cgroup limit was
396exceeded.
397
KAMEZAWA Hiroyukidfc05c22008-02-07 00:14:41 -0800398The memory.stat file gives accounting information. Now, the number of
399caches, RSS and Active pages/Inactive pages are shown.
400
Balbir Singh1b6df3a2008-02-07 00:13:46 -08004014. Testing
402
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700403For testing features and implementation, see memcg_test.txt.
404
405Performance test is also important. To see pure memory controller's overhead,
406testing on tmpfs will give you good numbers of small overheads.
407Example: do kernel make on tmpfs.
408
409Page-fault scalability is also important. At measuring parallel
410page fault test, multi-process test may be better than multi-thread
411test because it has noise of shared objects/status.
412
413But the above two are testing extreme situations.
414Trying usual test under memory controller is always helpful.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800415
4164.1 Troubleshooting
417
418Sometimes a user might find that the application under a cgroup is
Michael Kerrisk1939c552012-10-08 16:33:09 -0700419terminated by the OOM killer. There are several causes for this:
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800420
4211. The cgroup limit is too low (just too low to do anything useful)
4222. The user is using anonymous memory and swap is turned off or too low
423
424A sync followed by echo 1 > /proc/sys/vm/drop_caches will help get rid of
425some of the pages cached in the cgroup (page cache pages).
426
Michael Kerrisk1939c552012-10-08 16:33:09 -0700427To know what happens, disabling OOM_Kill as per "10. OOM Control" (below) and
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700428seeing what happens will be helpful.
429
Balbir Singh1b6df3a2008-02-07 00:13:46 -08004304.2 Task migration
431
Francis Galieguea33f3222010-04-23 00:08:02 +0200432When a task migrates from one cgroup to another, its charge is not
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800433carried forward by default. The pages allocated from the original cgroup still
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800434remain charged to it, the charge is dropped when the page is freed or
435reclaimed.
436
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700437You can move charges of a task along with task migration.
438See 8. "Move charges at task migration"
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800439
Balbir Singh1b6df3a2008-02-07 00:13:46 -08004404.3 Removing a cgroup
441
442A cgroup can be removed by rmdir, but as discussed in sections 4.1 and 4.2, a
443cgroup might have some charge associated with it, even though all
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700444tasks have migrated away from it. (because we charge against pages, not
445against tasks.)
446
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700447We move the stats to root (if use_hierarchy==0) or parent (if
448use_hierarchy==1), and no change on the charge except uncharging
449from the child.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800450
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800451Charges recorded in swap information is not updated at removal of cgroup.
452Recorded information is discarded and a cgroup which uses swap (swapcache)
453will be charged as a new owner of it.
454
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700455About use_hierarchy, see Section 6.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800456
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -08004575. Misc. interfaces.
458
4595.1 force_empty
460 memory.force_empty interface is provided to make cgroup's memory usage empty.
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800461 When writing anything to this
462
463 # echo 0 > memory.force_empty
464
Michal Hockof61c42a72014-05-12 16:34:17 +0200465 the cgroup will be reclaimed and as many pages reclaimed as possible.
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800466
Michael Kerrisk1939c552012-10-08 16:33:09 -0700467 The typical use case for this interface is before calling rmdir().
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800468 Because rmdir() moves all pages to parent, some out-of-use page caches can be
469 moved to the parent. If you want to avoid that, force_empty will be useful.
470
Glauber Costad5bdae72012-12-18 14:22:22 -0800471 Also, note that when memory.kmem.limit_in_bytes is set the charges due to
472 kernel pages will still be seen. This is not considered a failure and the
473 write will still return success. In this case, it is expected that
474 memory.kmem.usage_in_bytes == memory.usage_in_bytes.
475
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700476 About use_hierarchy, see Section 6.
477
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -08004785.2 stat file
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800479
Johannes Weiner185efc02011-09-14 16:21:58 -0700480memory.stat file includes following statistics
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800481
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700482# per-memory cgroup local status
Bharata B Raoc863d832009-04-13 14:40:15 -0700483cache - # of bytes of page cache memory.
David Rientjesb070e652013-05-07 16:18:09 -0700484rss - # of bytes of anonymous and swap cache memory (includes
485 transparent hugepages).
486rss_huge - # of bytes of anonymous transparent hugepages.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700487mapped_file - # of bytes of mapped file (includes tmpfs/shmem)
Ying Han0527b692012-01-12 17:18:27 -0800488pgpgin - # of charging events to the memory cgroup. The charging
489 event happens each time a page is accounted as either mapped
490 anon page(RSS) or cache page(Page Cache) to the cgroup.
491pgpgout - # of uncharging events to the memory cgroup. The uncharging
492 event happens each time a page is unaccounted from the cgroup.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700493swap - # of bytes of swap usage
Sha Zhengju9cb2dc12013-09-12 15:13:54 -0700494writeback - # of bytes of file/anon cache that are queued for syncing to
495 disk.
Aaro Koskinena15e4192013-06-19 15:34:29 +0300496inactive_anon - # of bytes of anonymous and swap cache memory on inactive
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700497 LRU list.
498active_anon - # of bytes of anonymous and swap cache memory on active
Aaro Koskinena15e4192013-06-19 15:34:29 +0300499 LRU list.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700500inactive_file - # of bytes of file-backed memory on inactive LRU list.
501active_file - # of bytes of file-backed memory on active LRU list.
Bharata B Raoc863d832009-04-13 14:40:15 -0700502unevictable - # of bytes of memory that cannot be reclaimed (mlocked etc).
503
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700504# status considering hierarchy (see memory.use_hierarchy settings)
505
506hierarchical_memory_limit - # of bytes of memory limit with regard to hierarchy
507 under which the memory cgroup is
508hierarchical_memsw_limit - # of bytes of memory+swap limit with regard to
509 hierarchy under which memory cgroup is.
510
Johannes Weinereb6332a2012-05-29 15:06:26 -0700511total_<counter> - # hierarchical version of <counter>, which in
512 addition to the cgroup's own value includes the
513 sum of all hierarchical children's values of
514 <counter>, i.e. total_cache
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700515
516# The following additional stats are dependent on CONFIG_DEBUG_VM.
Bharata B Raoc863d832009-04-13 14:40:15 -0700517
Bharata B Raoc863d832009-04-13 14:40:15 -0700518recent_rotated_anon - VM internal parameter. (see mm/vmscan.c)
519recent_rotated_file - VM internal parameter. (see mm/vmscan.c)
520recent_scanned_anon - VM internal parameter. (see mm/vmscan.c)
521recent_scanned_file - VM internal parameter. (see mm/vmscan.c)
522
523Memo:
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700524 recent_rotated means recent frequency of LRU rotation.
525 recent_scanned means recent # of scans to LRU.
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800526 showing for better debug please see the code for meanings.
527
Bharata B Raoc863d832009-04-13 14:40:15 -0700528Note:
529 Only anonymous and swap cache memory is listed as part of 'rss' stat.
530 This should not be confused with the true 'resident set size' or the
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700531 amount of physical memory used by the cgroup.
532 'rss + file_mapped" will give you resident set size of cgroup.
533 (Note: file and shmem may be shared among other cgroups. In that case,
534 file_mapped is accounted only when the memory cgroup is owner of page
535 cache.)
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800536
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -08005375.3 swappiness
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -0800538
Michal Hocko688eb982014-06-06 14:38:15 -0700539Overrides /proc/sys/vm/swappiness for the particular group. The tunable
540in the root cgroup corresponds to the global swappiness setting.
Johannes Weiner3dae7fe2014-06-04 16:07:01 -0700541
Michal Hocko688eb982014-06-06 14:38:15 -0700542Please note that unlike during the global reclaim, limit reclaim
543enforces that 0 swappiness really prevents from any swapping even if
544there is a swap storage available. This might lead to memcg OOM killer
545if there are no file pages to reclaim.
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -0800546
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -07005475.4 failcnt
548
549A memory cgroup provides memory.failcnt and memory.memsw.failcnt files.
550This failcnt(== failure count) shows the number of times that a usage counter
551hit its limit. When a memory cgroup hits a limit, failcnt increases and
552memory under it will be reclaimed.
553
554You can reset failcnt by writing 0 to failcnt file.
555# echo 0 > .../memory.failcnt
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -0800556
Daisuke Nishimuraa111c962011-04-27 15:26:48 -07005575.5 usage_in_bytes
558
559For efficiency, as other kernel components, memory cgroup uses some optimization
560to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
Michael Kerrisk1939c552012-10-08 16:33:09 -0700561method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz
Daisuke Nishimuraa111c962011-04-27 15:26:48 -0700562value for efficient access. (Of course, when necessary, it's synchronized.)
563If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
564value in memory.stat(see 5.2).
565
Ying Han50c35e52011-06-15 15:08:16 -07005665.6 numa_stat
567
568This is similar to numa_maps but operates on a per-memcg basis. This is
569useful for providing visibility into the numa locality information within
570an memcg since the pages are allowed to be allocated from any physical
Michael Kerrisk1939c552012-10-08 16:33:09 -0700571node. One of the use cases is evaluating application performance by
572combining this information with the application's CPU allocation.
Ying Han50c35e52011-06-15 15:08:16 -0700573
Ying Han071aee12013-11-12 15:07:41 -0800574Each memcg's numa_stat file includes "total", "file", "anon" and "unevictable"
575per-node page counts including "hierarchical_<counter>" which sums up all
576hierarchical children's values in addition to the memcg's own value.
577
Masanari Iida8173d5a2013-12-22 00:57:33 +0900578The output format of memory.numa_stat is:
Ying Han50c35e52011-06-15 15:08:16 -0700579
580total=<total pages> N0=<node 0 pages> N1=<node 1 pages> ...
581file=<total file pages> N0=<node 0 pages> N1=<node 1 pages> ...
582anon=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...
583unevictable=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...
Ying Han071aee12013-11-12 15:07:41 -0800584hierarchical_<counter>=<counter pages> N0=<node 0 pages> N1=<node 1 pages> ...
Ying Han50c35e52011-06-15 15:08:16 -0700585
Ying Han071aee12013-11-12 15:07:41 -0800586The "total" count is sum of file + anon + unevictable.
Ying Han50c35e52011-06-15 15:08:16 -0700587
Balbir Singh52bc0d82009-01-07 18:08:03 -08005886. Hierarchy support
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800589
Balbir Singh52bc0d82009-01-07 18:08:03 -0800590The memory controller supports a deep hierarchy and hierarchical accounting.
591The hierarchy is created by creating the appropriate cgroups in the
592cgroup filesystem. Consider for example, the following cgroup filesystem
593hierarchy
594
Jörg Sommer67de0162011-06-15 13:00:47 -0700595 root
Balbir Singh52bc0d82009-01-07 18:08:03 -0800596 / | \
Jörg Sommer67de0162011-06-15 13:00:47 -0700597 / | \
598 a b c
599 | \
600 | \
601 d e
Balbir Singh52bc0d82009-01-07 18:08:03 -0800602
603In the diagram above, with hierarchical accounting enabled, all memory
604usage of e, is accounted to its ancestors up until the root (i.e, c and root),
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700605that has memory.use_hierarchy enabled. If one of the ancestors goes over its
Balbir Singh52bc0d82009-01-07 18:08:03 -0800606limit, the reclaim algorithm reclaims from the tasks in the ancestor and the
607children of the ancestor.
608
6096.1 Enabling hierarchical accounting and reclaim
610
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700611A memory cgroup by default disables the hierarchy feature. Support
Balbir Singh52bc0d82009-01-07 18:08:03 -0800612can be enabled by writing 1 to memory.use_hierarchy file of the root cgroup
613
614# echo 1 > memory.use_hierarchy
615
616The feature can be disabled by
617
618# echo 0 > memory.use_hierarchy
619
Greg Thelen689bca32011-02-16 17:51:23 -0800620NOTE1: Enabling/disabling will fail if either the cgroup already has other
621 cgroups created below it, or if the parent cgroup has use_hierarchy
622 enabled.
Balbir Singh52bc0d82009-01-07 18:08:03 -0800623
KAMEZAWA Hiroyukidaaf1e62010-03-10 15:22:32 -0800624NOTE2: When panic_on_oom is set to "2", the whole system will panic in
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700625 case of an OOM event in any cgroup.
Balbir Singh52bc0d82009-01-07 18:08:03 -0800626
Balbir Singha6df6362009-09-23 15:56:34 -07006277. Soft limits
628
629Soft limits allow for greater sharing of memory. The idea behind soft limits
630is to allow control groups to use as much of the memory as needed, provided
631
632a. There is no memory contention
633b. They do not exceed their hard limit
634
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700635When the system detects memory contention or low memory, control groups
Balbir Singha6df6362009-09-23 15:56:34 -0700636are pushed back to their soft limits. If the soft limit of each control
637group is very high, they are pushed back as much as possible to make
638sure that one control group does not starve the others of memory.
639
Michael Kerrisk1939c552012-10-08 16:33:09 -0700640Please note that soft limits is a best-effort feature; it comes with
Balbir Singha6df6362009-09-23 15:56:34 -0700641no guarantees, but it does its best to make sure that when memory is
642heavily contended for, memory is allocated based on the soft limit
Michael Kerrisk1939c552012-10-08 16:33:09 -0700643hints/setup. Currently soft limit based reclaim is set up such that
Balbir Singha6df6362009-09-23 15:56:34 -0700644it gets invoked from balance_pgdat (kswapd).
645
6467.1 Interface
647
648Soft limits can be setup by using the following commands (in this example we
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700649assume a soft limit of 256 MiB)
Balbir Singha6df6362009-09-23 15:56:34 -0700650
651# echo 256M > memory.soft_limit_in_bytes
652
653If we want to change this to 1G, we can at any time use
654
655# echo 1G > memory.soft_limit_in_bytes
656
657NOTE1: Soft limits take effect over a long period of time, since they involve
658 reclaiming memory for balancing between memory cgroups
659NOTE2: It is recommended to set the soft limit always below the hard limit,
660 otherwise the hard limit will take precedence.
661
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -08006628. Move charges at task migration
663
664Users can move charges associated with a task along with task migration, that
665is, uncharge task's pages from the old cgroup and charge them to the new cgroup.
Daisuke Nishimura02491442010-03-10 15:22:17 -0800666This feature is not supported in !CONFIG_MMU environments because of lack of
667page tables.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800668
6698.1 Interface
670
Masanari Iida8173d5a2013-12-22 00:57:33 +0900671This feature is disabled by default. It can be enabled (and disabled again) by
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800672writing to memory.move_charge_at_immigrate of the destination cgroup.
673
674If you want to enable it:
675
676# echo (some positive value) > memory.move_charge_at_immigrate
677
678Note: Each bits of move_charge_at_immigrate has its own meaning about what type
679 of charges should be moved. See 8.2 for details.
Michael Kerrisk1939c552012-10-08 16:33:09 -0700680Note: Charges are moved only when you move mm->owner, in other words,
681 a leader of a thread group.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800682Note: If we cannot find enough space for the task in the destination cgroup, we
683 try to make space by reclaiming memory. Task migration may fail if we
684 cannot make enough space.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700685Note: It can take several seconds if you move charges much.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800686
687And if you want disable it again:
688
689# echo 0 > memory.move_charge_at_immigrate
690
Michael Kerrisk1939c552012-10-08 16:33:09 -07006918.2 Type of charges which can be moved
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800692
Michael Kerrisk1939c552012-10-08 16:33:09 -0700693Each bit in move_charge_at_immigrate has its own meaning about what type of
694charges should be moved. But in any case, it must be noted that an account of
695a page or a swap can be moved only when it is charged to the task's current
696(old) memory cgroup.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800697
698 bit | what type of charges would be moved ?
699 -----+------------------------------------------------------------------------
Michael Kerrisk1939c552012-10-08 16:33:09 -0700700 0 | A charge of an anonymous page (or swap of it) used by the target task.
701 | You must enable Swap Extension (see 2.4) to enable move of swap charges.
Daisuke Nishimura87946a72010-05-26 14:42:39 -0700702 -----+------------------------------------------------------------------------
Michael Kerrisk1939c552012-10-08 16:33:09 -0700703 1 | A charge of file pages (normal file, tmpfs file (e.g. ipc shared memory)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700704 | and swaps of tmpfs file) mmapped by the target task. Unlike the case of
Michael Kerrisk1939c552012-10-08 16:33:09 -0700705 | anonymous pages, file pages (and swaps) in the range mmapped by the task
Daisuke Nishimura87946a72010-05-26 14:42:39 -0700706 | will be moved even if the task hasn't done page fault, i.e. they might
707 | not be the task's "RSS", but other task's "RSS" that maps the same file.
Michael Kerrisk1939c552012-10-08 16:33:09 -0700708 | And mapcount of the page is ignored (the page can be moved even if
709 | page_mapcount(page) > 1). You must enable Swap Extension (see 2.4) to
Daisuke Nishimura87946a72010-05-26 14:42:39 -0700710 | enable move of swap charges.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800711
7128.3 TODO
713
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800714- All of moving charge operations are done under cgroup_mutex. It's not good
715 behavior to hold the mutex too long, so we may need some trick.
716
Kirill A. Shutemov2e72b632010-03-10 15:22:24 -08007179. Memory thresholds
718
Michael Kerrisk1939c552012-10-08 16:33:09 -0700719Memory cgroup implements memory thresholds using the cgroups notification
Kirill A. Shutemov2e72b632010-03-10 15:22:24 -0800720API (see cgroups.txt). It allows to register multiple memory and memsw
721thresholds and gets notifications when it crosses.
722
Michael Kerrisk1939c552012-10-08 16:33:09 -0700723To register a threshold, an application must:
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700724- create an eventfd using eventfd(2);
725- open memory.usage_in_bytes or memory.memsw.usage_in_bytes;
726- write string like "<event_fd> <fd of memory.usage_in_bytes> <threshold>" to
727 cgroup.event_control.
Kirill A. Shutemov2e72b632010-03-10 15:22:24 -0800728
729Application will be notified through eventfd when memory usage crosses
730threshold in any direction.
731
732It's applicable for root and non-root cgroup.
733
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -070073410. OOM Control
735
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700736memory.oom_control file is for OOM notification and other controls.
737
Michael Kerrisk1939c552012-10-08 16:33:09 -0700738Memory cgroup implements OOM notifier using the cgroup notification
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700739API (See cgroups.txt). It allows to register multiple OOM notification
740delivery and gets notification when OOM happens.
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700741
Michael Kerrisk1939c552012-10-08 16:33:09 -0700742To register a notifier, an application must:
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700743 - create an eventfd using eventfd(2)
744 - open memory.oom_control file
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700745 - write string like "<event_fd> <fd of memory.oom_control>" to
746 cgroup.event_control
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700747
Michael Kerrisk1939c552012-10-08 16:33:09 -0700748The application will be notified through eventfd when OOM happens.
749OOM notification doesn't work for the root cgroup.
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700750
Michael Kerrisk1939c552012-10-08 16:33:09 -0700751You can disable the OOM-killer by writing "1" to memory.oom_control file, as:
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700752
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700753 #echo 1 > memory.oom_control
754
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700755If OOM-killer is disabled, tasks under cgroup will hang/sleep
756in memory cgroup's OOM-waitqueue when they request accountable memory.
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700757
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700758For running them, you have to relax the memory cgroup's OOM status by
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700759 * enlarge limit or reduce usage.
760To reduce usage,
761 * kill some tasks.
762 * move some tasks to other group with account migration.
763 * remove some files (on tmpfs?)
764
765Then, stopped tasks will work again.
766
767At reading, current status of OOM is shown.
768 oom_kill_disable 0 or 1 (if 1, oom-killer is disabled)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700769 under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700770 be stopped.)
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700771
Anton Vorontsov70ddf632013-04-29 15:08:31 -070077211. Memory Pressure
773
774The pressure level notifications can be used to monitor the memory
775allocation cost; based on the pressure, applications can implement
776different strategies of managing their memory resources. The pressure
777levels are defined as following:
778
779The "low" level means that the system is reclaiming memory for new
780allocations. Monitoring this reclaiming activity might be useful for
781maintaining cache level. Upon notification, the program (typically
782"Activity Manager") might analyze vmstat and act in advance (i.e.
783prematurely shutdown unimportant services).
784
785The "medium" level means that the system is experiencing medium memory
786pressure, the system might be making swap, paging out active file caches,
787etc. Upon this event applications may decide to further analyze
788vmstat/zoneinfo/memcg or internal memory usage statistics and free any
789resources that can be easily reconstructed or re-read from a disk.
790
791The "critical" level means that the system is actively thrashing, it is
792about to out of memory (OOM) or even the in-kernel OOM killer is on its
793way to trigger. Applications should do whatever they can to help the
794system. It might be too late to consult with vmstat or any other
795statistics, so it's advisable to take an immediate action.
796
797The events are propagated upward until the event is handled, i.e. the
798events are not pass-through. Here is what this means: for example you have
799three cgroups: A->B->C. Now you set up an event listener on cgroups A, B
800and C, and suppose group C experiences some pressure. In this situation,
801only group C will receive the notification, i.e. groups A and B will not
802receive it. This is done to avoid excessive "broadcasting" of messages,
803which disturbs the system and which is especially bad if we are low on
804memory or thrashing. So, organize the cgroups wisely, or propagate the
805events manually (or, ask us to implement the pass-through events,
806explaining why would you need them.)
807
808The file memory.pressure_level is only used to setup an eventfd. To
809register a notification, an application must:
810
811- create an eventfd using eventfd(2);
812- open memory.pressure_level;
813- write string like "<event_fd> <fd of memory.pressure_level> <level>"
814 to cgroup.event_control.
815
816Application will be notified through eventfd when memory pressure is at
817the specific level (or higher). Read/write operations to
818memory.pressure_level are no implemented.
819
820Test:
821
822 Here is a small script example that makes a new cgroup, sets up a
823 memory limit, sets up a notification in the cgroup and then makes child
824 cgroup experience a critical pressure:
825
826 # cd /sys/fs/cgroup/memory/
827 # mkdir foo
828 # cd foo
829 # cgroup_event_listener memory.pressure_level low &
830 # echo 8000000 > memory.limit_in_bytes
831 # echo 8000000 > memory.memsw.limit_in_bytes
832 # echo $$ > tasks
833 # dd if=/dev/zero | read x
834
835 (Expect a bunch of notifications, and eventually, the oom-killer will
836 trigger.)
837
83812. TODO
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800839
Li Zefanf968ef12013-07-03 15:02:25 -07008401. Make per-cgroup scanner reclaim not-shared pages first
8412. Teach controller to account for shared-pages
8423. Start reclamation in the background when the limit is
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800843 not yet hit but the usage is getting closer
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800844
845Summary
846
847Overall, the memory controller has been a stable controller and has been
848commented and discussed quite extensively in the community.
849
850References
851
8521. Singh, Balbir. RFC: Memory Controller, http://lwn.net/Articles/206697/
8532. Singh, Balbir. Memory Controller (RSS Control),
854 http://lwn.net/Articles/222762/
8553. Emelianov, Pavel. Resource controllers based on process cgroups
856 http://lkml.org/lkml/2007/3/6/198
8574. Emelianov, Pavel. RSS controller based on process cgroups (v2)
Li Zefan2324c5d2008-02-23 15:24:12 -0800858 http://lkml.org/lkml/2007/4/9/78
Balbir Singh1b6df3a2008-02-07 00:13:46 -08008595. Emelianov, Pavel. RSS controller based on process cgroups (v3)
860 http://lkml.org/lkml/2007/5/30/244
8616. Menage, Paul. Control Groups v10, http://lwn.net/Articles/236032/
8627. Vaidyanathan, Srinivasan, Control Groups: Pagecache accounting and control
863 subsystem (v3), http://lwn.net/Articles/235534/
Li Zefan2324c5d2008-02-23 15:24:12 -08008648. Singh, Balbir. RSS controller v2 test results (lmbench),
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800865 http://lkml.org/lkml/2007/5/17/232
Li Zefan2324c5d2008-02-23 15:24:12 -08008669. Singh, Balbir. RSS controller v2 AIM9 results
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800867 http://lkml.org/lkml/2007/5/18/1
Li Zefan2324c5d2008-02-23 15:24:12 -080086810. Singh, Balbir. Memory controller v6 test results,
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800869 http://lkml.org/lkml/2007/8/19/36
Li Zefan2324c5d2008-02-23 15:24:12 -080087011. Singh, Balbir. Memory controller introduction (v6),
871 http://lkml.org/lkml/2007/8/17/69
Balbir Singh1b6df3a2008-02-07 00:13:46 -080087212. Corbet, Jonathan, Controlling memory use in cgroups,
873 http://lwn.net/Articles/243795/