[PATCH] mm: never ClearPageLRU released pages
If vmscan finds a zero refcount page on the lru list, never ClearPageLRU
it. This means the release code need not hold ->lru_lock to stabilise
PageLRU, so that lock may be skipped entirely when releasing !PageLRU pages
(because we know PageLRU won't have been temporarily cleared by vmscan,
which was previously guaranteed by holding the lock to synchronise against
vmscan).
Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 4fe7e3a..acb7611 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1085,21 +1085,25 @@
page = lru_to_page(src);
prefetchw_prev_lru_page(page, src, flags);
- if (!TestClearPageLRU(page))
- BUG();
list_del(&page->lru);
- if (get_page_testone(page)) {
+ if (unlikely(get_page_testone(page))) {
/*
* It is being freed elsewhere
*/
__put_page(page);
- SetPageLRU(page);
list_add(&page->lru, src);
continue;
- } else {
- list_add(&page->lru, dst);
- nr_taken++;
}
+
+ /*
+ * Be careful not to clear PageLRU until after we're sure
+ * the page is not being freed elsewhere -- the page release
+ * code relies on it.
+ */
+ if (!TestClearPageLRU(page))
+ BUG();
+ list_add(&page->lru, dst);
+ nr_taken++;
}
*scanned = scan;