diff options
| author | Hugh Dickins <hughd@google.com> | 2025-09-08 15:16:53 -0700 |
|---|---|---|
| committer | Greg Kroah-Hartman <gregkh@linuxfoundation.org> | 2025-10-02 13:42:49 +0200 |
| commit | d37ec803b281363120af797ee8b4432e2dac307c (patch) | |
| tree | 5b6c09f6d2ada7309db26013c7d18b895bdfcba3 /mm | |
| parent | 768c44cc8b6386b9f4ec7b90ef5311d2be953894 (diff) | |
| download | linux-d37ec803b281363120af797ee8b4432e2dac307c.tar.gz linux-d37ec803b281363120af797ee8b4432e2dac307c.tar.bz2 linux-d37ec803b281363120af797ee8b4432e2dac307c.zip | |
mm/gup: local lru_add_drain() to avoid lru_add_drain_all()
[ Upstream commit a09a8a1fbb374e0053b97306da9dbc05bd384685 ]
In many cases, if collect_longterm_unpinnable_folios() does need to drain
the LRU cache to release a reference, the cache in question is on this
same CPU, and much more efficiently drained by a preliminary local
lru_add_drain(), than the later cross-CPU lru_add_drain_all().
Marked for stable, to counter the increase in lru_add_drain_all()s from
"mm/gup: check ref_count instead of lru before migration". Note for clean
backports: can take 6.16 commit a03db236aebf ("gup: optimize longterm
pin_user_pages() for large folio") first.
Link: https://lkml.kernel.org/r/66f2751f-283e-816d-9530-765db7edc465@google.com
Signed-off-by: Hugh Dickins <hughd@google.com>
Acked-by: David Hildenbrand <david@redhat.com>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Keir Fraser <keirf@google.com>
Cc: Konstantin Khlebnikov <koct9i@gmail.com>
Cc: Li Zhe <lizhe.67@bytedance.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Peter Xu <peterx@redhat.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shivank Garg <shivankg@amd.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Wei Xu <weixugc@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: yangge <yangge1116@126.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Yu Zhao <yuzhao@google.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
[ Resolved minor conflicts ]
Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
Diffstat (limited to 'mm')
| -rw-r--r-- | mm/gup.c | 15 |
1 files changed, 11 insertions, 4 deletions
@@ -1953,7 +1953,7 @@ static unsigned long collect_longterm_unpinnable_pages( { unsigned long i, collected = 0; struct folio *prev_folio = NULL; - bool drain_allow = true; + int drained = 0; for (i = 0; i < nr_pages; i++) { struct folio *folio = page_folio(pages[i]); @@ -1975,10 +1975,17 @@ static unsigned long collect_longterm_unpinnable_pages( continue; } - if (drain_allow && folio_ref_count(folio) != - folio_expected_ref_count(folio) + 1) { + if (drained == 0 && + folio_ref_count(folio) != + folio_expected_ref_count(folio) + 1) { + lru_add_drain(); + drained = 1; + } + if (drained == 1 && + folio_ref_count(folio) != + folio_expected_ref_count(folio) + 1) { lru_add_drain_all(); - drain_allow = false; + drained = 2; } if (!folio_isolate_lru(folio)) |