linux.git/include/asm-generic/pgalloc.h, branch v6.18.21

mm: actually mark kernel page table pages

2026-01-23T10:21:35+00:00

commit 977870522af34359b461060597ee3a86f27450d6 upstream.

Now that the API is in place, mark kernel page table pages just after they
are allocated.  Unmark them just before they are freed.

Note: Unconditionally clearing the 'kernel' marking (via
ptdesc_clear_kernel()) would be functionally identical to what is here.
But having the if() makes it logically clear that this function can be
used for kernel and non-kernel page tables.

Link: https://lkml.kernel.org/r/20251022082635.2462433-4-baolu.lu@linux.intel.com
Signed-off-by: Dave Hansen 
Signed-off-by: Lu Baolu 
Reviewed-by: Jason Gunthorpe 
Reviewed-by: Kevin Tian 
Acked-by: David Hildenbrand 
Acked-by: Mike Rapoport (Microsoft) 
Cc: Alistair Popple 
Cc: Andy Lutomirski 
Cc: Borislav Betkov 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Jean-Philippe Brucker 
Cc: Joerg Roedel 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Peter Zijlstra 
Cc: Robin Murohy 
Cc: Thomas Gleinxer 
Cc: "Uladzislau Rezki (Sony)" 
Cc: Vasant Hegde 
Cc: Vinicius Costa Gomes 
Cc: Vlastimil Babka 
Cc: Will Deacon 
Cc: Yi Lai 
Signed-off-by: Andrew Morton 
Signed-off-by: Greg Kroah-Hartman

mm: call ctor/dtor for kernel PTEs

2025-05-12T00:48:21+00:00

Since [1], constructors/destructors are expected to be called for all page
table pages, at all levels and for both user and kernel pgtables.  There
is however one glaring exception: kernel PTEs are managed via separate
helpers (pte_alloc_kernel/pte_free_kernel), which do not call the [cd]tor,
at least not in the generic implementation.

The most obvious reason for this anomaly is that init_mm is special-cased
not to use split page table locks.  As a result calling ptlock_init() for
PTEs associated with init_mm would be wasteful, potentially resulting in
dynamic memory allocation.  However, pgtable [cd]tors perform other
actions - currently related to accounting/statistics, and potentially more
functionally significant in the future.

Now that pagetable_pte_ctor() is passed the associated mm, we can make it
skip the call to ptlock_init() for init_mm; this allows us to call the
ctor from pte_alloc_one_kernel() too.  This is matched by a call to the
pgtable destructor in pte_free_kernel(); no special-casing is needed on
that path, as ptlock_free() is already called unconditionally. 
(ptlock_free() is a no-op unless a ptlock was allocated for the given
PTP.)

This patch ensures that all architectures that rely on
 call the [cd]tor for kernel PTEs. 
pte_free_kernel() cannot be overridden so changing the generic
implementation is sufficient.  pte_alloc_one_kernel() can be overridden
using __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL, and a few architectures implement
it by calling the page allocator directly.  We amend those so that they
call the generic __pte_alloc_one_kernel() instead, if possible, ensuring
that the ctor is called.

A few architectures do not use ; those will be
taken care of separately.

[1] https://lore.kernel.org/linux-mm/20250103184415.2744423-1-kevin.brodsky@arm.com/

Link: https://lkml.kernel.org/r/20250408095222.860601-4-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky 
Reviewed-by: Alexander Gordeev  # s390
Cc: Albert Ou 
Cc: Andreas Larsson 
Cc: Catalin Marinas 
Cc: David S. Miller 
Cc: Geert Uytterhoeven 
Cc: Linus Waleij 
Cc: Madhavan Srinivasan 
Cc: Mark Rutland 
Cc: Matthew Wilcox (Oracle) 
Cc: Michael Ellerman 
Cc: Mike Rapoport 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Peter Zijlstra 
Cc: Qi Zheng 
Cc: Ryan Roberts 
Cc: Will Deacon 
Cc: 
Cc: Yang Shi 
Cc: Dave Hansen 
Signed-off-by: Andrew Morton

mm: pass mm down to pagetable_{pte,pmd}_ctor

2025-05-12T00:48:21+00:00

Patch series "Always call constructor for kernel page tables", v2.

There has been much confusion around exactly when page table
constructors/destructors (pagetable_*_[cd]tor) are supposed to be called. 
They were initially introduced for user PTEs only (to support split page
table locks), then at the PMD level for the same purpose.  Accounting was
added later on, starting at the PTE level and then moving to higher levels
(PMD, PUD).  Finally, with my earlier series "Account page tables at all
levels" [1], the ctor/dtor is run for all levels, all the way to PGD.

I thought this was the end of the story, and it hopefully is for user
pgtables, but I was wrong for what concerns kernel pgtables.  The current
situation there makes very little sense:

* At the PTE level, the ctor/dtor is not called (at least in the generic
  implementation).  Specific helpers are used for kernel pgtables at this
  level (pte_{alloc,free}_kernel()) and those have never called the
  ctor/dtor, most likely because they were initially irrelevant in the
  kernel case.

* At all other levels, the ctor/dtor is normally called.  This is
  potentially wasteful at the PMD level (more on that later).

This series aims to ensure that the ctor/dtor is always called for kernel
pgtables, as it already is for user pgtables.  Besides consistency, the
main motivation is to guarantee that ctor/dtor hooks are systematically
called; this makes it possible to insert hooks to protect page tables [2],
for instance.  There is however an extra challenge: split locks are not
used for kernel pgtables, and it would therefore be wasteful to initialise
them (ptlock_init()).

It is worth clarifying exactly when split locks are used.  They clearly
are for user pgtables, but as illustrated in commit 61444cde9170 ("ARM:
8591/1: mm: use fully constructed struct pages for EFI pgd allocations"),
they also are for special page tables like efi_mm.  The one case where
split locks are definitely unused is pgtables owned by init_mm; this is
consistent with the behaviour of apply_to_pte_range().

The approach chosen in this series is therefore to pass the mm associated
to the pgtables being constructed to pagetable_{pte,pmd}_ctor() (patch 1),
and skip ptlock_init() if mm == &init_mm (patch 3 and 7).  This makes it
possible to call the PTE ctor/dtor from pte_{alloc,free}_kernel() without
unintended consequences (patch 3).  As a result the accounting functions
are now called at all levels for kernel pgtables, and split locks are
never initialised.

In configurations where ptlocks are dynamically allocated (32-bit,
PREEMPT_RT, etc.) and ARCH_ENABLE_SPLIT_PMD_PTLOCK is selected, this
series results in the removal of a kmem_cache allocation for every kernel
PMD.  Additionally, for certain architectures that do not use
 such as s390, the same optimisation occurs at the
PTE level.

===

Things get more complicated when it comes to special pgtable allocators
(patch 8-12).  All architectures need such allocators to create initial
kernel pgtables; we are not concerned with those as the ctor cannot be
called so early in the boot sequence.  However, those allocators may also
be used later in the boot sequence or during normal operations.  There are
two main use-cases:

1. Mapping EFI memory: efi_mm (arm, arm64, riscv)
2. arch_add_memory(): init_mm

The ctor is already explicitly run (at the PTE/PMD level) in the first
case, as required for pgtables that are not associated with init_mm. 
However the same allocators may also be used for the second use-case (or
others), and this is where it gets messy.  Patch 1 calls the ctor with
NULL as mm in those situations, as the actual mm isn't available. 
Practically this means that ptlocks will be unconditionally initialised. 
This is fine on arm - create_mapping_late() is only used for the EFI
mapping.  On arm64, __create_pgd_mapping() is also used by
arch_add_memory(); patch 8/9/11 ensure that ctors are called at all levels
with the appropriate mm.  The situation is similar on riscv, but
propagating the mm down to the ctor would require significant refactoring.
Since they are already called unconditionally, this series leaves riscv
no worse off - patch 10 adds comments to clarify the situation.

From a cursory look at other architectures implementing arch_add_memory(),
s390 and x86 may also need a similar treatment to add constructor calls. 
This is to be taken care of in a future version or as a follow-up.

===

The complications in those special pgtable allocators beg the question:
does it really make sense to treat efi_mm and init_mm differently in e.g. 
apply_to_pte_range()?  Maybe what we really need is a way to tell if an mm
corresponds to user memory or not, and never use split locks for non-user
mm's.  Feedback and suggestions welcome!


This patch (of 12):

In preparation for calling constructors for all kernel page tables while
eliding unnecessary ptlock initialisation, let's pass down the associated
mm to the PTE/PMD level ctors.  (These are the two levels where ptlocks
are used.)

In most cases the mm is already around at the point of calling the ctor so
we simply pass it down.  This is however not the case for special page
table allocators:

* arch/arm/mm/mmu.c
* arch/arm64/mm/mmu.c
* arch/riscv/mm/init.c

In those cases, the page tables being allocated are either for standard
kernel memory (init_mm) or special page directories, which may not be
associated to any mm.  For now let's pass NULL as mm; this will be refined
where possible in future patches.

No functional change in this patch.

Link: https://lore.kernel.org/linux-mm/20250103184415.2744423-1-kevin.brodsky@arm.com/ [1]
Link: https://lore.kernel.org/linux-hardening/20250203101839.1223008-1-kevin.brodsky@arm.com/ [2]
Link: https://lkml.kernel.org/r/20250408095222.860601-1-kevin.brodsky@arm.com
Link: https://lkml.kernel.org/r/20250408095222.860601-2-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky 
Reviewed-by: Alexander Gordeev 	[s390]
Cc: Albert Ou 
Cc: Andreas Larsson 
Cc: Catalin Marinas 
Cc: David S. Miller 
Cc: Geert Uytterhoeven 
Cc: Kevin Brodsky 
Cc: Linus Waleij 
Cc: Madhavan Srinivasan 
Cc: Mark Rutland 
Cc: Matthew Wilcox (Oracle) 
Cc: Michael Ellerman 
Cc: Mike Rapoport 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Peter Zijlstra 
Cc: Qi Zheng 
Cc: Ryan Roberts 
Cc: Will Deacon 
Cc: Yang Shi 
Cc: 
Cc: Dave Hansen 
Signed-off-by: Andrew Morton

mm: introduce ctor/dtor at PGD level

2025-01-26T04:22:24+00:00

Following on from the introduction of P4D-level ctor/dtor, let's finish
the job and introduce ctor/dtor at PGD level.  The incurred improvement in
page accounting is minimal - the main motivation is to create a single,
generic place where construction/destruction hooks can be added for all
page table pages.

This patch should cover all architectures and all configurations where
PGDs are one or more regular pages.  This excludes any configuration where
PGDs are allocated from a kmem_cache object.

Link: https://lkml.kernel.org/r/20250103184415.2744423-7-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky 
Acked-by: Dave Hansen 
Acked-by: Qi Zheng 
Cc: Andy Lutomirski 
Cc: Catalin Marinas 
Cc: Ingo Molnar 
Cc: Linus Walleij 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Rapoport (Microsoft) 
Cc: Peter Zijlstra 
Cc: Ryan Roberts 
Cc: Thomas Gleixner 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

asm-generic: pgalloc: provide generic __pgd_{alloc,free}

2025-01-26T04:22:24+00:00

We already have a generic implementation of alloc/free up to P4D level, as
well as pgd_free().  Let's finish the work and add a generic PGD-level
alloc helper as well.

Unlike at lower levels, almost all architectures need some specific magic
at PGD level (typically initialising PGD entries), so introducing a
generic pgd_alloc() isn't worth it.  Instead we introduce two new helpers,
__pgd_alloc() and __pgd_free(), and make use of them in the arch-specific
pgd_alloc() and pgd_free() wherever possible.  To accommodate as many arch
as possible, __pgd_alloc() takes a page allocation order.

Because pagetable_alloc() allocates zeroed pages, explicit zeroing in
pgd_alloc() becomes redundant and we can get rid of it.  Some trivial
implementations of pgd_free() also become unnecessary once __pgd_alloc()
is used; remove them.

Another small improvement is consistent accounting of PGD pages by using
GFP_PGTABLE_{USER,KERNEL} as appropriate.

Not all PGD allocations can be handled by the generic helpers.  In
particular, multiple architectures allocate PGDs from a kmem_cache, and
those PGDs may not be page-sized.

Link: https://lkml.kernel.org/r/20250103184415.2744423-6-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky 
Acked-by: Dave Hansen 
Acked-by: Qi Zheng 
Cc: Andy Lutomirski 
Cc: Catalin Marinas 
Cc: Ingo Molnar 
Cc: Linus Walleij 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Rapoport (Microsoft) 
Cc: Peter Zijlstra 
Cc: Ryan Roberts 
Cc: Thomas Gleixner 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: pgtable: introduce generic pagetable_dtor_free()

2025-01-26T04:22:23+00:00

The pte_free(), pmd_free(), __pud_free() and __p4d_free() in
asm-generic/pgalloc.h and the generic __tlb_remove_table() are basically
the same, so let's introduce pagetable_dtor_free() to deduplicate them.

In addition, the pagetable_dtor_free() in s390 does the same thing, so
let's s390 also calls generic pagetable_dtor_free().

Link: https://lkml.kernel.org/r/1663a0565aca881d1338ceb7d1db4aa9c333abd6.1736317725.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Suggested-by: Peter Zijlstra (Intel) 
Reviewed-by: Kevin Brodsky 
Acked-by: Alexander Gordeev 	[s390]
Cc: Alexandre Ghiti 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Aneesh Kumar K.V (Arm) 
Cc: Arnd Bergmann 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jann Horn 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Rapoport (Microsoft) 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Ryan Roberts 
Cc: Thomas Gleixner 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm: pgtable: introduce pagetable_dtor()

2025-01-26T04:22:22+00:00

The pagetable_p*_dtor() are exactly the same except for the handling of
ptlock.  If we make ptlock_free() handle the case where ptdesc->ptl is
NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify
pagetable_p*_dtor() into one function.  Let's introduce pagetable_dtor()
to do this.

Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that
ptlock and page table pages can be freed together (regardless of whether
RCU is used).  This prevents the use-after-free problem where the ptlock
is freed immediately but the page table pages is freed later via RCU.

Link: https://lkml.kernel.org/r/47f44fff9dc68d9d9e9a0d6c036df275f820598a.1736317725.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Originally-by: Peter Zijlstra (Intel) 
Reviewed-by: Kevin Brodsky 
Acked-by: Alexander Gordeev 	[s390]
Cc: Alexandre Ghiti 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Aneesh Kumar K.V (Arm) 
Cc: Arnd Bergmann 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jann Horn 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Rapoport (Microsoft) 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Ryan Roberts 
Cc: Thomas Gleixner 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm: pgtable: add statistics for P4D level page table

2025-01-26T04:22:21+00:00

Like other levels of page tables, add statistics for P4D level page table.

Link: https://lkml.kernel.org/r/d55fe3c286305aae84457da9e1066df99b3de125.1736317725.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Originally-by: Peter Zijlstra (Intel) 
Reviewed-by: Kevin Brodsky 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Aneesh Kumar K.V (Arm) 
Cc: Arnd Bergmann 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jann Horn 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Rapoport (Microsoft) 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Ryan Roberts 
Cc: Thomas Gleixner 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

asm-generic: pgalloc: provide generic p4d_{alloc_one,free}

2025-01-26T04:22:21+00:00

Four architectures currently implement 5-level pgtables: arm64, riscv, x86
and s390.  The first three have essentially the same implementation for
p4d_alloc_one() and p4d_free(), so we've got an opportunity to reduce
duplication like at the lower levels.

Provide a generic version of p4d_alloc_one() and p4d_free(), and make use
of it on those architectures.

Their implementation is the same as at PUD level, except that p4d_free()
performs a runtime check by calling mm_p4d_folded().  5-level pgtables
depend on a runtime-detected hardware feature on all supported
architectures, so we might as well include this check in the generic
implementation.  No runtime check is required in p4d_alloc_one() as the
top-level p4d_alloc() already does the required check.

Link: https://lkml.kernel.org/r/26d69c74a29183ecc335b9b407040d8e4cd70c6a.1736317725.git.zhengqi.arch@bytedance.com
Signed-off-by: Kevin Brodsky 
Signed-off-by: Qi Zheng 
Acked-by: Dave Hansen 
Acked-by: Arnd Bergmann 		[asm-generic]
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Aneesh Kumar K.V (Arm) 
Cc: David Hildenbrand 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jann Horn 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Rapoport (Microsoft) 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Peter Zijlstra (Intel) 
Cc: Ryan Roberts 
Cc: Thomas Gleixner 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm: change inlined allocation helpers to account at the call site

2024-04-26T03:55:59+00:00

Main goal of memory allocation profiling patchset is to provide accounting
that is cheap enough to run in production.  To achieve that we inject
counters using codetags at the allocation call sites to account every time
allocation is made.  This injection allows us to perform accounting
efficiently because injected counters are immediately available as opposed
to the alternative methods, such as using _RET_IP_, which would require
counter lookup and appropriate locking that makes accounting much more
expensive.  This method requires all allocation functions to inject
separate counters at their call sites so that their callers can be
individually accounted.  Counter injection is implemented by allocation
hooks which should wrap all allocation functions.

Inlined functions which perform allocations but do not use allocation
hooks are directly charged for the allocations they perform.  In most
cases these functions are just specialized allocation wrappers used from
multiple places to allocate objects of a specific type.  It would be more
useful to do the accounting at their call sites instead.  Instrument these
helpers to do accounting at the call site.  Simple inlined allocation
wrappers are converted directly into macros.  More complex allocators or
allocators with documentation are converted into _noprof versions and
allocation hooks are added.  This allows memory allocation profiling
mechanism to charge allocations to the callers of these functions.

Link: https://lkml.kernel.org/r/20240415020731.1152108-1-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Acked-by: Jan Kara 		[jbd2]
Cc: Anna Schumaker 
Cc: Arnd Bergmann 
Cc: Benjamin Tissoires 
Cc: Christoph Lameter 
Cc: David Rientjes 
Cc: David S. Miller 
Cc: Dennis Zhou 
Cc: Eric Dumazet 
Cc: Herbert Xu 
Cc: Jakub Kicinski 
Cc: Jakub Sitnicki 
Cc: Jiri Kosina 
Cc: Joerg Roedel 
Cc: Joonsoo Kim 
Cc: Kent Overstreet 
Cc: Matthew Wilcox (Oracle) 
Cc: Paolo Abeni 
Cc: Pekka Enberg 
Cc: Tejun Heo 
Cc: Theodore Ts'o 
Cc: Trond Myklebust 
Cc: Vlastimil Babka 
Cc: Will Deacon 
Signed-off-by: Andrew Morton