linux.git/include/asm-generic/pgalloc.h, branch v6.12.80

mm: change inlined allocation helpers to account at the call site

2024-04-26T03:55:59+00:00

Main goal of memory allocation profiling patchset is to provide accounting
that is cheap enough to run in production.  To achieve that we inject
counters using codetags at the allocation call sites to account every time
allocation is made.  This injection allows us to perform accounting
efficiently because injected counters are immediately available as opposed
to the alternative methods, such as using _RET_IP_, which would require
counter lookup and appropriate locking that makes accounting much more
expensive.  This method requires all allocation functions to inject
separate counters at their call sites so that their callers can be
individually accounted.  Counter injection is implemented by allocation
hooks which should wrap all allocation functions.

Inlined functions which perform allocations but do not use allocation
hooks are directly charged for the allocations they perform.  In most
cases these functions are just specialized allocation wrappers used from
multiple places to allocate objects of a specific type.  It would be more
useful to do the accounting at their call sites instead.  Instrument these
helpers to do accounting at the call site.  Simple inlined allocation
wrappers are converted directly into macros.  More complex allocators or
allocators with documentation are converted into _noprof versions and
allocation hooks are added.  This allows memory allocation profiling
mechanism to charge allocations to the callers of these functions.

Link: https://lkml.kernel.org/r/20240415020731.1152108-1-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Acked-by: Jan Kara 		[jbd2]
Cc: Anna Schumaker 
Cc: Arnd Bergmann 
Cc: Benjamin Tissoires 
Cc: Christoph Lameter 
Cc: David Rientjes 
Cc: David S. Miller 
Cc: Dennis Zhou 
Cc: Eric Dumazet 
Cc: Herbert Xu 
Cc: Jakub Kicinski 
Cc: Jakub Sitnicki 
Cc: Jiri Kosina 
Cc: Joerg Roedel 
Cc: Joonsoo Kim 
Cc: Kent Overstreet 
Cc: Matthew Wilcox (Oracle) 
Cc: Paolo Abeni 
Cc: Pekka Enberg 
Cc: Tejun Heo 
Cc: Theodore Ts'o 
Cc: Trond Myklebust 
Cc: Vlastimil Babka 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: add statistics for PUD level pagetable

2023-10-06T21:44:10+00:00

Recently, we found that cross-die access to pagetable pages on ARM64
machines can cause performance fluctuations in our business.  Currently,
there are no PMU events available to track this situation on our ARM64
machines, so accurate pagetable accounting can help to analyze this issue,
but now the PUD level pagetable accounting is missed.

So introduce pagetable_pud_ctor/dtor() to help to get accurate PUD
pagetable accounting, as well as converting the architectures which use
generic PUD pagetable allocation to add corresponding PUD pagetable
accounting.  Moreover this patch will mark the PUD level pagetable with
PG_table flag, which will help to do sanity validation in
unpoison_memory().

On my testing machine, I can see more pagetables statistics after the patch
with page-types tool:

Before patch:
        flags           page-count      MB  symbolic-flags                     long-symbolic-flags
0x0000000004000000           27326      106  __________________________g_________________       pgtable
After patch:
0x0000000004000000           27541      107  __________________________g_________________       pgtable

Link: https://lkml.kernel.org/r/876c71c03a7e69c17722a690e3225a4f7b172fb2.1695017383.git.baolin.wang@linux.alibaba.com
Signed-off-by: Baolin Wang 
Acked-by: Mike Rapoport (IBM) 
Acked-by: Vishal Moola (Oracle) 
Cc: Andy Lutomirski 
Cc: Aneesh Kumar K.V 
Cc: Arnd Bergmann 
Cc: Borislav Petkov 
Cc: Catalin Marinas 
Cc: Dave Hansen 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Matthew Wilcox (Oracle) 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleixner 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

pgalloc: convert various functions to use ptdescs

2023-08-21T20:37:55+00:00

As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.

Some of the functions use the *get*page*() helper functions.  Convert
these to use pagetable_alloc() and ptdesc_address() instead to help
standardize page tables further.

Link: https://lkml.kernel.org/r/20230807230513.102486-17-vishal.moola@gmail.com
Signed-off-by: Vishal Moola (Oracle) 
Cc: Arnd Bergmann 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: Claudio Imbrenda 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: "David S. Miller" 
Cc: Dinh Nguyen 
Cc: Geert Uytterhoeven 
Cc: Geert Uytterhoeven 
Cc: Guo Ren 
Cc: Huacai Chen 
Cc: Hugh Dickins 
Cc: John Paul Adrian Glaubitz 
Cc: Jonas Bonn 
Cc: Matthew Wilcox 
Cc: Mike Rapoport (IBM) 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Richard Weinberger 
Cc: Thomas Bogendoerfer 
Cc: Yoshinori Sato 
Signed-off-by: Andrew Morton

mm: prefer xxx_page() alloc/free functions for order-0 pages

2023-03-28T23:20:16+00:00

Update instances of alloc_pages(..., 0), __get_free_pages(..., 0) and
__free_pages(..., 0) to use alloc_page(), __get_free_page() and
__free_page() respectively in core code.

Link: https://lkml.kernel.org/r/50c48ca4789f1da2a65795f2346f5ae3eff7d665.1678710232.git.lstoakes@gmail.com
Signed-off-by: Lorenzo Stoakes 
Reviewed-by: David Hildenbrand 
Reviewed-by: Mike Rapoport (IBM) 
Acked-by: Mel Gorman 
Cc: Arnd Bergmann 
Cc: Christoph Hellwig 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra 
Cc: Uladzislau Rezki (Sony) 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

asm-generic: Prepare for riscv use of pud_alloc_one and pud_free

2022-01-20T01:54:08+00:00

In the following commits, riscv will almost use the generic versions of
pud_alloc_one and pud_free but an additional check is required since those
functions are only relevant when using at least a 4-level page table, which
will be determined at runtime on riscv.

So move the content of those functions into other functions that riscv
can use without duplicating code.

Signed-off-by: Alexandre Ghiti 
Signed-off-by: Palmer Dabbelt

asm-generic: pgalloc.h: use correct #ifdef to enable pud_alloc_one()

2020-08-15T02:56:55+00:00

The #ifdef statement that guards the generic version of pud_alloc_one() by
mistake used __HAVE_ARCH_PUD_FREE instead of __HAVE_ARCH_PUD_ALLOC_ONE.

Fix it.

Fixes: d9e8b929670b ("asm-generic: pgalloc: provide generic pud_alloc_one() and pud_free_one()")
Reported-by: kernel test robot 
Signed-off-by: Mike Rapoport 
Signed-off-by: Andrew Morton 
Link: http://lkml.kernel.org/r/20200812191415.GE163101@linux.ibm.com
Signed-off-by: Linus Torvalds

asm-generic: pgalloc: provide generic pgd_free()

2020-08-07T18:33:26+00:00

Most architectures define pgd_free() as a wrapper for free_page().

Provide a generic version in asm-generic/pgalloc.h and enable its use for
most architectures.

Signed-off-by: Mike Rapoport 
Signed-off-by: Andrew Morton 
Reviewed-by: Pekka Enberg 
Acked-by: Geert Uytterhoeven 	[m68k]
Cc: Abdul Haleem 
Cc: Andy Lutomirski 
Cc: Arnd Bergmann 
Cc: Christophe Leroy 
Cc: Joerg Roedel 
Cc: Joerg Roedel 
Cc: Max Filippov 
Cc: Peter Zijlstra (Intel) 
Cc: Satheesh Rajendran 
Cc: Stafford Horne 
Cc: Stephen Rothwell 
Cc: Steven Rostedt 
Cc: Matthew Wilcox 
Link: http://lkml.kernel.org/r/20200627143453.31835-7-rppt@kernel.org
Signed-off-by: Linus Torvalds

asm-generic: pgalloc: provide generic pud_alloc_one() and pud_free_one()

2020-08-07T18:33:26+00:00

Several architectures define pud_alloc_one() as a wrapper for
__get_free_page() and pud_free() as a wrapper for free_page().

Provide a generic implementation in asm-generic/pgalloc.h and use it where
appropriate.

Signed-off-by: Mike Rapoport 
Signed-off-by: Andrew Morton 
Reviewed-by: Pekka Enberg 
Cc: Abdul Haleem 
Cc: Andy Lutomirski 
Cc: Arnd Bergmann 
Cc: Christophe Leroy 
Cc: Joerg Roedel 
Cc: Joerg Roedel 
Cc: Max Filippov 
Cc: Peter Zijlstra (Intel) 
Cc: Satheesh Rajendran 
Cc: Stafford Horne 
Cc: Stephen Rothwell 
Cc: Steven Rostedt 
Cc: Geert Uytterhoeven 
Cc: Matthew Wilcox 
Link: http://lkml.kernel.org/r/20200627143453.31835-6-rppt@kernel.org
Signed-off-by: Linus Torvalds

asm-generic: pgalloc: provide generic pmd_alloc_one() and pmd_free_one()

2020-08-07T18:33:26+00:00

For most architectures that support >2 levels of page tables,
pmd_alloc_one() is a wrapper for __get_free_pages(), sometimes with
__GFP_ZERO and sometimes followed by memset(0) instead.

More elaborate versions on arm64 and x86 account memory for the user page
tables and call to pgtable_pmd_page_ctor() as the part of PMD page
initialization.

Move the arm64 version to include/asm-generic/pgalloc.h and use the
generic version on several architectures.

The pgtable_pmd_page_ctor() is a NOP when ARCH_ENABLE_SPLIT_PMD_PTLOCK is
not enabled, so there is no functional change for most architectures
except of the addition of __GFP_ACCOUNT for allocation of user page
tables.

The pmd_free() is a wrapper for free_page() in all the cases, so no
functional change here.

Signed-off-by: Mike Rapoport 
Signed-off-by: Andrew Morton 
Reviewed-by: Pekka Enberg 
Cc: Matthew Wilcox 
Cc: Abdul Haleem 
Cc: Andy Lutomirski 
Cc: Arnd Bergmann 
Cc: Christophe Leroy 
Cc: Joerg Roedel 
Cc: Joerg Roedel 
Cc: Max Filippov 
Cc: Peter Zijlstra (Intel) 
Cc: Satheesh Rajendran 
Cc: Stafford Horne 
Cc: Stephen Rothwell 
Cc: Steven Rostedt 
Cc: Geert Uytterhoeven 
Link: http://lkml.kernel.org/r/20200627143453.31835-5-rppt@kernel.org
Signed-off-by: Linus Torvalds

mm: treewide: clarify pgtable_page_{ctor,dtor}() naming

2019-09-26T17:10:44+00:00

The naming of pgtable_page_{ctor,dtor}() seems to have confused a few
people, and until recently arm64 used these erroneously/pointlessly for
other levels of page table.

To make it incredibly clear that these only apply to the PTE level, and to
align with the naming of pgtable_pmd_page_{ctor,dtor}(), let's rename them
to pgtable_pte_page_{ctor,dtor}().

These changes were generated with the following shell script:

----
git grep -lw 'pgtable_page_.tor' | while read FILE; do
    sed -i '{s/pgtable_page_ctor/pgtable_pte_page_ctor/}' $FILE;
    sed -i '{s/pgtable_page_dtor/pgtable_pte_page_dtor/}' $FILE;
done
----

... with the documentation re-flowed to remain under 80 columns, and
whitespace fixed up in macros to keep backslashes aligned.

There should be no functional change as a result of this patch.

Link: http://lkml.kernel.org/r/20190722141133.3116-1-mark.rutland@arm.com
Signed-off-by: Mark Rutland 
Reviewed-by: Mike Rapoport 
Acked-by: Geert Uytterhoeven 	[m68k]
Cc: Anshuman Khandual 
Cc: Matthew Wilcox 
Cc: Michal Hocko 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds