linux.git/mm/dmapool.c, branch v6.12.80

mm/mempool/dmapool: remove CONFIG_DEBUG_SLAB ifdefs

2023-12-05T10:17:58+00:00

CONFIG_DEBUG_SLAB is going away with CONFIG_SLAB, so remove dead ifdefs
in mempool and dmapool code.

Reviewed-by: Kees Cook 
Acked-by: David Rientjes 
Tested-by: David Rientjes 
Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Vlastimil Babka

dmapool: create/destroy cleanup

2023-06-09T23:25:17+00:00

Set the 'empty' bool directly from the result of the function that
determines its value instead of adding additional logic.

Link: https://lkml.kernel.org/r/20230126215125.4069751-13-kbusch@meta.com
Fixes: 2d55c16c0c54 ("dmapool: create/destroy cleanup")
Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Cc: Matthew Wilcox 
Cc: Tony Battersby 
Signed-off-by: Andrew Morton

dmapool: link blocks across pages

2023-05-06T17:33:38+00:00

The allocated dmapool pages are never freed for the lifetime of the pool. 
There is no need for the two level list+stack lookup for finding a free
block since nothing is ever removed from the list.  Just use a simple
stack, reducing time complexity to constant.

The implementation inserts the stack linking elements and the dma handle
of the block within itself when freed.  This means the smallest possible
dmapool block is increased to at most 16 bytes to accommodate these
fields, but there are no exisiting users requesting a dma pool smaller
than that anyway.

Removing the list has a significant change in performance. Using the
kernel's micro-benchmarking self test:

Before:

  # modprobe dmapool_test
  dmapool test: size:16   blocks:8192   time:57282
  dmapool test: size:64   blocks:8192   time:172562
  dmapool test: size:256  blocks:8192   time:789247
  dmapool test: size:1024 blocks:2048   time:371823
  dmapool test: size:4096 blocks:1024   time:362237

After:

  # modprobe dmapool_test
  dmapool test: size:16   blocks:8192   time:24997
  dmapool test: size:64   blocks:8192   time:26584
  dmapool test: size:256  blocks:8192   time:33542
  dmapool test: size:1024 blocks:2048   time:9022
  dmapool test: size:4096 blocks:1024   time:6045

The module test allocates quite a few blocks that may not accurately
represent how these pools are used in real life.  For a more marco level
benchmark, running fio high-depth + high-batched on nvme, this patch shows
submission and completion latency reduced by ~100usec each, 1% IOPs
improvement, and perf record's time spent in dma_pool_alloc/free were
reduced by half.

[kbusch@kernel.org: push new blocks in ascending order]
  Link: https://lkml.kernel.org/r/20230221165400.1595247-1-kbusch@meta.com
Link: https://lkml.kernel.org/r/20230126215125.4069751-12-kbusch@meta.com
Fixes: 2d55c16c0c54 ("dmapool: create/destroy cleanup")
Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Tested-by: Bryan O'Donoghue 
Cc: Matthew Wilcox 
Cc: Tony Battersby 
Signed-off-by: Andrew Morton

dmapool: don't memset on free twice

2023-05-06T17:33:38+00:00

If debug is enabled, dmapool will poison the range, so no need to clear it
to 0 immediately before writing over it.

Link: https://lkml.kernel.org/r/20230126215125.4069751-11-kbusch@meta.com
Fixes: 2d55c16c0c54 ("dmapool: create/destroy cleanup")
Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Cc: Matthew Wilcox 
Cc: Tony Battersby 
Signed-off-by: Andrew Morton

dmapool: simplify freeing

2023-05-06T17:33:37+00:00

The actions for busy and not busy are mostly the same, so combine these
and remove the unnecessary function.  Also, the pool is about to be freed
so there's no need to poison the page data since we only check for poison
on alloc, which can't be done on a freed pool.

Link: https://lkml.kernel.org/r/20230126215125.4069751-10-kbusch@meta.com
Fixes: 2d55c16c0c54 ("dmapool: create/destroy cleanup")
Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Cc: Matthew Wilcox 
Cc: Tony Battersby 
Signed-off-by: Andrew Morton

dmapool: consolidate page initialization

2023-05-06T17:33:37+00:00

Various fields of the dma pool are set in different places. Move it all
to one function.

Link: https://lkml.kernel.org/r/20230126215125.4069751-9-kbusch@meta.com
Fixes: 2d55c16c0c54 ("dmapool: create/destroy cleanup")
Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Cc: Matthew Wilcox 
Cc: Tony Battersby 
Signed-off-by: Andrew Morton

dmapool: rearrange page alloc failure handling

2023-05-06T17:33:37+00:00

Handle the error in a condition so the good path can be in the normal
flow.

Link: https://lkml.kernel.org/r/20230126215125.4069751-8-kbusch@meta.com
Fixes: 2d55c16c0c54 ("dmapool: create/destroy cleanup")
Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Cc: Matthew Wilcox 
Cc: Tony Battersby 
Signed-off-by: Andrew Morton

dmapool: move debug code to own functions

2023-05-06T17:33:37+00:00

Clean up the normal path by moving the debug code outside it.

Link: https://lkml.kernel.org/r/20230126215125.4069751-7-kbusch@meta.com
Fixes: 2d55c16c0c54 ("dmapool: create/destroy cleanup")
Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Cc: Matthew Wilcox 
Cc: Tony Battersby 
Signed-off-by: Andrew Morton

dmapool: speedup DMAPOOL_DEBUG with init_on_alloc

2023-05-06T17:33:36+00:00

Avoid double-memset of the same allocated memory in dma_pool_alloc() when
both DMAPOOL_DEBUG is enabled and init_on_alloc=1.

Link: https://lkml.kernel.org/r/20230126215125.4069751-6-kbusch@meta.com
Fixes: 2d55c16c0c54 ("dmapool: create/destroy cleanup")
Signed-off-by: Tony Battersby 
Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Cc: Matthew Wilcox 
Signed-off-by: Andrew Morton

dmapool: cleanup integer types

2023-05-06T17:33:36+00:00

To represent the size of a single allocation, dmapool currently uses
'unsigned int' in some places and 'size_t' in other places.  Standardize
on 'unsigned int' to reduce overhead, but use 'size_t' when counting all
the blocks in the entire pool.

Link: https://lkml.kernel.org/r/20230126215125.4069751-5-kbusch@meta.com
Fixes: 2d55c16c0c54 ("dmapool: create/destroy cleanup")
Signed-off-by: Tony Battersby 
Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Cc: Matthew Wilcox 
Signed-off-by: Andrew Morton