linux.git/kernel/rcu/srcutree.c, branch v6.14-rc6

srcu: Remove redundant GP sequence checks in srcu_funnel_gp_start

2024-12-14T16:13:24+00:00

We will perform GP sequence checking at the beginning of srcu_gp_start,
thus making it safe to remove duplicate GP sequence checks prior to
calling srcu_gp_start.

Signed-off-by: Feng Lee <379943137@qq.com>
Signed-off-by: Paul E. McKenney 
Signed-off-by: Uladzislau Rezki (Sony)

srcu: Guarantee non-negative return value from srcu_read_lock()

2024-12-14T16:13:09+00:00

For almost 20 years, the int return value from srcu_read_lock() has
been always either zero or one.  This commit therefore documents the
fact that it will be non-negative, and does the same for the underlying
__srcu_read_lock().

[ paulmck: Apply Andrii Nakryiko feedback. ]

Signed-off-by: Paul E. McKenney 
Acked-by: Andrii Nakryiko 
Acked-by: Peter Zijlstra (Intel) 
Signed-off-by: Uladzislau Rezki (Sony)

srcu: Unconditionally record srcu_read_lock_lite() in ->srcu_reader_flavor

2024-11-15T21:13:37+00:00

Currently, srcu_read_lock_lite() uses the SRCU_READ_FLAVOR_LITE bit in
->srcu_reader_flavor to communicate to the grace-period processing in
srcu_readers_active_idx_check() that the smp_mb() must be replaced by a
synchronize_rcu().  Unfortunately, ->srcu_reader_flavor is not updated
unless the kernel is built with CONFIG_PROVE_RCU=y.  Therefore in all
kernels built with CONFIG_PROVE_RCU=n, srcu_readers_active_idx_check()
incorrectly uses smp_mb() instead of synchronize_rcu() for srcu_struct
structures whose readers use srcu_read_lock_lite().

This commit therefore causes Tree SRCU srcu_read_lock_lite()
to unconditionally update ->srcu_reader_flavor so that
srcu_readers_active_idx_check() can make the correct choice.

Reported-by: Neeraj Upadhyay 
Closes: https://lore.kernel.org/all/d07e8f4a-d5ff-4c8e-8e61-50db285c57e9@amd.com/
Fixes: c0f08d6b5a61 ("srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite()")
Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Reviewed-by: Neeraj Upadhyay 
Signed-off-by: Frederic Weisbecker

srcu: Check for srcu_read_lock_lite() across all CPUs

2024-11-12T22:31:28+00:00

If srcu_read_lock_lite() is used on a given srcu_struct structure, then
the grace-period processing must do synchronize_rcu() instead of smp_mb()
between the scans of the ->srcu_unlock_count[] and ->srcu_lock_count[]
counters.  Currently, it does that by testing the SRCU_READ_FLAVOR_LITE
bit of the ->srcu_reader_flavor mask, which works well.  But only if
the CPU running that srcu_struct structure's grace period has previously
executed srcu_read_lock_lite(), which might not be the case, especially
just after that srcu_struct structure has been created and initialized.

This commit therefore updates the srcu_readers_unlock_idx() function
to OR together the ->srcu_reader_flavor masks from all CPUs, and
then make the srcu_readers_active_idx_check() function that test the
SRCU_READ_FLAVOR_LITE bit in the resulting mask.

Note that the srcu_readers_unlock_idx() function is already scanning all
the CPUs to sum up the ->srcu_unlock_count[] fields and that this is on
the grace-period slow path, hence no concerns about the small amount of
extra work.

Reported-by: Neeraj Upadhyay 
Closes: https://lore.kernel.org/all/d07e8f4a-d5ff-4c8e-8e61-50db285c57e9@amd.com/
Fixes: c0f08d6b5a61 ("srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite()")
Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Reviewed-by: Neeraj Upadhyay 
Signed-off-by: Frederic Weisbecker

srcu: Allow inlining of __srcu_read_{,un}lock_lite()

2024-11-12T14:44:03+00:00

This commit moves __srcu_read_lock_lite() and __srcu_read_unlock_lite()
into include/linux/srcu.h and marks them "static inline" so that they
can be inlined into srcu_read_lock_lite() and srcu_read_unlock_lite(),
respectively.  They are not hand-inlined due to Tree SRCU and Tiny SRCU
having different implementations.

The earlier removal of smp_mb() combined with the inlining produce
significant single-percentage performance wins.

Link: https://lore.kernel.org/all/CAEf4BzYgiNmSb=ZKQ65tm6nJDi1UX2Gq26cdHSH1mPwXJYZj5g@mail.gmail.com/

Reported-by: Alexei Starovoitov 
Signed-off-by: Paul E. McKenney 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Peter Zijlstra 
Cc: Kent Overstreet 
Cc: 
Reviewed-by: Neeraj Upadhyay 
Signed-off-by: Frederic Weisbecker

srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite()

2024-11-12T14:43:34+00:00

This patch adds srcu_read_lock_lite() and srcu_read_unlock_lite(), which
dispense with the read-side smp_mb() but also are restricted to code
regions that RCU is watching.  If a given srcu_struct structure uses
srcu_read_lock_lite() and srcu_read_unlock_lite(), it is not permitted
to use any other SRCU read-side marker, before, during, or after.

Another price of light-weight readers is heavier weight grace periods.
Such readers mean that SRCU grace periods on srcu_struct structures
used by light-weight readers will incur at least two calls to
synchronize_rcu().  In addition, normal SRCU grace periods for
light-weight-reader srcu_struct structures never auto-expedite.
Note that expedited SRCU grace periods for light-weight-reader
srcu_struct structures still invoke synchronize_rcu(), not
synchronize_srcu_expedited().  Something about wishing to keep
the IPIs down to a dull roar.

The srcu_read_lock_lite() and srcu_read_unlock_lite() functions may not
(repeat, *not*) be used from NMI handlers, but if this is needed, an
additional flavor of SRCU reader can be added by some future commit.

[ paulmck: Apply Alexei Starovoitov expediting feedback. ]
[ paulmck: Apply kernel test robot feedback. ]

Signed-off-by: Paul E. McKenney 
Tested-by: kernel test robot 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Peter Zijlstra 
Cc: Kent Overstreet 
Cc: 
Reviewed-by: Neeraj Upadhyay 
Signed-off-by: Frederic Weisbecker

srcu: Create CPP macros for normal and NMI-safe SRCU readers

2024-11-12T14:43:21+00:00

This commit creates SRCU_READ_FLAVOR_NORMAL and SRCU_READ_FLAVOR_NMI
C-preprocessor macros for srcu_read_lock() and srcu_read_lock_nmisafe(),
respectively.  These replace the old true/false values that were
previously passed to srcu_check_read_flavor().  In addition, the
srcu_check_read_flavor() function itself requires a bit of rework to
handle bitmasks instead of true/false values.

Signed-off-by: Paul E. McKenney 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Peter Zijlstra 
Cc: Kent Overstreet 
Cc: 
Signed-off-by: Paul E. McKenney 
Reviewed-by: Neeraj Upadhyay 
Signed-off-by: Frederic Weisbecker

srcu: Standardize srcu_data pointers to "sdp" and similar

2024-11-12T14:42:41+00:00

This commit changes a few "cpuc" variables to "sdp" to align with usage
elsewhere.

[ paulmck: Apply Neeraj Upadhyay feedback. ]

Signed-off-by: Paul E. McKenney 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Peter Zijlstra 
Cc: Kent Overstreet 
Cc: 
Reviewed-by: Neeraj Upadhyay 
Signed-off-by: Frederic Weisbecker

srcu: Bit manipulation changes for additional reader flavor

2024-11-12T14:42:20+00:00

Currently, there are only two flavors of readers, normal and NMI-safe.
Very straightforward state updates suffice to check for erroneous
mixing of reader flavors on a given srcu_struct structure.  This commit
upgrades the checking in preparation for the addition of light-weight
(as in memory-barrier-free) readers.

Signed-off-by: Paul E. McKenney 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Peter Zijlstra 
Cc: Kent Overstreet 
Cc: 
Reviewed-by: Neeraj Upadhyay 
Signed-off-by: Frederic Weisbecker

srcu: Renaming in preparation for additional reader flavor

2024-11-12T14:41:30+00:00

Currently, there are only two flavors of readers, normal and NMI-safe.
A number of fields, functions, and types reflect this restriction.
This renaming-only commit prepares for the addition of light-weight
(as in memory-barrier-free) readers.  OK, OK, there is also a drive-by
white-space fixeup!

Signed-off-by: Paul E. McKenney 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Peter Zijlstra 
Cc: Kent Overstreet 
Cc: 
Reviewed-by: Neeraj Upadhyay 
Signed-off-by: Frederic Weisbecker