linux.git/kernel/locking/percpu-rwsem.c, branch v6.18.21

locking/percpu-rwsem: add freezable alternative to down_read

2025-04-07T07:37:16+00:00

Percpu-rwsems are used for superblock locking.  However, we know the
read percpu-rwsem we take for sb_start_write() on a frozen filesystem
needs not to inhibit system from suspending or hibernating.  That
means it needs to wait with TASK_UNINTERRUPTIBLE | TASK_FREEZABLE.

Introduce a new percpu_down_read_freezable() that allows us to control
whether TASK_FREEZABLE is added to the wait flags.

Signed-off-by: James Bottomley 
Link: https://lore.kernel.org/r/20250327140613.25178-2-James.Bottomley@HansenPartnership.com
Signed-off-by: Christian Brauner

percpu: use TYPEOF_UNQUAL() in variable declarations

2025-03-17T05:05:53+00:00

Use TYPEOF_UNQUAL() to declare variables as a corresponding type without
named address space qualifier to avoid "`__seg_gs' specified for auto
variable `var'" errors.

Link: https://lkml.kernel.org/r/20250127160709.80604-4-ubizjak@gmail.com
Signed-off-by: Uros Bizjak 
Acked-by: Nadav Amit 
Acked-by: Christoph Lameter 
Cc: Dennis Zhou 
Cc: Tejun Heo 
Cc: Andy Lutomirski 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: Borislav Petkov 
Cc: Dave Hansen 
Cc: "H. Peter Anvin" 
Cc: Kent Overstreet 
Cc: Arnd Bergmann 
Cc: "David S. Miller" 
Cc: Eric Dumazet 
Cc: Jakub Kicinski 
Cc: Paolo Abeni 
Cc: Peter Zijlstra 
Cc: Will Deacon 
Cc: Waiman Long 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Brian Gerst 
Cc: Denys Vlasenko 
Signed-off-by: Andrew Morton

locking/percpu-rwsem: Trigger contention tracepoints only if contended

2024-02-28T12:10:29+00:00

We mistakenly always fire lock contention tracepoints in the writer path,
while it should be conditional on the trylock result.

Signed-off-by: Namhyung Kim 
Signed-off-by: Ingo Molnar 
Reviewed-by: Waiman Long 
Link: https://lore.kernel.org/r/20231108215322.2845536-1-namhyung@kernel.org

locking/percpu-rwsem: Add percpu_is_write_locked() and percpu_is_read_locked()

2022-08-30T08:56:23+00:00

Implement simple accessors to probe percpu-rwsem's locked state:
percpu_is_write_locked(), percpu_is_read_locked().

Signed-off-by: Marco Elver 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Dmitry Vyukov 
Acked-by: Ian Rogers 
Link: https://lore.kernel.org/r/20220829124719.675715-11-elver@google.com

locking: Apply contention tracepoints in the slow path

2022-04-05T08:24:35+00:00

Adding the lock contention tracepoints in various lock function slow
paths.  Note that each arch can define spinlock differently, I only
added it only to the generic qspinlock for now.

Signed-off-by: Namhyung Kim 
Signed-off-by: Peter Zijlstra (Intel) 
Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Link: https://lkml.kernel.org/r/20220322185709.141236-3-namhyung@kernel.org

locking: Add missing __sched attributes

2022-02-11T11:13:55+00:00

This patch adds __sched attributes to a few missing places
to show blocked function rather than locking function
in get_wchan.

Signed-off-by: Minchan Kim 
Signed-off-by: Peter Zijlstra (Intel) 
Link: https://lkml.kernel.org/r/20220115231657.84828-1-minchan@kernel.org

locking/percpu-rwsem: Use this_cpu_{inc,dec}() for read_count

2020-09-16T14:26:56+00:00

The __this_cpu*() accessors are (in general) IRQ-unsafe which, given
that percpu-rwsem is a blocking primitive, should be just fine.

However, file_end_write() is used from IRQ context and will cause
load-store issues on architectures where the per-cpu accessors are not
natively irq-safe.

Fix it by using the IRQ-safe this_cpu_*() for operations on
read_count. This will generate more expensive code on a number of
platforms, which might cause a performance regression for some of the
other percpu-rwsem users.

If any such is reported, we can consider alternative solutions.

Fixes: 70fe2f48152e ("aio: fix freeze protection of aio writes")
Signed-off-by: Hou Tao 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Will Deacon 
Acked-by: Oleg Nesterov 
Link: https://lkml.kernel.org/r/20200915140750.137881-1-houtao1@huawei.com

locking/percpu-rwsem: Fix a task_struct refcount

2020-04-08T10:05:06+00:00

The following commit:

  7f26482a872c ("locking/percpu-rwsem: Remove the embedded rwsem")

introduced task_struct memory leaks due to messing up the task_struct
refcount.

At the beginning of percpu_rwsem_wake_function(), it calls get_task_struct(),
but if the trylock failed, it will remain in the waitqueue. However, it
will run percpu_rwsem_wake_function() again with get_task_struct() to
increase the refcount but then only call put_task_struct() once the trylock
succeeded.

Fix it by adjusting percpu_rwsem_wake_function() a bit to guard against
when percpu_rwsem_wait() observing !private, terminating the wait and
doing a quick exit() while percpu_rwsem_wake_function() then doing
wake_up_process(p) as a use-after-free.

Fixes: 7f26482a872c ("locking/percpu-rwsem: Remove the embedded rwsem")
Suggested-by: Peter Zijlstra 
Signed-off-by: Qian Cai 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Link: https://lkml.kernel.org/r/20200330213002.2374-1-cai@lca.pw

rcuwait: Add @state argument to rcuwait_wait_event()

2020-03-21T15:00:22+00:00

Extend rcuwait_wait_event() with a state variable so that it is not
restricted to UNINTERRUPTIBLE waits.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Thomas Gleixner 
Signed-off-by: Peter Zijlstra (Intel) 
Link: https://lkml.kernel.org/r/20200321113241.824030968@linutronix.de

locking/percpu-rwsem: Add might_sleep() for writer locking

2020-02-11T12:11:02+00:00

We are missing this annotation in percpu_down_write(). Correct
this.

Signed-off-by: Davidlohr Bueso 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Acked-by: Will Deacon 
Acked-by: Waiman Long 
Link: https://lkml.kernel.org/r/20200108013305.7732-1-dave@stgolabs.net