linux.git/kernel/unwind/deferred.c, branch v6.19.12

unwind: Make unwind_task_info::unwind_mask consistent

2025-10-29T09:29:57+00:00

The unwind_task_info::unwind_mask was manipulated using a mixture of:

  regular store
  WRITE_ONCE()
  try_cmpxchg()
  set_bit()
  atomic_long_*()

Clean up and make it consistently atomic_long_t.

Signed-off-by: Peter Zijlstra (Intel) 
Link: https://patch.msgid.link/20250924080119.384384486@infradead.org

unwind: Simplify unwind_user_faultable()

2025-10-29T09:29:56+00:00

Signed-off-by: Peter Zijlstra (Intel) 
Link: https://patch.msgid.link/20250924080119.271671514@infradead.org

unwind: Clarify calling context

2025-10-29T09:29:56+00:00

The get_cookie() function hard relies on IRQs being disabled, but this
isn't immediately obvious when reading the function.

Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Steven Rostedt (Google) 
Link: https://patch.msgid.link/20250924080119.122507632@infradead.org

unwind: Fix unwind_deferred_request() vs NMI

2025-10-29T09:29:56+00:00

task_work_add(RWA_RESUME) isn't NMI-safe, use TWA_NMI_CURRENT when
used from NMI context.

Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Steven Rostedt (Google) 
Link: https://patch.msgid.link/20250924080119.005422353@infradead.org

unwind: Finish up unwind when a task exits

2025-07-31T14:20:11+00:00

On do_exit() when a task is exiting, if a unwind is requested and the
deferred user stacktrace is deferred via the task_work, the task_work
callback is called after exit_mm() is called in do_exit(). This means that
the user stack trace will not be retrieved and an empty stack is created.

Instead, add a function unwind_deferred_task_exit() and call it just
before exit_mm() so that the unwinder can call the requested callbacks
with the user space stack.

Cc: Masami Hiramatsu 
Cc: Mathieu Desnoyers 
Cc: Josh Poimboeuf 
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182406.504259474@kernel.org
Signed-off-by: Steven Rostedt (Google)

unwind deferred: Use SRCU unwind_deferred_task_work()

2025-07-31T14:20:11+00:00

Instead of using the callback_mutex to protect the link list of callbacks
in unwind_deferred_task_work(), use SRCU instead. This gets called every
time a task exits that has to record a stack trace that was requested.
This can happen for many tasks on several CPUs at the same time. A mutex
is a bottleneck and can cause a bit of contention and slow down performance.

As the callbacks themselves are allowed to sleep, regular RCU cannot be
used to protect the list. Instead use SRCU, as that still allows the
callbacks to sleep and the list can be read without needing to hold the
callback_mutex.

Link: https://lore.kernel.org/all/ca9bd83a-6c80-4ee0-a83c-224b9d60b755@efficios.com/

Cc: "Paul E. McKenney" 
Cc: Masami Hiramatsu 
Cc: Josh Poimboeuf 
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182406.331548065@kernel.org
Suggested-by: Mathieu Desnoyers 
Signed-off-by: Steven Rostedt (Google)

unwind: Add USED bit to only have one conditional on way back to user space

2025-07-31T14:20:11+00:00

On the way back to user space, the function unwind_reset_info() is called
unconditionally (but always inlined). It currently has two conditionals.
One that checks the unwind_mask which is set whenever a deferred trace is
called and is used to know that the mask needs to be cleared. The other
checks if the cache has been allocated, and if so, it resets the
nr_entries so that the unwinder knows it needs to do the work to get a new
user space stack trace again (it only does it once per entering the
kernel).

Use one of the bits in the unwind mask as a "USED" bit that gets set
whenever a trace is created. This will make it possible to only check the
unwind_mask in the unwind_reset_info() to know if it needs to do work or
not and eliminates a conditional that happens every time the task goes
back to user space.

Cc: Masami Hiramatsu 
Cc: Mathieu Desnoyers 
Cc: Josh Poimboeuf 
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182406.155422551@kernel.org
Signed-off-by: Steven Rostedt (Google)

unwind deferred: Add unwind_completed mask to stop spurious callbacks

2025-07-31T14:20:10+00:00

If there's more than one registered tracer to the unwind deferred
infrastructure, it is currently possible that one tracer could cause extra
callbacks to happen for another tracer if the former requests a deferred
stacktrace after the latter's callback was executed and before the task
went back to user space.

Here's an example of how this could occur:

  [Task enters kernel]
    tracer 1 request -> add cookie to its buffer
    tracer 1 request -> add cookie to its buffer
    <..>
    [ task work executes ]
    tracer 1 callback -> add trace + cookie to its buffer

    [tracer 2 requests and triggers the task work again]
    [ task work executes again ]
    tracer 1 callback -> add trace + cookie to its buffer
    tracer 2 callback -> add trace + cookie to its buffer
 [Task exits back to user space]

This is because the bit for tracer 1 gets set in the task's unwind_mask
when it did its request and does not get cleared until the task returns
back to user space. But if another tracer were to request another deferred
stacktrace, then the next task work will executed all tracer's callbacks
that have their bits set in the task's unwind_mask.

To fix this issue, add another mask called unwind_completed and place it
into the task's info->cache structure. The cache structure is allocated
on the first occurrence of a deferred stacktrace and this unwind_completed
mask is not needed until then. It's better to have it in the cache than to
permanently waste space in the task_struct.

After a tracer's callback is executed, it's bit gets set in this
unwind_completed mask. When the task_work enters, it will AND the task's
unwind_mask with the inverse of the unwind_completed which will eliminate
any work that already had its callback executed since the task entered the
kernel.

When the task leaves the kernel, it will reset this unwind_completed mask
just like it resets the other values as it enters user space.

Link: https://lore.kernel.org/all/20250716142609.47f0e4a5@batman.local.home/

Cc: Masami Hiramatsu 
Cc: Mathieu Desnoyers 
Cc: Josh Poimboeuf 
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182405.989222722@kernel.org
Signed-off-by: Steven Rostedt (Google)

unwind deferred: Use bitmask to determine which callbacks to call

2025-07-31T14:20:10+00:00

In order to know which registered callback requested a stacktrace for when
the task goes back to user space, add a bitmask to keep track of all
registered tracers. The bitmask is the size of long, which means that on a
32 bit machine, it can have at most 32 registered tracers, and on 64 bit,
it can have at most 64 registered tracers. This should not be an issue as
there should not be more than 10 (unless BPF can abuse this?).

When a tracer registers with unwind_deferred_init() it will get a bit
number assigned to it. When a tracer requests a stacktrace, it will have
its bit set within the task_struct. When the task returns back to user
space, it will call the callbacks for all the registered tracers where
their bits are set in the task's mask.

When a tracer is removed by the unwind_deferred_cancel() all current tasks
will clear the associated bit, just in case another tracer gets registered
immediately afterward and then gets their callback called unexpectedly.

To prevent live locks from happening if an event that happens between the
task_work and when the task goes back to user space, triggers the deferred
unwind, have the unwind_mask get cleared on exit to user space and not
after the callback is made.

Move the pending bit from a value on the task_struct to bit zero of the
unwind_mask (saves space on the task_struct). This will allow modifying
the pending bit along with the work bits atomically.

Instead of clearing a work's bit after its callback is called, it is
delayed until exit. If the work is requested again, the task_work is not
queued again and the request will be notified that the task has already been
called by returning a positive number (the same as if it was already
pending).

The pending bit is cleared before calling the callback functions but the
current work bits remain. If one of the called works registers again, it
will not trigger a task_work if its bit is still present in the task's
unwind_mask.

If a new work requests a deferred unwind, then it will set both the
pending bit and its own bit. Note this will also cause any work that was
previously queued and had their callback already executed to be executed
again. Future work will remove these spurious callbacks.

The use of atomic_long bit operations were suggested by Peter Zijlstra:
Link: https://lore.kernel.org/all/20250715102912.GQ1613200@noisy.programming.kicks-ass.net/
The unwind_mask could not be converted to atomic_long_t do to atomic_long
not having all the bit operations needed by unwind_mask. Instead it
follows other use cases in the kernel and just typecasts the unwind_mask
to atomic_long_t when using the two atomic_long functions.

Cc: Masami Hiramatsu 
Cc: Mathieu Desnoyers 
Cc: Josh Poimboeuf 
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182405.822789300@kernel.org
Signed-off-by: Steven Rostedt (Google)

unwind_user/deferred: Make unwind deferral requests NMI-safe

2025-07-31T14:20:10+00:00

Make unwind_deferred_request() NMI-safe so tracers in NMI context can
call it and safely request a user space stacktrace when the task exits.

Note, this is only allowed for architectures that implement a safe
cmpxchg. If an architecture requests a deferred stack trace from NMI
context that does not support a safe NMI cmpxchg, it will get an -EINVAL
and trigger a warning. For those architectures, they would need another
method (perhaps an irqwork), to request a deferred user space stack trace.
That can be dealt with later if one of theses architectures require this
feature.

Cc: Masami Hiramatsu 
Cc: Mathieu Desnoyers 
Cc: Josh Poimboeuf 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182405.657072238@kernel.org
Suggested-by: Peter Zijlstra 
Signed-off-by: Steven Rostedt (Google)