linux.git/arch/powerpc/kernel/watchdog.c, branch v5.4.211

powerpc/watchdog: Fix missed watchdog reset due to memory ordering race

2022-01-27T08:19:47+00:00

[ Upstream commit 5dad4ba68a2483fc80d70b9dc90bbe16e1f27263 ]

It is possible for all CPUs to miss the pending cpumask becoming clear,
and then nobody resetting it, which will cause the lockup detector to
stop working. It will eventually expire, but watchdog_smp_panic will
avoid doing anything if the pending mask is clear and it will never be
reset.

Order the cpumask clear vs the subsequent test to close this race.

Add an extra check for an empty pending mask when the watchdog fires and
finds its bit still clear, to try to catch any other possible races or
bugs here and keep the watchdog working. The extra test in
arch_touch_nmi_watchdog is required to prevent the new warning from
firing off.

Signed-off-by: Nicholas Piggin 
Reviewed-by: Laurent Dufour 
Debugged-by: Laurent Dufour 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20211110025056.2084347-2-npiggin@gmail.com
Signed-off-by: Sasha Levin

powerpc/watchdog: Use hrtimers for per-CPU heartbeat

2019-04-30T01:31:02+00:00

Using a jiffies timer creates a dependency on the tick_do_timer_cpu
incrementing jiffies. If that CPU has locked up and jiffies is not
incrementing, the watchdog heartbeat timer for all CPUs stops and
creates false positives and confusing warnings on local CPUs, and
also causes the SMP detector to stop, so the root cause is never
detected.

Fix this by using hrtimer based timers for the watchdog heartbeat,
like the generic kernel hardlockup detector.

Cc: Gautham R. Shenoy 
Reported-by: Ravikumar Bangoria 
Signed-off-by: Nicholas Piggin 
Tested-by: Ravi Bangoria 
Reported-by: Ravi Bangoria 
Reviewed-by: Gautham R. Shenoy 
Signed-off-by: Michael Ellerman

powerpc: NMI IPI make NMI IPIs fully sychronous

2018-07-24T12:03:14+00:00

There is an asynchronous aspect to smp_send_nmi_ipi. The caller waits
for all CPUs to call in to the handler, but it does not wait for
completion of the handler. This is a needless complication, so remove
it and always wait synchronously.

The synchronous wait allows the caller to easily time out and clear
the wait for completion (zero nmi_ipi_busy_count) in the case of badly
behaved handlers. This would have prevented the recent smp_send_stop
NMI IPI bug from causing the system to hang.

Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman

powerpc/watchdog: provide more data in watchdog messages

2018-05-10T13:25:11+00:00

Provide timebase and timebase of last heartbeat in watchdog lockup
messages. Also provide a stack trace of when a CPU becomes un-stuck,
which can be useful -- it could be where irqs are re-enabled, so it
may be the end of the critical section which is responsible for the
latency which is useful information.

Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman

powerpc/watchdog: don't update the watchdog timestamp if a lockup is detected

2018-05-10T13:25:11+00:00

The watchdog heartbeat timestamp is updated when the local heartbeat
timer fires (or touch_nmi_watchdog() is called).

This is an interesting data point, so don't overwrite it when the
soft-NMI interrupt detects a hard lockup. That code came from a pre-
merge version to prevent hard lockup messages flood, but that's taken
care of with the stuck CPU logic now, so there is no reason to
update the heartbeat timestamp here.

Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman

powerpc/watchdog: fix typo 'can by' to 'can be'

2018-05-10T13:25:05+00:00

Signed-off-by: Wolfram Sang 
Signed-off-by: Michael Ellerman

powerpc/watchdog: Print the NIP in soft_nmi_interrupt()

2018-01-28T06:08:29+00:00

When a CPU detects its locked up via soft_nmi_interrupt() we have
pt_regs, so print the regs->nip, which points to where we took the
soft-NMI.

Signed-off-by: Michael Ellerman

powerpc/watchdog: regs can't be null in soft_nmi_interrupt()

2018-01-28T06:08:28+00:00

soft_nmi_interrupt() is called directly from the asm exception
handling code, which passes regs as a pointer to the stack. So regs
can't be NULL, it may be full of junk, but that's a separate problem.

Signed-off-by: Michael Ellerman

powerpc/watchdog: Tweak watchdog printks

2018-01-28T06:08:26+00:00

Use pr_fmt() in the watchdog code, so we don't have to say "Watchdog"
so many times.

Rather than "CPU:%d" just spell it "CPU %d", "Hard" doesn't need a
capital in the middle of a sentence, and "LOCKUP other CPUS" should be
"LOCKUP on other CPUS".

Also make it clear when a CPU self detects a lockup by spelling it
out.

Signed-off-by: Michael Ellerman

powerpc/watchdog: improve watchdog comments

2018-01-21T04:06:26+00:00

The overview comments in the powerpc watchdog are out of date after
several iterations and changes of the code. Bring them up to date.

Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman