linux.git/include/asm-generic/bitops, branch v6.18.21

bitops: Add __attribute_const__ to generic ffs()-family implementations

2025-09-08T21:58:50+00:00

While tracking down a problem where constant expressions used by
BUILD_BUG_ON() suddenly stopped working[1], we found that an added static
initializer was convincing the compiler that it couldn't track the state
of the prior statically initialized value. Tracing this down found that
ffs() was used in the initializer macro, but since it wasn't marked with
__attribute__const__, the compiler had to assume the function might
change variable states as a side-effect (which is not true for ffs(),
which provides deterministic math results).

Add missing __attribute_const__ annotations to generic implementations of
ffs(), __ffs(), fls(), and __fls() functions. These are pure mathematical
functions that always return the same result for the same input with no
side effects, making them eligible for compiler optimization.

Build tested with x86_64 defconfig using GCC 14.2.0, which should validate
the implementations when used by ARM, ARM64, LoongArch, Microblaze,
NIOS2, and SPARC32 architectures.

Link: https://github.com/KSPP/linux/issues/364 [1]
Acked-by: Peter Zijlstra (Intel) 
Link: https://lore.kernel.org/r/20250804164417.1612371-2-kees@kernel.org
Signed-off-by: Kees Cook

bitops: Change function return types from long to int

2024-05-03T15:04:50+00:00

Change the return types of bitops functions (ffs, fls, and fns) from
long to int. The expected return values are in the range [0, 64], for
which int is sufficient.

Additionally, int aligns well with the return types of the corresponding
__builtin_* functions, potentially reducing overall type conversions.

Many of the existing bitops functions already return an int and don't
need to be changed. The bitops functions in arch/ should be considered
separately.

Adjust some return variables to match the function return types.

With GCC 13 and defconfig, these changes reduced the size of a test
kernel image by 5,432 bytes on arm64 and by 248 bytes on riscv; there
were no changes in size on x86_64, powerpc, or m68k.

Signed-off-by: Thorsten Blum 
Signed-off-by: Arnd Bergmann

riscv: Avoid code duplication with generic bitops implementation

2024-01-25T01:25:36+00:00

There's code duplication between the fallback implementation for bitops
__ffs/__fls/ffs/fls API and the generic C implementation in
include/asm-generic/bitops/. To avoid this duplication, this patch renames
the generic C implementation by adding a "generic_" prefix to them, then we
can use these generic APIs as fallback.

Suggested-by: Geert Uytterhoeven 
Signed-off-by: Xiao Wang 
Reviewed-by: Charlie Jenkins 
Link: https://lore.kernel.org/r/20231112094421.4014931-1-xiao.w.wang@intel.com
Signed-off-by: Palmer Dabbelt

mm: delete checks for xor_unlock_is_negative_byte()

2023-10-18T21:34:17+00:00

Architectures which don't define their own use the one in
asm-generic/bitops/lock.h.  Get rid of all the ifdefs around "maybe we
don't have it".

Link: https://lkml.kernel.org/r/20231004165317.1061855-15-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Acked-by: Geert Uytterhoeven 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Andreas Dilger 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Heiko Carstens 
Cc: Ivan Kokshaysky 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Richard Henderson 
Cc: Sven Schnelle 
Cc: "Theodore Ts'o" 
Cc: Thomas Bogendoerfer 
Cc: Vasily Gorbik 
Signed-off-by: Andrew Morton

bitops: add xor_unlock_is_negative_byte()

2023-10-18T21:34:16+00:00

Replace clear_bit_and_unlock_is_negative_byte() with
xor_unlock_is_negative_byte().  We have a few places that like to lock a
folio, set a flag and unlock it again.  Allow for the possibility of
combining the latter two operations for efficiency.  We are guaranteed
that the caller holds the lock, so it is safe to unlock it with the xor. 
The caller must guarantee that nobody else will set the flag without
holding the lock; it is not safe to do this with the PG_dirty flag, for
example.

Link: https://lkml.kernel.org/r/20231004165317.1061855-8-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Andreas Dilger 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Geert Uytterhoeven 
Cc: Heiko Carstens 
Cc: Ivan Kokshaysky 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Richard Henderson 
Cc: Sven Schnelle 
Cc: "Theodore Ts'o" 
Cc: Thomas Bogendoerfer 
Cc: Vasily Gorbik 
Signed-off-by: Andrew Morton

locking/atomic: treewide: use raw_atomic*_()

2023-06-05T07:57:20+00:00

Now that we have raw_atomic*_() definitions, there's no need to use
arch_atomic*_() definitions outside of the low-level atomic
definitions.

Move treewide users of arch_atomic*_() over to the equivalent
raw_atomic*_().

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Kees Cook 
Link: https://lore.kernel.org/r/20230605070124.3741859-19-mark.rutland@arm.com

wait_on_bit: add an acquire memory barrier

2022-08-26T16:30:25+00:00

There are several places in the kernel where wait_on_bit is not followed
by a memory barrier (for example, in drivers/md/dm-bufio.c:new_read).

On architectures with weak memory ordering, it may happen that memory
accesses that follow wait_on_bit are reordered before wait_on_bit and
they may return invalid data.

Fix this class of bugs by introducing a new function "test_bit_acquire"
that works like test_bit, but has acquire memory ordering semantics.

Signed-off-by: Mikulas Patocka 
Acked-by: Will Deacon 
Cc: stable@vger.kernel.org
Signed-off-by: Linus Torvalds

locking/atomic: Make test_and_*_bit() ordered on failure

2022-08-16T16:19:43+00:00

These operations are documented as always ordered in
include/asm-generic/bitops/instrumented-atomic.h, and producer-consumer
type use cases where one side needs to ensure a flag is left pending
after some shared data was updated rely on this ordering, even in the
failure case.

This is the case with the workqueue code, which currently suffers from a
reproducible ordering violation on Apple M1 platforms (which are
notoriously out-of-order) that ends up causing the TTY layer to fail to
deliver data to userspace properly under the right conditions.  This
change fixes that bug.

Change the documentation to restrict the "no order on failure" story to
the _lock() variant (for which it makes sense), and remove the
early-exit from the generic implementation, which is what causes the
missing barrier semantics in that case.  Without this, the remaining
atomic op is fully ordered (including on ARM64 LSE, as of recent
versions of the architecture spec).

Suggested-by: Linus Torvalds 
Cc: stable@vger.kernel.org
Fixes: e986a0d6cb36 ("locking/atomics, asm-generic/bitops/atomic.h: Rewrite using atomic_*() APIs")
Fixes: 61e02392d3c7 ("locking/atomic/bitops: Document and clarify ordering semantics for failed test_and_{}_bit()")
Signed-off-by: Hector Martin 
Acked-by: Will Deacon 
Reviewed-by: Arnd Bergmann 
Signed-off-by: Linus Torvalds

bitops: wrap non-atomic bitops with a transparent macro

2022-07-01T02:52:41+00:00

In preparation for altering the non-atomic bitops with a macro, wrap
them in a transparent definition. This requires prepending one more
'_' to their names in order to be able to do that seamlessly. It is
a simple change, given that all the non-prefixed definitions are now
in asm-generic.
sparc32 already has several triple-underscored functions, so I had
to rename them ('___' -> 'sp32_').

Signed-off-by: Alexander Lobakin 
Reviewed-by: Marco Elver 
Reviewed-by: Andy Shevchenko 
Signed-off-by: Yury Norov

bitops: define const_*() versions of the non-atomics

2022-07-01T02:52:41+00:00

Define const_*() variants of the non-atomic bitops to be used when
the input arguments are compile-time constants, so that the compiler
will be always able to resolve those to compile-time constants as
well. Those are mostly direct aliases for generic_*() with one
exception for const_test_bit(): the original one is declared
atomic-safe and thus doesn't discard the `volatile` qualifier, so
in order to let optimize code, define it separately disregarding
the qualifier.
Add them to the compile-time type checks as well just in case.

Suggested-by: Marco Elver 
Signed-off-by: Alexander Lobakin 
Reviewed-by: Marco Elver 
Reviewed-by: Andy Shevchenko 
Signed-off-by: Yury Norov