diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2024-09-25 10:25:40 -0700 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2024-09-25 10:25:40 -0700 |
| commit | 570172569238c66a482ec3eb5d766cc9cf255f69 (patch) | |
| tree | 615e935d737b3f8ec9d3a49cb1895c9cac81d679 | |
| parent | 684a64bf32b6e488004e0ad7f0d7e922798f65b6 (diff) | |
| parent | a2f11547052001bd448ccec81dd1e68409078fbb (diff) | |
| download | linux-570172569238c66a482ec3eb5d766cc9cf255f69.tar.gz linux-570172569238c66a482ec3eb5d766cc9cf255f69.tar.bz2 linux-570172569238c66a482ec3eb5d766cc9cf255f69.zip | |
Merge tag 'rust-6.12' of https://github.com/Rust-for-Linux/linux
Pull Rust updates from Miguel Ojeda:
"Toolchain and infrastructure:
- Support 'MITIGATION_{RETHUNK,RETPOLINE,SLS}' (which cleans up
objtool warnings), teach objtool about 'noreturn' Rust symbols and
mimic '___ADDRESSABLE()' for 'module_{init,exit}'. With that, we
should be objtool-warning-free, so enable it to run for all Rust
object files.
- KASAN (no 'SW_TAGS'), KCFI and shadow call sanitizer support.
- Support 'RUSTC_VERSION', including re-config and re-build on
change.
- Split helpers file into several files in a folder, to avoid
conflicts in it. Eventually those files will be moved to the right
places with the new build system. In addition, remove the need to
manually export the symbols defined there, reusing existing
machinery for that.
- Relax restriction on configurations with Rust + GCC plugins to just
the RANDSTRUCT plugin.
'kernel' crate:
- New 'list' module: doubly-linked linked list for use with reference
counted values, which is heavily used by the upcoming Rust Binder.
This includes 'ListArc' (a wrapper around 'Arc' that is guaranteed
unique for the given ID), 'AtomicTracker' (tracks whether a
'ListArc' exists using an atomic), 'ListLinks' (the prev/next
pointers for an item in a linked list), 'List' (the linked list
itself), 'Iter' (an iterator over a 'List'), 'Cursor' (a cursor
into a 'List' that allows to remove elements), 'ListArcField' (a
field exclusively owned by a 'ListArc'), as well as support for
heterogeneous lists.
- New 'rbtree' module: red-black tree abstractions used by the
upcoming Rust Binder.
This includes 'RBTree' (the red-black tree itself), 'RBTreeNode' (a
node), 'RBTreeNodeReservation' (a memory reservation for a node),
'Iter' and 'IterMut' (immutable and mutable iterators), 'Cursor'
(bidirectional cursor that allows to remove elements), as well as
an entry API similar to the Rust standard library one.
- 'init' module: add 'write_[pin_]init' methods and the
'InPlaceWrite' trait. Add the 'assert_pinned!' macro.
- 'sync' module: implement the 'InPlaceInit' trait for 'Arc' by
introducing an associated type in the trait.
- 'alloc' module: add 'drop_contents' method to 'BoxExt'.
- 'types' module: implement the 'ForeignOwnable' trait for
'Pin<Box<T>>' and improve the trait's documentation. In addition,
add the 'into_raw' method to the 'ARef' type.
- 'error' module: in preparation for the upcoming Rust support for
32-bit architectures, like arm, locally allow Clippy lint for
those.
Documentation:
- https://rust.docs.kernel.org has been announced, so link to it.
- Enable rustdoc's "jump to definition" feature, making its output a
bit closer to the experience in a cross-referencer.
- Debian Testing now also provides recent Rust releases (outside of
the freeze period), so add it to the list.
MAINTAINERS:
- Trevor is joining as reviewer of the "RUST" entry.
And a few other small bits"
* tag 'rust-6.12' of https://github.com/Rust-for-Linux/linux: (54 commits)
kasan: rust: Add KASAN smoke test via UAF
kbuild: rust: Enable KASAN support
rust: kasan: Rust does not support KHWASAN
kbuild: rust: Define probing macros for rustc
kasan: simplify and clarify Makefile
rust: cfi: add support for CFI_CLANG with Rust
cfi: add CONFIG_CFI_ICALL_NORMALIZE_INTEGERS
rust: support for shadow call stack sanitizer
docs: rust: include other expressions in conditional compilation section
kbuild: rust: replace proc macros dependency on `core.o` with the version text
kbuild: rust: rebuild if the version text changes
kbuild: rust: re-run Kconfig if the version text changes
kbuild: rust: add `CONFIG_RUSTC_VERSION`
rust: avoid `box_uninit_write` feature
MAINTAINERS: add Trevor Gross as Rust reviewer
rust: rbtree: add `RBTree::entry`
rust: rbtree: add cursor
rust: rbtree: add mutable iterator
rust: rbtree: add iterator
rust: rbtree: add red-black tree implementation backed by the C version
...
63 files changed, 3870 insertions, 394 deletions
diff --git a/Documentation/rust/general-information.rst b/Documentation/rust/general-information.rst index e3f388ef4ee4..6146b49b6a98 100644 --- a/Documentation/rust/general-information.rst +++ b/Documentation/rust/general-information.rst @@ -15,6 +15,8 @@ but not `std <https://doc.rust-lang.org/std/>`_. Crates for use in the kernel must opt into this behavior using the ``#![no_std]`` attribute. +.. _rust_code_documentation: + Code documentation ------------------ @@ -22,10 +24,17 @@ Rust kernel code is documented using ``rustdoc``, its built-in documentation generator. The generated HTML docs include integrated search, linked items (e.g. types, -functions, constants), source code, etc. They may be read at (TODO: link when -in mainline and generated alongside the rest of the documentation): +functions, constants), source code, etc. They may be read at: + + https://rust.docs.kernel.org + +For linux-next, please see: + + https://rust.docs.kernel.org/next/ - http://kernel.org/ +There are also tags for each main release, e.g.: + + https://rust.docs.kernel.org/6.10/ The docs can also be easily generated and read locally. This is quite fast (same order as compiling the code itself) and no special tools or environment @@ -75,7 +84,7 @@ should provide as-safe-as-possible abstractions as needed. .. code-block:: rust/bindings/ - (rust/helpers.c) + (rust/helpers/) include/ -----+ <-+ | | @@ -112,7 +121,7 @@ output files in the ``rust/bindings/`` directory. For parts of the C header that ``bindgen`` does not auto generate, e.g. C ``inline`` functions or non-trivial macros, it is acceptable to add a small -wrapper function to ``rust/helpers.c`` to make it available for the Rust side as +wrapper function to ``rust/helpers/`` to make it available for the Rust side as well. Abstractions @@ -142,3 +151,11 @@ configuration: #[cfg(CONFIG_X="y")] // Enabled as a built-in (`y`) #[cfg(CONFIG_X="m")] // Enabled as a module (`m`) #[cfg(not(CONFIG_X))] // Disabled + +For other predicates that Rust's ``cfg`` does not support, e.g. expressions with +numerical comparisons, one may define a new Kconfig symbol: + +.. code-block:: kconfig + + config RUSTC_VERSION_MIN_107900 + def_bool y if RUSTC_VERSION >= 107900 diff --git a/Documentation/rust/index.rst b/Documentation/rust/index.rst index 46d35bd395cf..55dcde9e9e7e 100644 --- a/Documentation/rust/index.rst +++ b/Documentation/rust/index.rst @@ -25,13 +25,27 @@ support is still in development/experimental, especially for certain kernel configurations. +Code documentation +------------------ + +Given a kernel configuration, the kernel may generate Rust code documentation, +i.e. HTML rendered by the ``rustdoc`` tool. + .. only:: rustdoc and html - You can also browse `rustdoc documentation <rustdoc/kernel/index.html>`_. + This kernel documentation was built with `Rust code documentation + <rustdoc/kernel/index.html>`_. .. only:: not rustdoc and html - This documentation does not include rustdoc generated information. + This kernel documentation was not built with Rust code documentation. + +A pregenerated version is provided at: + + https://rust.docs.kernel.org + +Please see the :ref:`Code documentation <rust_code_documentation>` section for +more details. .. toctree:: :maxdepth: 1 diff --git a/Documentation/rust/quick-start.rst b/Documentation/rust/quick-start.rst index 8e3ad9678719..2d107982c87b 100644 --- a/Documentation/rust/quick-start.rst +++ b/Documentation/rust/quick-start.rst @@ -39,8 +39,8 @@ of the box, e.g.:: Debian ****** -Debian Unstable (Sid), outside of the freeze period, provides recent Rust -releases and thus it should generally work out of the box, e.g.:: +Debian Testing and Debian Unstable (Sid), outside of the freeze period, provide +recent Rust releases and thus they should generally work out of the box, e.g.:: apt install rustc rust-src bindgen rustfmt rust-clippy diff --git a/MAINTAINERS b/MAINTAINERS index 7bfef98226d9..098d13ab6b34 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -20144,6 +20144,7 @@ R: Björn Roy Baron <bjorn3_gh@protonmail.com> R: Benno Lossin <benno.lossin@proton.me> R: Andreas Hindborg <a.hindborg@kernel.org> R: Alice Ryhl <aliceryhl@google.com> +R: Trevor Gross <tmgross@umich.edu> L: rust-for-linux@vger.kernel.org S: Supported W: https://rust-for-linux.com @@ -645,9 +645,11 @@ endif # The expansion should be delayed until arch/$(SRCARCH)/Makefile is included. # Some architectures define CROSS_COMPILE in arch/$(SRCARCH)/Makefile. -# CC_VERSION_TEXT is referenced from Kconfig (so it needs export), -# and from include/config/auto.conf.cmd to detect the compiler upgrade. +# CC_VERSION_TEXT and RUSTC_VERSION_TEXT are referenced from Kconfig (so they +# need export), and from include/config/auto.conf.cmd to detect the compiler +# upgrade. CC_VERSION_TEXT = $(subst $(pound),,$(shell LC_ALL=C $(CC) --version 2>/dev/null | head -n 1)) +RUSTC_VERSION_TEXT = $(subst $(pound),,$(shell $(RUSTC) --version 2>/dev/null)) ifneq ($(findstring clang,$(CC_VERSION_TEXT)),) include $(srctree)/scripts/Makefile.clang @@ -668,7 +670,7 @@ ifdef config-build # KBUILD_DEFCONFIG may point out an alternative default configuration # used for 'make defconfig' include $(srctree)/arch/$(SRCARCH)/Makefile -export KBUILD_DEFCONFIG KBUILD_KCONFIG CC_VERSION_TEXT +export KBUILD_DEFCONFIG KBUILD_KCONFIG CC_VERSION_TEXT RUSTC_VERSION_TEXT config: outputmakefile scripts_basic FORCE $(Q)$(MAKE) $(build)=scripts/kconfig $@ @@ -924,6 +926,7 @@ ifdef CONFIG_SHADOW_CALL_STACK ifndef CONFIG_DYNAMIC_SCS CC_FLAGS_SCS := -fsanitize=shadow-call-stack KBUILD_CFLAGS += $(CC_FLAGS_SCS) +KBUILD_RUSTFLAGS += -Zsanitizer=shadow-call-stack endif export CC_FLAGS_SCS endif @@ -948,6 +951,16 @@ endif ifdef CONFIG_CFI_CLANG CC_FLAGS_CFI := -fsanitize=kcfi +ifdef CONFIG_CFI_ICALL_NORMALIZE_INTEGERS + CC_FLAGS_CFI += -fsanitize-cfi-icall-experimental-normalize-integers +endif +ifdef CONFIG_RUST + # Always pass -Zsanitizer-cfi-normalize-integers as CONFIG_RUST selects + # CONFIG_CFI_ICALL_NORMALIZE_INTEGERS. + RUSTC_FLAGS_CFI := -Zsanitizer=kcfi -Zsanitizer-cfi-normalize-integers + KBUILD_RUSTFLAGS += $(RUSTC_FLAGS_CFI) + export RUSTC_FLAGS_CFI +endif KBUILD_CFLAGS += $(CC_FLAGS_CFI) export CC_FLAGS_CFI endif diff --git a/arch/Kconfig b/arch/Kconfig index 405c85ab86f2..98157b38f5cf 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -835,6 +835,22 @@ config CFI_CLANG https://clang.llvm.org/docs/ControlFlowIntegrity.html +config CFI_ICALL_NORMALIZE_INTEGERS + bool "Normalize CFI tags for integers" + depends on CFI_CLANG + depends on $(cc-option,-fsanitize=kcfi -fsanitize-cfi-icall-experimental-normalize-integers) + help + This option normalizes the CFI tags for integer types so that all + integer types of the same size and signedness receive the same CFI + tag. + + The option is separate from CONFIG_RUST because it affects the ABI. + When working with build systems that care about the ABI, it is + convenient to be able to turn on this flag first, before Rust is + turned on. + + This option is necessary for using CFI with Rust. If unsure, say N. + config CFI_PERMISSIVE bool "Use CFI in permissive mode" depends on CFI_CLANG diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 49f054dcd4de..3e29b44d2d7b 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -235,7 +235,7 @@ config ARM64 select HAVE_FUNCTION_ARG_ACCESS_API select MMU_GATHER_RCU_TABLE_FREE select HAVE_RSEQ - select HAVE_RUST if CPU_LITTLE_ENDIAN + select HAVE_RUST if RUSTC_SUPPORTS_ARM64 select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES @@ -270,6 +270,18 @@ config ARM64 help ARM 64-bit (AArch64) Linux support. +config RUSTC_SUPPORTS_ARM64 + def_bool y + depends on CPU_LITTLE_ENDIAN + # Shadow call stack is only supported on certain rustc versions. + # + # When using the UNWIND_PATCH_PAC_INTO_SCS option, rustc version 1.80+ is + # required due to use of the -Zfixed-x18 flag. + # + # Otherwise, rustc version 1.82+ is required due to use of the + # -Zsanitizer=shadow-call-stack flag. + depends on !SHADOW_CALL_STACK || RUSTC_VERSION >= 108200 || RUSTC_VERSION >= 108000 && UNWIND_PATCH_PAC_INTO_SCS + config CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS def_bool CC_IS_CLANG # https://github.com/ClangBuiltLinux/linux/issues/1507 diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index f6bc3da1ef11..b058c4803efb 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -57,9 +57,11 @@ KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +KBUILD_RUSTFLAGS += -Cforce-unwind-tables=n else KBUILD_CFLAGS += -fasynchronous-unwind-tables KBUILD_AFLAGS += -fasynchronous-unwind-tables +KBUILD_RUSTFLAGS += -Cforce-unwind-tables=y -Zuse-sync-unwind=n endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) @@ -114,6 +116,7 @@ endif ifeq ($(CONFIG_SHADOW_CALL_STACK), y) KBUILD_CFLAGS += -ffixed-x18 +KBUILD_RUSTFLAGS += -Zfixed-x18 endif ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index b6d515db869b..22dc5ea4196c 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -177,7 +177,7 @@ config RISCV select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RETHOOK if !XIP_KERNEL select HAVE_RSEQ - select HAVE_RUST if 64BIT + select HAVE_RUST if RUSTC_SUPPORTS_RISCV select HAVE_SAMPLE_FTRACE_DIRECT select HAVE_SAMPLE_FTRACE_DIRECT_MULTI select HAVE_STACKPROTECTOR @@ -209,6 +209,13 @@ config RISCV select USER_STACKTRACE_SUPPORT select ZONE_DMA32 if 64BIT +config RUSTC_SUPPORTS_RISCV + def_bool y + depends on 64BIT + # Shadow call stack requires rustc version 1.82+ due to use of the + # -Zsanitizer=shadow-call-stack flag. + depends on !SHADOW_CALL_STACK || RUSTC_VERSION >= 108200 + config CLANG_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_CLANG # https://github.com/ClangBuiltLinux/linux/issues/1817 diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 801fd85c3ef6..cd75e78a06c1 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -24,11 +24,15 @@ RETPOLINE_CFLAGS += $(call cc-option,-mindirect-branch-cs-prefix) ifdef CONFIG_MITIGATION_RETHUNK RETHUNK_CFLAGS := -mfunction-return=thunk-extern +RETHUNK_RUSTFLAGS := -Zfunction-return=thunk-extern RETPOLINE_CFLAGS += $(RETHUNK_CFLAGS) +RETPOLINE_RUSTFLAGS += $(RETHUNK_RUSTFLAGS) endif export RETHUNK_CFLAGS +export RETHUNK_RUSTFLAGS export RETPOLINE_CFLAGS +export RETPOLINE_RUSTFLAGS export RETPOLINE_VDSO_CFLAGS # For gcc stack alignment is specified with -mpreferred-stack-boundary, @@ -218,9 +222,10 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables # Avoid indirect branches in kernel to deal with Spectre ifdef CONFIG_MITIGATION_RETPOLINE KBUILD_CFLAGS += $(RETPOLINE_CFLAGS) + KBUILD_RUSTFLAGS += $(RETPOLINE_RUSTFLAGS) # Additionally, avoid generating expensive indirect jumps which # are subject to retpolines for small number of switch cases. - # clang turns off jump table generation by default when under + # LLVM turns off jump table generation by default when under # retpoline builds, however, gcc does not for x86. This has # only been fixed starting from gcc stable version 8.4.0 and # onwards, but not for older ones. See gcc bug #86952. @@ -237,6 +242,10 @@ ifdef CONFIG_CALL_PADDING PADDING_CFLAGS := -fpatchable-function-entry=$(CONFIG_FUNCTION_PADDING_BYTES),$(CONFIG_FUNCTION_PADDING_BYTES) KBUILD_CFLAGS += $(PADDING_CFLAGS) export PADDING_CFLAGS + +PADDING_RUSTFLAGS := -Zpatchable-function-entry=$(CONFIG_FUNCTION_PADDING_BYTES),$(CONFIG_FUNCTION_PADDING_BYTES) +KBUILD_RUSTFLAGS += $(PADDING_RUSTFLAGS) +export PADDING_RUSTFLAGS endif KBUILD_LDFLAGS += -m elf_$(UTS_MACHINE) diff --git a/init/Kconfig b/init/Kconfig index b05467014041..fbd0cb06a50a 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -60,6 +60,13 @@ config LLD_VERSION default $(ld-version) if LD_IS_LLD default 0 +config RUSTC_VERSION + int + default $(shell,$(srctree)/scripts/rustc-version.sh $(RUSTC)) + help + It does not depend on `RUST` since that one may need to use the version + in a `depends on`. + config RUST_IS_AVAILABLE def_bool $(success,$(srctree)/scripts/rust_is_available.sh) help @@ -1935,12 +1942,14 @@ config RUST bool "Rust support" depends on HAVE_RUST depends on RUST_IS_AVAILABLE - depends on !CFI_CLANG depends on !MODVERSIONS - depends on !GCC_PLUGINS + depends on !GCC_PLUGIN_RANDSTRUCT depends on !RANDSTRUCT - depends on !SHADOW_CALL_STACK depends on !DEBUG_INFO_BTF || PAHOLE_HAS_LANG_EXCLUDE + depends on !CFI_CLANG || RUSTC_VERSION >= 107900 && $(cc-option,-fsanitize=kcfi -fsanitize-cfi-icall-experimental-normalize-integers) + select CFI_ICALL_NORMALIZE_INTEGERS if CFI_CLANG + depends on !CALL_PADDING || RUSTC_VERSION >= 108000 + depends on !KASAN_SW_TAGS help Enables Rust support in the kernel. @@ -1957,7 +1966,9 @@ config RUST config RUSTC_VERSION_TEXT string depends on RUST - default "$(shell,$(RUSTC) --version 2>/dev/null)" + default "$(RUSTC_VERSION_TEXT)" + help + See `CC_VERSION_TEXT`. config BINDGEN_VERSION_TEXT string diff --git a/mm/kasan/Makefile b/mm/kasan/Makefile index 7634dd2a6128..b88543e5c0cc 100644 --- a/mm/kasan/Makefile +++ b/mm/kasan/Makefile @@ -44,7 +44,8 @@ ifndef CONFIG_CC_HAS_KASAN_MEMINTRINSIC_PREFIX CFLAGS_KASAN_TEST += -fno-builtin endif -CFLAGS_kasan_test.o := $(CFLAGS_KASAN_TEST) +CFLAGS_kasan_test_c.o := $(CFLAGS_KASAN_TEST) +RUSTFLAGS_kasan_test_rust.o := $(RUSTFLAGS_KASAN) CFLAGS_kasan_test_module.o := $(CFLAGS_KASAN_TEST) obj-y := common.o report.o @@ -52,5 +53,10 @@ obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o report_generic.o shadow.o quaran obj-$(CONFIG_KASAN_HW_TAGS) += hw_tags.o report_hw_tags.o tags.o report_tags.o obj-$(CONFIG_KASAN_SW_TAGS) += init.o report_sw_tags.o shadow.o sw_tags.o tags.o report_tags.o +kasan_test-objs := kasan_test_c.o +ifdef CONFIG_RUST + kasan_test-objs += kasan_test_rust.o +endif + obj-$(CONFIG_KASAN_KUNIT_TEST) += kasan_test.o obj-$(CONFIG_KASAN_MODULE_TEST) += kasan_test_module.o diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index fb2b9ac0659a..f438a6cdc964 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -555,6 +555,12 @@ static inline bool kasan_arch_is_ready(void) { return true; } void kasan_kunit_test_suite_start(void); void kasan_kunit_test_suite_end(void); +#ifdef CONFIG_RUST +char kasan_test_rust_uaf(void); +#else +static inline char kasan_test_rust_uaf(void) { return '\0'; } +#endif + #else /* CONFIG_KASAN_KUNIT_TEST */ static inline void kasan_kunit_test_suite_start(void) { } diff --git a/mm/kasan/kasan_test.c b/mm/kasan/kasan_test_c.c index 567d33b493e2..a181e4780d9d 100644 --- a/mm/kasan/kasan_test.c +++ b/mm/kasan/kasan_test_c.c @@ -1944,6 +1944,16 @@ static void match_all_mem_tag(struct kunit *test) kfree(ptr); } +/* + * Check that Rust performing a use-after-free using `unsafe` is detected. + * This is a smoke test to make sure that Rust is being sanitized properly. + */ +static void rust_uaf(struct kunit *test) +{ + KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_RUST); + KUNIT_EXPECT_KASAN_FAIL(test, kasan_test_rust_uaf()); +} + static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(kmalloc_oob_right), KUNIT_CASE(kmalloc_oob_left), @@ -2017,6 +2027,7 @@ static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(match_all_not_assigned), KUNIT_CASE(match_all_ptr_tag), KUNIT_CASE(match_all_mem_tag), |