diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2020-06-01 15:45:27 -0700 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2020-06-01 15:45:27 -0700 |
| commit | b23c4771ff62de8ca9b5e4a2d64491b2fb6f8f69 (patch) | |
| tree | 3ff6b2bdfec161fbc383bba06bab6329e81b02f7 /Documentation/locking | |
| parent | c2b0fc847f3122e5a4176c3772626a7a8facced0 (diff) | |
| parent | e35b5a4c494a75a683ddf4901a43e0a128d5bfe3 (diff) | |
| download | linux-b23c4771ff62de8ca9b5e4a2d64491b2fb6f8f69.tar.gz linux-b23c4771ff62de8ca9b5e4a2d64491b2fb6f8f69.tar.bz2 linux-b23c4771ff62de8ca9b5e4a2d64491b2fb6f8f69.zip | |
Merge tag 'docs-5.8' of git://git.lwn.net/linux
Pull documentation updates from Jonathan Corbet:
"A fair amount of stuff this time around, dominated by yet another
massive set from Mauro toward the completion of the RST conversion. I
*really* hope we are getting close to the end of this. Meanwhile,
those patches reach pretty far afield to update document references
around the tree; there should be no actual code changes there. There
will be, alas, more of the usual trivial merge conflicts.
Beyond that we have more translations, improvements to the sphinx
scripting, a number of additions to the sysctl documentation, and lots
of fixes"
* tag 'docs-5.8' of git://git.lwn.net/linux: (130 commits)
Documentation: fixes to the maintainer-entry-profile template
zswap: docs/vm: Fix typo accept_threshold_percent in zswap.rst
tracing: Fix events.rst section numbering
docs: acpi: fix old http link and improve document format
docs: filesystems: add info about efivars content
Documentation: LSM: Correct the basic LSM description
mailmap: change email for Ricardo Ribalda
docs: sysctl/kernel: document unaligned controls
Documentation: admin-guide: update bug-hunting.rst
docs: sysctl/kernel: document ngroups_max
nvdimm: fixes to maintainter-entry-profile
Documentation/features: Correct RISC-V kprobes support entry
Documentation/features: Refresh the arch support status files
Revert "docs: sysctl/kernel: document ngroups_max"
docs: move locking-specific documents to locking/
docs: move digsig docs to the security book
docs: move the kref doc into the core-api book
docs: add IRQ documentation at the core-api book
docs: debugging-via-ohci1394.txt: add it to the core-api book
docs: fix references for ipmi.rst file
...
Diffstat (limited to 'Documentation/locking')
| -rw-r--r-- | Documentation/locking/futex-requeue-pi.rst | 132 | ||||
| -rw-r--r-- | Documentation/locking/hwspinlock.rst | 485 | ||||
| -rw-r--r-- | Documentation/locking/index.rst | 7 | ||||
| -rw-r--r-- | Documentation/locking/locktorture.rst | 2 | ||||
| -rw-r--r-- | Documentation/locking/percpu-rw-semaphore.rst | 28 | ||||
| -rw-r--r-- | Documentation/locking/pi-futex.rst | 122 | ||||
| -rw-r--r-- | Documentation/locking/preempt-locking.rst | 144 | ||||
| -rw-r--r-- | Documentation/locking/robust-futex-ABI.rst | 184 | ||||
| -rw-r--r-- | Documentation/locking/robust-futexes.rst | 221 | ||||
| -rw-r--r-- | Documentation/locking/rt-mutex.rst | 2 |
10 files changed, 1325 insertions, 2 deletions
diff --git a/Documentation/locking/futex-requeue-pi.rst b/Documentation/locking/futex-requeue-pi.rst new file mode 100644 index 000000000000..14ab5787b9a7 --- /dev/null +++ b/Documentation/locking/futex-requeue-pi.rst @@ -0,0 +1,132 @@ +================ +Futex Requeue PI +================ + +Requeueing of tasks from a non-PI futex to a PI futex requires +special handling in order to ensure the underlying rt_mutex is never +left without an owner if it has waiters; doing so would break the PI +boosting logic [see rt-mutex-desgin.txt] For the purposes of +brevity, this action will be referred to as "requeue_pi" throughout +this document. Priority inheritance is abbreviated throughout as +"PI". + +Motivation +---------- + +Without requeue_pi, the glibc implementation of +pthread_cond_broadcast() must resort to waking all the tasks waiting +on a pthread_condvar and letting them try to sort out which task +gets to run first in classic thundering-herd formation. An ideal +implementation would wake the highest-priority waiter, and leave the +rest to the natural wakeup inherent in unlocking the mutex +associated with the condvar. + +Consider the simplified glibc calls:: + + /* caller must lock mutex */ + pthread_cond_wait(cond, mutex) + { + lock(cond->__data.__lock); + unlock(mutex); + do { + unlock(cond->__data.__lock); + futex_wait(cond->__data.__futex); + lock(cond->__data.__lock); + } while(...) + unlock(cond->__data.__lock); + lock(mutex); + } + + pthread_cond_broadcast(cond) + { + lock(cond->__data.__lock); + unlock(cond->__data.__lock); + futex_requeue(cond->data.__futex, cond->mutex); + } + +Once pthread_cond_broadcast() requeues the tasks, the cond->mutex +has waiters. Note that pthread_cond_wait() attempts to lock the +mutex only after it has returned to user space. This will leave the +underlying rt_mutex with waiters, and no owner, breaking the +previously mentioned PI-boosting algorithms. + +In order to support PI-aware pthread_condvar's, the kernel needs to +be able to requeue tasks to PI futexes. This support implies that +upon a successful futex_wait system call, the caller would return to +user space already holding the PI futex. The glibc implementation +would be modified as follows:: + + + /* caller must lock mutex */ + pthread_cond_wait_pi(cond, mutex) + { + lock(cond->__data.__lock); + unlock(mutex); + do { + unlock(cond->__data.__lock); + futex_wait_requeue_pi(cond->__data.__futex); + lock(cond->__data.__lock); + } while(...) + unlock(cond->__data.__lock); + /* the kernel acquired the mutex for us */ + } + + pthread_cond_broadcast_pi(cond) + { + lock(cond->__data.__lock); + unlock(cond->__data.__lock); + futex_requeue_pi(cond->data.__futex, cond->mutex); + } + +The actual glibc implementation will likely test for PI and make the +necessary changes inside the existing calls rather than creating new +calls for the PI cases. Similar changes are needed for +pthread_cond_timedwait() and pthread_cond_signal(). + +Implementation +-------------- + +In order to ensure the rt_mutex has an owner if it has waiters, it +is necessary for both the requeue code, as well as the waiting code, +to be able to acquire the rt_mutex before returning to user space. +The requeue code cannot simply wake the waiter and leave it to +acquire the rt_mutex as it would open a race window between the +requeue call returning to user space and the waiter waking and +starting to run. This is especially true in the uncontended case. + +The solution involves two new rt_mutex helper routines, +rt_mutex_start_proxy_lock() and rt_mutex_finish_proxy_lock(), which +allow the requeue code to acquire an uncontended rt_mutex on behalf +of the waiter and to enqueue the waiter on a contended rt_mutex. +Two new system calls provide the kernel<->user interface to +requeue_pi: FUTEX_WAIT_REQUEUE_PI and FUTEX_CMP_REQUEUE_PI. + +FUTEX_WAIT_REQUEUE_PI is called by the waiter (pthread_cond_wait() +and pthread_cond_timedwait()) to block on the initial futex and wait +to be requeued to a PI-aware futex. The implementation is the +result of a high-speed collision between futex_wait() and +futex_lock_pi(), with some extra logic to check for the additional +wake-up scenarios. + +FUTEX_CMP_REQUEUE_PI is called by the waker +(pthread_cond_broadcast() and pthread_cond_signal()) to requeue and +possibly wake the waiting tasks. Internally, this system call is +still handled by futex_requeue (by passing requeue_pi=1). Before +requeueing, futex_requeue() attempts to acquire the requeue target +PI futex on behalf of the top waiter. If it can, this waiter is +woken. futex_requeue() then proceeds to requeue the remaining +nr_wake+nr_requeue tasks to the PI futex, calling +rt_mutex_start_proxy_lock() prior to each requeue to prepare the +task as a waiter on the underlying rt_mutex. It is possible that +the lock can be acquired at this stage as well, if so, the next +waiter is woken to finish the acquisition of the lock. + +FUTEX_CMP_REQUEUE_PI accepts nr_wake and nr_requeue as arguments, but +their sum is all that really matters. futex_requeue() will wake or +requeue up to nr_wake + nr_requeue tasks. It will wake only as many +tasks as it can acquire the lock for, which in the majority of cases +should be 0 as good programming practice dictates that the caller of +either pthread_cond_broadcast() or pthread_cond_signal() acquire the +mutex prior to making the call. FUTEX_CMP_REQUEUE_PI requires that +nr_wake=1. nr_requeue should be INT_MAX for broadcast and 0 for +signal. diff --git a/Documentation/locking/hwspinlock.rst b/Documentation/locking/hwspinlock.rst new file mode 100644 index 000000000000..6f03713b7003 --- /dev/null +++ b/Documentation/locking/hwspinlock.rst @@ -0,0 +1,485 @@ +=========================== +Hardware Spinlock Framework +=========================== + +Introduction +============ + +Hardware spinlock modules provide hardware assistance for synchronization +and mutual exclusion between heterogeneous processors and those not operating +under a single, shared operating system. + +For example, OMAP4 has dual Cortex-A9, dual Cortex-M3 and a C64x+ DSP, +each of which is running a different Operating System (the master, A9, +is usually running Linux and the slave processors, the M3 and the DSP, +are running some flavor of RTOS). + +A generic hwspinlock framework allows platform-independent drivers to use +the hwspinlock device in order to access data structures that are shared +between remote processors, that otherwise have no alternative mechanism +to accomplish synchronization and mutual exclusion operations. + +This is necessary, for example, for Inter-processor communications: +on OMAP4, cpu-intensive multimedia tasks are offloaded by the host to the +remote M3 and/or C64x+ slave processors (by an IPC subsystem called Syslink). + +To achieve fast message-based communications, a minimal kernel support +is needed to deliver messages arriving from a remote processor to the +appropriate user process. + +This communication is based on simple data structures that is shared between +the remote processors, and access to it is synchronized using the hwspinlock +module (remote processor directly places new messages in this shared data +structure). + +A common hwspinlock interface makes it possible to have generic, platform- +independent, drivers. + +User API +======== + +:: + + struct hwspinlock *hwspin_lock_request(void); + +Dynamically assign an hwspinlock and return its address, or NULL +in case an unused hwspinlock isn't available. Users of this +API will usually want to communicate the lock's id to the remote core +before it can be used to achieve synchronization. + +Should be called from a process context (might sleep). + +:: + + struct hwspinlock *hwspin_lock_request_specific(unsigned int id); + +Assign a specific hwspinlock id and return its address, or NULL +if that hwspinlock is already in use. Usually board code will +be calling this function in order to reserve specific hwspinlock +ids for predefined purposes. + +Should be called from a process context (might sleep). + +:: + + int of_hwspin_lock_get_id(struct device_node *np, int index); + +Retrieve the global lock id for an OF phandle-based specific lock. +This function provides a means for DT users of a hwspinlock module +to get the global lock id of a specific hwspinlock, so that it can +be requested using the normal hwspin_lock_request_specific() API. + +The function returns a lock id number on success, -EPROBE_DEFER if +the hwspinlock device is not yet registered with the core, or other +error values. + +Should be called from a process context (might sleep). + +:: + + int hwspin_lock_free(struct hwspinlock *hwlock); + +Free a previously-assigned hwspinlock; returns 0 on success, or an +appropriate error code on failure (e.g. -EINVAL if the hwspinlock +is already free). + +Should be called from a process context (might sleep). + +:: + + int hwspin_lock_timeout(struct hwspinlock *hwlock, unsigned int timeout); + +Lock a previously-assigned hwspinlock with a timeout limit (specified in +msecs). If the hwspinlock is already taken, the function will busy loop +waiting for it to be released, but give up when the timeout elapses. +Upon a successful return from this function, preemption is disabled so +the caller must not sleep, and is advised to release the hwspinlock as +soon as possible, in order to minimize remote cores polling on the +hardware interconnect. + +Returns 0 when successful and an appropriate error code otherwise (most +notably -ETIMEDOUT if the hwspinlock is still busy after timeout msecs). +The function will never sleep. + +:: + + int hwspin_lock_timeout_irq(struct hwspinlock *hwlock, unsigned int timeout); + +Lock a previously-assigned hwspinlock with a timeout limit (specified in +msecs). If the hwspinlock is already taken, the function will busy loop +waiting for it to be released, but give up when the timeout elapses. +Upon a successful return from this function, preemption and the local +interrupts are disabled, so the caller must not sleep, and is advised to +release the hwspinlock as soon as possible. + +Returns 0 when successful and an appropriate error code otherwise (most +notably -ETIMEDOUT if the hwspinlock is still busy after timeout msecs). +The function will never sleep. + +:: + + int hwspin_lock_timeout_irqsave(struct hwspinlock *hwlock, unsigned int to, + unsigned long *flags); + +Lock a previously-assigned hwspinlock with a timeout limit (specified in +msecs). If the hwspinlock is already taken, the function will busy loop +waiting for it to be released, but give up when the timeout elapses. +Upon a successful return from this function, preemption is disabled, +local interrupts are disabled and their previous state is saved at the +given flags placeholder. The caller must not sleep, and is advised to +release the hwspinlock as soon as possible. + +Returns 0 when successful and an appropriate error code otherwise (most +notably -ETIMEDOUT if the hwspinlock is still busy after timeout msecs). + +The function will never sleep. + +:: + + int hwspin_lock_timeout_raw(struct hwspinlock *hwlock, unsigned int timeout); + +Lock a previously-assigned hwspinlock with a timeout limit (specified in +msecs). If the hwspinlock is already taken, the function will busy loop +waiting for it to be released, but give up when the timeout elapses. + +Caution: User must protect the routine of getting hardware lock with mutex +or spinlock to avoid dead-lock, that will let user can do some time-consuming +or sleepable operations under the hardware lock. + +Returns 0 when successful and an appropriate error code otherwise (most +notably -ETIMEDOUT if the hwspinlock is still busy after timeout msecs). + +The function will never sleep. + +:: + + int hwspin_lock_timeout_in_atomic(struct hwspinlock *hwlock, unsigned int to); + +Lock a previously-assigned hwspinlock with a timeout limit (specified in +msecs). If the hwspinlock is already taken, the function will busy loop +waiting for it to be released, but give up when the timeout elapses. + +This function shall be called only from an atomic context and the timeout +value shall not exceed a few msecs. + +Returns 0 when successful and an appropriate error code otherwise (most +notably -ETIMEDOUT if the hwspinlock is still busy after timeout msecs). + +The function will never sleep. + +:: + + int hwspin_trylock(struct hwspinlock *hwlock); + + +Attempt to lock a previously-assigned hwspinlock, but immediately fail if +it is already taken. + +Upon a successful return from this function, preemption is disabled so +caller must not sleep, and is advised to release the hwspinlock as soon as +possible, in order to minimize remote cores polling on the hardware +interconnect. + +Returns 0 on success and an appropriate error code otherwise (most +notably -EBUSY if the hwspinlock was already taken). +The function will never sleep. + +:: + + int hwspin_trylock_irq(struct hwspinlock *hwlock); + + +Attempt to lock a previously-assigned hwspinlock, but immediately fail if +it is already taken. + +Upon a successful return from this function, preemption and the local +interrupts are disabled so caller must not sleep, and is advised to +release the hwspinlock as soon as possible. + +Returns 0 on success and an appropriate error code otherwise (most +notably -EBUSY if the hwspinlock was already taken). + +The function will never sleep. + +:: + + int hwspin_trylock_irqsave(struct hwspinlock *hwlock, unsigned long *flags); + +Attempt to lock a previously-assigned hwspinlock, but immediately fail if +it is already taken. + +Upon a successful return from this function, preemption is disabled, +the local interrupts are disabled and their previous state is saved +at the given flags placeholder. The caller must not sleep, and is advised +to release the hwspinlock as soon as possible. + +Returns 0 on success and an appropriate error code otherwise (most +notably -EBUSY if the hwspinlock was already taken). +The function will never sleep. + +:: + + int hwspin_trylock_raw(struct hwspinlock *hwlock); + +Attempt to lock a previously-assigned hwspinlock, but immediately fail if +it is already taken. + +Caution: User must protect the routine of getting hardware lock with mutex +or spinlock to avoid dead-lock, that will let user can do some time-consuming +or sleepable operations under the hardware lock. + +Returns 0 on success and an appropriate error code otherwise (most +notably -EBUSY if the hwspinlock was already taken). +The function will never sleep. + +:: + + int hwspin_trylock_in_atomic(struct hwspinlock *hwlock); + +Attempt to lock a previously-assigned hwspinlock, but immediately fail if +it is already taken. + +This function shall be called only from an atomic context. + +Returns 0 on success and an appropriate error code otherwise (most +notably -EBUSY if the hwspinlock was already taken). +The function will never sleep. + +:: + + void hwspin_unlock(struct hwspinlock *hwlock); + +Unlock a previously-locked hwspinlock. Always succeed, and can be called +from any context (the function never sleeps). + +.. note:: + + code should **never** unlock an hwspinlock which is already unlocked + (there is no protection against this). + +:: + + void hwspin_unlock_irq(struct hwspinlock *hwlock); + +Unlock a previously-locked hwspinlock and enable local interrupts. +The caller should **never** unlock an hwspinlock which is already unlocked. + +Doing so is considered a bug (there is no protection against this). +Upon a successful return from this function, preemption and local +interrupts are enabled. This function will never sleep. + +:: + + void + hwspin_unlock_irqrestore(struct hwspinlock *hwlock, unsigned long *flags); + +Unlock a previously-locked hwspinlock. + +The caller should **never** unlock an hwspinlock which is already unlocked. +Doing so is considered a bug (there is no protection against this). +Upon a successful return from this function, preemption is reenabled, +and the state of the local interrupts is restored to the state saved at +the given flags. This function will never sleep. + +:: + + void hwspin_unlock_raw(struct hwspinlock *hwlock); + +Unlock a previously-locked hwspinlock. + +The caller should **never** unlock an hwspinlock which is already unlocked. +Doing so is considered a bug (there is no protection against this). +This function will never sleep. + +:: + + void hwspin_unlock_in_atomic(struct hwspinlock *hwlock); + +Unlock a previously-locked hwspinlock. + +The caller should **never** unlock an hwspinlock which is already unlocked. +Doing so is considered a bug (there is no protection against this). +This function will never sleep. + +:: + + int hwspin_lock_get_id(struct hwspinlock *hwlock); + +Retrieve id number of a given hwspinlock. This is needed when an +hwspinlock is dynamically assigned: before it can be used to achieve +mutual exclusion with a remote cpu, the id number should be communicated +to the remote task with which we want to synchronize. + +Returns the hwspinlock id number, or -EINVAL if hwlock is null. + +Typical usage +============= + +:: + + #include <linux/hwspinlock.h> + #include <linux/err.h> + + int hwspinlock_example1(void) + { + struct hwspinlock *hwlock; + int ret; + + /* dynamically assign a hwspinlock */ + hwlock = hwspin_lock_request(); + if (!hwlock) + ... + + id = hwspin_lock_get_id(hwlock); + /* probably need to communicate id to a remote processor now */ + + /* take the lock, spin for 1 sec if it's already taken */ + ret = hwspin_lock_timeout(hwlock, 1000); + if (ret) + ... + + /* + * we took the lock, do our thing now, but do NOT sleep + */ + + /* release the lock */ + hwspin_unlock(hwlock); + + /* free the lock */ + ret = hwspin_lock_free(hwlock); + if (ret) + ... + + return ret; + } + + int hwspinlock_example2(void) + { + struct hwspinlock *hwlock; + int ret; + + /* + * assign a specific hwspinlock id - this should be called early + * by board init code. + */ + hwlock = hwspin_lock_request_specific(PREDEFINED_LOCK_ID); + if (!hwlock) + ... + + /* try to take it, but don't spin on it */ + ret = hwspin_trylock(hwlock); + if (!ret) { + pr_info("lock is already taken\n"); + return -EBUSY; + } + + /* + * we took the lock, do our thing now, but do NOT sleep + */ + + /* release the lock */ + hwspin_unlock(hwlock); + + /* free the lock */ + ret = hwspin_lock_free(hwlock); + if (ret) + ... + + return ret; + } + + +API for implementors +==================== + +:: + + int hwspin_lock_register(struct hwspinlock_device *bank, struct device *dev, + const struct hwspinlock_ops *ops, int base_id, int num_locks); + +To be called from the underlying platform-specific implementation, in +order to register a new hwspinlock device (which is usually a bank of +numerous locks). Should be called from a process context (this function +might sleep). + +Returns 0 on success, or appropriate error code on failure. + +:: + + int hwspin_lock_unregister(struct hwspinlock_device *bank); + +To be called from the underlying vendor-specific implementation, in order +to unregister an hwspinlock device (which is usually a bank of numerous +locks). + +Should be called from a process context (this function might sleep). + +Returns the address of hwspinlock on success, or NULL on error (e.g. +if the hwspinlock is still in use). + +Important structs +================= + +struct hwspinlock_device is a device which usually contains a bank +of hardware locks. It is registered by the underlying hwspinlock +implementation using the hwspin_lock_register() API. + +:: + + /** + * struct hwspinlock_device - a device which usually spans numerous hwspinlocks + * @dev: underlying device, will be used to invoke runtime PM api + * @ops: platform-specific hwspinlock handlers + * @base_id: id index of the first lock in this device + * @num_locks: number of locks in this device + * @lock: dynamically allocated array of 'struct hwspinlock' + */ + struct hwspinlock_device { + struct device *dev; + const struct hwspinlock_ops *ops; + int base_id; + int num_locks; + struct hwspinlock lock[0]; + }; + +struct hwspinlock_device contains an array of hwspinlock structs, each +of which represents a single hardware lock:: + + /** + * struct hwspinlock - this struct represents a single hwspinlock instance + * @bank: the hwspinlock_device structure which owns this lock + * @lock: initialized and used by hwspinlock core + * @priv: private data, owned by the underlying platform-specific hwspinlock drv + */ + struct hwspinlock { + struct hwspinlock_device *bank; + spinlock_t lock; + void *priv; + }; + +When registering a bank of locks, the hwspinlock driver only needs to +set the priv members of the locks. The rest of the members are set and +initialized by the hwspinlock core itself. + +Implementation callbacks +======================== + +There are three possible callbacks defined in 'struct hwspinlock_ops':: + + struct hwspinlock_ops { + int (*trylock)(struct hwspinlock *lock); + void (*unlock)(struct hwspinlock *lock); + void (*relax)(struct hwspinlock *lock); + }; + +The first two callbacks are mandatory: + +The ->trylock() callback should make a single attempt to take the lock, and +return 0 on failure and 1 on success. This callback may **not** sleep. + +The ->unlock() callback releases the lock. It always succeed, and it, too, +may **not** sleep. + +The ->relax() callback is optional. It is called by hwspinlock core while +spinning on a lock, and can be used by the underlying implementation to force +a delay between two successive invocations of ->trylock(). It may **not** sleep. diff --git a/Documentation/locking/index.rst b/Documentation/locking/index.rst index 5d6800a723dc..d785878cad65 100644 --- a/Documentation/locking/index.rst +++ b/Documentation/locking/index.rst @@ -16,6 +16,13 @@ locking rt-mutex spinlocks ww-mutex-design + preempt-locking + pi-futex + futex-requeue-pi + hwspinlock + percpu-rw-semaphore + robust-futexes + robust-futex-ABI .. only:: subproject and html diff --git a/Documentation/locking/locktorture.rst b/Documentation/locking/locktorture.rst index 5bcb99ba7bd9..8012a74555e7 100644 --- a/Documentation/locking/locktorture.rst +++ b/Documentation/locking/locktorture.rst @@ -110,7 +110,7 @@ stutter same period of time. Defaults to "stutter=5", so as to run and pause for (roughly) five-second intervals. Specifying "stutter=0" causes the test to run continuously - without pausing, which is the old default behavior. + without pausing. shuffle_interval The number of seconds to keep the test threads affinitied diff --git a/Documentation/locking/percpu-rw-semaphore.rst b/Documentation/locking/percpu-rw-semaphore.rst new file mode 100644 index 000000000000..247de6410855 --- /dev/null +++ b/Documentation/locking/percpu-rw-semaphore.rst @@ -0,0 +1,28 @@ +==================== +Percpu rw semaphores +==================== + +Percpu rw semaphores is a new read-write semaphore design that is +optimized for locking for reading. + +The problem with traditional read-write semaphores is that when multiple +cores take the lock for reading, the cache line containing the semaphore +is bouncing between L1 caches of the cores, causing performance +degradation. + +Locking for reading is very fast, it uses RCU and it avoids any atomic +instruction in the lock and unlock path. On the other hand, locking for +writing is very expensive, it calls synchronize_rcu() that can take +hundreds of milliseconds. + +The lock is declared with "struct percpu_rw_semaphore" type. +The lock is initialized percpu_init_rwsem, it returns 0 on success and +-ENOMEM on allocation failure. +The lock must be freed with percpu_free_rwsem to avoid memory leak. + +The lock is locked for read with percpu_down_read, percpu_up_read and +for write with percpu_down_write, percpu_up_write. + +The idea of using RCU for optimized rw-lock was introduced by +Eric Dumazet <eric.dumazet@gmail.com>. +The code was written by Mikulas Patocka <mpatocka@redhat.com> diff --git a/Documentation/locking/pi-futex.rst b/Documentation/locking/pi-futex.rst new file mode 100644 index 000000000000..c33ba2befbf8 --- /dev/null +++ b/Documentation/locking/pi-futex.rst @@ -0,0 +1,122 @@ +====================== +Lightweight PI-futexes +====================== + +We are calling them lightweight for 3 reasons: + + - in the user-space fastpath a PI-enabled futex involves no kernel work + (or any other PI complexity) at all. No registration, no extra kernel + calls - just pure fast atomic ops in userspace. + + - even in the slowpath, the system call and scheduling pattern is very + similar to normal futexes. + + - the in-kernel PI implementation is streamlined around the mutex + abstraction, with strict rules that keep the implementation + relatively simple: only a single owner may own a lock (i.e. no + read-write lock support), only the owner may unlock a lock, no + recursive locking, etc. + +Priority Inheritance - why? +--------------------------- + +The short reply: user-space PI helps achieving/improving determinism for +user-space applications. In the best-case, it can help achieve +determinism and well-bound latencies. Even in the worst-case, PI will +improve the statistical distribution of locking related application +delays. + +The longer reply +---------------- + +Firstly, sharing locks between multiple tasks is a common programming +technique that often cannot be replaced with lockless algorithms. As we +can see it in the kernel [which is a quite complex program in itself], +lockless structures are rather the exception than the norm - the current +ratio of lockless vs. locky code for shared data structures is somewhere +between 1:10 and 1:100. Lockless is hard, and the complexity of lockless +algorithms often endangers to ability to do robust reviews of said code. +I.e. critical RT apps often choose lock structures to protect critical +data structures, instead of lockless algorithms. Furthermore, there are +cases (like shared hardware, or other resource limits) where lockless +access is mathematically impossible. + +Media players (such as Jack) are an example of reasonable application +design with multiple tasks (with multiple priority levels) sharing +short-held locks: for example, a highprio audio playback thread is +combined with medium-prio construct-audio-data threads and low-prio +display-colory-stuff threads. Add video and decoding to the mix and +we've got even more priority levels. + +So once we accept that synchronization objects (locks) are an +unavoidable fact of life, and once we accept that multi-task userspace +apps have a very fair expectation of being able to use locks, we've got +to think about how to offer the option of a deterministic locking +implementation to user-space. + +Most of the technical counter-arguments against doing priority +inheritance only apply to kernel-space locks. But user-space locks are +different, there we cannot disable interrupts or make the task +non-preemptible in a critical section, so the 'use spinlocks' argument +does not apply (user-space spinlocks have the same priority inversion +problems as other user-space locking constructs). Fact is, pretty much +the only technique that currently enables good determinism for userspace +locks (such as futex-based pthread mutexes) is priority inheritance: + +Currently (without PI), if a high-prio and a low-prio task shares a lock +[this is a quite common scenario for most non-trivial RT applications], +even if all critical sections are coded carefully to be deterministic +(i.e. all critical sections are short in duration and only execute a +limited number of instructions), the kernel cannot guarantee any +deterministic execution of the high-prio task: any medium-priority task +could preempt the low-prio task while it holds the shared lock and +executes the critical section, and could delay it indefinitely. + +Implementation +-------------- + +As mentioned before, the userspace fastpath of PI-enabled pthread +mutexes involves no kernel work at all - they behave quite similarly to +normal futex-based locks: a 0 value means unlocked, and a value==TID +means locked. (This is the same method as used by list-based robust +futexes.) Userspace uses atomic ops to lock/unlock these mutexes without +entering the kernel. + +To handle the slowpath, we have added two new futex ops: + + - FUTEX_LOCK_PI + - FUTEX_UNLOCK_PI + +If the lock-acquire fastpath fails, [i.e. an atomic transition from 0 to +TID fails], then FUTEX_LOCK_PI is called. The kernel does all the +remaining work: if there is no futex-queue attached to the futex address +yet then the code looks up the task that owns the futex [it has put its +own TID into the futex value], and attaches a 'PI state' structure to +the futex-queue. The pi_state includes an rt-mutex, which is a PI-aware, +kernel-based synchronization object. The 'other' task is made the owner +of the rt-mutex, and the FUTEX_WAITERS bit is atomically set in the +futex value. Then this task tries to lock the rt-mutex, on which it +blocks. Once it returns, it has the mutex acquired, and it sets the +futex value to its own TID and returns. Userspace has no other work to +perform - it now owns the lock, and futex value contains +FUTEX_WAITERS|TID. + +If the unlock side fastpath succeeds, [i.e. userspace manages to do a +TID -> 0 atomic transition of the futex value], then no kernel work is +triggered. + +If the unlock fastpath fails (because the FUTEX_WAITERS bit is set), +then FUTEX_UNLOCK_PI is called, and the kernel unlocks the futex on the +behalf of userspace - and it also unlocks the attached +pi_state->rt_mutex and thus wakes up any potential waiters. + +Note that under this approach, contrary to previous PI-futex approaches, +there is no prior 'registration' of a PI-futex. [which is not quite +possible anyway, due to existing ABI properties of pthread mutexes.] + +Also, under this scheme, 'robustness' and 'PI' are two orthogonal +properties of futexes, and all four combinations are possible: futex, +robust-futex, PI-futex, robust+PI-futex. + +More details about priority inheritance can be found in +Documentation/locking/rt-mutex.rst. diff --git a/Documentation/locking/preempt-locking.rst b/Documentation/locking/preempt-locking.rst new file mode 100644 index 000000000000..dce336134e54 --- /dev/null +++ b/Documentation/locking/preempt-locking.rst @@ -0,0 +1,144 @@ +=========================================================================== +Proper Locking Under a Preemptible Kernel: Keeping Kernel Code Preempt-Safe +=========================================================================== + +:Author: Robert Love <rml@tech9.net> + + +Introduction +============ + + +A preemptible kernel creates new locking issues. The issues are the same as +those under SMP: concurrency and reentrancy. Thankfully, the Linux preemptible +kernel model leverages existing SMP locking mechanisms. Thus, the kernel +requires explicit additional locking for very few additional situations. + |
