diff options
| author | Pavel Begunkov <asml.silence@gmail.com> | 2023-01-09 14:46:10 +0000 |
|---|---|---|
| committer | Greg Kroah-Hartman <gregkh@linuxfoundation.org> | 2024-12-14 19:54:52 +0100 |
| commit | 3ab9326f93ec4471cab6f2107ecdf0cf6a8615aa (patch) | |
| tree | 5a4d6a50251ecb8b9e4c19f7afe254cf5e9f1130 | |
| parent | 7d5c04d26d35e7b1b2aa5db8bf2f3ca96d71cd51 (diff) | |
| download | linux-3ab9326f93ec4471cab6f2107ecdf0cf6a8615aa.tar.gz linux-3ab9326f93ec4471cab6f2107ecdf0cf6a8615aa.tar.bz2 linux-3ab9326f93ec4471cab6f2107ecdf0cf6a8615aa.zip | |
io_uring: wake up optimisations
commit 3181e22fb79910c7071e84a43af93ac89e8a7106 upstream.
Flush completions is done either from the submit syscall or by the
task_work, both are in the context of the submitter task, and when it
goes for a single threaded rings like implied by ->task_complete, there
won't be any waiters on ->cq_wait but the master task. That means that
there can be no tasks sleeping on cq_wait while we run
__io_submit_flush_completions() and so waking up can be skipped.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/60ad9768ec74435a0ddaa6eec0ffa7729474f69f.1673274244.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
| -rw-r--r-- | io_uring/io_uring.c | 12 |
1 files changed, 11 insertions, 1 deletions
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4f0ae938b146..0b1361663267 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -582,6 +582,16 @@ static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) io_cqring_ev_posted(ctx); } +static inline void __io_cq_unlock_post_flush(struct io_ring_ctx *ctx) + __releases(ctx->completion_lock) +{ + io_commit_cqring(ctx); + spin_unlock(&ctx->completion_lock); + io_commit_cqring_flush(ctx); + if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN)) + __io_cqring_wake(ctx); +} + void io_cq_unlock_post(struct io_ring_ctx *ctx) { __io_cq_unlock_post(ctx); @@ -1339,7 +1349,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) if (!(req->flags & REQ_F_CQE_SKIP)) __io_fill_cqe_req(ctx, req); } - __io_cq_unlock_post(ctx); + __io_cq_unlock_post_flush(ctx); io_free_batch_list(ctx, state->compl_reqs.first); INIT_WQ_LIST(&state->compl_reqs); |