Commit 3181e22f authored by Pavel Begunkov's avatar Pavel Begunkov Committed by Jens Axboe
Browse files

io_uring: wake up optimisations 



Flush completions is done either from the submit syscall or by the
task_work, both are in the context of the submitter task, and when it
goes for a single threaded rings like implied by ->task_complete, there
won't be any waiters on ->cq_wait but the master task. That means that
there can be no tasks sleeping on cq_wait while we run
__io_submit_flush_completions() and so waking up can be skipped.

Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/60ad9768ec74435a0ddaa6eec0ffa7729474f69f.1673274244.git.asml.silence@gmail.com


Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
parent bca39f39

io_uring/io_uring.c

+20 −1
Original line number Diff line number Diff line
@@ -621,6 +621,25 @@ static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) 
	io_cqring_wake(ctx);

}



static inline void __io_cq_unlock_post_flush(struct io_ring_ctx *ctx)

	__releases(ctx->completion_lock)

{

	io_commit_cqring(ctx);

	__io_cq_unlock(ctx);

	io_commit_cqring_flush(ctx);



	/*

	 * As ->task_complete implies that the ring is single tasked, cq_wait

	 * may only be waited on by the current in io_cqring_wait(), but since

	 * it will re-check the wakeup conditions once we return we can safely

	 * skip waking it up.

	 */

	if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN)) {

		smp_mb();

		__io_cqring_wake(ctx);

	}

}



void io_cq_unlock_post(struct io_ring_ctx *ctx)

	__releases(ctx->completion_lock)

{
@@ -1480,7 +1499,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) 
			}

		}

	}

	__io_cq_unlock_post(ctx);

	__io_cq_unlock_post_flush(ctx);



	if (!wq_list_empty(&ctx->submit_state.compl_reqs)) {

		io_free_batch_list(ctx, state->compl_reqs.first);