- Sep 13, 2019
-
-
Daniel Jordan authored
Padata binds the parallel part of a job to a single CPU and round-robins over all CPUs in the system for each successive job. Though the serial parts rely on per-CPU queues for correct ordering, they're not necessary for parallel work, and it improves performance to run the job locally on NUMA machines and let the scheduler pick the CPU within a node on a busy system. So, make the parallel workqueue unbound. Update the parallel workqueue's cpumask when the instance's parallel cpumask changes. Now that parallel jobs no longer run on max_active=1 workqueues, two or more parallel works that hash to the same CPU may run simultaneously, finish out of order, and so be serialized out of order. Prevent this by keeping the works sorted on the reorder list by sequence number and checking that in the reordering logic. padata_get_next becomes padata_find_next so it can be reused for the end of padata_reorder, where it's used to avoid uselessly queueing work when the next job by sequence number isn't finished yet but a later job that hashed to the same CPU has. The ENODATA case in padata_find_next no longer makes sense because parallel jobs aren't bound to specific CPUs. The EINPROGRESS case takes care of the scenario where a parallel job is potentially running on the same CPU as padata_find_next, and with only one error code left, just use NULL instead. Signed-off-by:
Daniel Jordan <daniel.m.jordan@oracle.com> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Lai Jiangshan <jiangshanlai@gmail.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Steffen Klassert <steffen.klassert@secunet.com> Cc: Tejun Heo <tj@kernel.org> Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by:
Herbert Xu <herbert@gondor.apana.org.au>
-
Daniel Jordan authored
padata currently uses one per-CPU workqueue per instance for all work. Prepare for running parallel jobs on an unbound workqueue by introducing dedicated workqueues for parallel and serial work. Signed-off-by:
Steffen Klassert <steffen.klassert@secunet.com> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Lai Jiangshan <jiangshanlai@gmail.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Tejun Heo <tj@kernel.org> Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by:
-
Daniel Jordan authored
With pcrypt's cpumask no longer used, take the CPU hotplug lock inside padata_alloc_possible. Useful later in the series for avoiding nested acquisition of the CPU hotplug lock in padata when padata_alloc_possible is allocating an unbound workqueue. Without this patch, this nested acquisition would happen later in the series: pcrypt_init_padata get_online_cpus alloc_padata_possible alloc_padata alloc_workqueue(WQ_UNBOUND) // later in the series alloc_and_link_pwqs apply_wqattrs_lock get_online_cpus // recursive rwsem acquisition Signed-off-by: -
Daniel Jordan authored
Now that padata_do_parallel takes care of finding an alternate callback CPU, there's no need for pcrypt's callback cpumask, so remove it and the notifier callback that keeps it in sync. Signed-off-by:
-
Daniel Jordan authored
padata_do_parallel currently returns -EINVAL if the callback CPU isn't in the callback cpumask. pcrypt tries to prevent this situation by keeping its own callback cpumask in sync with padata's and checks that the callback CPU it passes to padata is valid. Make padata handle this instead. padata_do_parallel now takes a pointer to the callback CPU and updates it for the caller if an alternate CPU is used. Overall behavior in terms of which callback CPUs are chosen stays the same. Prepares for removal of the padata cpumask notifier in pcrypt, which will fix a lockdep complaint about nested acquisition of the CPU hotplug lock later in the series. Signed-off-by:
-
Daniel Jordan authored
Change the calling convention for apply_workqueue_attrs to require CPU hotplug read exclusion. Avoids lockdep complaints about nested calls to get_online_cpus in a future patch where padata calls apply_workqueue_attrs when changing other CPU-hotplug-sensitive data structures with the CPU read lock already held. Signed-off-by:
Tejun Heo <tj@kernel.org> Acked-by:
-
Daniel Jordan authored
padata will use these these interfaces in a later patch, so unconfine them. Signed-off-by:
-
Daniel Jordan authored
Move workqueue allocation inside of padata to prepare for further changes to how padata uses workqueues. Guarantees the workqueue is created with max_active=1, which padata relies on to work correctly. No functional change. Signed-off-by:
-
Andrey Smirnov authored
Add node for CAAM - Cryptographic Acceleration and Assurance Module. Signed-off-by:
Horia Geantă <horia.geanta@nxp.com> Signed-off-by:
Andrey Smirnov <andrew.smirnov@gmail.com> Cc: Cory Tusar <cory.tusar@zii.aero> Cc: Chris Healy <cphealy@gmail.com> Cc: Lucas Stach <l.stach@pengutronix.de> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Shawn Guo <shawnguo@kernel.org> Cc: Iuliana Prodan <iuliana.prodan@nxp.com> Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org Acked-by:
Li Yang <leoyang.li@nxp.com> Signed-off-by:
-
- Sep 09, 2019
-
-
Stephen Boyd authored
Sebastian reports that after commit ff296293 ("random: Support freezable kthreads in add_hwgenerator_randomness()") we can call might_sleep() when the task state is TASK_INTERRUPTIBLE (state=1). This leads to the following warning. do not call blocking ops when !TASK_RUNNING; state=1 set at [<00000000349d1489>] prepare_to_wait_event+0x5a/0x180 WARNING: CPU: 0 PID: 828 at kernel/sched/core.c:6741 __might_sleep+0x6f/0x80 Modules linked in: CPU: 0 PID: 828 Comm: hwrng Not tainted 5.3.0-rc7-next-20190903+ #46 RIP: 0010:__might_sleep+0x6f/0x80 Call Trace: kthread_freezable_should_stop+0x1b/0x60 add_hwgenerator_randomness+0xdd/0x130 hwrng_fillfn+0xbf/0x120 kthread+0x10c/0x140 ret_from_fork+0x27/0x50 We shouldn't call kthread_freezable_should_stop() from deep within the wait_event code because the task state is still set as TASK_INTERRUPTIBLE instead of TASK_RUNNING and kthread_freezable_should_stop() will try to call into the freezer with the task in the wrong state. Use wait_event_freezable() instead so that it calls schedule() in the right place and tries to enter the freezer when the task state is TASK_RUNNING instead. Reported-by:
Sebastian Andrzej Siewior <bigeasy@linutronix.de> Tested-by:
Stephen Boyd <swboyd@chromium.org> Signed-off-by:
-
Herbert Xu authored
This patch fixes a number of warnings encountered when this driver is built on a 64-bit platform with COMPILE_TEST. Signed-off-by: -
Ard Biesheuvel authored
The RFC4106 key derivation code instantiates an AES cipher transform to encrypt only a single block before it is freed again. Switch to the new AES library which is more suitable for such use cases. Signed-off-by:
Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by:
-
Wei Yongjun authored
When using single_open() for opening, single_release() should be used instead of seq_release(), otherwise there is a memory leak. Fixes: 09ae5d37 ("crypto: zip - Add Compression/Decompression statistics") Cc: <stable@vger.kernel.org> Signed-off-by:
Wei Yongjun <weiyongjun1@huawei.com> Signed-off-by:
-
zhong jiang authored
Use kzfree instead of memset() + kfree(). Signed-off-by:
zhong jiang <zhongjiang@huawei.com> Signed-off-by:
-
Andrey Smirnov authored
With IRQ requesting being managed by devres we need to make sure that we dispose of IRQ mapping after and not before it is free'd (otherwise we'll end up with a warning from the kernel). To achieve that simply convert IRQ mapping to rely on devres as well. Fixes: f314f12db65c ("crypto: caam - convert caam_jr_init() to use devres") Signed-off-by: -
Andrey Smirnov authored
Irq_of_parse_and_map will return zero in case of error, so add a error check for that. Signed-off-by:
-
Andrey Smirnov authored
Use devres to unmap memory and drop explicit de-initialization code. NOTE: There's no corresponding unmapping code in caam_jr_remove which seems like a resource leak. Signed-off-by:
-
Andrey Smirnov authored
In order to access IP block's registers we need to enable appropriate clocks first, otherwise we are risking hanging the CPU. The problem becomes very apparent when trying to use CAAM driver built as a kernel module. In that case caam_probe() gets called after clk_disable_unused() which means all of the necessary clocks are guaranteed to be disabled. Coincidentally, this change also fixes iomap leak introduced by early return (instead of "goto iounmap_ctrl") in commit 41fc54afae70 ("crypto: caam - simplfy clock initialization") Tested on ZII i.MX6Q+ RDU2 Fixes: 176435ad ("crypto: caam - defer probing until QMan is available") Fixes: 41fc54afae70 ("crypto: caam - simplfy clock initialization") Signed-off-by: -
Ard Biesheuvel authored
Instead of relying on the CTS template to wrap the accelerated CBC skcipher, implement the ciphertext stealing part directly. Signed-off-by:
-
Ard Biesheuvel authored
Update the AES-XTS implementation based on NEON instructions so that it can deal with inputs whose size is not a multiple of the cipher block size. This is part of the original XTS specification, but was never implemented before in the Linux kernel. Signed-off-by:
-
Ard Biesheuvel authored
Update the AES-XTS implementation based on AES instructions so that it can deal with inputs whose size is not a multiple of the cipher block size. This is part of the original XTS specification, but was never implemented before in the Linux kernel. Signed-off-by:
-
Ard Biesheuvel authored
Update the AES-XTS implementation based on NEON instructions so that it can deal with inputs whose size is not a multiple of the cipher block size. This is part of the original XTS specification, but was never implemented before in the Linux kernel. Since the bit slicing driver is only faster if it can operate on at least 7 blocks of input at the same time, let's reuse the alternate path we are adding for CTS to process any data tail whose size is not a multiple of 128 bytes. Signed-off-by:
-
Ard Biesheuvel authored
Add the missing support for ciphertext stealing in the implementation of AES-XTS, which is part of the XTS specification but was omitted up until now due to lack of a need for it. The asm helpers are updated so they can deal with any input size, as long as the last full block and the final partial block are presented at the same time. The glue code is updated so that the common case of operating on a sector or page is mostly as before. When CTS is needed, the walk is split up into two pieces, unless the entire input is covered by a single step. Signed-off-by:
-
Ard Biesheuvel authored
Since the CTS-CBC code completes synchronously, there is no point in keeping part of the scratch data it uses in the request context, so move it to the stack instead. Signed-off-by:
-
Ard Biesheuvel authored
Optimize away one of the tbl instructions in the decryption path, which turns out to be unnecessary. Signed-off-by:
-
Ard Biesheuvel authored
After starting a skcipher walk, the only way to ensure that all resources it has tied up are released is to complete it. In some cases, it will be useful to be able to abort a walk cleanly after it has started, so add this ability to the skcipher walk API. Signed-off-by:
-
Ard Biesheuvel authored
The pure NEON AES implementation predates the bit-slicing one, and is generally slower, unless the algorithm in question can only execute sequentially. So advertising the skciphers that the bit-slicing driver implements as well serves no real purpose, and we can just disable them. Note that the bit-slicing driver also has a link time dependency on the pure NEON driver, for CBC encryption and for XTS tweak calculation, so we still need both drivers on systems that do not implement the Crypto Extensions. At the same time, expose those modaliases for the AES instruction based driver. This is necessary since otherwise, we may end up loading the wrong driver when any of the skciphers are instantiated before the CPU capability based module loading has completed. Finally, add the missing modalias for cts(cbc(aes)) so requests for this algorithm will autoload the correct module. Signed-off-by:
-
Ard Biesheuvel authored
Replace the vector load from memory sequence with a simple instruction sequence to compose the tweak vector directly. Signed-off-by:
-
Ard Biesheuvel authored
Replace the vector load from memory sequence with a simple instruction sequence to compose the tweak vector directly. Signed-off-by:
-
Ard Biesheuvel authored
Replace the vector load from memory sequence with a simple instruction sequence to compose the tweak vector directly. Signed-off-by:
-
Ard Biesheuvel authored
When the ARM AES instruction based crypto driver was introduced, there were no known implementations that could benefit from a 4-way interleave, and so a 3-way interleave was used instead. Since we have sufficient space in the SIMD register file, let's switch to a 4-way interleave to align with the 64-bit driver, and to ensure that we can reach optimum performance when running under emulation on high end 64-bit cores. Signed-off-by:
-
Ard Biesheuvel authored
Reduce the scope of the kernel_neon_begin/end regions so that the SIMD unit is released (and thus preemption re-enabled) if the crypto operation cannot be completed in a single scatterwalk step. This avoids scheduling blackouts due to preemption being enabled for unbounded periods, resulting in a more responsive system. After this change, we can also permit the cipher_walk infrastructure to sleep, so set the 'atomic' parameter to skcipher_walk_virt() to false as well. Signed-off-by:
-
Ard Biesheuvel authored
The AES round keys are arrays of u32s in native endianness now, so update the function prototypes accordingly. Signed-off-by:
-
Herbert Xu authored
skcipher_walk_done may be called with an error by internal or external callers. For those internal callers we shouldn't unmap pages but for external callers we must unmap any pages that are in use. This patch distinguishes between the two cases by checking whether walk->nbytes is zero or not. For internal callers, we now set walk->nbytes to zero prior to the call. For external callers, walk->nbytes has always been non-zero (as zero is used to indicate the termination of a walk). Reported-by:
-
zhong jiang authored
PTR_ERR_OR_ZERO contains if(IS_ERR(...)) + PTR_ERR. It is better to use it directly. hence just replace it. Signed-off-by:
Will Deacon <will@kernel.org> Signed-off-by:
-
- Sep 05, 2019
-
-
Hans de Goede authored
lib/crypto/sha256.c and include/crypto/sha256_base.h define 99% identical functions to init a sha256_state struct for sha224 or sha256 use. This commit moves the functions from lib/crypto/sha256.c to include/crypto/sha.h (making them static inline) and makes the sha224/256_base_init static inline functions from include/crypto/sha256_base.h wrappers around the now also static inline include/crypto/sha.h functions. Signed-off-by:
Hans de Goede <hdegoede@redhat.com> Signed-off-by:
-
Hans de Goede authored
The generic sha256 implementation from lib/crypto/sha256.c uses data structs defined in crypto/sha.h, so lets move the function prototypes there too. Signed-off-by:
-
Hans de Goede authored
Rename the sha*_init arrays to n2_sha*_init so that they do not conflict with the functions declared in crypto/sha256.h. Also rename md5_init to n2_md5_init for consistency. This is a preparation patch for folding crypto/sha256.h into crypto/sha.h. Signed-off-by:
-
Hans de Goede authored
Rename the sha*_init arrays to chcr_sha*_init so that they do not conflict with the functions declared in crypto/sha256.h. This is a preparation patch for folding crypto/sha256.h into crypto/sha.h. Signed-off-by:
-
Hans de Goede authored
Rename the algo_init arrays to cc_algo_init so that they do not conflict with the functions declared in crypto/sha256.h. This is a preparation patch for folding crypto/sha256.h into crypto/sha.h. Signed-off-by:
Gilad Ben-Yossef <gilad@benyossef.com> Signed-off-by:
-
