- Apr 30, 2021
-
-
kernel test robot authored
Use min and max to make the effect more clear. Generated by: scripts/coccinelle/misc/minmax.cocci CC: Denis Efremov <efremov@linux.com> Reported-by:
kernel test robot <lkp@intel.com> Signed-off-by:
Julia Lawall <julia.lawall@inria.fr> Acked-by:
Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/alpine.DEB.2.22.394.2104292246300.16899@hadrien [catalin.marinas@arm.com: include <linux/minmax.h> explicitly] Signed-off-by:
Catalin Marinas <catalin.marinas@arm.com>
-
Mark Rutland authored
We removed the terminal frame records in commit: 6106e111 ("arm64: remove EL0 exception frame record") ... on the assumption that as we no longer used them to find the pt_regs at exception boundaries, they were no longer necessary. However, Leo reports that as an unintended side-effect, this causes traces which cross secondary_start_kernel to terminate one entry too late, with a spurious "0" entry. There are a few ways we could sovle this, but as we're planning to use terminal records for RELIABLE_STACKTRACE, let's revert the logic change for now, keeping the update comments and accounting for the changes in commit: 3c026001 ("arm64: stacktrace: Report when we reach the end of the stack") This is effectively a partial revert of commit: 6106e111 ("arm64: remove EL0 exception frame record") Signed-off-by:
Leo Yan <leo.yan@linaro.org> Tested-by:
-
Bill Wendling authored
The arm64 assembler in binutils 2.32 and above generates a program property note in a note section, .note.gnu.property, to encode used x86 ISAs and features. But the kernel linker script only contains a single NOTE segment: PHDRS { text PT_LOAD FLAGS(5) FILEHDR PHDRS; /* PF_R|PF_X */ dynamic PT_DYNAMIC FLAGS(4); /* PF_R */ note PT_NOTE FLAGS(4); /* PF_R */ } The NOTE segment generated by the vDSO linker script is aligned to 4 bytes. But the .note.gnu.property section must be aligned to 8 bytes on arm64. $ readelf -n vdso64.so Displaying notes found in: .note Owner Data size Description Linux 0x00000004 Unknown note type: (0x00000000) description data: 06 00 00 00 readelf: Warning: note with invalid namesz and/or descsz found at offset 0x20 readelf: Warning: type: 0x78, namesize: 0x00000100, descsize: 0x756e694c, alignment: 8 Since the note.gnu.property section in the vDSO is not checked by the dynamic linker, discard the .note.gnu.property sections in the vDSO. Similar to commit 4caffe6a ("x86/vdso: Discard .note.gnu.property sections in vDSO"), but for arm64. Signed-off-by:Bill Wendling <morbo@google.com> Reviewed-by:
Kees Cook <keescook@chromium.org> Acked-by:
Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20210423205159.830854-1-morbo@google.com Signed-off-by:
-
- Apr 23, 2021
-
-
Matthew Wilcox (Oracle) authored
Displaying two registers per line takes 15 lines. That improves to just 10 lines if we display three registers per line, which reduces the amount of information lost when oopses are cut off. It stays within 80 columns and matches x86-64. Signed-off-by:
Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by:
Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/20210420172245.3679077-1-willy@infradead.org Signed-off-by:
-
Jisheng Zhang authored
After commit 9fb7410f ("arm64/BUG: Use BRK instruction for generic BUG traps"), arm64 has switched to generic BUG implementation, so there's no need to select HAVE_DEBUG_BUGVERBOSE. Signed-off-by:
Jisheng Zhang <jszhang@kernel.org> Acked-by:
-
Mark Rutland authored
In __apply_alternatives() we take a pointer to void which we later assign to a pointer to struct alt_region. As all callers are passing a pointer to struct alt_region to begin with, it's simpler and more robust to take a pointer to struct alt region, so let's do so and avoid the need for a temporary variable. Signed-off-by:
-
Catalin Marinas authored
Currently arm64 allows a choice of FLATMEM, SPARSEMEM and SPARSEMEM_VMEMMAP. However, only the latter is tested regularly. FLATMEM does not seem to boot in certain configurations (guest under KVM with Qemu as a VMM). Since the reduction of the SECTION_SIZE_BITS to 27 (4K pages) or 29 (64K page), there's little argument against the memory wasted by the mem_map array with SPARSEMEM. Make SPARSEMEM_VMEMMAP the only available option, non-selectable, and remove the corresponding #ifdefs under arch/arm64/. Signed-off-by:
Marc Zyngier <maz@kernel.org> Reviewed-by:
Anshuman Khandual <anshuman.khandual@arm.com> Acked-by:
Mike Rapoport <rppt@linux.ibm.com> Link: https://lore.kernel.org/r/20210420093559.23168-1-catalin.marinas@arm.com Signed-off-by:
-
Nick Desaulniers authored
Clang can assemble these files just fine; this is a relic from the top level Makefile conditionally adding this. We no longer need --prefix, --gcc-toolchain, or -Qunused-arguments flags either with this change, so remove those too. To test building: $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \ CROSS_COMPILE_COMPAT=arm-linux-gnueabi- make LLVM=1 LLVM_IAS=1 \ defconfig arch/arm64/kernel/vdso32/ Suggested-by:
Nathan Chancellor <nathan@kernel.org> Signed-off-by:
Nick Desaulniers <ndesaulniers@google.com> Reviewed-by:
Vincenzo Frascino <vincenzo.frascino@arm.com> Tested-by:
Stephen Boyd <swboyd@chromium.org> Acked-by:
-
- Apr 15, 2021
-
-
Mark Brown authored
The FPSIMD code was relying on IS_ENABLED() checks in system_suppors_sve() to cause the compiler to delete references to SVE functions in some places, add explicit IS_ENABLED() checks back. Fixes: ef9c5d09 ("arm64/sve: Remove redundant system_supports_sve() tests") Reported-by:
Mark Brown <broonie@kernel.org> Link: https://lore.kernel.org/r/20210415121742.36628-1-broonie@kernel.org Signed-off-by:
-
- Apr 13, 2021
-
-
zhouchuangao authored
It can be optimized at compile time. Signed-off-by:
zhouchuangao <zhouchuangao@vivo.com> Reviewed-by:
Masami Hiramatsu <mhiramat@kernel.org> Link: https://lore.kernel.org/r/1617105472-6081-1-git-send-email-zhouchuangao@vivo.com Signed-off-by:
-
Peter Collingbourne authored
The kernel does not use any keys besides IA so we don't need to install IB/DA/DB/GA on kernel exit if we arrange to install them on task switch instead, which we can expect to happen an order of magnitude less often. Furthermore we can avoid installing the user IA in the case where the user task has IA disabled and just leave the kernel IA installed. This also lets us avoid needing to install IA on kernel entry. On an Apple M1 under a hypervisor, the overhead of kernel entry/exit has been measured to be reduced by 15.6ns in the case where IA is enabled, and 31.9ns in the case where IA is disabled. Signed-off-by:
Peter Collingbourne <pcc@google.com> Link: https://linux-review.googlesource.com/id/Ieddf6b580d23c9e0bed45a822dabe72d2ffc9a8e Link: https://lore.kernel.org/r/2d653d055f38f779937f2b92f8ddd5cf9e4af4f4.1616123271.git.pcc@google.com Signed-off-by:
-
Peter Collingbourne authored
This change introduces a prctl that allows the user program to control which PAC keys are enabled in a particular task. The main reason why this is useful is to enable a userspace ABI that uses PAC to sign and authenticate function pointers and other pointers exposed outside of the function, while still allowing binaries conforming to the ABI to interoperate with legacy binaries that do not sign or authenticate pointers. The idea is that a dynamic loader or early startup code would issue this prctl very early after establishing that a process may load legacy binaries, but before executing any PAC instructions. This change adds a small amount of overhead to kernel entry and exit due to additional required instruction sequences. On a DragonBoard 845c (Cortex-A75) with the powersave governor, the overhead of similar instruction sequences was measured as 4.9ns when simulating the common case where IA is left enabled, or 43.7ns when simulating the uncommon case where IA is disabled. These numbers can be seen as the worst case scenario, since in more realistic scenarios a better performing governor would be used and a newer chip would be used that would support PAC unlike Cortex-A75 and would be expected to be faster than Cortex-A75. On an Apple M1 under a hypervisor, the overhead of the entry/exit instruction sequences introduced by this patch was measured as 0.3ns in the case where IA is left enabled, and 33.0ns in the case where IA is disabled. Signed-off-by:
Dave Martin <Dave.Martin@arm.com> Link: https://linux-review.googlesource.com/id/Ibc41a5e6a76b275efbaa126b31119dc197b927a5 Link: https://lore.kernel.org/r/d6609065f8f40397a4124654eb68c9f490b4d477.1616123271.git.pcc@google.com Signed-off-by:
-
Peter Collingbourne authored
In an upcoming change we are going to introduce per-task SCTLR_EL1 bits for PAC. Move the existing per-task SCTLR_EL1 field out of the MTE-specific code so that we will be able to use it from both the PAC and MTE code paths and make the task switching code more efficient. Signed-off-by:
-
Mark Brown authored
Currently there are a number of places in the SVE code where we check both system_supports_sve() and TIF_SVE. This is a bit redundant given that we should never get into a situation where we have set TIF_SVE without having SVE support and it is not clear that silently ignoring a mistakenly set TIF_SVE flag is the most sensible error handling approach. For now let's just drop the system_supports_sve() checks since this will at least reduce overhead a little. Signed-off-by:
-
- Apr 12, 2021
-
-
Ard Biesheuvel authored
Kernel mode NEON can be used in task or softirq context, but only in a non-nesting manner, i.e., softirq context is only permitted if the interrupt was not taken at a point where the kernel was using the NEON in task context. This means all users of kernel mode NEON have to be aware of this limitation, and either need to provide scalar fallbacks that may be much slower (up to 20x for AES instructions) and potentially less safe, or use an asynchronous interface that defers processing to a later time when the NEON is guaranteed to be available. Given that grabbing and releasing the NEON is cheap, we can relax this restriction, by increasing the granularity of kernel mode NEON code, and always disabling softirq processing while the NEON is being used in task context. Signed-off-by:
-
Ard Biesheuvel authored
The AArch64 asm syntax has this slightly tedious property that the names used in mnemonics to refer to registers depend on whether the opcode in question targets the entire 64-bits (xN), or only the least significant 8, 16 or 32 bits (wN). When writing parameterized code such as macros, this can be annoying, as macro arguments don't lend themselves to indexed lookups, and so generating a reference to wN in a macro that receives xN as an argument is problematic. For instance, an upcoming patch that modifies the implementation of the cond_yield macro to be able to refer to 32-bit registers would need to modify invocations such as cond_yield 3f, x8 to cond_yield 3f, 8 so that the second argument can be token pasted after x or w to emit the correct register reference. Unfortunately, this interferes with the self documenting nature of the first example, where the second argument is obviously a register, whereas in the second example, one would need to go and look at the code to find out what '8' means. So let's fix this by defining wxN aliases for all xN registers, which resolve to the 32-bit alias of each respective 64-bit register. This allows the macro implementation to paste the xN reference after a w to obtain the correct register name. Signed-off-by:
-
Ard Biesheuvel authored
The users of the conditional NEON yield macros have all been switched to the simplified cond_yield macro, and so the NEON specific ones can be removed. Signed-off-by:
-
- Apr 11, 2021
-
-
Andrey Konovalov authored
This change adds KASAN-KUnit tests support for the async HW_TAGS mode. In async mode, tag fault aren't being generated synchronously when a bad access happens, but are instead explicitly checked for by the kernel. As each KASAN-KUnit test expect a fault to happen before the test is over, check for faults as a part of the test handler. Acked-by:
Andrey Konovalov <andreyknvl@google.com> Tested-by:
-
Vincenzo Frascino authored
When MTE async mode is enabled TFSR_EL1 contains the accumulative asynchronous tag check faults for EL1 and EL0. During the suspend/resume operations the firmware might perform some operations that could change the state of the register resulting in a spurious tag check fault report. Report asynchronous tag faults before suspend and clear the TFSR_EL1 register after resume to prevent this to happen. Cc: Will Deacon <will@kernel.org> Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> Reviewed-by:
Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> Acked-by:
-
Vincenzo Frascino authored
MTE provides a mode that asynchronously updates the TFSR_EL1 register when a tag check exception is detected. To take advantage of this mode the kernel has to verify the status of the register at: 1. Context switching 2. Return to user/EL0 (Not required in entry from EL0 since the kernel did not run) 3. Kernel entry from EL1 4. Kernel exit to EL1 If the register is non-zero a trace is reported. Add the required features for EL1 detection and reporting. Note: ITFSB bit is set in the SCTLR_EL1 register hence it guaranties that the indirect writes to TFSR_EL1 are synchronized at exception entry to EL1. On the context switch path the synchronization is guarantied by the dsb() in __switch_to(). The dsb(nsh) in mte_check_tfsr_exit() is provisional pending confirmation by the architects. Cc: Will Deacon <will@kernel.org> Reviewed-by:
-
Vincenzo Frascino authored
mte_enable_kernel_*() are not needed if KASAN_HW is disabled. Add ash defines around the functions to conditionally compile the functions. Signed-off-by:
-
Vincenzo Frascino authored
load_unaligned_zeropad() and __get/put_kernel_nofault() functions can read past some buffer limits which may include some MTE granule with a different tag. When MTE async mode is enabled, the load operation crosses the boundaries and the next granule has a different tag the PE sets the TFSR_EL1.TF1 bit as if an asynchronous tag fault is happened. Enable Tag Check Override (TCO) in these functions before the load and disable it afterwards to prevent this to happen. Note: The same condition can be hit in MTE sync mode but we deal with it through the exception handling. In the current implementation, mte_async_mode flag is set only at boot time but in future kasan might acquire some runtime features that that change the mode dynamically, hence we disable it when sync mode is selected for future proof. Cc: Will Deacon <will@kernel.org> Reported-by:
Branislav Rankov <Branislav.Rankov@arm.com> Tested-by:
-
Vincenzo Frascino authored
arch_enable_tagging() was left in memory.h after the introduction of async mode to not break the bysectability of the KASAN KUNIT tests. Remove the function now that KASAN has been fully converted. Cc: Will Deacon <will@kernel.org> Acked-by:
-
Vincenzo Frascino authored
MTE provides an asynchronous mode for detecting tag exceptions. In particular instead of triggering a fault the arm64 core updates a register which is checked by the kernel after the asynchronous tag check fault has occurred. Add support for MTE asynchronous mode. The exception handling mechanism will be added with a future patch. Note: KASAN HW activates async mode via kasan.mode kernel parameter. The default mode is set to synchronous. The code that verifies the status of TFSR_EL1 will be added with a future patch. Cc: Will Deacon <will@kernel.org> Reviewed-by:
-
- Apr 08, 2021
-
-
Marc Zyngier authored
CONFIG_ARM64_VHE was introduced with ARMv8.1 (some 7 years ago), and has been enabled by default for almost all that time. Given that newer systems that are VHE capable are finally becoming available, and that some systems are even incapable of not running VHE, drop the configuration altogether. Anyone willing to stick to non-VHE on VHE hardware for obscure reasons should use the 'kvm-arm.mode=nvhe' command-line option. Suggested-by:
-
Marc Zyngier authored
It seems that the CPUs part of the SoC known as Apple M1 have the terrible habit of being stuck with HCR_EL2.E2H==1, in violation of the architecture. Try and work around this deplorable state of affairs by detecting the stuck bit early and short-circuit the nVHE dance. Additional filtering code ensures that attempts at switching to nVHE from the command-line are also ignored. It is still unknown whether there are many more such nuggets to be found... Reported-by:
Hector Martin <marcan@marcan.st> Acked-by:
-
Marc Zyngier authored
Some CPUs are broken enough that some overrides need to be rejected at the earliest opportunity. In some cases, that's right at cpu feature override time. Provide the necessary infrastructure to filter out overrides, and to report such filtered out overrides to the core cpufeature code. Acked-by:
-
Mark Brown authored
The arm64 FEAT_FGT extension introduces a set of traps to EL2 for accesses to small sets of registers and instructions from EL1 and EL0. Currently Linux makes no use of this feature, ensure that it is not active at boot by disabling the traps during EL2 setup. Signed-off-by:
-
Vincenzo Frascino authored
mte_assign_mem_tag_range() was added in commit 85f49cae ("arm64: mte: add in-kernel MTE helpers") in 5.11 but moved out of mte.S by commit 2cb34276 ("arm64: kasan: simplify and inline MTE functions") in 5.12 and renamed to mte_set_mem_tag_range(). 2cb34276 did not delete the old function prototypes in mte.h. Remove the unused prototype from mte.h. Cc: Will Deacon <will@kernel.org> Reported-by:
Derrick McKee <derrick.mckee@gmail.com> Signed-off-by:
-
Jisheng Zhang authored
They are not needed after booting, so mark them as __init to move them to the .init section. Signed-off-by:
Jisheng Zhang <Jisheng.Zhang@synaptics.com> Reviewed-by:
Steven Price <steven.price@arm.com> Link: https://lore.kernel.org/r/20210330135449.4dcffd7f@xhacker.debian Signed-off-by:
-
Mark Brown authored
When we enable SVE usage in userspace after taking a SVE access trap we need to ensure that the portions of the register state that are not shared with the FPSIMD registers are zeroed. Currently we do this by forcing the FPSIMD registers to be saved to the task struct and converting them there. This is wasteful in the common case where the task state is loaded into the registers and we will immediately return to userspace since we can initialise the SVE state directly in registers instead of accessing multiple copies of the register state in memory. Instead in that common case do the conversion in the registers and update the task metadata so that we can return to userspace without spilling the register state to memory unless there is some other reason to do so. Signed-off-by:
-
- Apr 01, 2021
-
-
Qi Liu authored
The initialization of value in function armv8pmu_read_hw_counter() and armv8pmu_read_counter() seem redundant, as they are soon updated. So, We can remove them. Signed-off-by:
Qi Liu <liuqi115@huawei.com> Link: https://lore.kernel.org/r/1617275801-1980-1-git-send-email-liuqi115@huawei.com Signed-off-by:
-
- Mar 29, 2021
-
-
Lecopzer Chen authored
Before this patch, someone who wants to use VMAP_STACK when KASAN_GENERIC enabled must explicitly select KASAN_VMALLOC. >From Will's suggestion [1]: > I would _really_ like to move to VMAP stack unconditionally, and > that would effectively force KASAN_VMALLOC to be set if KASAN is in use Because VMAP_STACK now depends on either HW_TAGS or KASAN_VMALLOC if KASAN enabled, in order to make VMAP_STACK selected unconditionally, we bind KANSAN_GENERIC and KASAN_VMALLOC together. Note that SW_TAGS supports neither VMAP_STACK nor KASAN_VMALLOC now, so this is the first step to make VMAP_STACK selected unconditionally. Bind KANSAN_GENERIC and KASAN_VMALLOC together is supposed to cost more memory at runtime, thus the alternative is using SW_TAGS KASAN instead. [1]: https://lore.kernel.org/lkml/20210204150100.GE20815@willie-the-truck/ Suggested-by:
Lecopzer Chen <lecopzer.chen@mediatek.com> Link: https://lore.kernel.org/r/20210324040522.15548-6-lecopzer.chen@mediatek.com Signed-off-by:
-
Lecopzer Chen authored
After KASAN_VMALLOC works in arm64, we can randomize module region into vmalloc area now. Test: VMALLOC area ffffffc010000000 fffffffdf0000000 before the patch: module_alloc_base/end ffffffc008b80000 ffffffc010000000 after the patch: module_alloc_base/end ffffffdcf4bed000 ffffffc010000000 And the function that insmod some modules is fine. Suggested-by:
-
Lecopzer Chen authored
We can backed shadow memory in vmalloc area after vmalloc area isn't populated at kasan_init(), thus make KASAN_VMALLOC selectable. Signed-off-by:
Andrey Konovalov <andreyknvl@gmail.com> Tested-by:
-
Lecopzer Chen authored
Arm64 provides defined macro for KERNEL_START and KERNEL_END, thus replace them by the abstration instead of using _text and _end. Signed-off-by:
-
Lecopzer Chen authored
Linux support KAsan for VMALLOC since commit 3c5c3cfb ("kasan: support backing vmalloc space with real shadow memory") Like how the MODULES_VADDR does now, just not to early populate the VMALLOC_START between VMALLOC_END. Before: MODULE_VADDR: no mapping, no zero shadow at init VMALLOC_VADDR: backed with zero shadow at init After: MODULE_VADDR: no mapping, no zero shadow at init VMALLOC_VADDR: no mapping, no zero shadow at init Thus the mapping will get allocated on demand by the core function of KASAN_VMALLOC. ----------- vmalloc_shadow_start | | | | | | <= non-mapping | | | | |-----------| |///////////|<- kimage shadow with page table mapping. |-----------| | | | | <= non-mapping | | ------------- vmalloc_shadow_end |00000000000| |00000000000| <= Zero shadow |00000000000| ------------- KASAN_SHADOW_END Signed-off-by:
-
Chen Lifu authored
In commit eb631bb5 ("arm64: Support arch_irq_work_raise() via self IPIs") a new function "arch_irq_work_raise" was added without a prototype. In commit d914d4d4 ("arm64: Implement panic_smp_self_stop()") a new function "panic_smp_self_stop" was added without a prototype. We get the following warnings on W=1: arch/arm64/kernel/smp.c:842:6: warning: no previous prototype for ‘arch_irq_work_raise’ [-Wmissing-prototypes] arch/arm64/kernel/smp.c:862:6: warning: no previous prototype for ‘panic_smp_self_stop’ [-Wmissing-prototypes] Fix the warnings by: 1. Adding the prototype for 'arch_irq_work_raise' in irq_work.h 2. Adding the prototype for 'panic_smp_self_stop' in smp.h Signed-off-by:
Chen Lifu <chenlifu@huawei.com> Link: https://lore.kernel.org/r/20210329034343.183974-1-chenlifu@huawei.com Signed-off-by:
-
- Mar 28, 2021
-
-
Mark Rutland authored
In __cpu_setup we conditionally manipulate the TCR_EL1 value in x10 after previously using x10 as a scratch register for unrelated temporary variables. To make this a bit clearer, let's move the TCR_EL1 value into a named register `tcr`. To simplify the register allocation, this is placed in the highest available caller-saved scratch register, tcr. Following the example of `mair`, we initialise the register with the default value prior to any feature discovery, and write it to MAIR_EL1 after all feature discovery is complete, which allows us to simplify the featuere discovery code. The existing `mte_tcr` register is no longer needed, and is replaced by the use of x10 as a temporary, matching the rest of the MTE feature discovery assembly in __cpu_setup. As x20 is no longer used, the function is now AAPCS compliant, as we've generally aimed for in our assembly functions. There should be no functional change as as a result of this patch. Signed-off-by:
-
Mark Rutland authored
In __cpu_setup we conditionally manipulate the MAIR_EL1 value in x5 before later reusing x5 as a scratch register for unrelated temporary variables. To make this a bit clearer, let's move the MAIR_EL1 value into a named register `mair`. To simplify the register allocation, this is placed in the highest available caller-saved scratch register, x17. As it is no longer clobbered by other usage, we can write the value to MAIR_EL1 at the end of the function as we do for TCR_EL1 rather than part-way though feature discovery. There should be no functional change as as a result of this patch. Signed-off-by:
-
