Skip to content
  1. Dec 05, 2019
  3. Dec 01, 2019
  5. Nov 29, 2019
    • Stanislav Fomichev's avatar
      bpf: Force .BTF section start to zero when dumping from vmlinux · df786c9b
      Stanislav Fomichev authored 
      

While trying to figure out why fentry_fexit selftest doesn't pass for me
(old pahole, broken BTF), I found out that my latest patch can break vmlinux
.BTF generation. objcopy preserves section start when doing --only-section,
so there is a chance (depending on where pahole inserts .BTF section) to
have leading empty zeroes. Let's explicitly force section offset to zero.

Before:

$ objcopy --set-section-flags .BTF=alloc -O binary \
	--only-section=.BTF vmlinux .btf.vmlinux.bin
$ xxd .btf.vmlinux.bin | head -n1
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................

After:

$ objcopy --change-section-address .BTF=0 \
	--set-section-flags .BTF=alloc -O binary \
	--only-section=.BTF vmlinux .btf.vmlinux.bin
$ xxd .btf.vmlinux.bin | head -n1
00000000: 9feb 0100 1800 0000 0000 0000 80e1 1c00  ................
          ^BTF magic

As part of this change, I'm also dropping '2>/dev/null' from objcopy
invocation to be able to catch possible other issues (objcopy doesn't
produce any warnings for me anymore, it did before with --dump-section).

Fixes: da5fb182 ("bpf: Support pre-2.25-binutils objcopy for vmlinux BTF")
Signed-off-by: default avatarStanislav Fomichev <sdf@google.com>
Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
Acked-by: default avatarJohn Fastabend <john.fastabend@gmail.com>
Cc: Andrii Nakryiko <andriin@fb.com>
Link: https://lore.kernel.org/bpf/20191127225759.39923-1-sdf@google.com
      df786c9b
  7. Nov 27, 2019
  9. Nov 25, 2019
  11. Nov 23, 2019
    • Masahiro Yamada's avatar
      modpost: respect the previous export when 'exported twice' is warned · 7ef9ab3b
      Masahiro Yamada authored 
      

When 'exported twice' is warned, let sym_add_exported() return without
updating the symbol info. This respects the previous export, which is
ordered first in modules.order

This simplifies the code too.

Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      7ef9ab3b
    • Masahiro Yamada's avatar
      modpost: do not set ->preloaded for symbols from Module.symvers · e4b26c9f
      Masahiro Yamada authored 
      

Now that there is no overwrap between symbols from ELF files and
ones from Module.symvers.

So, the 'exported twice' warning should be reported irrespective
of where the symbol in question came from.

The exceptional case is external module; in some cases, we build
an external module to provide a different version/variant of the
corresponding in-kernel module, overriding the same set of exported
symbols.

You can see this use-case in upstream; tools/testing/nvdimm/libnvdimm.ko
replaces drivers/nvdimm/libnvdimm.ko in order to link it against mocked
version of core kernel symbols.

So, let's relax the 'exported twice' warning when building external
modules. The multiple export from external modules is warned only
when the previous one is from vmlinux or itself.

With this refactoring, the ugly preloading goes away.

Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      e4b26c9f
    • Masahiro Yamada's avatar
      modpost: stop symbol preloading for modversion CRC · 1743694e
      Masahiro Yamada authored 
      

It is complicated to add mocked-up symbols for pre-handling CRC.
Handle CRC after all the export symbols in the relevant module
are registered.

Call handle_modversion() after the handle_symbol() iteration.

In some cases, I see atand-alone __crc_* without __ksymtab_*.
For example, ARCH=arm allyesconfig produces __crc_ccitt_veneer and
__crc_itu_t_veneer. I guess they come from crc_ccitt, crc_itu_t,
respectively. Since __*_veneer are auto-generated symbols, just
ignore them.

Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      1743694e
    • Masahiro Yamada's avatar
      modpost: rename handle_modversions() to handle_symbol() · 9bd2a099
      Masahiro Yamada authored 
      

This function handles not only modversions, but also unresolved
symbols, export symbols, etc.

Rename it to a more proper function name.

While I was here, I also added the 'const' qualifier to *sym.

Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      9bd2a099
    • Masahiro Yamada's avatar
      modpost: refactor namespace_from_kstrtabns() to not hard-code section name · e84f9fbb
      Masahiro Yamada authored 
      

Currently, namespace_from_kstrtabns() relies on the fact that
namespace strings are recorded in the __ksymtab_strings section.
Actually, it is coded in include/linux/export.h, but modpost does
not need to hard-code the section name.

Elf_Sym::st_shndx holds the index of the relevant section. Using it is
a more portable way to get the namespace string.

Make namespace_from_kstrtabns() simply call sym_get_data(), and delete
the info->ksymtab_strings .

While I was here, I added more 'const' qualifiers to pointers.

Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      e84f9fbb
    • Masahiro Yamada's avatar
      modpost: add a helper to get data pointed by a symbol · afa0459d
      Masahiro Yamada authored 
      

When CONFIG_MODULE_REL_CRCS is enabled, the value of __crc_* is not
an absolute value, but the address to the CRC data embedded in the
.rodata section.

Getting the data pointed by the symbol value is somewhat complex.
Split it out into a new helper, sym_get_data().

I will reuse it to refactor namespace_from_kstrtabns() in the next
commit.

Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      afa0459d
  13. Nov 22, 2019
  15. Nov 14, 2019
  17. Nov 13, 2019
  19. Nov 11, 2019
    • Masahiro Yamada's avatar
      kbuild: rename any-prereq to newer-prereqs · eba19032
      Masahiro Yamada authored 
      

GNU Make manual says:

  $?
      The names of all the prerequisites that are newer than the target,
      with spaces between them.

To reflect this, rename any-prereq to newer-prereqs, which is clearer
and more intuitive.

Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      eba19032
    • Masahiro Yamada's avatar
      kbuild: drop $(wildcard $^) check in if_changed* for faster rebuild · 2d3b1b8f
      Masahiro Yamada authored 
      The incremental build of Linux kernel is pretty slow when lots of
objects are compiled. The rebuild of allmodconfig may take a few
minutes even when none of the objects needs to be rebuilt.

The time-consuming part in the incremental build is the evaluation of
if_changed* macros since they are used in the recipes to compile C and
assembly source files into objects.

I notice the following code in if_changed* is expensive:

  $(filter-out $(PHONY) $(wildcard $^),$^)

In the incremental build, every object has its .*.cmd file, which
contains the auto-generated list of included headers. So, $^ are
expanded into the long list of the source file + included headers,
and $(wildcard $^) checks whether they exist.

It may not be clear why this check exists there.

Here is the record of my research.

[1] The first code addition into Kbuild

This code dates back to 2002. It is the pre-git era. So, I copy-pasted
it from the historical git tree.

| commit 4a6db0791528c220655b063cf13fefc8470dbfee (HEAD)
| Author: Kai Germaschewski <kai@tp1.ruhr-uni-bochum.de>
| Date:   Mon Jun 17 00:22:37 2002 -0500
|
|     kbuild: Handle removed headers
|
|     New and old way to handle dependencies would choke when a file
|     #include'd by other files was removed, since the dependency on it was
|     still recorded, but since it was gone, make has no idea what to do about
|     it (and would complain with "No rule to make <file> ...")
|
|     We now add targets for all the previously included files, so make will
|     just ignore them if they disappear.
|
| diff --git a/Rules.make b/Rules.make
| index 6ef827d3df39..7db5301ea7db 100644
| --- a/Rules.make
| +++ b/Rules.make
| @@ -446,7 +446,7 @@ if_changed = $(if $(strip $? \
|  # execute the command and also postprocess generated .d dependencies
|  # file
|
| -if_changed_dep = $(if $(strip $? \
| +if_changed_dep = $(if $(strip $? $(filter-out FORCE $(wildcard $^),$^)\
|                           $(filter-out $(cmd_$(1)),$(cmd_$@))\
|                           $(filter-out $(cmd_$@),$(cmd_$(1)))),\
|         @set -e; \
| diff --git a/scripts/fixdep.c b/scripts/fixdep.c
| index b5d7bee8efc7..db45bd1888c0 100644
| --- a/scripts/fixdep.c
| +++ b/scripts/fixdep.c
| @@ -292,7 +292,7 @@ void parse_dep_file(void *map, size_t len)
|                 exit(1);
|         }
|         memcpy(s, m, p-m); s[p-m] = 0;
| -       printf("%s: \\\n", target);
| +       printf("deps_%s := \\\n", target);
|         m = p+1;
|
|         clear_config();
| @@ -314,7 +314,8 @@ void parse_dep_file(void *map, size_t len)
|                 }
|                 m = p + 1;
|         }
| -       printf("\n");
| +       printf("\n%s: $(deps_%s)\n\n", target, target);
| +       printf("$(deps_%s):\n", target);
|  }
|
|  void print_deps(void)

The "No rule to make <file> ..." error can be solved by passing -MP to
the compiler, but I think the detection of header removal is a good
feature. When a header is removed, all source files that previously
included it should be re-compiled. This makes sure we has correctly
got rid of #include directives of it.

This is also related with the behavior of $?. The GNU Make manual says:

  $?
      The names of all the prerequisites that are newer than the target,
      with spaces between them.

This does not explain whether a non-existent prerequisite is considered
to be newer than the target.

At this point of time, GNU Make 3.7x was used, where the $? did not
include non-existent prerequisites. Therefore,

  $(filter-out FORCE $(wildcard $^),$^)

was useful to detect the header removal, and to rebuild the related
objects if it is the case.

[2] Change of $? behavior

Later, the behavior of $? was changed (fixed) to include prerequisites
that did not exist.

First, GNU Make commit 64e16d6c00a5 ("Various changes getting ready for
the release of 3.81.") changed it, but in the release test of 3.81, it
turned out to break the kernel build.

See these:

 - http://lists.gnu.org/archive/html/bug-make/2006-03/msg00003.html
 - https://savannah.gnu.org/bugs/?16002
 - https://savannah.gnu.org/bugs/?16051

Then, GNU Make commit 6d8d9b74d9c5 ("Numerous updates to tests for
issues found on Cygwin and Windows.") reverted it for the 3.81 release
to give Linux kernel time to adjust to the new behavior.

After the 3.81 release, GNU Make commit 7595f38f62af ("Fixed a number
of documentation bugs, plus some build/install issues:") re-added it.

[3] Adjustment to the new $? behavior on Kbuild side

Meanwhile, the kernel build was changed by commit 4f193362 ("kbuild:
change kbuild to not rely on incorrect GNU make behavior") to adjust to
the new $? behavior.

[4] GNU Make 3.82 released in 2010

GNU Make 3.82 was the first release that integrated the correct $?
behavior. At this point, Kbuild dealt with GNU Make versions with
different $? behaviors.

 3.81 or older:
    $? does not contain any non-existent prerequisite.
    $(filter-out $(PHONY) $(wildcard $^),$^) was useful to detect
    removed include headers.

 3.82 or newer:
    $? contains non-existent prerequisites. When a header is removed,
    it appears in $?. $(filter-out $(PHONY) $(wildcard $^),$^) became
    a redundant check.

With the correct $? behavior, we could have dropped the expensive
check for 3.82 or later, but we did not. (Maybe nobody noticed this
optimization.)

[5] The .SECONDARY special target trips up $?

Some time later, I noticed $? did not work as expected under some
circumstances. As above, $? should contain non-existent prerequisites,
but the ones specified as SECONDARY do not appear in $?.

I asked this in GNU Make ML, and it seems a bug:

  https://lists.gnu.org/archive/html/bug-make/2019-01/msg00001.html



Since commit 8e9b61b2 ("kbuild: move .SECONDARY special target to
Kbuild.include"), all files, including headers listed in .*.cmd files,
are treated as secondary.

So, we are back into the incorrect $? behavior.

If we Kbuild want to react to the header removal, we need to keep
$(filter-out $(PHONY) $(wildcard $^),$^) but this makes the rebuild
so slow.

[Summary]

 - I believe noticing the header removal and recompiling related objects
   is a nice feature for the build system.

 - If $? worked correctly, $(filter-out $(PHONY),$?) would be enough
   to detect the header removal.

 - Currently, $? does not work correctly when used with .SECONDARY,
   and Kbuild is hit by this bug.

 - I filed a bug report for this, but not fixed yet as of writing.

 - Currently, the header removal is detected by the following expensive
   code:

    $(filter-out $(PHONY) $(wildcard $^),$^)

 - I do not want to revert commit 8e9b61b2 ("kbuild: move
   .SECONDARY special target to Kbuild.include"). Specifying
   .SECONDARY globally is clean, and it matches to the Kbuild policy.

This commit proactively removes the expensive check since it makes the
incremental build faster. A downside is Kbuild will no longer be able
to notice the header removal.

You can confirm it by the full-build followed by a header removal, and
then re-build.

  $ make defconfig all
    [ full build ]
  $ rm include/linux/device.h
  $ make
    CALL    scripts/checksyscalls.sh
    CALL    scripts/atomic/check-atomics.sh
    DESCEND  objtool
    CHK     include/generated/compile.h
  Kernel: arch/x86/boot/bzImage is ready  (#11)
    Building modules, stage 2.
    MODPOST 12 modules

Previously, Kbuild noticed a missing header and emits a build error.
Now, Kbuild is fine with it. This is an unusual corner-case, not a big
deal. Once the $? bug is fixed in GNU Make, everything will work fine.

Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      2d3b1b8f
    • Masahiro Yamada's avatar
      modpost: remove unneeded local variable in contains_namespace() · 76b54cf0
      Masahiro Yamada authored 
      

The local variable, ns_entry, is unneeded.

While I was here, I also cleaned up the comparison with NULL or 0.

Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: default avatarMatthias Maennich <maennich@google.com>
      76b54cf0
    • Masahiro Yamada's avatar
      scripts/nsdeps: support nsdeps for external module builds · bc35d4bd
      Masahiro Yamada authored 
      

scripts/nsdeps is written to take care of only in-tree modules.
Perhaps, this is not a bug, but just a design. At least,
Documentation/core-api/symbol-namespaces.rst focuses on in-tree modules.

Having said that, some people already tried nsdeps for external modules.
So, it would be nice to support it.

Reported-by: default avatarSteve French <smfrench@gmail.com>
Reported-by: default avatarJessica Yu <jeyu@kernel.org>
Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
Tested-by: default avatarJessica Yu <jeyu@kernel.org>
Acked-by: default avatarJessica Yu <jeyu@kernel.org>
Reviewed-by: default avatarMatthias Maennich <maennich@google.com>
Tested-by: default avatarMatthias Maennich <maennich@google.com>
      bc35d4bd
    • Masahiro Yamada's avatar
      modpost: dump missing namespaces into a single modules.nsdeps file · bbc55bde
      Masahiro Yamada authored 
      The modpost, with the -d option given, generates per-module .ns_deps
files.

Kbuild generates per-module .mod files to carry module information.
This is convenient because Make handles multiple jobs in parallel
when the -j option is given.

On the other hand, the modpost always runs as a single thread.
I do not see a strong reason to produce separate .ns_deps files.

This commit changes the modpost to generate just one file,
modules.nsdeps, each line of which has the following format:

  <module_name>: <list of missing namespaces>

Please note it contains *missing* namespaces instead of required ones.
So, modules.nsdeps is empty if the namespace dependency is all good.

This will work more efficiently because spatch will no longer process
already imported namespaces. I removed the '(if needed)' from the
nsdeps log since spatch is invoked only when needed.

This also solves the stale .ns_deps problem reported by Jessica Yu:

  https://lkml.org/lkml/2019/10/28/467



Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
Tested-by: default avatarJessica Yu <jeyu@kernel.org>
Acked-by: default avatarJessica Yu <jeyu@kernel.org>
Reviewed-by: default avatarMatthias Maennich <maennich@google.com>
Tested-by: default avatarMatthias Maennich <maennich@google.com>
      bbc55bde