-r325627 of head: mergemaster: Creating objdir after objdir after .

Post by Mark Millard
~/sys_build_scripts.aarch64-host/mergemaster_cortexA53-aarch64-host.sh -FUPi -D/mnt
# more ~/sys_build_scripts.aarch64-host/mergemaster_cortexA53-aarch64-host.sh
kldload -n filemon && \
script ~/sys_typescripts/typescript_mergemaster_cortexA53_clang_bootstrap_clang-aarch64-host-$(date +%Y-%m-%d:%H:%M:%S) \
env __MAKE_CONF="/root/src.configs/make.conf" SRCCONF="/dev/null" SRC_ENV_CONF="/root/src.configs/src.conf.cortexA53-clang-bootstrap.aarch64-host" \
mergemaster -A aarch64 $*
in a context where /usr/obj/usr does not exist
Script started, output file is /root/sys_typescripts/typescript_mergemaster_cortexA53_clang_bootstrap_clang-aarch64-host-2017-11-09:23:57:04
*** Creating the temporary root environment in /var/tmp/temproot
*** /var/tmp/temproot ready for use
*** Creating and populating directory structure in /var/tmp/temproot
[Creating objdir /usr/obj/usr/src/arm64.aarch64/share/termcap...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/etc/syslog.d...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/usr.sbin/rmt...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/etc/pam.d...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/csu...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/csu/aarch64...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libc...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libc_nonshared...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libcompiler_rt...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libclang_rt...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libc++...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libc++experimental...]
[Creating nested objdir /usr/obj/usr/src/arm64.aarch64/lib/libc++experimental/filesystem...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libcxxrt...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libelf...]
[Creating nested objdir /usr/obj/usr/src/arm64.aarch64/lib/libelf/sys...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/msun...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libalias...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/lib/libalias/libalias...]
. . . (long list) . . .
[Creating objdir /usr/obj/usr/src/arm64.aarch64/usr.sbin/wpa/hostapd...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/usr.sbin/wpa/hostapd_cli...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/usr.sbin/wpa/ndis_events...]
So a /usr/obj/usr/src/arm64.aarch64/ directory tree
ends up being created.

Hah, not what we want. I'll fix that.

However from reading mergemaster.sh it seems that _at least_
/usr/obj/usr/src/etc/sendmail would be created before my changes. Can
someone confirm that on stable/ or something?

Post by Mark Millard
(MAKEOBJDIRPREFIX= does control the path-prefix used
if specified in the env list before mergemaster.)
===
Mark Millard
markmi at dsl-only.net

--
Regards,
Bryan Drewery

Bryan Drewery

2017-11-10 16:30:44 UTC

Hah, not what we want. I'll fix that.

In fact it's similar to my META_MODE whitelist in the top-level
Makefile. There's quite a few targets we don't care for AUTO_OBJ on,
like distribute*, installworld, installkernel, etc.

Post by Bryan Drewery
However from reading mergemaster.sh it seems that _at least_
/usr/obj/usr/src/etc/sendmail would be created before my changes. Can
someone confirm that on stable/ or something?

Post by Mark Millard
(MAKEOBJDIRPREFIX= does control the path-prefix used
if specified in the env list before mergemaster.)
===
Mark Millard
markmi at dsl-only.net

--
Regards,
Bryan Drewery

Mark Millard

2017-11-11 08:51:06 UTC

Hah, not what we want. I'll fix that.

In fact it's similar to my META_MODE whitelist in the top-level
Makefile. There's quite a few targets we don't care for AUTO_OBJ on,
like distribute*, installworld, installkernel, etc.

r325697 should fix it.

Most of the messages are gone in -r325700 . But there was:

*** Creating the temporary root environment in /var/tmp/temproot
*** /var/tmp/temproot ready for use
*** Creating and populating directory structure in /var/tmp/temproot

[Creating objdir /usr/obj/usr/src/arm64.aarch64...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/etc...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/etc/sendmail...]

(No more objdir lines after that.)

Post by Mark Millard
(MAKEOBJDIRPREFIX= does control the path-prefix used
if specified in the env list before mergemaster.)

===
Mark Millard
markmi at dsl-only.net

Mark Millard

2017-11-11 08:58:14 UTC

===
Mark Millard

Hah, not what we want. I'll fix that.

In fact it's similar to my META_MODE whitelist in the top-level
Makefile. There's quite a few targets we don't care for AUTO_OBJ on,
like distribute*, installworld, installkernel, etc.

r325697 should fix it.

*** Creating the temporary root environment in /var/tmp/temproot
*** /var/tmp/temproot ready for use
*** Creating and populating directory structure in /var/tmp/temproot
[Creating objdir /usr/obj/usr/src/arm64.aarch64...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/etc...]
[Creating objdir /usr/obj/usr/src/arm64.aarch64/etc/sendmail...]
(No more objdir lines after that.)

Post by Mark Millard
(MAKEOBJDIRPREFIX= does control the path-prefix used
if specified in the env list before mergemaster.)

FYI a check-old (with MAKEOBJDIRPREFIX="/usr/obj/cortexA53_clang/arm64.aarch64"
in use) got me a:

[Creating objdir /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64...]

(Just the one objdir line.)

/usr/obj/cortexA53_clang/ did not exist when check-old
was started.

So, the activity does track MAKEOBJDIRPREFIX when it
is in use.

===
Mark Millard
markmi at dsl-only.net

Bryan Drewery

2017-11-11 16:47:01 UTC

Hah, not what we want. I'll fix that.

In fact it's similar to my META_MODE whitelist in the top-level
Makefile. There's quite a few targets we don't care for AUTO_OBJ on,
like distribute*, installworld, installkernel, etc.

r325697 should fix it.

Yea this is expected. Mergemaster runs ‘make obj’ in etc/.

The top-level check-old objdir creation is unavoidable right now... you can use -DNO_OBJ if you want to avoid it.

Post by Mark Millard
(MAKEOBJDIRPREFIX= does control the path-prefix used
if specified in the env list before mergemaster.)

===
Mark Millard
markmi at dsl-only.net

Mark Millard

2017-11-11 22:25:06 UTC

Post by Bryan Drewery
. . .
In fact it's similar to my META_MODE whitelist in the top-level
Makefile. There's quite a few targets we don't care for AUTO_OBJ on,
like distribute*, installworld, installkernel, etc.

r325697 should fix it.

Yea this is expected. Mergemaster runs ‘make obj’ in etc/.

Hmm. I looking I see the:

${MM_MAKE} _obj SUBDIR_OVERRIDE=etc >/dev/null &&
${MM_MAKE} everything SUBDIR_OVERRIDE=etc >/dev/null &&

Its too bad that the mergemaster man page makes no reference
to depending on MAKEOBJDIRPREFIX (or its default) and the
tree contents that it points to. If one has more than one
tree around then one should be picking the right one --but
nothing in the man page suggests that.

It also would not play well with not having that build tree
available at the time of a mergemaster.

The 20130425 UPDATING entry does note use of
MAKEOBJDIRPREFIX. It notes that mergemaster makes use of the
specific, bootstrapped mtree and install. Does that mean
that a cross-build needs mergemaster to be executed on
the cross-build host instead of on the target system? I do
not see the man page as well-covering such questions for
the proper usage technique.

Dependency on /usr/src (by default or an alternate via -m) is
clear from the man page. So, having a proper vintage of such
and having mergemaster use it was clear. But such seems to not
be sufficient, which was not clear.

Post by Bryan Drewery
The top-level check-old objdir creation is unavoidable right now... you can use -DNO_OBJ if you want to avoid it.

Post by Mark Millard
(MAKEOBJDIRPREFIX= does control the path-prefix used
if specified in the env list before mergemaster.)

===
Mark Millard
markmi at dsl-only.net

Mark Millard

2017-11-16 21:15:50 UTC

===
Mark Millard

. . .

Can you test this patch please in context of this problem please?
It resolves read-only objdirs and should avoid more of the objdir
creations at mergemaster/installworld time. It probably will still
create the etc/sendmail one.
https://people.freebsd.org/~bdrewery/patches/top-level-objdirs.diff

. . .

I applied the patch.

The 3 experiments below only showed the sendmail related
"Creating objdir" lines, no other "Creating objdir" lines.

Based on using ( /tmp/usr/obj/armv7_clang not existing at the time):

# more ~/sys_build_scripts.armv7-host/mergemaster_armv7-armv7-host.sh
kldload -n filemon && \
script ~/sys_typescripts/typescript_mergemaster_armv7_clang_bootstrap_clang-armv7-host-$(date +%Y-%m-%d:%H:%M:%S) \
env __MAKE_CONF="/root/src.configs/make.conf" SRCCONF="/dev/null" SRC_ENV_CONF="/root/src.configs/src.conf.armv7-clang-bootstrap.armv7-host" \
MAKEOBJDIRPREFIX="/tmp/usr/obj/armv7_clang/arm.armv7" \
mergemaster -A armv7 $*

The merge master produced just the 3 objdir lines related to sendmail:

# ~/sys_build_scripts.armv7-host/mergemaster_armv7-armv7-host.sh -FUPi
Script started, output file is /root/sys_typescripts/typescript_mergemaster_armv7_clang_bootstrap_clang-armv7-host-2017-11-16:12:31:48

*** Creating the temporary root environment in /var/tmp/temproot
*** /var/tmp/temproot ready for use
*** Creating and populating directory structure in /var/tmp/temproot

[Creating objdir /tmp/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7...]
[Creating objdir /tmp/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/etc...]
[Creating objdir /tmp/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/etc/sendmail...]

. . . (other stuff) . . .

Based on ( /usr/obj/bpim3_clang not existing at the time [renamed]):

# more ~/sys_build_scripts.armv7-host/make_bpim3_nodebug_clang_bootstrap-armv7-host.sh
kldload -n filemon && \
script ~/sys_typescripts/typescript_make_bpim3_nodebug_clang_bootstrap-armv7-host-$(date +%Y-%m-%d:%H:%M:%S) \
env __MAKE_CONF="/root/src.configs/make.conf" SRCCONF="/dev/null" SRC_ENV_CONF="/root/src.configs/src.conf.armv7-clang-bootstrap.armv7-host" \
WITH_META_MODE=yes \
WORLD_FLAGS="${WORLD_FLAGS} UBLDR_LOADADDR=0x42000000" \
MAKEOBJDIRPREFIX="/usr/obj/bpim3_clang/arm.armv7" \
make $*

the check-old did not produce any objdir lines:

# ~/sys_build_scripts.armv7-host/make_bpim3_nodebug_clang_bootstrap-armv7-host.sh check-old
Script started, output file is /root/sys_typescripts/typescript_make_bpim3_nodebug_clang_bootstrap-armv7-host-2017-11-16:12:44:43

Checking for old files
Checking for old libraries
Checking for old directories

To remove old files and directories run 'make delete-old'.
To remove old libraries run 'make delete-old-libs'.

Script done, output file is /root/sys_typescripts/typescript_make_bpim3_nodebug_clang_bootstrap-armv7-host-2017-11-16:12:44:43

As for installworld installkernel creating a new directory tree. . .
(same .sh file for check-old above but with /usr/obj/bpim3_clang back in place)

# ~/sys_build_scripts.armv7-host/make_bpim3_nodebug_clang_bootstrap-armv7-host.sh -j4 installworld installkernel DESTDIR=/usr/obj/DESTDIRs/clang-armv7-installworld-objdir-test
Script started, output file is /root/sys_typescripts/typescript_make_bpim3_nodebug_clang_bootstrap-armv7-host-2017-11-16:12:54:04
--- installworld ---
make[1]: "/usr/obj/bpim3_clang/arm.armv7/usr/src/arm.armv7/compiler-metadata.mk" line 1: Using cached compiler metadata from build at bpim3 on Tue Nov 14 22:25:57 PST 2017
--- __installcheck_UGID ---
--- installworld ---
. . .
--- _kmodinstall ---
install -T release -o root -g wheel -m 555 g_mouse.ko /usr/obj/DESTDIRs/clang-armv7-installworld-objdir-test/boot/kernel/
install -T debug -o root -g wheel -m 555 g_mouse.ko.debug /usr/obj/DESTDIRs/clang-armv7-installworld-objdir-test/usr/lib/debug/boot/kernel/
--- afterinstall ---
kldxref /usr/obj/DESTDIRs/clang-armv7-installworld-objdir-test/boot/kernel

Script done, output file is /root/sys_typescripts/typescript_make_bpim3_nodebug_clang_bootstrap-armv7-host-2017-11-16:12:54:04

# grep "Creating objdir" /root/sys_typescripts/typescript_make_bpim3_nodebug_clang_bootstrap-armv7-host-2017-11-16:12:54:04 | more
#

(So: None found in the typescript file.)

===
Mark Millard
markmi at dsl-only.net

Bryan Drewery

2017-11-16 21:53:38 UTC