Discussion:
LUA boot loader coming very soon
(too old to reply)
Warner Losh
2018-02-12 15:27:27 UTC
Permalink
Greetings,

As you may know, the Lua (http://www.lua.org) boot loader has been in the
works for some time. It started out life as a GSoC in 2014 by Pedro Souza
mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch to
try to integrate it. I rebased that effort into a github branch which Pedro
Arthur fixed up. Over the past year, I've been cleaning up the boot loader
for other reasons, and found the time was ripe to start integrating this
into the tree. However, those integration efforts have taken a while as my
day-job work on the boot loader took priority. In the mean time, Ed Maste
and the FreeBSD Foundation funded Zakary Nafziger to enhance the original
GSoC Lua scripts to bring it closer to parity with the evolution of the
FORTH menu system since the GSoC project started.

I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later today. This
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a few weeks
and I didn't want to delay pushing this into the tree for that.

To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do this,
you can do it in src/stand and install the result (be sure to have the
options for both the build and the install). This will replace your current
/boot/loader that is scripted with FORTH to one that's scripted with Lua.
It will install the lua scripts in /boot/lua. The boot is scripted with
/boot/lua/loader.lua instead of /boot/loader.rc. You are strongly advised
to create a backup copy of /boot/loader before testing (eg cp /boot/loader
/boot/loader_forth), since you'll need to boot that from boot2 if something
goes wrong. I've tested it extensively, though, with userboot.so and it's
test program, so all the initial kinks of finding the lua scripts, etc have
been worked out.

While it's possible to build all the /boot/loader variants with Lua, I've
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the instructions for
testing some of them may be rather tedious (especially UEFI, if you want a
simple path to back out). Since there's not been full convergence testing,
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.

I put MFC after a month on the commit. Due to the heroic (dare I say
almost crazy) work of Kyle Evans on merging all the revs from -current to
11, I'm planning a MFC to 11 after the co-existence issues are hammered
out. In 11, FORTH will be the default, and Lua will be built by default,
but users will have to do something to use it. 12, both FORTH and Lua will
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine we'll make
the switch to Lua by default shortly after that. In 13, FORTH will be
removed unless there's a really really compelling case made to keep it.

So please give it a spin and give me any feedback, documentation updates
and/or bug fixes. I'm especially interested in reviews from people that
have embedded Lua in other projects or experts in Lua that can improve the
robustness of the menu code.

Warner
Kris Moore
2018-02-12 16:26:33 UTC
Permalink
Post by Warner Losh
Greetings,
As you may know, the Lua (http://www.lua.org) boot loader has been in the
works for some time. It started out life as a GSoC in 2014 by Pedro Souza
mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch to
try to integrate it. I rebased that effort into a github branch which Pedro
Arthur fixed up. Over the past year, I've been cleaning up the boot loader
for other reasons, and found the time was ripe to start integrating this
into the tree. However, those integration efforts have taken a while as my
day-job work on the boot loader took priority. In the mean time, Ed Maste
and the FreeBSD Foundation funded Zakary Nafziger to enhance the original
GSoC Lua scripts to bring it closer to parity with the evolution of the
FORTH menu system since the GSoC project started.
I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later today. This
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a few weeks
and I didn't want to delay pushing this into the tree for that.
To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do this,
you can do it in src/stand and install the result (be sure to have the
options for both the build and the install). This will replace your current
/boot/loader that is scripted with FORTH to one that's scripted with Lua.
It will install the lua scripts in /boot/lua. The boot is scripted with
/boot/lua/loader.lua instead of /boot/loader.rc. You are strongly advised
to create a backup copy of /boot/loader before testing (eg cp /boot/loader
/boot/loader_forth), since you'll need to boot that from boot2 if something
goes wrong. I've tested it extensively, though, with userboot.so and it's
test program, so all the initial kinks of finding the lua scripts, etc have
been worked out.
While it's possible to build all the /boot/loader variants with Lua, I've
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the instructions for
testing some of them may be rather tedious (especially UEFI, if you want a
simple path to back out). Since there's not been full convergence testing,
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.
I put MFC after a month on the commit. Due to the heroic (dare I say
almost crazy) work of Kyle Evans on merging all the revs from -current to
11, I'm planning a MFC to 11 after the co-existence issues are hammered
out. In 11, FORTH will be the default, and Lua will be built by default,
but users will have to do something to use it. 12, both FORTH and Lua will
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine we'll make
the switch to Lua by default shortly after that. In 13, FORTH will be
removed unless there's a really really compelling case made to keep it.
So please give it a spin and give me any feedback, documentation updates
and/or bug fixes. I'm especially interested in reviews from people that
have embedded Lua in other projects or experts in Lua that can improve the
robustness of the menu code.
Warner
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current
Warner & Co,

Great job all! Been wanting this for years, very excited to see it start
landing :)

One question, if we switch to Lua loader in TrueOS, do you know if the
ZFS boot-environment menus have been implemented / tested yet?
--
Kris Moore
Vice President of Engineering
iXsystems
Enterprise Storage & Servers Driven By Open Source
Warner Losh
2018-02-12 16:31:58 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Greetings,
As you may know, the Lua (http://www.lua.org) boot loader has been in
the
Post by Warner Losh
works for some time. It started out life as a GSoC in 2014 by Pedro Souza
mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch to
try to integrate it. I rebased that effort into a github branch which
Pedro
Post by Warner Losh
Arthur fixed up. Over the past year, I've been cleaning up the boot
loader
Post by Warner Losh
for other reasons, and found the time was ripe to start integrating this
into the tree. However, those integration efforts have taken a while as
my
Post by Warner Losh
day-job work on the boot loader took priority. In the mean time, Ed Maste
and the FreeBSD Foundation funded Zakary Nafziger to enhance the original
GSoC Lua scripts to bring it closer to parity with the evolution of the
FORTH menu system since the GSoC project started.
I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later today. This
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a few weeks
and I didn't want to delay pushing this into the tree for that.
To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do this,
you can do it in src/stand and install the result (be sure to have the
options for both the build and the install). This will replace your
current
Post by Warner Losh
/boot/loader that is scripted with FORTH to one that's scripted with Lua.
It will install the lua scripts in /boot/lua. The boot is scripted with
/boot/lua/loader.lua instead of /boot/loader.rc. You are strongly advised
to create a backup copy of /boot/loader before testing (eg cp
/boot/loader
Post by Warner Losh
/boot/loader_forth), since you'll need to boot that from boot2 if
something
Post by Warner Losh
goes wrong. I've tested it extensively, though, with userboot.so and it's
test program, so all the initial kinks of finding the lua scripts, etc
have
Post by Warner Losh
been worked out.
While it's possible to build all the /boot/loader variants with Lua, I've
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the instructions
for
Post by Warner Losh
testing some of them may be rather tedious (especially UEFI, if you want
a
Post by Warner Losh
simple path to back out). Since there's not been full convergence
testing,
Post by Warner Losh
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.
I put MFC after a month on the commit. Due to the heroic (dare I say
almost crazy) work of Kyle Evans on merging all the revs from -current to
11, I'm planning a MFC to 11 after the co-existence issues are hammered
out. In 11, FORTH will be the default, and Lua will be built by default,
but users will have to do something to use it. 12, both FORTH and Lua
will
Post by Warner Losh
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine we'll make
the switch to Lua by default shortly after that. In 13, FORTH will be
removed unless there's a really really compelling case made to keep it.
So please give it a spin and give me any feedback, documentation updates
and/or bug fixes. I'm especially interested in reviews from people that
have embedded Lua in other projects or experts in Lua that can improve
the
Post by Warner Losh
robustness of the menu code.
Warner
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current
freebsd.org"
Warner & Co,
Great job all! Been wanting this for years, very excited to see it start
landing :)
One question, if we switch to Lua loader in TrueOS, do you know if the
ZFS boot-environment menus have been implemented / tested yet?
I don't think that they have been implemented yet. I've certainly not
tested them.

Warner
Kris Moore
2018-02-12 16:45:28 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Greetings,
As you may know, the Lua (http://www.lua.org) boot loader has
been in the
Post by Warner Losh
works for some time. It started out life as a GSoC in 2014 by
Pedro Souza
Post by Warner Losh
mentored by Wojciech A. Koszek. Rui Paulo created a svn project
branch to
Post by Warner Losh
try to integrate it. I rebased that effort into a github branch
which Pedro
Post by Warner Losh
Arthur fixed up. Over the past year, I've been cleaning up the
boot loader
Post by Warner Losh
for other reasons, and found the time was ripe to start
integrating this
Post by Warner Losh
into the tree. However, those integration efforts have taken a
while as my
Post by Warner Losh
day-job work on the boot loader took priority. In the mean time,
Ed Maste
Post by Warner Losh
and the FreeBSD Foundation funded Zakary Nafziger to enhance the
original
Post by Warner Losh
GSoC Lua scripts to bring it closer to parity with the evolution
of the
Post by Warner Losh
FORTH menu system since the GSoC project started.
I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later
today. This
Post by Warner Losh
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a
few weeks
Post by Warner Losh
and I didn't want to delay pushing this into the tree for that.
To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to
do this,
Post by Warner Losh
you can do it in src/stand and install the result (be sure to
have the
Post by Warner Losh
options for both the build and the install). This will replace
your current
Post by Warner Losh
/boot/loader that is scripted with FORTH to one that's scripted
with Lua.
Post by Warner Losh
It will install the lua scripts in /boot/lua. The boot is
scripted with
Post by Warner Losh
/boot/lua/loader.lua instead of /boot/loader.rc. You are
strongly advised
Post by Warner Losh
to create a backup copy of /boot/loader before testing (eg cp
/boot/loader
Post by Warner Losh
/boot/loader_forth), since you'll need to boot that from boot2
if something
Post by Warner Losh
goes wrong. I've tested it extensively, though, with userboot.so
and it's
Post by Warner Losh
test program, so all the initial kinks of finding the lua
scripts, etc have
Post by Warner Losh
been worked out.
While it's possible to build all the /boot/loader variants with
Lua, I've
Post by Warner Losh
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the
instructions for
Post by Warner Losh
testing some of them may be rather tedious (especially UEFI, if
you want a
Post by Warner Losh
simple path to back out). Since there's not been full
convergence testing,
Post by Warner Losh
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.
I put  MFC after a month on the commit. Due to the heroic (dare
I say
Post by Warner Losh
almost crazy) work of Kyle Evans on merging all the revs from
-current to
Post by Warner Losh
11, I'm planning a MFC to 11 after the co-existence issues are
hammered
Post by Warner Losh
out. In 11, FORTH will be the default, and Lua will  be built by
default,
Post by Warner Losh
but users will have to do something to use it. 12, both FORTH
and Lua will
Post by Warner Losh
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine
we'll make
Post by Warner Losh
the switch to Lua by default shortly after that. In 13, FORTH
will be
Post by Warner Losh
removed unless there's a really really compelling case made to
keep it.
Post by Warner Losh
So please give it a spin and give me any feedback, documentation
updates
Post by Warner Losh
and/or bug fixes. I'm especially interested in reviews from
people that
Post by Warner Losh
have embedded Lua in other projects or experts in Lua that can
improve the
Post by Warner Losh
robustness of the menu code.
Warner
_______________________________________________
mailing list
Post by Warner Losh
https://lists.freebsd.org/mailman/listinfo/freebsd-current
<https://lists.freebsd.org/mailman/listinfo/freebsd-current>
Post by Warner Losh
To unsubscribe, send any mail to
Warner & Co,
Great job all! Been wanting this for years, very excited to see it start
landing :)
One question, if we switch to Lua loader in TrueOS, do you know if the
ZFS boot-environment menus have been implemented / tested yet?
I don't think that they have been implemented yet. I've certainly not
tested them.
Warner 
Ok, I'll have to mess about with it. I recall there was some library
which created to expose ZFS in limited functionality over to the FORTH
code. That's probably a good place to start and see if that still works
and Lua can probe the pool for BE information.
--
Kris Moore
Vice President of Engineering
iXsystems
Enterprise Storage & Servers Driven By Open Source
Warner Losh
2018-02-12 16:51:04 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
Greetings,
As you may know, the Lua (http://www.lua.org) boot loader has been in
the
Post by Warner Losh
works for some time. It started out life as a GSoC in 2014 by Pedro
Souza
Post by Warner Losh
mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch
to
Post by Warner Losh
try to integrate it. I rebased that effort into a github branch which
Pedro
Post by Warner Losh
Arthur fixed up. Over the past year, I've been cleaning up the boot
loader
Post by Warner Losh
for other reasons, and found the time was ripe to start integrating this
into the tree. However, those integration efforts have taken a while as
my
Post by Warner Losh
day-job work on the boot loader took priority. In the mean time, Ed
Maste
Post by Warner Losh
and the FreeBSD Foundation funded Zakary Nafziger to enhance the
original
Post by Warner Losh
GSoC Lua scripts to bring it closer to parity with the evolution of the
FORTH menu system since the GSoC project started.
I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later today. This
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a few
weeks
Post by Warner Losh
and I didn't want to delay pushing this into the tree for that.
To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do
this,
Post by Warner Losh
you can do it in src/stand and install the result (be sure to have the
options for both the build and the install). This will replace your
current
Post by Warner Losh
/boot/loader that is scripted with FORTH to one that's scripted with
Lua.
Post by Warner Losh
It will install the lua scripts in /boot/lua. The boot is scripted with
/boot/lua/loader.lua instead of /boot/loader.rc. You are strongly
advised
Post by Warner Losh
to create a backup copy of /boot/loader before testing (eg cp
/boot/loader
Post by Warner Losh
/boot/loader_forth), since you'll need to boot that from boot2 if
something
Post by Warner Losh
goes wrong. I've tested it extensively, though, with userboot.so and
it's
Post by Warner Losh
test program, so all the initial kinks of finding the lua scripts, etc
have
Post by Warner Losh
been worked out.
While it's possible to build all the /boot/loader variants with Lua,
I've
Post by Warner Losh
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the instructions
for
Post by Warner Losh
testing some of them may be rather tedious (especially UEFI, if you
want a
Post by Warner Losh
simple path to back out). Since there's not been full convergence
testing,
Post by Warner Losh
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.
I put MFC after a month on the commit. Due to the heroic (dare I say
almost crazy) work of Kyle Evans on merging all the revs from -current
to
Post by Warner Losh
11, I'm planning a MFC to 11 after the co-existence issues are hammered
out. In 11, FORTH will be the default, and Lua will be built by
default,
Post by Warner Losh
but users will have to do something to use it. 12, both FORTH and Lua
will
Post by Warner Losh
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine we'll
make
Post by Warner Losh
the switch to Lua by default shortly after that. In 13, FORTH will be
removed unless there's a really really compelling case made to keep it.
So please give it a spin and give me any feedback, documentation updates
and/or bug fixes. I'm especially interested in reviews from people that
have embedded Lua in other projects or experts in Lua that can improve
the
Post by Warner Losh
robustness of the menu code.
Warner
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current
reebsd.org"
Warner & Co,
Great job all! Been wanting this for years, very excited to see it start
landing :)
One question, if we switch to Lua loader in TrueOS, do you know if the
ZFS boot-environment menus have been implemented / tested yet?
I don't think that they have been implemented yet. I've certainly not
tested them.
Warner
Ok, I'll have to mess about with it. I recall there was some library which
created to expose ZFS in limited functionality over to the FORTH code.
That's probably a good place to start and see if that still works and Lua
can probe the pool for BE information.
OK. There's no such functionality in FORTH in the tree today. However, zfs
sets a number of env vars that loader.getenv() can get in lua.

Warner
Warner Losh
2018-02-12 16:58:56 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
Greetings,
As you may know, the Lua (http://www.lua.org) boot loader has been in
the
Post by Warner Losh
works for some time. It started out life as a GSoC in 2014 by Pedro
Souza
Post by Warner Losh
mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch
to
Post by Warner Losh
try to integrate it. I rebased that effort into a github branch which
Pedro
Post by Warner Losh
Arthur fixed up. Over the past year, I've been cleaning up the boot
loader
Post by Warner Losh
for other reasons, and found the time was ripe to start integrating
this
Post by Warner Losh
into the tree. However, those integration efforts have taken a while
as my
Post by Warner Losh
day-job work on the boot loader took priority. In the mean time, Ed
Maste
Post by Warner Losh
and the FreeBSD Foundation funded Zakary Nafziger to enhance the
original
Post by Warner Losh
GSoC Lua scripts to bring it closer to parity with the evolution of the
FORTH menu system since the GSoC project started.
I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later today. This
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a few
weeks
Post by Warner Losh
and I didn't want to delay pushing this into the tree for that.
To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do
this,
Post by Warner Losh
you can do it in src/stand and install the result (be sure to have the
options for both the build and the install). This will replace your
current
Post by Warner Losh
/boot/loader that is scripted with FORTH to one that's scripted with
Lua.
Post by Warner Losh
It will install the lua scripts in /boot/lua. The boot is scripted with
/boot/lua/loader.lua instead of /boot/loader.rc. You are strongly
advised
Post by Warner Losh
to create a backup copy of /boot/loader before testing (eg cp
/boot/loader
Post by Warner Losh
/boot/loader_forth), since you'll need to boot that from boot2 if
something
Post by Warner Losh
goes wrong. I've tested it extensively, though, with userboot.so and
it's
Post by Warner Losh
test program, so all the initial kinks of finding the lua scripts, etc
have
Post by Warner Losh
been worked out.
While it's possible to build all the /boot/loader variants with Lua,
I've
Post by Warner Losh
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the
instructions for
Post by Warner Losh
testing some of them may be rather tedious (especially UEFI, if you
want a
Post by Warner Losh
simple path to back out). Since there's not been full convergence
testing,
Post by Warner Losh
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.
I put MFC after a month on the commit. Due to the heroic (dare I say
almost crazy) work of Kyle Evans on merging all the revs from -current
to
Post by Warner Losh
11, I'm planning a MFC to 11 after the co-existence issues are hammered
out. In 11, FORTH will be the default, and Lua will be built by
default,
Post by Warner Losh
but users will have to do something to use it. 12, both FORTH and Lua
will
Post by Warner Losh
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine we'll
make
Post by Warner Losh
the switch to Lua by default shortly after that. In 13, FORTH will be
removed unless there's a really really compelling case made to keep it.
So please give it a spin and give me any feedback, documentation
updates
Post by Warner Losh
and/or bug fixes. I'm especially interested in reviews from people that
have embedded Lua in other projects or experts in Lua that can improve
the
Post by Warner Losh
robustness of the menu code.
Warner
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current
reebsd.org"
Warner & Co,
Great job all! Been wanting this for years, very excited to see it start
landing :)
One question, if we switch to Lua loader in TrueOS, do you know if the
ZFS boot-environment menus have been implemented / tested yet?
I don't think that they have been implemented yet. I've certainly not
tested them.
Warner
Ok, I'll have to mess about with it. I recall there was some library
which created to expose ZFS in limited functionality over to the FORTH
code. That's probably a good place to start and see if that still works and
Lua can probe the pool for BE information.
OK. There's no such functionality in FORTH in the tree today. However, zfs
sets a number of env vars that loader.getenv() can get in lua.
There's also a 'reloadbe' boot loader command that's independent of
interpreter. Sadly, it's reimplemented three times identically, with a
single comment being the only difference. Now that's something that's ripe
for cleanup...

Warner
John Baldwin
2018-02-12 17:12:27 UTC
Permalink
Post by Warner Losh
Greetings,
As you may know, the Lua (http://www.lua.org) boot loader has been in the
works for some time. It started out life as a GSoC in 2014 by Pedro Souza
mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch to
try to integrate it. I rebased that effort into a github branch which Pedro
Arthur fixed up. Over the past year, I've been cleaning up the boot loader
for other reasons, and found the time was ripe to start integrating this
into the tree. However, those integration efforts have taken a while as my
day-job work on the boot loader took priority. In the mean time, Ed Maste
and the FreeBSD Foundation funded Zakary Nafziger to enhance the original
GSoC Lua scripts to bring it closer to parity with the evolution of the
FORTH menu system since the GSoC project started.
I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later today. This
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a few weeks
and I didn't want to delay pushing this into the tree for that.
To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do this,
you can do it in src/stand and install the result (be sure to have the
options for both the build and the install). This will replace your current
/boot/loader that is scripted with FORTH to one that's scripted with Lua.
It will install the lua scripts in /boot/lua. The boot is scripted with
/boot/lua/loader.lua instead of /boot/loader.rc. You are strongly advised
to create a backup copy of /boot/loader before testing (eg cp /boot/loader
/boot/loader_forth), since you'll need to boot that from boot2 if something
goes wrong. I've tested it extensively, though, with userboot.so and it's
test program, so all the initial kinks of finding the lua scripts, etc have
been worked out.
While it's possible to build all the /boot/loader variants with Lua, I've
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the instructions for
testing some of them may be rather tedious (especially UEFI, if you want a
simple path to back out). Since there's not been full convergence testing,
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.
I put MFC after a month on the commit. Due to the heroic (dare I say
almost crazy) work of Kyle Evans on merging all the revs from -current to
11, I'm planning a MFC to 11 after the co-existence issues are hammered
out. In 11, FORTH will be the default, and Lua will be built by default,
but users will have to do something to use it. 12, both FORTH and Lua will
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine we'll make
the switch to Lua by default shortly after that. In 13, FORTH will be
removed unless there's a really really compelling case made to keep it.
So please give it a spin and give me any feedback, documentation updates
and/or bug fixes. I'm especially interested in reviews from people that
have embedded Lua in other projects or experts in Lua that can improve the
robustness of the menu code.
Do you have some memory usage numbers for LUA vs forth for the different
BIOS loaders (text/data/bss sizes)? For the EFI case we probably have lots
of room, but for the non-EFI case we are constrained to 0xa0000 - 0xa000
for the text/data/bss and stack (in some cases like PXE booting the top
can be lower than 0xa0000). I'm not sure if we have any other platforms
with similar memory constraints.
--
John Baldwin
Warner Losh
2018-02-12 21:31:46 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Greetings,
As you may know, the Lua (http://www.lua.org) boot loader has been in
the
Post by Warner Losh
works for some time. It started out life as a GSoC in 2014 by Pedro Souza
mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch to
try to integrate it. I rebased that effort into a github branch which
Pedro
Post by Warner Losh
Arthur fixed up. Over the past year, I've been cleaning up the boot
loader
Post by Warner Losh
for other reasons, and found the time was ripe to start integrating this
into the tree. However, those integration efforts have taken a while as
my
Post by Warner Losh
day-job work on the boot loader took priority. In the mean time, Ed Maste
and the FreeBSD Foundation funded Zakary Nafziger to enhance the original
GSoC Lua scripts to bring it closer to parity with the evolution of the
FORTH menu system since the GSoC project started.
I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later today. This
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a few weeks
and I didn't want to delay pushing this into the tree for that.
To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do this,
you can do it in src/stand and install the result (be sure to have the
options for both the build and the install). This will replace your
current
Post by Warner Losh
/boot/loader that is scripted with FORTH to one that's scripted with Lua.
It will install the lua scripts in /boot/lua. The boot is scripted with
/boot/lua/loader.lua instead of /boot/loader.rc. You are strongly advised
to create a backup copy of /boot/loader before testing (eg cp
/boot/loader
Post by Warner Losh
/boot/loader_forth), since you'll need to boot that from boot2 if
something
Post by Warner Losh
goes wrong. I've tested it extensively, though, with userboot.so and it's
test program, so all the initial kinks of finding the lua scripts, etc
have
Post by Warner Losh
been worked out.
While it's possible to build all the /boot/loader variants with Lua, I've
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the instructions
for
Post by Warner Losh
testing some of them may be rather tedious (especially UEFI, if you want
a
Post by Warner Losh
simple path to back out). Since there's not been full convergence
testing,
Post by Warner Losh
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.
I put MFC after a month on the commit. Due to the heroic (dare I say
almost crazy) work of Kyle Evans on merging all the revs from -current to
11, I'm planning a MFC to 11 after the co-existence issues are hammered
out. In 11, FORTH will be the default, and Lua will be built by default,
but users will have to do something to use it. 12, both FORTH and Lua
will
Post by Warner Losh
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine we'll make
the switch to Lua by default shortly after that. In 13, FORTH will be
removed unless there's a really really compelling case made to keep it.
So please give it a spin and give me any feedback, documentation updates
and/or bug fixes. I'm especially interested in reviews from people that
have embedded Lua in other projects or experts in Lua that can improve
the
Post by Warner Losh
robustness of the menu code.
Do you have some memory usage numbers for LUA vs forth for the different
BIOS loaders (text/data/bss sizes)? For the EFI case we probably have lots
of room, but for the non-EFI case we are constrained to 0xa0000 - 0xa000
for the text/data/bss and stack (in some cases like PXE booting the top
can be lower than 0xa0000). I'm not sure if we have any other platforms
with similar memory constraints.
Lua is about 60-70k larger than FORTH for this application:

Forth based:
-r-xr-xr-x 1 root wheel 385024 Feb 12 14:14
/home/imp/roots/amd64/boot/loader
-r-xr-xr-x 1 root wheel 450560 Feb 12 14:14
/home/imp/roots/amd64/boot/zfsloader
Lua based:
-r-xr-xr-x 1 root wheel 450560 Feb 12 14:20
/home/imp/roots/amd64/boot/loader
-r-xr-xr-x 1 root wheel 520192 Feb 12 14:20
/home/imp/roots/amd64/boot/zfsloader

So with Lua:
% size loader.bin
text data bss dec hex filename
411840 22264 48304 482408 0x75c68 loader.bin
% size zfsloader.bin
text data bss dec hex filename
478712 22968 52368 554048 0x87440 zfsloader.bin

so we're at 555k out of 640k before we look at heap usage. Of course, these
numbers are only going to get a lot worse as we pile-in new crypto
features... I know that the normal loader works, but I've not tried the
zfsloader.

We might be able to do some slimming of LUA. I've honestly not tried
super-hard to do that, but do presently omit a lot of the standard
libraries which saves some space.

Warner
John Baldwin
2018-02-12 21:53:59 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
Greetings,
As you may know, the Lua (http://www.lua.org) boot loader has been in
the
Post by Warner Losh
works for some time. It started out life as a GSoC in 2014 by Pedro Souza
mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch to
try to integrate it. I rebased that effort into a github branch which
Pedro
Post by Warner Losh
Arthur fixed up. Over the past year, I've been cleaning up the boot
loader
Post by Warner Losh
for other reasons, and found the time was ripe to start integrating this
into the tree. However, those integration efforts have taken a while as
my
Post by Warner Losh
day-job work on the boot loader took priority. In the mean time, Ed Maste
and the FreeBSD Foundation funded Zakary Nafziger to enhance the original
GSoC Lua scripts to bring it closer to parity with the evolution of the
FORTH menu system since the GSoC project started.
I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later today. This
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a few weeks
and I didn't want to delay pushing this into the tree for that.
To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do this,
you can do it in src/stand and install the result (be sure to have the
options for both the build and the install). This will replace your
current
Post by Warner Losh
/boot/loader that is scripted with FORTH to one that's scripted with Lua.
It will install the lua scripts in /boot/lua. The boot is scripted with
/boot/lua/loader.lua instead of /boot/loader.rc. You are strongly advised
to create a backup copy of /boot/loader before testing (eg cp
/boot/loader
Post by Warner Losh
/boot/loader_forth), since you'll need to boot that from boot2 if
something
Post by Warner Losh
goes wrong. I've tested it extensively, though, with userboot.so and it's
test program, so all the initial kinks of finding the lua scripts, etc
have
Post by Warner Losh
been worked out.
While it's possible to build all the /boot/loader variants with Lua, I've
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the instructions
for
Post by Warner Losh
testing some of them may be rather tedious (especially UEFI, if you want
a
Post by Warner Losh
simple path to back out). Since there's not been full convergence
testing,
Post by Warner Losh
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.
I put MFC after a month on the commit. Due to the heroic (dare I say
almost crazy) work of Kyle Evans on merging all the revs from -current to
11, I'm planning a MFC to 11 after the co-existence issues are hammered
out. In 11, FORTH will be the default, and Lua will be built by default,
but users will have to do something to use it. 12, both FORTH and Lua
will
Post by Warner Losh
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine we'll make
the switch to Lua by default shortly after that. In 13, FORTH will be
removed unless there's a really really compelling case made to keep it.
So please give it a spin and give me any feedback, documentation updates
and/or bug fixes. I'm especially interested in reviews from people that
have embedded Lua in other projects or experts in Lua that can improve
the
Post by Warner Losh
robustness of the menu code.
Do you have some memory usage numbers for LUA vs forth for the different
BIOS loaders (text/data/bss sizes)? For the EFI case we probably have lots
of room, but for the non-EFI case we are constrained to 0xa0000 - 0xa000
for the text/data/bss and stack (in some cases like PXE booting the top
can be lower than 0xa0000). I'm not sure if we have any other platforms
with similar memory constraints.
-r-xr-xr-x 1 root wheel 385024 Feb 12 14:14
/home/imp/roots/amd64/boot/loader
-r-xr-xr-x 1 root wheel 450560 Feb 12 14:14
/home/imp/roots/amd64/boot/zfsloader
-r-xr-xr-x 1 root wheel 450560 Feb 12 14:20
/home/imp/roots/amd64/boot/loader
-r-xr-xr-x 1 root wheel 520192 Feb 12 14:20
/home/imp/roots/amd64/boot/zfsloader
% size loader.bin
text data bss dec hex filename
411840 22264 48304 482408 0x75c68 loader.bin
% size zfsloader.bin
text data bss dec hex filename
478712 22968 52368 554048 0x87440 zfsloader.bin
so we're at 555k out of 640k before we look at heap usage. Of course, these
numbers are only going to get a lot worse as we pile-in new crypto
features... I know that the normal loader works, but I've not tried the
zfsloader.
Woof, on stable/11 the forth ones are quite a bit smaller:

-r-xr-xr-x 1 root wheel 315392 Dec 16 13:39 /boot/loader
-r-xr-xr-x 1 root wheel 356352 Dec 16 13:39 /boot/zfsloader

% size /usr/obj/usr/src/sys/boot/i386/loader/loader.bin
text data bss dec hex filename
274913 22624 46560 344097 0x54021 /usr/obj/usr/src/sys/boot/i386/loader/loader.bin
% size /usr/obj/usr/src/sys/boot/i386/zfsloader/zfsloader.bin
text data bss dec hex filename
313865 23184 50624 387673 0x5ea59 /usr/obj/usr/src/sys/boot/i386/zfsloader/zfsloader.bin

Note that it's not 640k, it's 0xa0000 (640k) - the start address of the
loader: 0xa000 (40k), so text/data/bss + stack has 600k of room. The heap
isn't in low memory anymore, so we don't have to worry about that. 555k for
ZFS loader leaves around 45k for stack. If the ZFS loader code is anything
like the kernel code then ZFS probably uses deeper stacks than UFS. :(

One of the larger uses of stack in the loader is probably gzipfs, so loading
a mfsroot.gz from the loader might be a good test. Unfortunately, there isn't
a good way to really detect stack overflow other than silent corruption in
bss. If we could create a 'volatile int canary' in the bss and force it to be
the last thing in the bss we could check the value of 'canary' in the loader's
main loop perhaps and complain if it is ever non-zero.
--
John Baldwin
Warner Losh
2018-02-12 22:25:42 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
Greetings,
As you may know, the Lua (http://www.lua.org) boot loader has been
in
Post by Warner Losh
Post by Warner Losh
the
Post by Warner Losh
works for some time. It started out life as a GSoC in 2014 by Pedro
Souza
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
mentored by Wojciech A. Koszek. Rui Paulo created a svn project
branch to
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
try to integrate it. I rebased that effort into a github branch which
Pedro
Post by Warner Losh
Arthur fixed up. Over the past year, I've been cleaning up the boot
loader
Post by Warner Losh
for other reasons, and found the time was ripe to start integrating
this
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
into the tree. However, those integration efforts have taken a while
as
Post by Warner Losh
Post by Warner Losh
my
Post by Warner Losh
day-job work on the boot loader took priority. In the mean time, Ed
Maste
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
and the FreeBSD Foundation funded Zakary Nafziger to enhance the
original
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
GSoC Lua scripts to bring it closer to parity with the evolution of
the
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
FORTH menu system since the GSoC project started.
I'm pleased to announce that all these threads of development have
converged and I'll be pushing the FreeBSD Lua Loader later today.
This
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
loader uses Lua as its scripting language instead of FORTH. While
co-existance is planned, the timeline for it is looking to be a few
weeks
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
and I didn't want to delay pushing this into the tree for that.
To try the loader, you'll need to build WITHOUT_FORTH=yes and
WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do
this,
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
you can do it in src/stand and install the result (be sure to have
the
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
options for both the build and the install). This will replace your
current
Post by Warner Losh
/boot/loader that is scripted with FORTH to one that's scripted with
Lua.
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
It will install the lua scripts in /boot/lua. The boot is scripted
with
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
/boot/lua/loader.lua instead of /boot/loader.rc. You are strongly
advised
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
to create a backup copy of /boot/loader before testing (eg cp
/boot/loader
Post by Warner Losh
/boot/loader_forth), since you'll need to boot that from boot2 if
something
Post by Warner Losh
goes wrong. I've tested it extensively, though, with userboot.so and
it's
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
test program, so all the initial kinks of finding the lua scripts,
etc
Post by Warner Losh
Post by Warner Losh
have
Post by Warner Losh
been worked out.
While it's possible to build all the /boot/loader variants with Lua,
I've
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
just tested a BIOS booting /boot/loader both with and without menus
enabled. I've not tested any of the other variants and the
instructions
Post by Warner Losh
Post by Warner Losh
for
Post by Warner Losh
testing some of them may be rather tedious (especially UEFI, if you
want
Post by Warner Losh
Post by Warner Losh
a
Post by Warner Losh
simple path to back out). Since there's not been full convergence
testing,
Post by Warner Losh
you'll almost certainly find bumps in this system. Also, all the
build-system APIs are likely not yet final.
I put MFC after a month on the commit. Due to the heroic (dare I say
almost crazy) work of Kyle Evans on merging all the revs from
-current to
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
11, I'm planning a MFC to 11 after the co-existence issues are
hammered
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
out. In 11, FORTH will be the default, and Lua will be built by
default,
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
but users will have to do something to use it. 12, both FORTH and Lua
will
Post by Warner Losh
be built and installed, with Lua as default (barring unforeseen
complications). Once the co-existence stuff goes in, I imagine we'll
make
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
the switch to Lua by default shortly after that. In 13, FORTH will be
removed unless there's a really really compelling case made to keep
it.
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
So please give it a spin and give me any feedback, documentation
updates
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
and/or bug fixes. I'm especially interested in reviews from people
that
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
have embedded Lua in other projects or experts in Lua that can
improve
Post by Warner Losh
Post by Warner Losh
the
Post by Warner Losh
robustness of the menu code.
Do you have some memory usage numbers for LUA vs forth for the
different
Post by Warner Losh
Post by Warner Losh
BIOS loaders (text/data/bss sizes)? For the EFI case we probably have
lots
Post by Warner Losh
Post by Warner Losh
of room, but for the non-EFI case we are constrained to 0xa0000 -
0xa000
Post by Warner Losh
Post by Warner Losh
for the text/data/bss and stack (in some cases like PXE booting the top
can be lower than 0xa0000). I'm not sure if we have any other
platforms
Post by Warner Losh
Post by Warner Losh
with similar memory constraints.
-r-xr-xr-x 1 root wheel 385024 Feb 12 14:14
/home/imp/roots/amd64/boot/loader
-r-xr-xr-x 1 root wheel 450560 Feb 12 14:14
/home/imp/roots/amd64/boot/zfsloader
-r-xr-xr-x 1 root wheel 450560 Feb 12 14:20
/home/imp/roots/amd64/boot/loader
-r-xr-xr-x 1 root wheel 520192 Feb 12 14:20
/home/imp/roots/amd64/boot/zfsloader
% size loader.bin
text data bss dec hex filename
411840 22264 48304 482408 0x75c68 loader.bin
% size zfsloader.bin
text data bss dec hex filename
478712 22968 52368 554048 0x87440 zfsloader.bin
so we're at 555k out of 640k before we look at heap usage. Of course,
these
Post by Warner Losh
numbers are only going to get a lot worse as we pile-in new crypto
features... I know that the normal loader works, but I've not tried the
zfsloader.
-r-xr-xr-x 1 root wheel 315392 Dec 16 13:39 /boot/loader
-r-xr-xr-x 1 root wheel 356352 Dec 16 13:39 /boot/zfsloader
% size /usr/obj/usr/src/sys/boot/i386/loader/loader.bin
text data bss dec hex filename
274913 22624 46560 344097 0x54021 /usr/obj/usr/src/sys/boot/
i386/loader/loader.bin
% size /usr/obj/usr/src/sys/boot/i386/zfsloader/zfsloader.bin
text data bss dec hex filename
313865 23184 50624 387673 0x5ea59 /usr/obj/usr/src/sys/boot/
i386/zfsloader/zfsloader.bin
Note that it's not 640k, it's 0xa0000 (640k) - the start address of the
loader: 0xa000 (40k), so text/data/bss + stack has 600k of room. The heap
isn't in low memory anymore, so we don't have to worry about that. 555k for
ZFS loader leaves around 45k for stack. If the ZFS loader code is anything
like the kernel code then ZFS probably uses deeper stacks than UFS. :(
Yea. We support more FS types than in 11 now I think (at least more than in
10),
so we might want to look to trimming that. I don't think we need 3
different compressed
filesystem formats compiled in by default, for example...
Post by Warner Losh
One of the larger uses of stack in the loader is probably gzipfs, so loading
a mfsroot.gz from the loader might be a good test. Unfortunately, there isn't
a good way to really detect stack overflow other than silent corruption in
bss. If we could create a 'volatile int canary' in the bss and force it to be
the last thing in the bss we could check the value of 'canary' in the loader's
main loop perhaps and complain if it is ever non-zero.
That's not a terrible idea... Assuming that the corruption is such that
it's right at the end of bss, and not something a bit before it...

Warner
David Wolfskill
2018-02-14 13:50:36 UTC
Permalink
Post by Warner Losh
...
So please give it a spin and give me any feedback, documentation updates
and/or bug fixes. I'm especially interested in reviews from people that
have embedded Lua in other projects or experts in Lua that can improve the
robustness of the menu code.
....
I was able to build, install, and (start to -- not loader's fault)
boot my laptop; I have placed screenshots at
<http://www.catwhisker.org/~david/FreeBSD/head/loader/>.

Some notes:
* For loader_1_lua.jpg, I had (reflexively) already hit "v" to boot
verbosely, so "Verbose :On" does NOT reflect the default. On the
other hand, "ACPI :off" DOES reflect the default, which is
opposite from the Forth loader (ref. loader_1_4th.jpg); I suspect that
this warrants changing.

* The lower left corner of the box seems to have an odd artifact for the
Lua loader. I haven't looked, but suspect it will be easy to fix.

Peace,
david
--
David H. Wolfskill ***@catwhisker.org
The circus around that memo helps confirm that Mr. Trump is unfit for office.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.
David Wolfskill
2018-02-14 15:12:04 UTC
Permalink
Also, after reverting r329225, I was able to build & boot the build
machine with the Lua loader... but I didn't even see any way of
interacting with it on the serial console. (I do get that opportunity
with the Forth loader.)

Peace,
david
--
David H. Wolfskill ***@catwhisker.org
The circus around that memo helps confirm that Mr. Trump is unfit for office.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.
Loading...