Here’s a summary of the basics, for anyone who doesn’t need all the grueling details:

This all matters for CBL because the build process for some of the necessary packages will result in programs that can’t find their shared libraries unless we use an RPATH or RUNPATH; and the build process for other packages sets an incorrect RPATH or RUNPATH that can cause problems unless we remove it.

Usually, ld isn’t executed directly by build processes, but is instead invoked by the gcc driver program. To get gcc to pass an option along to ld, you can give it the option -Wl, which should be followed by additional words separated by commas; gcc will replace the commas with spaces and pass the resulting arguments on to ld. So to set an RPATH of /some/dir and /another/dir, you can give gcc the argument -Wl,-rpath,/some/dir:/another/dir.

An alternative that might work is to set the environment variable LD_RUN_PATH to the desired RPATH before linking — the ld documentation suggests this will work, but I haven’t tried it.

This might make your eyes glaze over a bit.

A lot of this discussion pertains only to programs in the Executable and Linkable Format, commonly abbreviated as ELF. This is the only program format used in modern GNU/Linux systems, but you should be aware that there are other executable program formats, like a.out and COFF, and a lot of the details here don’t apply to those formats.

ELF programs consist of an ELF header followed by some number of segments of various types. You can see the full structure of ELF programs by using the program readelf, which is part of the GNU binutils package.

When a program is executed, the Linux kernel looks in its .interp section to find the path for its interpreter — in our case, this will be the full path of ld.so. That interpreter then looks in dynamic segments for the names of shared library files that are needed by the program and tries to find them. If any shared library can’t be found, the dynamic linker will terminate with an error.

There’s a bit of additional complexity: shared library files are also in ELF format and can also declare that they are dependent on other shared libraries. This can result in a chain of library dependencies, potentially a lengthy one! When the dynamic linker is trying to find a library, its behavior is partly determined by which object is having its references resolved: the original program, or one of the libraries it depends on, or one of their libraries, and so on. Whichever ELF file is having its references resolved is called the "loading object" here.

The rest of this section describes the procedure used by the dynamic linker to locate shared library files.

Shared library names can contain slash characters, although they usually do not. (I don’t even know how to get ld to create a program with library dependencies that specify a full path.) If a shared library name does contain any slash characters, it is treated as a relative or absolute path and the dynamic linker will only look for the library at that path.

In the common case, when a shared library is specified without any slash characters, the dynamic linker looks for it in a variety of locations. The algorithm it uses is poorly documented and there’s a lot of contradictory information about it on the web. After looking around for quite a while, I found a helpful blog post on qt.io that asserted that the relevant code is _dl_map_object in the glibc file elf/dl-load.c, and a review of that code revealed the following:

As soon as the shared library is found in any of those locations, that’s the version that will be used. If it’s not found in any of those locations, the dynamic linker will give up and crash with an error message. And, in case this was not clear, in all the things that look like PATH — LD_LIBRARY_PATH, RPATH, RUNPATH — you can specify any number of directories separated by colons, just like the PATH environment variable.

A historical note: RPATH has been around longer than RUNPATH. RUNPATH was implemented, along with all the logic about skipping RPATH when a RUNPATH is present, because people realized that when someone specifies an LD_LIBRARY_PATH it’s usually because they really do know what they want the dynamic linker to do, and it’s rude to override that desire at program compilation time.

When you give ld the -rpath option by itself, it just creates an RPATH in the resulting program or library. When you add the option --enable-new-dtags, it still creates an RPATH (in case you run the program with an old dynamic linker that doesn’t understand RUNPATH), but it also creates a RUNPATH so that modern dynamic linkers will ignore the RPATH.

As a reward for reading this far, here’s one last option you can use if none of the above suits your purpose: before it does anything else, the dynamic linker loads any .o object files, or .so or .a library files, that are named in the environment variable LD_PRELOAD or in the /etc/ld.so.preload file. Any functions defined in those files are used in preference to any other function definitions found later in the process. That lets you override individual function definitions, if you want to override some part of the program without replacing an entire library with a different version.

You can read more about all this stuff in the ld and ld.so info and man pages, and in the "Program Library HOWTO" in the Linux Documentation Project.

The GNU binary utilities package contains a plethora of programs and libraries that can be used to produce, manipulate, and otherwise operate on (compiled and assembled) object files. I’ll briefly describe them all here, but don’t worry! Not only will there not be a test on any of this, you won’t usually need to invoke any of these programs manually. The gcc driver program will invoke as and ld as necessary to do its work, and several of the other utility programs are similary used internally during the build process of other system components but you won’t need to use them yourself.

The most important binutils programs are as, the assembler, which transforms assembly source code (.s files) into binary or "object" code (.o files); and ld, the link editor (ld is usually just called the linker, but it’s hard to see why it’s called ld without knowing the other term), which combines multiple object files into an executable program.

There are actually two different ld programs in current GNU binutils: the original version uses the binary file descriptor (BFD) library and can always be found at ld.bfd. There’s also a newer program, "gold," that doesn’t use BFD and only works for binaries that are in the "Executable and Linkable Format," aka "ELF"; it’s always available at ld.gold. One or the other program is linked so it can be executed as ld.

The other programs are:

There are a couple of shared libraries used by those programs and available to others, as well: libbfd and libopcodes. All of the utility programs are documented much mor thoroughly in man pages and the binutils info file.

As I mentioned above, you don’t need to run any of those programs directly because the gcc driver program streamlines the simple case of building executables — to compile a "Hello, World" program, you just need to run gcc hello.c, and let the gcc driver program run as to assemble compiled source into object files, ld to link multiple object files together to produce an executable, and maybe other programs if it needs to for some reason.

The downside of using a driver program is that it can make complex builds (when, for example, specific options need to be passed to the assembler, compiler, and linker) a lot more complicated and fussy — as you’ll see, from time to time, during the CBL process.

When configuring binutils, we always specify --enable-64-bit-bfd. This is needed to enable 64-bit support, which is important when the host or target systems have a 64-bit userspace. It’s unimportant for entirely 32-bit builds (for example, an i686-to-mipsel CBL build), but doesn’t cause any problems when it’s used in one of those builds.

When cross-compiling, a bug introduced in GCC 7 (which was bug 81840 in its bugzilla bug-tracking system, but then bugzilla crashed and that bug report was lost) causes an incorrect warning when compiling tc-i386.c in the gas source tree in at least some cross-compilation scenarios. That’s not a big deal, except that the binutils build treats all warnings as errors and terminates the build when it sees them. This patch works around the GCC bug so it doesn’t produce the problematic warnings.

Binutils, like some other parts of the toolchain, should be built in a separate directory from the source. As with other components that we build several times, CBL puts each distinct build in a separate location to avoid any need to clean things up to a pristine state for the next build.

We build a cross-binutils using the "sysroot" framework (which you’ll read more about shortly). That framework isn’t particularly well-documented, but the important thing at this point is that we need to specify the configure options --with-sysroot and --with-build-sysroot to inform the build machinery of the desired sysroot directory.

All of the toolchain components should be installed in the same filesystem location. The CROSSTOOLS parameter lets you specify that location.

Some of the warning messages present in GCC 8 and later present problems when compiling the binutils. We can tweak the generated Makefiles so those warnings won’t be converted to errors.

For some reason, the binutils installation process doesn’t copy the libiberty header file. libiberty is a support library used by several components in the GNU toolchain, distributed both as part of binutils and gcc; later builds will want to use functions defined in it. So we install that ourselves.

GMP, the GNU Multi-Precision arithmetic library, is a component in — or perhaps it is more properly called a dependency of — the GNU toolchain. It allows arithmetic operations to be performed with levels of precision other than the standard integer and floating-point types. Applications can use GMP to provide arithmetic with thousands or millions of digits of precision if that’s what they need. GMP also provides support for rational-number arithmetic, as well as integer and floating-point.

If you’re really interested in high-precision floating-point arithmetic, you might want to look into MPFR rather than GMP! The GMP people say it’s much more complete.

GMP has been needed by the Fortran GCC front-end for some time, but starting with release 4.3.0 of GCC it is needed for C (and C++) as well.

MPC, another dependency of GCC, requires a GMP built with C++ support, so we need to specify that at configure time.

This package is often built in-tree as part of GCC, rather than separately — that’s especially true when the only reason you’re buiilding GMP is because GCC requires it. When using an in-tree build, this blueprint is pretty much irrelevant. However, as of 2015-09-27, there’s an issue with in-tree builds of GMP in some cross-toolchain builds, so for CBL we build it separately.

This is not necessary, because the system or host-prerequisites version of GMP can be used just fine by the cross-compiler. If things break down here and you haven’t built the Trustworthy Host-System Programs, you should probably do that.

MPFR is a library for arbitrary-precision floating-point arithmetic. It stands for "Multiple-Precision Floating-point Rounding," I think, but that’s not really clear from their site so I might be wrong. It uses GMP internally, but provides any level of precision (including very small precision) and provides the four rounding modes from the IEEE 754-1985 standard.

Like GMP, MPFR is a component in, or dependency of, the GNU toolchain. It has been needed by the Fortran GCC front-end for some time, but starting with release 4.3.0 of GCC, MPFR is needed for C and C++ as well. GCC uses MPFR to pre-calculate the result of some mathematic functions when those functions have constant arguments, and produces the same results regardless of the math library or floating point engine used on the runtime system. This occurs in what is called the GCC "middle-end," which is kind of a silly name since it’s not an end.

This package is often built in-tree as part of GCC, rather than separately. However, as of September 2015, in-tree builds of the dependencies have some issues in certain circumstances, so in CBL we step away from the in-tree build facility altogether.

As with GMP, this is not necessary because the system or host-prerequisites version of MPFR can be used.

MPC is a C library for arbitrary-precision arithmetic on complex numbers providing correct rounding. It can be thought of an extension to the MPFR library.

Like GMP and MPFR, MPC is a component in, or dependency of, the GNU toolchain. I haven’t been able to find any description of what features of MPC are actually needed by the GNU toolchain, but MPC is a hard build-time dependency of GCC.

If your system already has MPC installed, this step can be skipped.

Like the other GCC library dependencies, this package is often built in-tree as part of GCC. As mentioned earlier, though, this sometimes introduces problems, so CBL doesn’t take advantage of the in-tree build machinery provided by GCC.

As with GMP, this is not necessary if there is a system or host-prerequisites version of MPC available.

The project homepage describes ISL as a library for manipulating sets and relations of integer points bounded by linear constraints. I’m not sure what that means. It sounds like math.

Unless you have some specific reason to want ISL for one of your own projects, the main benefit it provides is as an optional dependency of GCC: the "graphite loop optimizations" in GCC (whatever those are) require ISL to be available.

As with GMP, this is not necessary if there is a system or host-prerequisites version of ISL available.

The kernel is Linux per se — the foundation of the operating system. Often, when people say "Linux," they mean the entire operating system that lets them use a computer; properly, though, Linux is just the kernel. Many — perhaps most — of the other components that make up the full operating system are pieces of the Free Software Foundation’s GNU project, which stands for "GNU’s Not UNIX"; almost all of the program-construction tools that form the foundation of the system are GNU components.

The job of the kernel — this is a critically important point and I say it a lot — is to initialize and manage all of the hardware on the computer (including CPUs, memory, and I/O devices) and provide services to userspace processes. That’s a big job, but it’s also a limited one! Everything you do with your computer is the responsibility of userspace processes.

The way that Linux performs its job — more generally, the way that UNIX kernels work — is conceptually very simple: when it is executed on a computer, Linux does some hardware initialization, mounts the root filesystem, and then starts a single userspace process. That process has process ID (PID) 1, and is conventionally called init.

The init process is responsible for starting all other userspace programs and getting the machine into a usable state; after the kernel starts init, it gets out of the way and waits for that process, or for other userspace programs, to request its services.

The complexity in the Linux kernel emerges mostly from the huge variety of hardware that it supports and the many layers of functionality that can be built into it. If you unpack the Linux source code, you’ll find that the whole thing adds up to about (as of the 5.1 kernel) 926 megabytes in total. Of that, about thirteen percent (127 megabytes) is specific to the twenty-six different CPU architectures that Linux supports, and well over half (547 megabytes) is device drivers. That means that for any given system, a large majority of the code that makes up the Linux kernel won’t ever be used.

In CBL, the kernel is kept as pristine as possible. The only patches applied here are intended to add new default configuration settings — which are more convenient than setting dozens of configuration options manually — and in some cases to add support for hardware devices and platforms that are not currently supported by the mainline kernel.

The way that programs make requests of the kernel is by invoking kernel functions known as "system calls." The Linux kernel sources include header files that define all of the system calls it makes available to userspace programs.

Userspace programs don’t usually invoke system calls themselves (although they can, and some do). Instead, they invoke library functions — particularly the ones defined in the C standard library, which on GNU/Linux systems is usually the GNU libc, glibc, but might be musl or uClibc or something else altogether. Those library functions invoke system calls as needed to accomplish their work.

In this step, we’re not actually building the Linux kernel; all we’re doing is installing the header files that specify those system calls. These header files are then used by the C library and other userspace libraries and programs to invoke the system calls as needed to perform their work.

The Linux source tree also includes a number of other, private, header files — these define data structures and functions that should be used only within the kernel itself, and are not intended to be visible to userspace programs. The Makefile target we use here, headers_install, installs only the public header files, not these private internal headers.

The makefile target mrproper puts the source tree into a completely pristine state. The name is a reference to the Proctor & Gamble cleaning product that people in the United States know as "Mr. Clean"; in many parts of Europe, it is marketed under the brand name "Mr. Proper."

The makefile target headers_check presumably makes sure that the public header files are OK. I haven’t really looked at it, though. If you know what it really does, and that’s not it, please tell us.

The install_headers target deletes the entire target directory before it does the installation. This is inconvenient, because we might possibly have header files there that we want to retain (such as from the binutils build). To avoid any issues, we’re going to install the headers to a temporary location and copy them to the real target location from there.

Notice that we’re actually installing the headers into the scaffolding directory underneath the sysroot directory. That’s common to the entire host-side portion of CBL: Everything that will be conveyed to the target system is under the scaffolding location. That way, when the final system is completed, we can get rid of /scaffolding and be fairly confident that everything remaining has been built entirely from source and (if it continues to work) has no dependencies on the scaffolding programs and libraries.

GCC is the GNU Compiler Collection. This is the single most important package in CBL: without a compiler, you can’t build any software. We use GCC to bootstrap everything, including itself.

Unfortunately, GCC is also probably the most complex package we need to build, and it has a very complex configuration and build process because it is tied so intricately to other packages. This is particularly true of glibc, the C standard library we use in CBL, which also has a complex build process of its own!

On the plus side, if you can get GCC built and working properly, you’re past the biggest hurdle of building a complete GNU/Linux system entirely from source.

The GCC installation process is documented in the "Installation" manual, found in the source distribution in the INSTALL directory. If you have problems getting GCC built, that’s an excellent resource for figuring out what’s going wrong. If you want to get a better understanding of the GCC configuration and build process — how it actually works — read the "Source Tree Structure and Build System" section of the GCC Internals document, found in the source distribution in gcc/doc/gccint.info.

The most important compilers in the collection, for purposes of bootstrapping a system, are the C and C++ compilers; but GCC also includes compilers for Fortran, Go, Ada, Objective C, and probably other languages as well. And it supports a huge number of machine architectures! That’s probably the most important aspect of GCC for our purposes.

The program you usually invoke to build C programs, gcc, is not actually a compiler. It’s just a driver that knows how to invoke other programs, using rules called "spec strings" that tell it exactly what other programs it needs to invoke, and what command-line arguments it should provide, to turn source code into an executable program. (We’ll talk more specifically about spec strings in Adjusting the GCC specs.)

I find it really important to keep that in mind throughout toolchain construction, so I’ll expand on that point: gcc just invokes other programs. Some of those programs — the C preprocessor cpp, cc1 (the C compiler itself), and internal utility programs like collect2 (which is sort of a first-stage linker, used to set up calls to constructors and other initialization routines as a program starts to run) — are part of the GCC package. Others, like the as and ld programs, are distributed separately (in the case of as and ld, that package is GNU binutils).

To get from source code to an executable C program, gcc actually performs a series of steps:

You can find out exactly what commands gcc is running by giving it the -v (for "verbose") command line argument. That’s a handy trick when things are going wrong and you’re not sure why!

The term "compilation" — which is the job of the compiler — actually refers only to the second of those steps: source code is compiled into assembly code. It’s not precisely correct to say that you’re "compiling" source code into an executable program! That’s a common conversational shorthand, but it masks the complete story about what’s going on. It would be more correct to say that you’re "pre-processing, compiling, assembling, and linking" source code to produce an executable program.

On the other hand, that’s a lot of words, and the complete story is seldom really something you need to keep in mind. This is probably why the shorthand term is so popular.

A few external dependencies were added to gcc in release 4.3: the GMP, MPFR, and MPC libraries are required for all compiler builds, and a few optimizations — the graphite loop optimizations — are only available if the Integer Set Library (ISL) is available. The graphite loop optimizations are not critically important, but there’s no reason not to include them if it’s not difficult to do.

Sources for the required — and optional — libraries can be included in the GCC sources in directories named gmp, isl, mpc, and mpfr; if they are, then they’ll be built automatically along with gcc. There’s actually a fairly large number of packages that the build machinery for GCC detects and incorporates into builds automatically. This is convenient, but restrictive: in-tree builds are only reliable when specific versions of the dependency libraries are present, and for CBL we prefer to use the latest stable release of everything.

There are also some cross-compilation scenarios in which the in-tree library builds actually do the wrong thing and produce a compiler that will not work right; that’s another reason that we avoid them here.

There’s a problem introduced in GCC 6.1 where C++ header paths are hard-coded, which prevents them from being found by the scaffolding compiler (which lives at a different filesystem location when the target system is booted).

When specifying locations for the dependency libraries (GMP, MPFR, etc), the specified library location should be used both at build time (with an -L linker directive) and at run time (with an -rpath link directive). This is not done in all cases by the normal GCC build process, but we can patch it easily to add that behavior.

The next few paragraphs discuss issues related to the way the dynamic linker works. If you’re not familiar with its operation, it might be a good idea to review the section A Word About The Dynamic Linker — or just skip ahead a bit, if you’re not interested in the details of a linking problem and the reason we apply this patch to work around it.

Even though the math libraries needed by GCC (GMP, ISL, MPC, and MPFR) have just been built here and GCC is configured to find them in their correct locations, some of the programs that make up GCC are built without an RPATH or RUNPATH, so the dynamic loader will look for those libraries at runtime in the normal host system library directories. This was filed as GCC bug 84153, but the GCC maintainers don’t consider it a problem.

To be fair, this really is not usually very much of a problem! The issue appears when the version of the library used in CBL is a major version later than the version installed on the host system. In that case, the GCC we’re building in this section won’t work because it won’t be able to find the version of the libraries it depends on.

This is the sort of situation that LD_LIBRARY_PATH is intended to resolve, but there’s a gotcha: in some of the host-scaffolding builds, the native toolchain is used to build some programs that are run as part of the build process itself, and the LD_LIBRARY_PATH set in the environment for the cross-build is unset when building those native programs.

That means that to get the this native gcc to be reliable, we need to set a RUNPATH or RPATH to tell the dynamic loader where to look for shared libraries. The way you normally do this is to set an LDFLAGS environment variable with a value like -Wl,-rpath,/usr/lib when running the configure script, so that ld would be told to build programs with that RPATH, but it turns out the GCC build doesn’t use the LDFLAGS linker arguments when constructing some of the programs that need the dependency libraries, like cc1, cc1plus, and lto1.

We can work around that by patching the configure script so that any time it’s given arguments like --with-gmp or --with-gmp-lib, the flags it collects will include a -Wl,-rpath option along with the -L option. That’s what this patch does.

An issue I’ve found in one scenario — building a 64-bit ARM to 64-bit x86 compiler — is that a necessary header file doesn’t get included because the getrusage function is not available. A trivial patch works around that problem.

An issue introduced in GCC 11.1, and reported both as bug 80196 and 100017, is that a file in the C++ standard library (which is distributed as part of GCC, rather than separately) is not able to find included headers when built as part of a canadian cross compiler or a target-native compiler (like the host-scaffolding GCC). A workaround for this is provided on bug 100017, and incorporated here as a patch.

Bootstrapping GCC as part of a cross-toolchain is tricky. You can build the compiler per se without a working C library (libc), but that compiler won’t be able to produce executable programs: GCC can only create programs if it has access to a set of C runtime object files that it expects to be provided by the C library — and, typically, it also needs a C standard library implementation to compile programs because essentially all C programs invoke functions also provided by that library.

Obviously, we can’t compile the C library without a compiler! So there’s a chicken-and-egg bootstrapping problem.

To work around that cycle of dependencies, we’re going to build just the parts of the compiler we really need at this point: a plain cross-compiler, and a minimal version of the libgcc support library that the compiler needs in order to function. (GCC needs libgcc because sometimes, while processing C code, the compiler generates references to functions defined in libgcc rather than generating assembler code.)

GCC can most easily be built as part of a cross-toolchain by using the "sysroot" framework — and, in fact, the GCC developers don’t support any other method for creating cross-toolchains. To perform a sysroot build, the configure options --with-sysroot and --with-build-sysroot must be specified; and when building GCC, the environment variables LDFLAGS_FOR_TARGET and CPPFLAGS_FOR_TARGET should be set to --sysroot=/home/lbl/work/sysroot.

…​At least, that’s what Carlos O’Donell said in a comment on GCC bug #35532. The documentation on sysroot builds is not particularly easy to find — or at least, it wasn’t when this was written. (If you know where sysroot builds are documented, please tell me!)

The sysroot concept is pretty clever, in fact. The basic idea is, when you build a cross-toolchain, you give it a local directory path — a directory that exists on the build system — and specify that that local directory will eventually become the root filesystem directory on the target system. The cross-toolchain knows about the sysroot location and knows to look there for system header files and the C library and so on. CBL uses the standard sysroot approach, in a slightly-nonstandard way: everything is installed not into the sysroot directory per se, but into a subdirectory of the sysroot called /scaffolding. That way, when we finally get booted into the target system, the root filesystem will be empty, except for the /scaffolding directory: everything we create after that and install outside of /scaffolding will become part of the final system.

Depending on the target, GCC has a variety of options that control how it operates. Generally, these can all be specified with command-line arguments beginning with -m. For many of these options, default values can also be specified when GCC is being configured.

For example, for many targets, GCC can build programs with a variety of application binary interfaces (ABIs) — this is the machine-code-level interface between programs, libraries, and the operating system; it defines things like the way that registers are used when invoking functions.

An example of a CPU that supports multiple ABIs is the 64-bit x86 architecture, which is called x86_64 or amd64. These processors can run programs in 64 bit mode (with 64-bit pointers and all AMD64 processor features enabled), 32 bit mode (32-bit pointers and only i686 processor features enabled — in this mode, fewer CPU registers are availble to programs, for example), or a hybrid "x32" mode (32-bit pointers but all AMD64 processor features enabled). You can specify the ABI to which GCC will compile by using an -m32, -m64, or -mx32 command-line argument.

You can also override the default ABI when configuring GCC by specifying a --with-abi configure directive; or, for some target architectures, other options, like --with-multilib-list or --enable-targets or probably a combination of those things. Confusingly, even though the way you set an x86 GCC to generate code for the 64-bit ABI is to use the configuration directive --with-abi, the runtime command line option -mabi doesn’t override that selection; it only selects between the sysv calling convention used by UNIX-ish systems and the ms convention used by Microsoft Windows.

This confusion is characteristic of the GCC configuration and usage options: a lot of the options that are available, and their meaning, depends on what architecture is targeted by the compiler, and the same options or terms can mean very different things for different targets.

There are a few ways to figure out what options are available for a specific target architecture: the installation manual lists some of them, mostly in the "configuration" section. The GCC manual has information in section 3.18 ("Machine-Dependent Options" — if you’ve built a set of trusted host tools specifically for the CBL build, the correct info file can be found at /usr/share/info/gcc.info). You can also look at the configuration script, gcc/config.gcc, to see what options are accepted for each type of target. And, finally, after building GCC for the target architecture, you can run gcc --help=target to see what options are available and what values they can have; and you can also find the actual compiler, cc1 (for C) or cc1plus (for C++) under the libexec/gcc/aarch64-cbl-linux-gnu/$VERSION directory, and run it with the --help command-line argument to see what options are enabled and disabled.

In addition to the selection of ABI, GCC can be instructed to optimize for specific CPUs or CPU families for many target types. The command-line arguments that control this behavior are similarly target-specific — look for -mtune, -march, -mcpu, and things like that.

In CBL, any set of these configure directives can be specified for the cross-toolchain in the TARGET_GCC_CONFIG parameter. It’s generally a good idea to specify default values for all of the options supported by the target platform. (If you don’t want to set any options at all, you can set this to an empty string.)

GCC is generally built with a large number of libraries included. Some of those fail in some circumstances — for example, x86 CPUs can’t build the libquadmath library when the C library being used is uClibc (or, at least, that was the case some time ago, the last time I tried); and the libsanitizer library fails to build when compiling for 64-bit Sparc machines. The easiest way to work around those problems at this stage is simply to disable those libraries, and that seems like a fine approach considering that the cross-toolchain we’re building here is just going to be used to build the ephemeral scaffolding programs. So if any libraries cause the build to fail, try adding an appropriate --disable directive to TARGET_GCC_CONFIG.

You might notice that, in this build, we’re using the same host-system builds of the various arithmetic dependency libraries as we used for the host-prerequisite GCC (or the same ones that are used for the native system GCC, if you’ve skipped the host-prerequisites). It’s totally unnecessary to build them again for this GCC — the cross-compiler we’re building here will only run on the host system, so the target architecture is irrelevant to it.

To build the minimal libgcc, we specify the configuration options --without-headers and --with-newlib. This is a bit sloppy — the first of those options is the only one that should be necessary — but the last time we tried building a static compiler without the newlib directive, it didn’t work. Then we’ll use that compiler and libgcc to build the C library; and then we can use the files provided by the C library to go back and build a full, functional GCC compiler.

The only targets we build at this point are all-gcc, which produces the plain compiler, and all-target-libgcc, which produces the minimal libgcc that compiler needs.

glibc is the C standard library produced as part of the GNU project. It contains the implementation for all of the functions that are assumed to be available in C programs — like printf and so on. It also provides a dynamic loader, which almost all programs use to find shared libraries at runtime, and a few miscellaneous utility programs.

Since all C and C++ userspace programs (except the Linux kernel itself) link against the C library, glibc is by far the most deeply-embedded component of a GNU/Linux installation. Upgrading the Linux kernel is a relatively trivial operation compared to upgrading the C library installed on a computer.

There are alternative C libraries that can be used instead of glibc. However, glibc is the C standard library used by the vast majority of GNU/Linux systems; using an alternative library like musl or uClibc-ng as the primary C library may cause problems somewhere down the line.

This builds a sysroot libc (for the target architecture), configured to install into the /scaffolding subdirectory of the sysroot.

That might be a little bit opaque, so let’s break it down a little bit. As mentioned earlier, the sysroot framework is all about setting up a path on the host system that will eventually become the root filesystem on the target system. And, also as mentioned earlier, the goal in the first stage of the CBL build process is to set up a kernel and minimal userspace for the target system, with the entirety of that userspace contained in a /scaffolding subdirectory rather than using the conventional filesystem paths (like /bin and /lib and the /usr directory structure and all the rest of that stuff you’re used to seeing).

What we’re building here is the libc that’s going to be used to build all the scaffolding programs and libraries — the stuff that we’re cross-compiling — so we want it to be contained in the scaffolding directory, and found by the scaffolding programs at runtime in that location. Hence, it’s a sysroot libc, with an extra prefix to move it from its usual normal /lib and /usr/lib directories to the /scaffolding/lib directory.

That means we configure it with a prefix of /scaffolding, but then when we install it we tell it that the root of the installation location is the sysroot directory /home/lbl/work/sysroot.

Simple, right?

If you ever want to set up a completely standard sysroot toolchain, by the way, it works pretty much the same way as this but you specify --prefix=/usr and --with-headers=/home/lbl/work/sysroot/usr/include. There is some magic in the glibc configuration or build machinery related to the --prefix directive: if you specify a prefix of /usr, the bits of glibc that are conventionally installed in /lib will be put there, rather than in /usr/lib.

Since this glibc is built for the target machine architecture, a number of tests run by the configure script won’t work right. The way we work around that, for glibc as with everything else that uses the GNU build system, is by setting the correct values in a config.cache ahead of time.

An option that we’re not using here is --enable-kernel, which can limit the amount of compatibility code built into glibc to support old Linux kernel versions. Since the glibc being built here is temporary and will be discarded in its entirety, saving a little bit of space here is kind of pointless. That option also makes the build more fragile, since the kernel version that will be checked by the code is the host system kernel; we don’t want to make any more presumptions about the host system than we must.

The glibc build process uses the makeinfo program to create the documentation, and the texinfo source file specifies a document encoding of UTF-8. When using some versions of perl, this leads to problems — it’s unclear why; perhaps this is because makeinfo only works right with UTF-8 documents when being run with UTF-8 localization settings, or maybe something else is going wrong.

Regardless of exactly what triggers the issue, there’s an easy way to work around it: just remove the @documentencoding directive from the libc manual source file.

As mentioned earlier, in gcc: The driver program, the gcc program we think of as a compiler really just runs other programs, and it uses a bunch of directives called "spec strings" to determine what programs to run and what options to give them. The format of specs strings is documented in the GCC documentation in section 3.15, "Specifying Subprocesses and the Switches to Pass to Them." Spec strings don’t have the most readable structure — I find it helpful to think of them as being written in a domain-specific language, because to me they look as much like line noise as complicated regular expressions do — but sometimes there’s no better way to figure out what is going on than to read the specs, and there is often no better way to adjust the behavior of gcc than to modify the specs it is using.

We have to do this — modify the specs — a few times throughout the CBL process, primarily because we need to control how gcc runs ld to link programs.

You can see the spec strings that gcc will use by running gcc with the -dumpspecs option. The default specs are built in to gcc, but you can provide your own specs to override the default behavior by using the -specs= command-line argument or by creating a specs file and putting it in a specific location in the filesystem.

We’re going to do the latter. The basic process is to first dump the specs file to the location where gcc will look for it:

Then we can modify it however we need to, and gcc will use the modified version.

Not to belabor the point, but remember that the whole purpose of this cross-toolchain is to let us build the scaffolding that will then allow us to build the final target system entirely from source code.

While we’re using the scaffolding tools to build the final system, we want to do a native build of everything, including glibc, so we want the scaffolding to be independent of any filesystem locations that will still be present in the final system. In particular, this includes /lib and /usr/lib. That way, once we’re done doing the target system build, we can delete the scaffolding directory altogether and be confident that there are no lingering host-system artifacts or ephemera on it.

When the standard GNU toolchain builds an executable, it almost always links it against the dynamic link library (sometimes called the "dynamic loader" or "program interpreter"; this is something like ld-linux.so.2 or ld.so.1, and is conventionally found in a top-level directory called /lib or a multilib variant like /lib32 or /lib64).^[5] That’s normally fine, but we want the scaffolding to be entirely independent of /lib. So we need to adjust our cross-toolchain so that the programs it builds look in the /lib directory under the scaffolding location for their libraries, including the dynamic link library. This is done by modifying the GCC specs file.

There’s another problem we need to work around, as well: gcc doesn’t provide any good way to tell it where to find some object files that need to be linked into every program: crti.o, crti1.o, and crtn.o. These are provided as a part of glibc, so like the rest of glibc they were installed into /home/lbl/work/sysroot/scaffolding, specifically into its /lib subdirectory. But, of course, that’s not a location where gcc normally expects to find them. So at this point if you were to try compiling a "Hello World" program with your cross-toolchain, it would complain that the cross-ld can’t find crt1.o or crti.o.

You can tell exactly what is going wrong by repeating the compile with -v, to get gcc to print out all the commands it’s running: cc1 to compile the code, then as to assemble it, then collect2 to link it. And apparently collect2 is running ld, which is producing the error message. (That’s weird, but you get used to weird stuff when you’re trying to figure out toolchain problems. There’s also no explicit execution of cpp; that’s because the C pre-processor is actually implemented in the libcpp library and is invoked as function calls to that library by the cc1 compiler program.)

Once you have the actual command line that’s failing, you can try adjusting it to see if there’s an easy way to get it to work. For example, the command line just names the startup files without any path components at all. After fussing around for a while looking for pleasant alternatives, the best solution I found was to specify the filenames with absolute paths. So that’s what we’re going to do in the specs file: replace every occurrence of the bare filename with absolute paths, for all the object files that appear in the scaffolding/lib directory.

We can now verify that the cross-toolchain is able to build programs successfully, and is set up to link against the sysroot glibc in the /scaffolding directory, by compiling any simple program (like "Hello World") and then running readelf on it — there will be a line in the program headers section that says "Requesting program interpreter:" and should contain the path to the dynamic link library in the scaffolding location.

After building a significant toolchain component, it’s a good idea to make sure that it works as intended. This is a simple smoke-test: it just compiles a "Hello, World" program and then inspects it to make sure it was built as expected and runs properly.

This verifies that the cross-toolchain and emulator work properly: look at the machine type and dynamic linker location for a compiled program, and then make sure that it runs in the userspace emulator. This proves that the cross-toolchain and QEMU were properly built with a compatible target architecture and so on.

This is compiled with:

To verify that it’s linked properly, use readelf.

The Machine line should indicate the target architecture, rather than the host architecture, and the program interpreter requested by the program should be under the /scaffolding/lib directory (which is where the dynamic loader will be found once we’re booted into the target system). If either of those is not the case, the grep commands will fail, causing the CBL build process to abort.

One last thing we can usefully do at this point is verify that the user-mode QEMU emulator can actually run programs for the target architecture.

This is a bit tricky when running the dynamically-linked program we just built, because it is expecting to find the program interpreter at /scaffolding/lib but in fact it’s actually at that location under the sysroot directory. Luckily, the user-mode QEMU emulator can be told where to find library files, using the -L command line argument or the QEMU_LD_PREFIX environment variable.

This will produce a friendly greeting, or — if something goes wrong — will, again, cause the build process to abort.

(If you’d like, you can try running the hello program outside of QEMU as well; that should produce an error message like "cannot execute binary file." And, again, if it doesn’t, that means something is horribly wrong!)

pkg-config is a program that makes it easy to find installed libraries and header files and things. Many programs use pkg-config in their build processes to find out if the libraries they depend on are present on the build system, and to find out what linker and include directives should be used to compile and link against them.

The original pkg-config program can be found at pkg-config.freedesktop.org. At some point in its development, its developers decided to use functions from the glib library; unfortunately, glib uses pkg-config to find its own dependencies (really, just zlib — glib doesn’t have a lot of dependencies), which introduced a cyclic dependency. That doesn’t cause any really intractable problems — a couple of environment variables can be defined when building glib so that it doesn’t have to use pkg-config to find zlib — but cyclic dependencies are always kind of horrifying: not to sound like a broken record, but the idea with CBL is to start with a minimal set of binaries and a pile of source code and turn that into a whole system. (Even worse than the glib dependency, pkg-config requires itself, so that it can find and link against the glib library, unless additional environment variables are provided.)

The situation has since been resolved, but it was resolved by bundling glib along with pkg-config. This isn’t a particularly elegant solution: the pkg-config distribution is about 11.5 megabytes of code, unpacked, and about 9.5 megabytes of that is glib.

A different approach was taken in a fork of pkg-config, "pkg-config-lite," which includes just the snippet of glib that is needed by pkg-config: that’s better, but still not ideal.

This brings us to pkgconf, a completely separate implementation of the pkg-config program. It has no external dependencies, doesn’t bundle any third-party code within its own distribution, and also has a design that I find preferable to the original pkg-config (it internally builds a directed acyclic graph of dependencies, rather than building an in-memory database of all known pkg-config files at runtime and then resolving dependencies from that database).

As distributed, the pkgconf program is missing a lot of files that are normally produced by the GNU autotools; the conventional way to build it is to start by running the autogen.sh script provided with the package distribution. However, one of the places the pkgconf program is built is as part of the host prerequisites; if that’s being done, it’s probably not a great idea to rely on the autotools being present at prerequisite build time. We can instead simply patch the source distribution to create the files that would normally be created via autogen.sh.

This is built here because pkg-config is already present on the host system (as a prerequisite, if nothing else), and it probably knows about a lot of host system libraries. While building the scaffolding, we need to ensure that their build processes don’t find (and try to link against) any of the host system libraries. Since the host system libraries are for a different machine architecture, this would cause build failures. We can do that by putting a new version of pkg-config, configured only to look in the scaffolding directory, at the head of the PATH while building them.

We could probably get by with a simple shell script that always just says "I couldn’t find anything!" when asked for dependencies. On the other hand, it takes around ten seconds to build pkgconf and set it up, so why not just do that?

In addition to the pkg-config symlink, we create a symbolic link aarch64-cbl-linux-gnu-pkg-config so that any build process that tries to find dependencies using a target-system pkg-config program will find ours.

Most Linux filesystems support "extended attributes" — arbitrary name/value pairs that can be associated with files or directories. These can be used for any purpose; for example, you might attach a "user.file_encoding" extended attribute to a text file, if it’s encoded unusually.

One of the primary uses of extended attributes is to implement access control lists or capabilities.

The attr package provides programs that allow extended attributes to be viewed and modified.

Since the configuration repository preserves extended attribute metadata as well as basic owner and mode, and since the package-users build script sets a user.package_owner attribute on all files that are installed by packages, we need the getfattr and setfattr commands early in the target system build.

Bash is a shell — it provides a command line prompt, parses commands and pipelines entered on the command line, executes programs based on those commands and pipelines, and then does it all again. If you’re interacting with a Linux system, and you’re not using a graphical environment like Xorg, you’re using a shell like bash. And, probably, you’re using bash — there are other shells, but they are not nearly as common.

The bash maintainers don’t provide all patchlevels for convenient download; you may have to download the tarfiles for the main version (like 4.3) and then separately download and apply all the update patches. The CBL repository, for convenience, has a tarfile that includes all of the patches. As of version 5.1.8, this may not be necessary — in addition to the 5.1 tarfile and the separate patches for it, I found a tarfile for 5.1.8 on the GNU FTP server.

Several of the tests run by the configure script don’t work right when bash is being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

Bash includes its own memory allocation routine, but historically it has not always been reliable. We use --without-bash-malloc to disable it so that the C standard library’s malloc is used instead. For the scaffolding version of bash, we also configure it to be linked statically — that reduces the number of pieces that are needed to get the minimal scaffolding-based system to boot.

M4 is a macro processor: it copies its standard input to standard output, expanding macros as it goes. It has a set of built-in macros that it understands, and it’s possible to add user-defined ones as well. M4 is used extensively in the GNU build system, primarily by Autoconf.

Bison is a parser generator. That means it takes a description of a language (in a specialized format called a "context-free grammar") and converts it into a parser for that language. This is mostly useful when writing compilers: the parser is the part of a compiler that takes a stream of tokens and figures out what syntax elements they represent. Parsers are fiddly and difficult to get correct, so it’s common to generate the code for parsers using tools like Bison.

The name "Bison" is kind of a joke. The original parser generator that was commonly available on UNIX systems is called "Yacc," which is an acronym for "Yet Another Compiler-Compiler." When the GNU project implemented their own parser generator, they called it "Bison" because "Yacc" sounds like "Yak."

Bison’s installation process also creates a program called yacc that simply runs bison in yacc emulation mode.

The absolute path to the m4 program is written into bison, so we need to make sure that the scaffolding bison will expect to find m4 in the scaffolding directory. This adds a little complexity, though: the bison configuration script checks to see whether lex is available and, if it is, whether it is flex. flex uses m4, so if we tell the bison configuration that m4 can be found in the scaffolding, flex will try to use that version of m4, which was built for the target system rather than the host system. As with other packages that use the GNU Build System, we can use config.cache to tell bison that flex is simply not available.

Bzip2 is a compression program like gzip, but uses the Burroughs-Wheeler block sorting algorithm and Huffman coding. It is a lot slower than gzip for both compressing and decompressing, but compresses data much better in most cases.

Bzip2 doesn’t use the GNU build system, so there isn’t a configure script.

There isn’t really a configuration step for bzip2, but we’ll use the configuration stage to modify the build process so it won’t try to run the tests — they won’t work when building with a cross-toolchain.

The regular Makefile only creates a static version of the library, which is fine; however, it doesn’t compile that library with -fPIC, which causes problems later on when other programs try to use it. We can add that directive directly in the Makefile.

Because bzip2 doesn’t use the GNU build system, we need to specify the cross-tools as arguments to make.

The Core Utilities are basic file, shell, and text manipulation utility programs: things like cat, chmod, chown, cp, …​ stuff like that. You use these all the time.

Version 8.32 of coreutils introduced a misfeature that breaks 64-bit ARM builds on Linux. Paul Eggert provided a patch that reverts that misfeature, so we apply that here.

A couple of the tests run by the configure script don’t work right when the coreutils are being cross-compiled. As with other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

Diffutils is a bundle of programs that find differences between files and help you to merge them together: cmp, diff, diff3, sdiff.

The most important program here is diff, which finds differences between files. It can find differences between two files, in the simplest case; but, more commonly, diff is used to find all the differences between all the files in two entire directory trees.

You can save the output from diff when used this way as a "patch file", and then use the patch program to take one of the directory trees and make it look just like the other one.

This is handy when distributing modified versions of software packages: if you’ve made a minor change to GCC, for example, and you want to submit that change to the GCC mailing list for consideration by the GCC maintainers, you can use diff between the original GCC source tree and your modified GCC source tree, and then include the output from diff in your email.

When cross-compiling, the configure script guesses that the getopt function provided by GNU glibc is not available and tries to use an internal one instead. This doesn’t work, for some reason (possibly because of behavior changes in GCC 7). Since the glibc getopt is available, we can just tell the configure script not to guess about it.

util-linux is a grab-bag of miscellaneous Linux utility programs, for all kinds of things: disk partitioning, filesystem creation and validation and mounting…​ all kinds of stuff like that. (I’d expect a lot of this stuff, like more, mount, and umount — to be a part of the GNU system, perhaps in the coreutils package, but that’s not the only thing I find surprising about the GNU/Linux world.)

We configure with an option that tells the build machinery not to chgrp the wall program to the tty group. This will probably make wall non-functional, but that’s perfectly okay — wall can be used to send messages simultaenously to all users logged on to the system, as for example if the system is going to be shut down or something of the sort, which is not very useful on single-user computers and certainly not necessary in the minimal scaffolding userspace.

You’ll notice that we’re configuring this package with a prefix of /home/lbl/work/sysroot/scaffolding instead of just /scaffolding (and installing without a DESTDIR). That’s because, when we configure and install util-linux the way we’re doing most of the scaffolding components, we’ve sometimes encountered problems where some components can’t find the libuuid library that is provided by this package. There are probably other ways to address that, and it’s not really clear whether those components are strictly necessary, but this non-standard configuration scheme doesn’t cause any problems and works fine.

The installation routine for util-linux changes the ownership and mode of some programs (like mount and umount) to be setuid to root. This doesn’t work when running the installation as a normal user, as is being done in this section, so we disable that.

The ext2fsprogs are utilities for managing the ext2 (and later) filesystems. ext2 was the first popular filesystem for Linux systems and its latest incarnation, ext4, is the most common. This package contains programs for creating filesystems, reconfiguring them, checking them for problems, that sort of thing.

There’s some overlap between e2fsprogs and util-linux: both provide libuuid and libblkid libraries, a uuidd daemon, and a fsck script. We skip those from e2fsprogs.

There is a glitch in the configuration logic that creates a file outside of the DESTDIR-specified path when there is no /etc/cron.d directory. This can be avoided by explicitly specifying that there is no crond directory at all.

Expat is a library that provides a stream-oriented XML parser. It’s used by all sorts of other programs.

file guesses file types by looking for particular characteristic byte values within them. If you’ve got a file and you’re not sure what it actually is, you can run file on it and be told things like "that’s a text file," or "that’s a JPEG image."

The Find Utilities are basic directory searching programs: mostly find and locate. This package also includes updatedb, which creates the file name database that locate uses; and xargs, which takes lists of file names produced by find and generates command lines that operate on all the files in those lists.

A couple of the tests run by the configure script don’t work right when the findutils are being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

AWK is a special-purpose programming language that makes it easy to do simple text manipulations. (The name is an acronym of the three people who designed the language: Alfred Aho, Peter Weinberger, and Brian Kernighan.) Gawk is the GNU implementation of the AWK language.

If you’re comfortable in perl or python or ruby, you probably won’t have much use for awk; on the other hand, if you’re working in a bash script, awk might turn out to be very handy.

Zlib is a compression library. It implements the same Lempel-Ziv compression algorithm as gzip and info-zip, which means it doesn’t compress data as effectively as most other algorithms (like the Burrows-Wheeler algorithm used by bzip2 or the LZMA algorithm used by xz-utils), but on the other hand it doesn’t use very much memory to compress, and it’s pretty fast.

There are a lot of programs that link against zlib to get basic compression capabilities. We’re not totally sure which components will fail to build if it’s not available, but zlib itself has no dependencies and is really fast and small to build, so making it available is no big deal.

Gettext is a set of tools that can be used by other programs to provide internationalization and localization capabilities. This lets a single program provide user interfaces and user-facing messages in a variety of languages without requiring it to be rebuilt.

One of the tests run by the configure script doesn’t work right when the coreutils are being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

If the emacs editor is available on the system, the gettext build will do something with it. I don’t really understand what it is, but it generates a couple of gigabytes worth of log output and makes the build process take orders of magnitude longer than it otherwise would. Luckily, it is easy to disable that with a configuration flag.

Grep searches input files for lines that match patterns called "regular expressions." It often prints those lines out as well.

We have heard that the program name "grep" was originally derived from the ed command "g/re/p", which does basically the same thing as the command-line grep program, and wikipedia makes this claim as well. Maybe it’s true!

The GNU libc library provides regular expression functions, but grep ignores them by default and uses its own bundled regex library. We use a configuration flag to override that behavior.

Gzip is a compression program that basically does the same thing as the zlib library, but with a command-line interface.

The default -Werror setting for gzip causes a format-truncation message produced by GCC 8 to abort the build. I am not all that concerned about format-truncation problems when printing error messages, so I just disable that setting.

The generated Makefiles for gzip include a -Wabi directive that GCC 8 complains about. We can just remove that directive.

The Kbd package contains key-table files and keyboard utilities.

The vlock program requires Pluggable Authentication Modules (Linux-PAM), which is not part of the basic CBL system, so we always disable it. Conversely, other programs that aren’t built by default might be helpful, so we enable those.

We need some of the programs provided by kbd (notably, openvt, which will let us run interactive shells on virtual terminals in the minimal target system) in the minimal userspace.

Libffi is a portable library that helps programs written in one language to invoke functions that were written in a different language.

LibreSSL is a fork of OpenSSL with the goals of modernizing the codebase, improving the reliability and security of the code, and applying better development practices.

LibreSSL should not be needed in the scaffolding, but the current version of rubygems has issues when used in a ruby installation without openssl support.

GNU make is a build automation program. To use make, you set up one or more configuration files — the primary one being called, by convention, Makefile — that declare recipes for producing intermediate and final program artifacts, and dependencies between artifacts, and the various targets that can be produced. At runtime, make looks to see what artifacts exist already and which source files have a timestamp indicating that they’ve changed after dependent artifacts were produced, figures out based on that analysis exactly which artifacts need to be rebuilt, and then executes the recipes that will produce those artifacts.

That’s all pretty awesome!

The downside is that Makefiles are not particularly clear or readable, and for large projects they get pretty big and complicated. That’s why there is a thing called "litbuild"!

But the vast majority of system components use make to automate their build process, so you really have to have it available regardless.

Ncurses ("new curses") is a library that provides terminal control features so programs can provide advanced text-based user interfaces. It’s a free-software version of a similar library (just called "curses") that was developed at Berkeley.

There are lots of programs, even just in the scaffolding we’re setting up, that use ncurses. bash and vim are among them.

Patches are available in a version subdirectory of the overall package download location — that is, patches for ncurses 6.2 are in the "6.2" subdirectory of the ncurses package directory. Patches need to be applied in order; there are instructions in the README file there, but the basic idea is that you find the latest patch bundle, called something like ncurses-6.1-20181020-patch.sh, and run that as a shell script; then you apply all the patch files later than that in sequence. I’ve compiled all of the patches for ncurses 6.2 up through the 20201219 patch into a branch-update patch, available in the freesa file repository.

Remember diffutils? The most important program included in diffutils is diff, which finds differences between two files or directory trees. Patch is the conceptual inverse of diff — it takes a file with all the differences found by diff as input (which is by convention called a "patch file"), and applies all of those differences as changes to files in a directory tree.

This is really handy. For example, the CBL repository contains a bunch of source tarfiles that were obtained directly from the project web sites; and it also contains patch files that can be applied to those source tarfiles to apply changes that we have found to be necessary or important when building a CBL system.

Pth is a portable threads library; it provides cooperative priority-based scheduling for multiple threads within a program.

The build system for pth doesn’t work well when parallelized, so we explicitly disable parallel make jobs by specifying -j1.

On some servers, the version of config.guess distributed with pth is too old to recognize the target triplet. It’s easy enough to update it to the version distributed with GCC.

In some cross-compiled scenarios, pth fails to compile because it considers versions of the Linux kernel later than 2.9 to be "braindead." I got a patch from https://bugzilla.redhat.com/attachment.cgi?id=591825 that corrects the problem.

Ruby is a programming language. There are several implementations of that language — rubinius and jruby are others — but the canonical one is the C implementation that was created by Yukihiro "Matz" Matsumoto, and that’s also the best one for our purposes in CBL because it has the fewest upstream dependencies.

CBL includes Ruby as a core component because CBL itself is designed to be used with the litbuild program — CBL consists entirely of litbuild blueprints — and litbuild is written in Ruby.

On 64-bit ARM systems, the Ruby build process exposes an issue with GCC (documented at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84521) when compiled with -fomit-frame-pointer and an optimization setting of -O1 or -O2. Since the default behavior for modern versions of GCC is to omit frame pointers, it’s important on such systems to specify -fno-omit-frame-pointer when building Ruby. It’s possible that other architectures have a simliar issue; if your ruby build crashes with a "stack smashing detected" error message, try adding that option to your builds as well. (The default configuration for CBL specifies -fno-omit-frame-pointer in the target-system CFLAGS.)

Once we boot into the target system, we’ll use litbuild to generate all the scripts that will build the final CBL components. That means we’ll need ruby available!

As with the file package, ruby can have issues in cross-compilation scenarios unless there is a native ruby installation of the same version. This doesn’t always happen, but the installation of a scaffolding ruby 2.7.0 failed when the system ruby was version 2.6.5; if something similar goes wrong for you, a version mismatch is a good thing to check.

The version of rubygems (about which see more below) distributed with ruby 2.7 really wants to load the ruby openssl library, which requires OpenSSL (or a fork of it, like LibreSSL).

Ruby includes a module called fiddle, which wraps the libffi library and allows ruby programs to call functions written in C or other languages. In some rare cases — the only one I’ve come across is when building ruby with the x32 ABI — the libffi build fails when it tries to assemble src/x86/win32.S with a cross-toolchain. That file probably allows libffi to call functions in Windows DDLs, or something; I am pretty sure it’s not something we’ll need to use. The sed command here removes it from the build process entirely.

Sed is kind of like a text editor, but instead of allowing you to modify text files interactively (like ed and vim do), sed acts as a filter: it takes a text file as an input stream, makes changes to that input stream, and produces the resulting modified text as an output stream. (The name, sed, means "Stream EDitor").

This is a fairly powerful and flexible thing to be able to do, but it’s not something people often need to do — like gawk, sed is most often handy in the context of a bash script or something like that, where a more powerful general-purpose language like ruby or python or perl isn’t convenient for whatever reason.

On the other hand, the automated build processes for some of the CBL components use sed, so you have to have it around anyway.

We’ve said this before, but it’s really important so it bears repeating: the job of the Linux kernel is to mount the root filesystem and then run a single program, conventionally located at /sbin/init, with process ID (PID) 1. It’s the responsibility of the init program to launch all the other userspace programs.

The sysvinit package provides an init program (along with other related programs), in the style of the init program that was used by UNIX System V — hence the name, "SysVInit". It was used by the vast majority of GNU/Linux systems for many years, although most modern distributions have switched to other init programs.

In the minimal "scaffolding" userspace, we don’t need anything very sophisticated for the init program. So we use sysvinit to manage the scaffolding userspace: it’s easy to cross-compile and set up, and it’s relatively simple to understand how it works.

This is the one of the few packages in CBL that is set up only in the scaffolding, and will not also be a part of the final system.

There are a few hard-coded paths we should adjust: the location of the init program itself, the location of the master configuration file (inittab), a script that init will use when running processes, and the shell that will be used to run that script.

The Makefile for this package is smart enough to add the linker flag -lcrypt if the C library it’s linking against is GNU libc…​ but the test for this is whether there is a file at /usr/lib*/libcrypt.a, which is not the case for all systems, and is a meaningless test for cross-compilation scenarios like this one. Since we are using GNU libc, we can just force the test to return true.

Tar is a program for manipulating archives of files. The name derives from its original purpose, back in the dawn of time, which was to manipulate archives on magnetic tape — ergo, "tar," for Tape ARchive. Actually storing archives on magnetic tape is pretty rare these days, but people still use tar-format files for all kinds of things.

These files are known as "tarfiles," for obvious reasons, or as "tarballs" for less obvious reasons. A speculation I found on Wikipedia is that the term references the tendency of actual tarballs (blobs of petroleum floating in the ocean) to get all sorts of things stuck to them; apparently, someone thought that was reminiscent of how the tar program collects a bunch of files together.

All of the source code distribution packages for CBL packages are compressed tarfiles, so the tar program is really important for CBL!.

The tar package also provides a program called "rmt" that lets you manipulate magnetic tape drives attached to other computers. This is an even more fringe thing to do in this day and age, but It only weighs in at about 50kb, so it’s not hurting anything.

Several of the tests run by the configure script don’t work right when tar is being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

Texinfo is the official documentation system for the GNU project. It uses input files in a standard format to produce a variety of outputs, both printed and online.

Texinfo is used in the automated build process of many of the CBL component projects.

When compiled against glibc 2.34, texinfo 6.8 runs into a problem due to an obsolete version of gnulib. We can patch gnulib to work around this issue.

When cross-compiling, the configure script makes some incorrect guesses about the availability of functions provided by glibc. We can override those guesses in a config.cache file.

Even after doing that, the presence of the gnulib version of getopt.h causes the build to break. That’s easy enough to work around simply by removing it.

The build process creates two scripts that expect perl to be in the location where it’s found on the host system, rather than the place it will be available in the initial target system, so we fix them up after installation.

Back in the early days of Unix, someone wrote a text editor called "ed" (which stood for, get this, "EDitor"). Ed is a line editor, which means that generally it lets you modify a line at a time. Ed is still available, and in fact still actively developed — GNU ed 1.14.1 was released in January of 2017 — but most people want a more interactive editor, and so one of the old proprietary Unixes eventually included an editor called "vi" ("VIsual editor") to fill this need.

Since vi is proprietary, various people implemented similar programs, which they released under open source licenses of various sorts. One of the people who decided to do this is Bram Moolenar, who created Vim; its name stands for "Vi IMproved," because it’s massively superior to vi.

Vim is distributed under a different software license than most of the packages that make up CBL: the Vim License. This license is GPL-compatible, but is described as "charityware" — Vim’s author urges users to make a donation for needy children in Uganda.

As far as that goes, you probably don’t need to install vim at all. There are plenty of other text editors that some people prefer. The default editor in some distributions is nano, and some people like their text editor to be their primary tool for interacting with the computer and use emacs. Whatever your preferences, though, you absolutely need a text editor of some sort, and vim is our favorite, so that’s the one that winds up in the basic CBL blueprints.

As with many scaffolding pieces, the configure script for vim doesn’t play nicely with cross-compilation, so we pre-fill a config.cache with a bunch of values that it won’t be able to discover on its own.

An interesting difference between vim and most other packages we’re building is that it automatically picks up entries from a config.cache file located in the src/auto directory, and apparently doesn’t support providing a reference to config.cache in the top-level configure run.

There are a lot of programs that expect a text editor called "vi" to exist, so we’ll create a symlink after installing vim.

The XZ Utils package provides a compression/decompression program with an interface similar to gzip, but using the LZMA algorithm. This is very slow when compressing, very fast when decompressing, and compresses more effectively than most of the other common compression programs, so it’s used fairly often when providing archive files for download.

The compression algorithm used by the XZ Utils is the same as that used by lzip, but the file format is more complex and the compressed output it produces is usually slightly larger than that produced by lzip.

Having constructed all the scaffolding we can build on the host system, it’s time to set up for the second (target) stage of the build. The target-side build will be run automatically when the target system is launched.

As mentioned elsewhere, you can add data to the entropy pool without any difficulty from any userspace process: all you have to do is write data to /dev/urandom. However, that doesn’t help to initialize the entropy pool, because it doesn’t increase the kernel’s estimate of how much entropy is actually available. To do that, we have to execute a system call that is only available to processes running as the superuser.

The system call in question is ioctl, which is kind of a catch-all that exposes arbitrary functionality in device drivers. ioctl itself is short for "Input-Output Control." The idea is: since there is a huge variety of input/output devices that might be available to your system, and there’s no way for Linux to provide specific system calls to exercise every distinct function of every one of those devices, any device driver can define a set of ioctl operations. Userspace applications can then use the ioctl system call to invoke any of those operations, using a file descriptor that references one of the device’s special files under /dev.

The device driver for the /dev/random and /dev/urandom special files happens to provide an ioctl that lets you add data to the entropy pool. We’re going to write a tiny C program that feeds some data to the kernel using that ioctl, so that the entropy pool will be fully initialized immediately.

Note that it is an extremely bad idea to use this program unless you are confident that the input data really is unpredictable, or unless (as in the case of the target-side CBL build) you are equally confident that there is no reason to be concerned about the quality of random data available to userspace processes.

In many cases, this program isn’t needed in a full working system — the rngd daemon from the rng-tools package feeds the kernel entropy from hardware random number generators, like the one built into modern x86-architecture CPUs. But CBL supports multiple computer architectures, and some of those don’t have anything suitable for rngd, so addentropy can be helpful in those cases.

The addentropy program is derived from ec2seed, at https://github.com/akkornel/ec2seed, which is available under the terms of the GNU GPL Version 3. Accordingly, there’s no issue with distributing the addentropy source as part of CBL; all the code within CBL is similarly licensed under the GNU GPL Version 3.

As with almost all C programs, we start by including header files.

We define a constant for the number of bytes we’re going to feed to the entropy pool. That’s really not all that important, but it makes some later parts of the program slightly clearer.

Before and after adding data to the entropy pool, we can obtain the kernel’s approximation of how much entropy it already contains, using another ioctl provided by the /dev/random device driver. Let’s define a function for that.

Now we can write the main routine for addentropy. As usual in C programs, we start by defining all of the variables we’ll use.

We need to open a file descriptor on the /dev/urandom special file so that we can invoke ioctl operations on it. And we need to allocate a buffer that we can read the data in to.

According to random.c, which is the source code for the random-number device driver, the entropy_count field of rand_pool_info structures is denominated in units that are an eighth of a bit each. I have no idea why. So maybe we should multiply RANDOM_BYTE_COUNT by 64 rather than by 8 here? But this appears to work perfectly well, so I’m not messing with it.

Now we need to obtain some data (which we will presume to be random). The way we’re going to do that is simply read it from the standard input stream. Since we’re using some low-level file manipulation functions, we have to deal with the possibility that any given call to read actually produces less data than we request; therefore, we set up a loop and read repeatedly until we have all the data we want.

Now we can add the data to the entropy pool, using the RNDADDENTROPY ioctl. We’ll also print out a description of what happened.

There’s nothing left to do other than clean up the resources we’ve allocated and terminate the program. Technically we don’t have to clean up the resources — Linux will do that automatically as the process is reaped — but it’s good practice to do it.

That’s it!

The command we use to compile the program, and the location where it will wind up, depends on whether we’re building this in the scaffolding or for the final target system.

For the scaffolding, we need to use the cross-compiler to compile addentropy, and it needs to wind up in the scaffolding bin directory.

The sysvinit program runs commands as directed by a file called inittab. It’s documented in a man page provided with the program (man 5 inittab to read it), so I’m not going to go into a lot of detail here; but I’ll explain a bit about what’s going on so you don’t have to look elsewhere. In this section, lines from inittab will be interspersed with the scripts that they execute, which will hopefully result in a clear narrative flow.

Sysvinit allows different sets of programs to be run, in case (for example) you sometimes want networking to be enabled and other times don’t; these are called "runlevels." For CBL we just use runlevel 2, which performs the full target side build. (Runlevels 0, 1, and 6 are reserved for shutdown, single-user mode, and reboot respectively.)

Each line of inittab has several fields separated by colon characters. In most cases, an inittab line defines a command that init will run under some circumstance. The first field is always a two-character identifier; the second field is usually a set of runlevels in which the command will be run; the third field specifies the way init should run the command; and the fourth field (when present) specifies the command that should be run.

The initdefault line doesn’t define a command, it just specifies the default runlevel for the system. In our case, as mentioned previously, this is 2.

A handy feature provided by sysvinit is the capability to use an "initscript" to run commands. This is a shell script that is used by init to run the commands it finds in inittab, instead of simply executing them. By setting some environment variables there, we’ll have them set automatically for all the scripts and interactive shells that init runs.

Since the processes spawned by init will be executed with a minimal environment, we need to start by setting environment variables like PATH — without that, the full path would need to be specified for every command run in the script or shell. We also set LD_LIBRARY_PATH, so all the shared libraries will be found by the program loader.

GNU/Linux systems have extensive support for internationalization and localization. Before we have the full system set up, it’s a good idea to specify a simple default setting for those features.

As the final system programs and libraries are built, they should be used in preference to the ones in /scaffolding, so we’ll put the directories where they’ll live in front of the /scaffolding directories.

The TARGET_SYSTEM_MAKEFLAGS parameter specifies the MAKEFLAGS that should be set for all target-side processes. (If a specific package has issues with parallel builds, it can be overridden to -j1 or unset entirely for those packages.)

The bash script automatically sets the HOME environment variable to be the current user’s home directory, and there are some programs that expect HOME to be set to a reasonable value. We can go ahead and set it here in case any of the programs we’re using is among those.

We also need to set environment variables for all the litbuild configuration parameters that might not use default parameter values. It’s not hard to define these properly because from this point onward we don’t need to worry about HOST and TARGET and so on: everything is just going to be a native build. We do need to pass a number of parameters from the host-side build along through to the target-side build, though!

Many of the parameters used in the litbuild blueprints will have static values defined by convention within CBL, so we don’t even need to pass the existing parameters defined for the host-side build along.

And of course we have to run the program that init is telling this script to run. The command line used to run the initscript is given four arguments, representing the first, second, third, and fourth inittab fields, respectively. The only one we need to use is the fourth one, which contains the command line specified in inittab.

If a bootwait line is present in inittab, the process on that line will be executed once during system boot, and init will wait to run any other processes until it completes. We can take advantage of that feature to run a script that gets the system to a baseline functional state — kernel filesystems are mounted, the root filesystem is writable, and litbuild is available to generate scripts from blueprints.

The script that will be run by the bootwait line is designed to be idempotent: that is to say, it can be run any number of times and will ensure that the system is in the expected state when it is complete, but won’t make unnecessary changes. This is helpful in cases where the target system doesn’t complete the build properly for some reason — if it’s running in a QEMU virtual machine, and QEMU crashes, for example, it will need to be restarted.

Most of the commands executed by startup-target-system are persistent across reboots, so we want to skip those commands if we restart the build. Accordingly, most of the commands executed by the script are enclosed in guard clauses that check whether it’s necessary to do anything; commands that don’t need to be run are skipped.

The script starts, as normal, with a "shebang" line that the kernel will use to determine what program will actually run it. In our case, that’s the scaffolding bash program.

Now we can start the build. The first thing to do is remount the root filesystem so we can write to it.

Linux systems have several magical filesystems. Two of these, /sys and /proc, don’t actually contain real files and directories, they just contain things that look like files and directories, but actually provide insight into the state of the kernel and system — and in many cases the ability to manipulate that state — using a file interface. For example, /proc/cmdline shows the command line that was passed to the kernel by the boot loader, and /proc/filesystems shows the set of filesystems that are supported by the kernel.

/dev is different — it’s the canonical location on the system for nodes, aka "special" files. These represent I/O devices and allow those devices to be accessed as though they were files. Linux has a feature called "devtmpfs" that automatically populates a RAM-based filesystem with nodes for all of the I/O devices it knows about.

Additional directories that should have RAM-based filesystems mounted on them are /run and /tmp. /run is a canonical location for files containing volatile system state information — things like the runtime s6 scan directory — and /tmp is the conventional location for temporary files that should not persist across reboots. It’s convenient to use a tmpfs for that purpose. This does limit the amount of data that can be written to /tmp: by default, a tmpfs filesystem is limited to half the physical RAM on the system. For small target systems, this can be a problematic constraint; if you wind up having any issues as a result, you can replace the mount command here with something like rm -rf /tmp; mkdir -m 1777 /tmp to ensure that everything from the previous boot (if any) has been cleared up but without using a (size-limited) tmpfs for it.

Now that /dev is mounted, we can reference paths within it meaningfully. The first device we need to use is the console device.

There are a couple of ways to reference terminals in Linux systems: virtual terminals are associated with numbered tty character special files ("tty" stands for "teletype": a historical artifact), like /dev/tty1 or /dev/tty2. The first of these, /dev/tty0, is special and always refers to the current virtual terminal. Serial ports are conventionally referred to using similar syntax with an extra "S" afer "tty": /dev/ttyS0 or /dev/ttyS1. /dev/tty is always a reference to whatever terminal opened the process that is looking at it…​ with the proviso that for some reason — possibly because /dev is not mounted at the time init runs? — the scripts started by init don’t seem to have an associated tty at all.

The system console, which is where the kernel’s boot messages are displayed, is always available at /dev/console. By default this points at /dev/tty0, but can be overridden by the boot loader using a console command-line directive, e.g. console=/dev/ttyS0. The console can also be multiplexed (written to and read from by multiple devices) by specifying multiple console directives.

Since we want to see this script’s output, and the script (for whatever reason) has no associated tty device, we need to redirect the script’s output to /dev/console. We also want to redirect input from /dev/console to the script, so that once the build process terminates (successfully or not) and leaves us at a shell prompt we will be able to type commands interactively.

A handy trick to do this is the exec shell builtin: when run without a command but with redirections, exec simply modifies the shell’s modifies its file descriptors as instructed.

Before that exec command runs, output from the startup script would cause the startup script to block until it was able to write the output somewhere — which would never happen, since init is patiently waiting for the bootwait script to complete. Now that it’s run, we can start logging what we’re doing.

There’s a little kludge we’re going to use to avoid long delays before the target-side build starts, but to explain what we’re doing I’m going to have to digress a few steps — you can skip this whole part if you’re not interested in some abstruse technical details.

Still with me? Brave soul. Okay, the Linux kernel has a built-in feature that lets you obtain cryptographically-strong random numbers. This is not something that computers are naturally good at, because computers are highly deterministic — there are plenty of pseudo-random number generation (PRNG) algorithms, but they all have to be seeded with some initial value and if you don’t have a source for that initial seed value that’s really random, you wind up with very predictable pseudo-random numbers. If you’re doing something involving cryptography, that’s no good! When a cryptographic algorithm calls for random bytes — in this context, sometimes this is called "entropy," although I remember hearing at some point that the terms aren’t strictly synonymous — the values of those bytes really need to be unpredictable, or else the strength of the algorithm collapses entirely.

Linux makes up for this by collecting some bits of entropy from unpredictable events. The most easily-observed of these events are human inputs, like keystrokes and mouse movements; by measuring the time between keystrokes and taking the least significant few bits from each interval, or measuring the interval between receiving network packets or some other kinds of hardware interrupts or other things like that, Linux can gather a fair amount of really random data. It mixes these random bits into an "entropy pool" it maintains. (You can read all about this in drivers/char/random.c in the Linux source tree.)

There are two easy ways that userspace processes can request random data from the entropy pool: they can read it out of the character device /dev/random, which only provides as much random data as Linux estimates to be available and blocks after that’s exhausted, and /dev/urandom, which is basically a PRNG that is periodically re-initialized with a really-random seed value — good enough for most purposes, and it never blocks. That latter source, /dev/urandom, is what the SecureRandom class in the Ruby standard library uses to initialize its pseudo-random number generator; it’s also used by lots of other programs that need random data.

Before Linux 4.16, that was the end of the story: there was /dev/random, which you would use if you wanted completely random bytes for things like cryptographic key generation and one-time pads; and /dev/urandom, which was perfect for getting more-or-less random data without any delays. In the 4.16 release series, though, this behavior changed slightly in order to address a security vulnerabliity: now, if any process tries to read from /dev/urandom before the kernel’s PRNG has been initialized with a really-random seed value, the read blocks until that initialization is complete.

This is still fine in most circumstances! But the CBL target-side build is unusual: it’s intended to be completely automated, and it’s not on a network because that would make things more complex than they need to be. So there aren’t any keystrokes or network packets to measure timings from! That means that initializing the kernel’s PRNG can take fifteen minutes or longer after booting the target system. It also turns out that some of the programs used in the target build — for example, the litbuild program used to generate the scripts for that build — wind up reading from one of the random devices, which causes the target-side build to hang — sometimes for quite a while — shortly after it starts.

You can feed entropy to the pool just by writing data to /dev/urandom (or, I think, /dev/random), but the kernel doesn’t trust that any data you write that way is really random, so this doesn’t help unblock the build. There’s a system call available to privileged processes (that is, processes that are running as root) that adds random data to the pool and assures the kernel that it really is random, though, so we can use a tiny C program to invoke that system call.

Of course, at this point we don’t have any reliable source of random data to use with that program — that’s the whole problem! But, luckily, we don’t actually need one. We’re not doing anything in the CBL build where the lack of cryptographically-strong random data is a problem. So we’re just going to lie to the kernel: we’ll write some non-random data to the entropy pool and assert that it is random.

Specifically, we’re going to pretend that the program code for the addentropy program is random, even though it’s really not at all!

We need to create the rest of the basic sytem directory structure. This doesn’t have to happen right now, but this is as good a time as any.

If the build process was interrupted and restarted, the directory structure and symbolic links will already exist, so we can skip this stuff. The same kind of logic applies in many later parts of this script, so there will be additional guard if statements around those blocks.

Also, all the multilib directories that the GNU toolchain components sometimes insist on using should be symbolic links to lib. This is still a kludge, but it’s the only way to avoid needing to specify some arbitrary set of lib directories in ld.so.conf.

If you’re building for an architecture that uses different multilib directory names, you might need to create additional symbolic links here. If you do that, you’ll probably also need to make a corresponding tweak in Create Symbolic Links For Scaffolding Lib Directories; also, look at the specs file when it’s being adjusted to see whether the multilib paths are present in the linking specs. If they are, you may need to make changes there as well.

Conventionally, information about filesystems that are currently mounted is available in the file /etc/mtab, and some packages therefore expect /etc/mtab to contain this information. The historical convention is that the mount and umount programs, which attach and detach filesystems from the filesystem hierarchy, would also add and remove corresponding lines from the mtab file.

The mtab file isn’t actually needed any more, though, because the /proc filesystem contains a file that always contains the kernel’s view of what filesystems are mounted. We can create a symbolic link to the conventional location, for the use of any packages that look for it there.

Users are defined in the file /etc/passwd. This file can also contain hashed versions of users' passwords, as well as other user account metadata — hence its name — but usually it does not! This is because passwd has to be readable by everyone (it’s not really important why this is the case, so I won’t get into that here) and shortly after UNIX systems became popular it became clear that it was a bad idea to allow any user to see even the hashed version of passwords.^[8]

So in most UNIX operating systems, the /etc/passwd file (confusingly) does not contain even the hashed version of passwords; those are found in a file called /etc/shadow, which can only be accessed by root. The extraction of passwords into /etc/shadow, and maintenance of them there, is done by the shadow package, which we’ll set up early in the target-side build.

These files have colon-delimited fields and newline-delimited records, so they’re pretty easy to read; man 5 passwd describes the file format. The second field is for the hashed password itself. If the field is blank, that account requires no password to log in; if it contains a single "x," that indicates that the password is stored in /etc/shadow as described a moment ago.

Users can belong to groups. Group definitions are similar to user definitions, but are in the file /etc/group — and, similarly to the passwd and shadow files, the password for groups that require passwords are usually found in /etc/gshadow.^[9]

Initially we’re just going to create the root user and group, and a few other system users and groups that are expected to exist by various programs.

There are some programs that write log data — like process resource usage data — to specific files, but only if those files already exist. Let’s create them.

The root user should have shell startup files just as other users do, and there’s no more-convenient place to set those up. Generally for the root user I like avoiding any significant shell startup activities, I only set environment variables, and I like having the same set of environment variables regardless of whether I’m using a login shell (for which the .bash_profile script is sourced) or a non-login shell (for which .bashrc is sourced), so I link those together.

More typically, .bash_profile can be used for any commands that should only be run at login time — like starting an ssh-agent process, for example — and .bashrc can be used for commands that should be run any time a new subshell is executed. (It’s common for .bash_profile to source in .bashrc as well, but this is by no means necessary.).

In addition to typical environment variables like PATH, we again want root’s environment to include all of the configuration parameters that will be used when running litbuild on the final system. Some of these, like PATCH_DIR and TARFILE_DIR, will be useful after the build is complete, so those will be written to .bash_profile. Others won’t, so we’ll write those to a .cblrc file that gets sourced in from .bashrc — once the build is complete, we can remove the command that sources that file from .bash_profile.

The code that sets these environment variables digs a little bit deeper into the bash bag of tricks than usual; the idea is, for each of the environment variables we want to set, we’ll echo both the name of the variable and the result of expanding the name of the variable as an environment variable into the .bash_profile script — an indirect reference. To do this we need to escape the first $, because otherwise bash expands $$ into its own process ID, which is not helpful.

I didn’t actually figure out how to do indirect references in bash; I did a something search and found everything I needed in chapter 28 of the "Advanced Bash-Scripting Guide," which is part of the Linux Documentation Project.

The Linux kernel can write pages of RAM to storage devices like disk partitions or files in the filesystem, which frees up those pages to be allocated to other programs; this is called "swapping", and the blocks of storage allocated to this purpose are referred to as "virtual memory" because they work like (very slow) RAM, or "swap space" because it’s space allocated to swapping. The terms are pretty much interchangeable.

It’s always a good idea to have some swap space available. That way, allocated memory that hasn’t been used in a long time can be written out to the swap area. If any process needs to access those pages, the kernel can re-read them from swap, so the only drawback is that access to those pages of memory takes longer than if they were already in RAM.

A configuration parameter, TARGET_SWAP_DEVICE, can be used to specify a block device to be used as virtual memory. If it exists and isn’t already in use, the following code will set it up. If the device exists but is something other than swap space, it will be ignored — just in case the parameter has an incorrect value and refers to an important filesystem or something.

CBL systems use s6 and s6-rc to manage the system state. We can set up the very first parts of that here — that is, the source directory that will contain service definitions, and the run-level bundle directories inside it. As other parts of the system are built, directories for other services will be created and, in many cases, added to the rl-default bundle.

We also set up an network-services bundle that will be part of the rl-default bundle if networking and network services should be part of the standard system run state.

This is all explained in much more detail in later sections: s6-rc, Configure the system initialization framework, and Construct the s6-rc service database.

The litbuild program needs to be installed so that we can use it to create scripts to build the rest of the system. We could simply install the rubygem-packaged version of litbuild, but it’s more consistent with the rest of CBL to install it from a source tarfile.

Of course, if the litbuild gem is already installed, we don’t need to build and install it.

Building the gem is a little bit fussy! When rubygems is used to build a gem package with gem build, it tries to compute checksums for the files as it adds them. This requires the ruby OpenSSL extension, which isn’t available in the scaffolding ruby; at least in Ruby 2.5.1, this causes the gem command to raise an exception and crash the build. It’s a bit of a kludge, but we can simply avoid raising the exception and everything will be fine.

To use the skip-upon-restart feature of litbuild throughout the target-side build, we can set up a litbuild database directory.

Now let’s proceed with the target-side build per se. We tell sysvinit to run this script when entering runlevel 2, but not restart it after it terminates: when this script terminates, the CBL build will (hopefully) be complete!

Once again, we need to do the same console-redirection trick we did earlier.

By default, bash keeps track of the location for programs it runs, which lets it skip looking through the PATH when it needs to find those programs again. As we construct the final system programs, we want those to be used in preference to the scaffolding versions, so we disable this caching behavior.

If anything goes wrong during the remainder of the build, we can bail out to an interactive shell. To do this, we’ll set the -e bash option (which causes any failing command to terminate the script) but set traps so that the termination — normal or unexpected — will cause the process to execute an interactive bash shell.

Now we can start the target-side build itself! This consists of running litbuild to produce scripts, then executing those scripts.

The first litbuild target builds the remainder of the scaffolding components — all the stuff we need for the real target-side build but which is difficult or impossible to cross-compile on the host system — and sets up the package-users framework.

Second, we build the final system components — these will automatically be generated as package-user-style build scripts, because the package-users framework is installed. This is a good time to switch to a final system litbuild database directory, since this is the point where we’ll start to build out the final system!

At this point, we have the bash program at the canonical system location, /bin/bash, so we can change the traps we set earlier to exec into that shell if the init script terminates.

And, finally, after building all of the programs and libraries needed on the system, we can finish up the system: tidy things up, remove any remaining references to the /scaffolding directory, configure the init system, and perhaps install the boot loader.

There are a couple more things we can do to tidy up before we declare the build complete.

Before shutting down the system, we should remount the root filesystem read-only. This is tricky because there may be some processes still lingering from the build process — I often see a few processes running as glibc, left over from its test suite, but there might be others as well.

Unfortunately, these processes are holding on to open file descriptors on the root filesystem, so we have to terminate them in order to remount that filesystem read-only. And unfortunately, that’s also tricky: killing those processes can, for some reason I find completely baffling, cause the target-side-build script to hang or crash.

If this happens, you won’t get a COMPLETE SUCCESS but it’s still safe to power-cycle the target system: we’re using a journaling filesystem, so we won’t need to do an extensive filesystem check on reboot or anything. It’s always a good idea to force all pending I/O operations to complete before proceeding, though.

If that worked, we should be able to make the root filesystem read-only and declare victory. But sometimes — honestly I have no idea what the deal is — I still find that Linux thinks the root filesystem has files open for writing. So we’ll put some explanations in the console messaging, just in case.

The target system build is complete. Nothing remains but to turn off the computer!

Under normal circumstances, the System V init program we’re using can be told to shut down using the shutdown or telinit programs, which are still present in the scaffolding directory. That won’t work at this point, though, because s6-linux-init has installed its own versions of shutdown, telinit, and other similar programs. We can’t use those as we typically would, either, because they’re expected to be run as part of the complete low-level userspace configuration set up by s6-linux-init-maker, which of course is not the case for the minimal target system.

In the typical running system circumstances, the init framework starts a long-running process running s6-linux-init-shutdownd, which listens on a fifo for instructions from other programs like s6-linux-init-shutdown and performs clean shutdown operations when told to do so. The very last thing it does is run s6-linux-init-hpr with an -f (for "force") option, which does a final filesystem sync operation and then uses the reboot system call to perform the actual shutdown (or halt, or reboot) operation. We can skip all the other complexity and go straight to the shutdown operation.

If anything goes wrong with the s6-linux-init-hpr command, the trap we set earlier will hopefully cause the script to drop into an interactive shell when it terminates.

If something goes wrong — or if we just want to examine the progress of the build as it proceeds — it’s helpful to have some interactive shells on other virtual terminals. This only works for target systems that support virtual terminals, obviously! Many emulated systems do not, but it doesn’t cause any problem to run some shell processes even on those systems, it just wastes a little bit of memory.

To do these, we’ll use the openvt command to specify which virtual consoles to run them on. The build scripts are typically running on the first virtual console, so we’ll run interactive shells on the second through sixth. (That’s a totally arbitrary decision; you can run as many as you like.)

Unlike the other init-run commands, these are set to be restarted if they terminate, by using the keyword respawn rather than once in the inittab directive. (That doesn’t actually work, though, for some reason; maybe the virtual terminals need to be closed before they can be re-opened.)

Of course, the scripts that are supposed to be executed by init need to be executable!

Lzip is a data compression program that uses the LZMA algorithm, just like XZ Utils does. Because it doesn’t have the same convoluted project history as XZ Utils, though, it seems to have a slightly tidier codebase and a simpler format for compressed files. (The lzip homepage observes, "The lzip manual provides the code of a simple decompressor along with a detailed explanation of how it works, so that with the only help of the lzip manual it would be possible for a digital archaeologist to extract the data from a lzip file long after quantum computers eventually render LZMA obsolete" — which seems like a good idea if you’re going to use a program for long-term archival storage.)

Also, it tends to compress files just a litle bit better than XZ Utils does, and substantially better than other common compression utilities.

All of the CBL source materials (except lzip) are compressed with lzip because it provides better space savings than any of the other compression programs.

Lzip doesn’t make any allowances for being cross-compiled: if host, target, and so on are specified, they are simply ignored. That means it really can’t be built until we are booted into the target system. We need to build it before we build anything else on the target system, though, because the other source code archives are all lzip-compressed!