Dear visitor,

it seems your are using Internet Explorer in version 11 or lower as your web browser. Due to a lack of supported modern web standards I do not support this browser anymore.

Please use an alternative web browser such as:

Fedora CoreOS - First Steps

Fedora CoreOS, the new operation system developed by the Fedora community and Red Hat, finally left the preview phase. As a user of the now deprecated Atomic Hosts (available for Fedora, CentOS, Red Hat Enterprise Linux) a good opportunity to try it out and give some hints how to configure your first working system.

What is it?

If you never heard about the Fedora CoreOS project, its predecessor Atomic Host or CoreOS Container Linux before, you may wonder what this new operating system is about:

Fedora CoreOS (abbreviated as FCOS) embraces the idea of an immutable operating system, specifically designed to enhance configuration consistency of multiple hosts in a distributed system and with containers as the primary building blocks to run applications and services. Therefore, container tooling like Kubernetes, Podman and Docker plays a large role in this approach.

When running a FCOS instance, the most important concept to understand is that the file system is now immutable, it is read only. To be precise: From the well known Linux file system only two folders are writable - /etc/ and /var/.

The idea of an immutable file system allows the introduction of a new concept: the so called OSTree. An OSTree represents a complete bootable file system tree that can be updated in an atomic or transactional manner. Instead of updating individual packages on the system by writing to the existing file system, a complete new version of the file system tree is checked out. OSTrees and the corresponding tooling therefore act like a version control for the whole operating system. This concept also enables the possibility to completely rollback the system to a previous working version in case something should go wrong.

Since, apart from a few exceptions, the file system is immutable, how to deal with applications and services I want to use? The answer is in most cases: utilizing containers. FCOS is shipped with support for Podman, Docker and the CRI-O container runtime out of the box. How this works in practice and further possibilities are topics of an upcoming article.

While FCOS is designed as a server operating system for single hosts or distributed systems, the concepts mentioned above can also be found in another project of the Fedora community: Fedora Silverblue. This is an immutable operating system with a container based workflow as well, but with the focus for desktop and workstation environments. It might be worth to check this out as well.

After this rough introduction of the basic concepts, in the following sections of this article I will explain in more detail how to setup a first working virtual machine with FCOS.

Ignition

FCOS, in contrast to classic operating systems, has no install-time configuration. Instead, a generic disk image is provided, which can be configured during the initial boot process. FCOS supports deployments on bare-metal machines, in a cloud environment (e.g. using services like Amazon EC2) and on local virtual machines, with corresponding possibilities to provide the initial configuration settings.

While the former Atomic Host project used cloud-init as their configuration system, Fedora CoreOS is now using Ignition, which originated from the CoreOS Container Linux project.

Ignition itself is a low-level provisioning tool that allows for early-boot configuration. The essential part is a configuration file which is read during the initial boot process. In the case of Fedora CoreOS this file is not written directly; a transpiler from the Fedora CoreOS Configuration Format (FCC) to the Ignition file format is used instead. If you were a user of CoreOS before, you maybe already know the corresponding transpiler ct. The FCC file itself is written in yaml syntax.

Writing the FCC configuration

In this section I will go through some basic options available. For a full reference please take a look at the official documentation.

FCC allows basically three types of configuration options:

User Settings

The minimal ignition configuration, along with some required meta data, needs to have information about the initial system users enabled to log into the system. Fedora CoreOS provides an initial user core which can be used to administrate the system. If you want to use this user the minimal configuration a least needs to assign a corresponding login password or an ssh-key.

An example could look like this:

variant: fcos
version: 1.0.0
passwd:
  users:
  - name: core
    ssh_authorized_keys:
    - "ssh-rsa AAAAB3Nza...."
  - name: matthias
    ssh_authorized_keys:
    - "ssh-rsa AAAAB4adsa...."
    password_hash: "$6$rounds=4096$yspKBgKn..."
    groups:
    - wheel

Here the system core user has a provided public ssh-key in the ssh_authorized_keys section and is therefore enabled to log into the system via ssh. The additionally created user matthias has a password_hash configured, so regular password based login is possible as well.

Obviously, allowing a regular password based login requires for a safe hash method (do not choose MD-5!). This can be done with the regular Linux tooling like:

mkpasswd --method=SHA-512 --rounds=4096

The hashed password can then be used for the FCC configuration. The example configuration also assigns the user matthias to the wheel group to allow privileged operations.

Apart from these standard settings, of course more sophisticated options like setting up system accounts without home directories or configuring the used login shell are available as well.

File System Settings

The storage section allows for various configuration of disk devices, creation of raids, handling of directories, files and symbolic links.

Handling Storage

Ignition allows setting up storage during the configuration phase. This includes partitioning, formatting and mounting of additional disk devices and corresponding file systems. To do this, first a list of disk devices has to be created. The absolute paths to the devices should usually be provided using the /dev/disk/by-… symbolic links, which are consistent and not prone to change. A list of partitions for each device can then be defined, with further options to wipe partition tables and pre-existing partitions.

Creating file systems on the configured disk devices is then easily done via listing them in the corresponding filesystems section. Parameters here include the device path, the desired file system and further settings like labels.

It is worth mentioning that, although Ignition can handle various file systems like ext4 and xfs, more sophisticated options like setting up LVM is not possible.

A minimal example of setting up storage could then look like this:

variant: fcos
version: 1.0.0
storage:
  disks:
  - device: <path to disk device>
    wipe_table: true
    partitions:
    - label: <unique partition label>
      number: 0
      wipe_partition_entry: true
  filesystems:
  - path: <mount path of device during configuration>
    device: <path to disk device>
    wipe_filesystem: true
    format: ext4
    label: <unique file system label>
Extended Configuration - Hostname

Ignition provides no possibility to run arbitrary shell commands during the configuration process. Therefore, the usual way to set the hostname using hostnamectl on systemd controlled systems is not working out of the box. There are two solutions to circumvent this constraint:

  1. Setting up a oneshot systemd service to execute a command
  2. Writing to the corresponding configuration file directly

In this example I will go with the second solution, which looks like this using the storage section of the configuration file:

variant: fcos
version: 1.0.0
storage:
  files:
  - path: /etc/hostname
    overwrite: true
    contents:
      inline: <your-hostname>

Systemd Settings

The systemd section of the FCC configuration allows the configuration of systemd units. It is possible to create new units as well as modifying existing ones with e.g. drop-in configuration files.

To refer back to the previous example, creating a new oneshot service to set up the hostname during the first boot can look like this:

variant: fcos
version: 1.0.0
systemd:
  units:
  - name: set-hostname.service
    enabled: true
    contents: |
      [Unit]
      Description=Set the hostname

      [Service]
      Type=oneshot
      ExecStart=/usr/bin/hostnamectl set-hostname your-hostname

      [Install]
      WantedBy=multi-user.target

Running Example

In this section I will go through a complete example to create a running virtual machine with Fedora CoreOS.

Example FCC / Ignition Configuration

The complete example configuration can look like this:

variant: fcos
version: 1.0.0
passwd:
  users:
  - name: core
    ssh_authorized_keys:
    - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDu4cjMWNFUz4YV9ooeoaBSwPhdoM9x8cqiWUrVhlbH24ixNyCgcat7cN1kbraIuTQTqymrfUG3rqQdjyCbiIRd0gafS7hEQ9NSh1gcVQVQAzNuvJvcTrrWstXdktM41NfX/K7GcGy4u4gN8mEP6WHytEcpazh3N5TgjeJkEpzgUXi0isyfBXLKxyYGQrFzD+QTjoeM8DvKHnJFvU+WF4SPWWDyhB23mXPDBwl9XpQaQmuL1/1nkhkEComq2igr30rr2Uhyw4tiworVel0mWOByKBIPATjXdXYbS/nUzspiQ0F1kOdYv/EOeeb2X9hZCHWhefdbQCbI1NfTkjbZFmFMTBfGX7e5wDNJpY9OdTacPGTN0PKV1XtaY68S5nYOi2jqj7l2iPaL+esMSdCllBGAdBZV6/PJ511dRzpxk/CunVUi/3oQORlnP02ip0WY9HmPdqfdoq5+jfIE/iB6jbSEW93tkMeeNaXbjeO2DSq9ihG7RAyPMPQNDXi/DHMnw6fzk5MNvBi14iXHJiykI0vG4yfYvgRA0ies8/3CH/a6SRwo+DKKm10zsDN5HM76kamlWx6Nd+Eq2Hx6q6InMkIsbjp5Aya2XSrLGXAI1RKPQEk0GKx0OP5+d4Npgbg/EbM0zUiFmpszFYlxfpCVqdc9lVMmKlaMe9HsRUXx7slq3w=="
  - name: matthias
    ssh_authorized_keys:
    - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDu4cjMWNFUz4YV9ooeoaBSwPhdoM9x8cqiWUrVhlbH24ixNyCgcat7cN1kbraIuTQTqymrfUG3rqQdjyCbiIRd0gafS7hEQ9NSh1gcVQVQAzNuvJvcTrrWstXdktM41NfX/K7GcGy4u4gN8mEP6WHytEcpazh3N5TgjeJkEpzgUXi0isyfBXLKxyYGQrFzD+QTjoeM8DvKHnJFvU+WF4SPWWDyhB23mXPDBwl9XpQaQmuL1/1nkhkEComq2igr30rr2Uhyw4tiworVel0mWOByKBIPATjXdXYbS/nUzspiQ0F1kOdYv/EOeeb2X9hZCHWhefdbQCbI1NfTkjbZFmFMTBfGX7e5wDNJpY9OdTacPGTN0PKV1XtaY68S5nYOi2jqj7l2iPaL+esMSdCllBGAdBZV6/PJ511dRzpxk/CunVUi/3oQORlnP02ip0WY9HmPdqfdoq5+jfIE/iB6jbSEW93tkMeeNaXbjeO2DSq9ihG7RAyPMPQNDXi/DHMnw6fzk5MNvBi14iXHJiykI0vG4yfYvgRA0ies8/3CH/a6SRwo+DKKm10zsDN5HM76kamlWx6Nd+Eq2Hx6q6InMkIsbjp5Aya2XSrLGXAI1RKPQEk0GKx0OP5+d4Npgbg/EbM0zUiFmpszFYlxfpCVqdc9lVMmKlaMe9HsRUXx7slq3w=="
    password_hash: "$6$rounds=4096$6MT9hwjAovop2bAq$bw/29axXXBX.l0NP7igGK67BbCU4t89UOmCHIkUS5nV9M7fgU4tlFJjRZrRYRF14hRgwF8jpCh1hlpqL5CLkJ/"
    groups:
    - wheel
storage:
  disks:
  - device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_hd1
    wipe_table: true
    partitions:
    - label: "disk.storage"
      number: 0
      wipe_partition_entry: true
  filesystems:
  - path: /var/storage
    device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_hd1
    wipe_filesystem: true
    format: ext4
    label: STORAGE
  files:
  - path: /etc/hostname
    overwrite: true
    mode: 0644
    contents:
      inline: fcos-server-test
systemd:
  units:
  - name: var-storage.mount
    enabled: true
    contents: |
      [Mount]
      What=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_hd1
      Where=/var/storage
      Type=ext4

      [Install]
      WantedBy=local-fs.target

So what is happening here:

Two users, core and matthias, are configured to be able to log in the system with ssh-keys, while the latter is also able to log in with a regular password (which here is test for demonstration purposes). The user matthias is also added to the wheel group.

The ssh-keys in this section were created with:

ssh-keygen -t rsa -b 4096

The generated public key is then used in the configuration file.

In the first part of the storage section an additional hard disk is configured. The corresponding device is used by its id and all partition tables will be wiped during the setup. Then the hard disk is partitioned with one partition named disk.storage and formatted with an ext4 file system. Furthermore, this section configures the hostname of the Fedora CoreOS instance with the method explained above.

Finally a systemd mount unit, which mounts the previous created file system on the additional hard disk at the location /var/storage, is configured.

Generating the Ignition File

In the next step it is necessary to run the FCC transpiler to create a valid Ignition file. Instead of downloading the transpiler I will use a provided container image, which makes the process somewhat easier. Running the transpiler can then be done via podman (or docker) with the following command:

podman run -i --rm quay.io/coreos/fcct:v0.2.0 -pretty -strict < fcos.fcc > fcos.ign

Launching a VM with FCOS

To get the configuration example running I use the provided Fedora CoreOS qcow2 images and QEMU to create a virtual machine.

As the FCC configuration is using an additional hard disk, first I have to create the corresponding image to be used with QEMU:

# Creating a one gigabyte hard disk
qemu-img create -f qcow2 storage.qcow2 1G

Then I can start the virtual machine with the Fedora CoreOS image, the additional hard disk and the Ignition configuration file:

qemu-system-x86_64 -machine accel=kvm \
                   -m 2048 -cpu host -nographic \
                   -device virtio-rng-pci \
                   -device virtio-scsi-pci,id=scsi \
                   -drive file=fcos.qcow2,if=none,id=hd0 -device scsi-hd,drive=hd0 \
                   -drive file=storage.qcow2,if=none,id=hd1 -device scsi-hd,drive=hd1 \
                   -fw_cfg name=opt/com.coreos/config,file=./fcos.ign

Note that the hard disks are configured to use the virtio-scsi storage backend along with the ids hd0 and hd1. Using this parameters, QEMU automatically makes the additional storage hard disk available via the symlink /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_hd1, which is part of the FCC configuration file above.

After a short wait time you should the be able to log into the system with the user matthias and the password test. Running the shell command hostname should reflect the correct value and you should be able see the mounted additional storage file system at /var/storage as well.

A full example bash script for all tasks mentioned is provided here.

References