Happy New Year! Yep, it's that time of year again. A time of new beginnings, cleaned slates, and Winter weather (in the Northern Hemisphere). So, bundle up near the warmth of your PCs exhaust fans, and bask in the glow of artificial light from it's connected display. Because, dear reader, it's time to start anew! And what better way to achieve that, than with a clean install of NixOS?

Sure, perhaps you vowed this would be the year you'd stop distrohopping¹, saving your SSD from near certain doom. But if that's the case, there's a good chance you might just find NixOS to be that final hop you've been looking for ... Well, only one way to find out, right?

Introduction

After taking some time to setup a fresh install of one of my favorite distros, I felt it was a good idea to post a write up about the somewhat extensive process on installing NixOS with Full Disk Encryption (FDE), enabling secure boot, and automagically unlocking the encrypted LUKS device(s) during the boot process with keys enrolled into the motherboard's TPM2 chip.

Now, I should state that I didn't come up with this guide alone. This guide is more of an amalgamation, based pretty heavily upon a lot of other guides and resource materials out there scattered across the net. As such, I encourage you to check out the reference materials^{2, 3, 4, 5, 6, 7, 8, 9} I used, linked in the footnotes below.

What is NixOS?

NixOS is a operating system (OS) based on Nix, a purely functional package management system. Nix (the package manager) treats packages like how values are treated in purely functional programming languages, like Haskell, and stores the resultant packages in the Nix store (/nix/store). Such packages are built from Nix Expressions, written in the purely functional Nix language.¹⁰

In NixOS specifically, the whole OS is built by the Nix package manager, from a Nix expression that declares the entire system's configuration, typically in a configuration.nix file. Because everything is declared in a purely functional manner, a new configuration cannot overwrite previous configurations. This results in reliable, atomic upgrades, (which can be atomically rolled back) and reproducible system configurations.¹⁰

Who is the guide for?

This guide is for anyone interested in using NixOS¹⁰, in a relatively secure fashion, with modern conveniences like using an onboard TPM2 chip to decrypt LUKS devices during boot up. Additionally, the level of technical knowledge in this guide, assumes a somewhat decent familiarity with Linux systems, at least to the point where one might feel comfortable enough to perform a manual install of either Arch Linux¹⁴ or Gentoo¹⁵.

While we won't be installing Arch or Gentoo today, we are going to be making heavy use of the command line and TUI based text editors as we perform a manual install of NixOS. If that's something you're up for, then this guide is written for you.

On the other hand, if this is your first exposure to Linux or even Unix based operating systems, let alone NixOS, then I encourage you to consider skimming through this guide, instead of actually performing any of the steps contained within it. That way, you can at least discover if anything I talk about today seems interesting to you, even if it seems beyond your current comfort level of technical expertise. Because, who knows, you might just find you've been bitten by the 'Nix bug before you know it! Also, should that happen, I highly recommend trying a traditional Linux distro like fedora workstation, just to get your feet wet, before diving headfirst into something that's far less user-friendly.

Goals of this Guide

By the end of this article, my hope is that you’ll be able to:

1. Install NixOS with Full Disk Encryption.
2. Use a flake.nix to configure your NixOS system.
3. Install and configure Lanzaboote to enable secure boot on your machine.
4. Use systemd-cryptenroll to unlock your LUKS devices via your motherboard's TPM chip.
5. Use home-manager to configure your shell and git/gh.

Part 01: From Zero to NixOS

Prerequisites

1. A machine with a UEFI motherboard
2. A machine with a TPM2 chip (This is only needed for the Unlock LUKS via TPM2 part of the guide, which uses tpm2-tss to enable systemd-cryptenroll to enroll your LUKS keys into the TPM chip.¹⁶)
3. A bootable USB (4 GB minimum) with the Minimal NixOS ISO¹⁷ loaded onto it

Booting the installer

1. Repeatedly mash F2 or F10 or DEL or whichever key let’s you into your system’s UEFI/Bios.
2. Disable secure boot (if enabled).
3. Disable the CSM/Legacy Support (if enabled/applicable).
   1. WARNING: If this was enabled, any legacy hardware you might have (e.g., a GTX 660 with a VBIOS lacking UEFI support) will immediately become incompatible/stop working (until you re-enable CSM/Legacy Support). If you continue, you acknowledge that you're comfortable with removing/losing any incompatible hardware from your current system.
4. Move the NixOS USB to the highest boot priority (if applicable)
5. Save your changes, and boot into your NixOS USB.
6. Once you've reached GRUB, feel free to either load the installer from the USB itself, or copy it to the RAM. You can also choose "no modeset" if you're having issues (usually graphical) reaching the installer's TTY.

Connect to the Internet

Before we proceed with the installation, it’s probably a good idea to make sure you can access the internet way before making any drastic changes (partitioning, formatting, etc.) that would affect your currently installed OS.

So, plug in an ethernet cable, or follow the steps below to setup WiFi, then try to ping a website: e.g., ping google.com. If you can see packets coming back, hit ctrl+c to stop the pinging, then proceed to partitioning.

Otherwise, stop here, and start troubleshooting. If you still can’t connect to the internet, it’s possible your network adapter is just too new, and thus isn’t supported by the Linux kernel at this time.

WiFi Setup

First, make sure your wireless network adapter is actually enabled. You can see which adapters are available with ip link

[nixos@nixos:'']$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
 link/ether 55:a5:b5:c5:d5:f5 brd ff:ff:ff:ff:ff:ff
3: wlp: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DORMANT group default qlen 1000
 link/ether f5:d5:c5:b5:af:f5 brd ff:ff:ff:ff:ff:ff

If your WiFi adapter is DOWN like mine (wlp) just run:

# ip link set <your-interface> up

If that results in an error:

RTNETLINK answers: Operation not possible due to RF-kill

Please see the note on RF-kill Unblocking below, otherwise skip to Getting Connected with wpa_supplicant.

[nixos@nixos:'']$ rfkill
ID TYPE      DEVICE               SOFT      HARD
 0 wlan      ideapad_wlan      blocked unblocked
 1 bluetooth ideapad_bluetooth blocked unblocked
 2 bluetooth hci0              blocked unblocked
 3 wlan      phy0              blocked unblocked

[nixos@nixos:'']$ sudo rfkill unblock wlan
ID TYPE      DEVICE                 SOFT      HARD
 0 wlan      ideapad_wlan      unblocked unblocked
 1 bluetooth ideapad_bluetooth   blocked unblocked
 2 bluetooth hci0                blocked unblocked
 3 wlan      phy0              unblocked unblocked

[nixos@nixos:'']$ sudo ip link set wlp up

[nixos@nixos:'']$ _

Getting Connected with wpa_supplicant

With the WiFi adapter actually available, we can now continue (mostly) from the official NixOS manual⁹ for getting connected to the internet run:

# systemctl start wpa_supplicant

Then bring up the CLI with: wpa_cli. From there, you can scan for networks:

> scan
OK
> scan_results
bssid / frequency / signal level / flags / ssid
1a:2b:34:56:70    5555  -53 [WPA2-PSK+SAE-CCMP][WPS][ESS] myhomenetwork
...

Then add, set, and enable your network:

> add_network
0
> set_network 0 ssid "myhomenetwork"
OK
> set_network 0 psk "mypassword"
OK
> set_network 0 key_mgmt WPA-PSK
OK
> enable_network 0
OK

Alternatively for an enterprise network:

> add_network
0
> set_network 0 ssid "eduroam"
OK
> set_network 0 identity "myname@example.com"
OK
> set_network 0 password "mypassword"
OK
> set_network 0 key_mgmt WPA-EAP
OK
> enable_network 0
OK

If everything went okay after the last command, you’ll something like:

...
> enable_network 0
OK
<3>CTRL-EVENT-SCAN-STARTED
<3>CTRL-EVENT-SCAN-RESULTS
<3>SME: Trying to authenticate with 1a:2b:34:56:70 (SSID='myhomenetwork' freq=5555 MHz)
<3>Associated with 1a:2b:34:56:70
<3>CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
<3>WPA: Key negotiation completed with 1a:2b:34:56:70  [PTK=CCMP GTK=CCMP]
<3>CTRL-EVENT-CONNECTED - Connection to 1a:2b:34:56:70 completed [id=0 id_str=]
>

Then all you have to do is run quit to return to the shell, and we can proceed with the next step.

Partitioning

Now that we’re absolutely positive we can connect to the internet, we’re going to use fdisk to partition the drive we want to install NixOS on. Running the ensuing fdisk commands will create both a boot partition and a LVM partition. The LVM partition will hold both a root partition and a swap partition.

We’ll use the fdisk CLI to perform the partitioning part of the process. But first, we can also use it to list our disks via # fdisk -l, so we can get the exact location of the disk we want to partition.

[nixos@nixos:'']$ sudo fdisk -l
Disk /dev/loop0: 1021.89 MiB, 1071525888 bytes, 2092824 sectors

Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: SAMSUNG M2VLB512
Units: sectors of 1 ** 512 = 512 butes
Sector Size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 09515620-9D73-11EF-BE8C-0800200C9A66

Device             Start        End   Sectors   Size Type
/dev/nvme0n1p1      2048     206847    204800   100M EFI System
/dev/nvme0n1p2    206848     239615     32768    16M Microsoft reserved
/dev/nvme0n1p3    239616  998639615 998400000 476.1G Microsoft basic data
/dev/nvme0n1p4 998639616 1000212479   1572864   768M Windows recovery environment

...

Once we've found our disk, we'll pass it's location as an input into fdisk to perform the actual partitioning steps:

# fdisk /dev/your-disk-to-partition

In my case the command looks like: # fdisk /dev/nvme0n1.

Create a Empty GPT Table

To begin, we’ll clean up any old partitions by creating a empty GPT partition table (g):

Command (m for help): g
Created a new GPT disklabel (GUID: 5213d2a2-a040-44c4-ba6a-02787812777e).

Command (m for help):

To verify this, you can print (p) out the new partition table, which should now be empty:

Command (m for help): p
Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: SAMSUNG M2VLB512
Units: sectors of 1 ** 512 = 512 butes
Sector Size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5213d2a2-a040-44c4-ba6a-02787812777e

Command (m for help):

Create the EFI System Partition (ESP)

We can then create the ESP like so:

1. n
2. (partition number; default) enter
3. (first sector: default) enter
4. (last sector) +1G
5. (remove signature if applicable) Y
6. t (change partition type)
7. (partition type or alias) 1 (EFI)

Command (m for help): n
First sector (2048-1000214527), default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-1000214527, default 1000215216): +1G

Created a new partition 1 of type 'Linux filesystem' and of size 1 GiB.

Command (m for help): t
Partition type or alias (type L to list all): 1
Changed type of partition 'Linux filesystem' to 'EFI'.

Command (m for help):

Create the LVM Partition

Next, we’ll add our LVM partition:

1. n
2. (partition number: default) enter
3. (first sector: default) enter
4. (last sector: default) enter
5. (remove signature if applicable) Y
6. t (change partition type)
7. (partition number: (1, 2, default 2)) enter
8. (partition type or alias:) 44 (Linux LVM)

Command (m for help): n
Partition number (1, 2, default 2):
First sector (2099200-1000214527), default 2099200):
Last sector, +sectors or +size{K,M,G,T,P} (2048-1000215216, default 1000215216):

Created a new partition 2 of type 'Linux filesystem' and of size 475.9 GiB.

Command (m for help): t
Partition number (1, 2, default 2):
Partition type or alias (type L to list all): 44
Changed type of partition 'Linux filesystem' to 'Linux LVM'.

Command (m for help):

Verifying the Partitions

If you print (p) out the partition table again, it'll probably look similar to this:

Command (m for help): p
Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: SAMSUNG M2VLB512
Units: sectors of 1 ** 512 = 512 butes
Sector Size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5213d2a2-a040-44c4-ba6a-02787812777e

Device           Start        End   Sectors   Size Type
/dev/nvme0n1p1    2048    2099199   2097152     1G EFI System
/dev/nvme0n1p2 2099200 1000214527 998115328 475.9G Linux LVM

Command (m for help):

if it looks right, then we can save (w) our changes and exit!

Now, running # lsblk should look something like this:

[nixos@nixos:'']$ sudo lsblk
NAME        MAJ:MIN RM   SIZE  RO TYPE MOUNTPOINTS
...
nvme0n1      259:0    0  476.9G  0 disk
|-nvme0n1p1 259:3    0      1G  0 part
|-nvme0n1p2 259:4    0  475.9G  0 part

[nixos@nixos:'']$

Encrypting with LUKS

We’ll be encrypting the LVM partition with LUKS (version 2). To do that, we just have to run the following command:

# cryptsetup -v -y --label=NIXLUKS luksFormat --type luks2 /dev/nvme0n1p2

The -v flag ensures a verbose output, -y ensures it’ll ask us for our encryption password twice to confirm, the label flag is simply assigned to NIXLUKS but you can call it whatever you want. luksFormat actually encrypts our disk, and the --type flag set to luks2.¹⁹ This is important because it's both newer, and a necessary requirement to use systemd-cryptenroll, which enables enrolling your LUKS keys into the TPM2.¹⁶

[nixos@nixos:'']$ sudo cryptsetup -v -y --label=NIXLUKS luksFormat --type luks2 /dev/nvme0n1p2

WARNING!
========
This will overwrite data on /dev/nvme0n1p2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase fpr /dev/nvme0n1p2:
verify passphrase:
Key slot 0 created.
Command successful.

[nixos@nixos:'']$

Verifying the LUKS Device

To verify the partition got encrypted, you can inspect the LUKS header using:

# cryptsetup luksDump /dev/nvme0n1p2

Which will give an output similar to this:

[nixos@nixos:'']$ sudo cryptsetup luksDump /dev/nvme0n1p2
LUKS header information
Version         2
...
UUID:           09515620-9D73-11EF-BE8C-0800200C9A66
label:          NIXLUKS
...

The most important thing to note is the version number and your label. If it’s “2”, and your label looks right, then we’re all good to go.

# cryptsetup luksHeaderBackup /dev/nvme0n1p2 --header-backup-file /path/to/backup.dat

Opening the LUKS Device

Now, we’re going to open our encrypted LUKS partition and create a reference to it on /dev/mapper/cryptroot with:

# cryptsetup luksOpen /dev/nvme0n1p2 cryptroot

You can then check our mapped partition exists with: ls /dev/mapper/cryptroot

This won't return anything really (because there's nothing in cryptroot yet), unless cryptroot doesn't exist.

[nixos@nixos:'']$ ls /dev/mapper/cryptroot
/dev/mapper/cryptroot

[nixos@nixos:'']$

LVM Partitioning

Before creating the root and swap (and other possible) logical volumes (LVs), we'll first need to create a Physical Volume on cryptroot, which can then be used to create a Logical Volume Group (LVG) named lvmroot.

# pvcreate /dev/mapper/cryptroot

# vgcreate lvmroot /dev/mapper/cryptroot

With that, we can create the root LV and the swap LV. You can of course create as many LVs as you want, such as a home LV. However, for my purposes, I’m going to keep it simple.

Create the swap Logical Volume

We can create the swap LV on our root LVG (lvmroot) with the following command:

# lvcreate -L24G lvmroot -n swap

Create the root Logical Volume

With the swap LV set, we can set the root LV to take up the rest of the available space:

# lvcreate -l 100%FREE lvmroot -n root

Formatting

This is a fairly standard procedure, the only thing of note here is adding labels, which makes mounting the partitions much easier.

1. Format the ESP partition:

   # mkfs.fat -F 32 -n NIXBOOT /dev/nvme0n1p1
   

2. Format the Root LV:

   # mkfs.ext4 -L NIXROOT /dev/mapper/lvmroot-root
   

3. Format the Swap LV (if you made one):

   # mkswap -L NIXSWAP /dev/mapper/lvmroot-swap
   

Mounting

Then we can mount the formatted partitions like so.

1. Mount the Root LV:

   # mount /dev/disk/by-label/NIXROOT /mnt
   

2. Create the ESP Directory:

   # mkdir /mnt/boot
   

3. Mount the ESP:

   # mount -o umask=0077 /dev/disk/by-label/NIXBOOT /mnt/boot
   

4. Turn on Swap:

   # swapon -L NIXSWAP
   

Verifying the Mounts

To verify everything is in the right place, running:

# lsblk -o name,label,size,type,mountpoints /dev/nvme0n1

should result in output similar to this:

[nixos@nixos:'']$ sudo lsblk -o name,label,size,type,mountpoints /dev/nvme0n1
NAME                LABEL     SIZE TYPE  MOUNTPOINTS
nvme0n1                     476.9G disk
|-nvme0n1p1         NIXBOOT     1G part  /mnt/boot
|-nvme0n1p2         NIXLUKS 475.9G part
   |-cryptroot              475.9G crypt
     |-lvmroot-swap NIXSWAP    24G lvm   [SWAP]
     |-lvmroot-root NIXROOT 451.9G lvm   /mnt

[nixos@nixos:'']$

Installing NixOS

We’ll first generate a minimal configuration file. Then we’ll edit it to support our LUKS encrypted partitions.

# nixos-generate-config --root /mnt

Then we can use Vim (or nano) prefixed with sudo to edit our generated config to mount our LUKS partition. If you forget to elevate your editor command, you won't be able to save your changes—don't ask me how I know that.

# vim /mnt/etc/nixos/hardware-configuration

hardware-configuration.nix

Our generated config currently looks something like this:

# Do not modify this file! It was generated by ■nixos-generate-config■
# and may be overwritten by future invocations. Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernerlModules = ["xhci_pci" "ahci" "nvme" "usb_storage" "usbhid" "sd_mod" ];
  boot.initrd.kernelModules = [ "dm-snapshot" ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    {
      device = "/dev/disk/by-uuid/f96d7825-6b61-452b-a6f9-456ad8ba308f";
      fsType = "ext4";
    };

  fileSystems."/boot" =
    {
      device = "/dev/disk/by-uuid/6335-2377";
      fsType = "vfat";
      options = [ "fmask=0077" "dmask=0077" ];
    };

  swapDevices =
    [{ device = "/dev/disk/by-uuid/b0849f28-36bf-41c1-b58f-108f4a51442a"; }
    ];

  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
  # (the default) this is the recommended approach. When using systemd-networkd it's
  # still possible to use this option, but it's recommended to use it in conjunction
  # with explicit per-interface declaration with `networking.interfaces.<interface>.useDHCP`.
  networking.useDHCP = lib.mkDefault true;

  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
  hardware.cpu.intel.updatedMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}

While it's nearly perfect, it doesn't take into account our LUKS device, and so we'll have to disregard the ominous warnings to make some minor changes if we want our machine to be useable. In short, all we have to do is:

• Add cryptd to boot.initrd.kernelModules.
• Define our LUKS partition as a LUKS device under boot.initrd.luks.devices."cryptroot".
• Update our filesystem mounts to use “by-label” for ease of use / readability.

This last change is optional, but I find it helpful for compatibility reasons:

• Set hardware.enableAllFirmware to true.

# Do not modify this file! It was generated by ■nixos-generate-config■
# and may be overwritten by future invocations. Please make changes
# to /etc/nixos/configuration.nix instead.
{config, lib, pkgs, modulesPath, ...}:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernerlModules = ["xhci_pci" "ahci" "nvme" "usb_storage" "usbhid" "sd_mod" ];
  boot.initrd.kernelModules = [ "dm-snapshot" "cryptd" ]; # add cryptd
  # Define our LUKS device
  boot.initrd.luks.devices."cryptroot".device = "/dev/disk/by-label/NIXLUKS";
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

  # Use labels instead of UUIDs to mount our LVs
  fileSystems."/" =
    {
      device = "/dev/disk/by-label/NIXROOT";
      fsType = "ext4";
    };

  fileSystems."/boot" =
    {
      device = "/dev/disk/by-label/NIXBOOT";
      fsType = "vfat";
      options = [ "fmask=0077" "dmask=0077" ];
    };

  swapDevices =
    [{ device = "/dev/disk/by-label/NIXSWAP"; }
    ];

  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
  # (the default) this is the recommended approach. When using systemd-networkd it's
  # still possible to use this option, but it's recommended to use it in conjunction
  # with explicit per-interface declaration with `networking.interfaces.<interface>.useDHCP`.
  networking.useDHCP = lib.mkDefault true;

  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
  hardware.enableAllFirmware = true; # Helps with compatibility
  hardware.cpu.intel.updatedMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}

With our edits in place, we can save and exit the hardware-configuration.nix file and move on to configuration.nix.

configuration.nix

configuration.nix is where you'll want to configure everything else about your machine (e.g., hostname, user accounts, networking, window managers, programs, etc.). As such, we'll want to make some edits from the minimal configuration to something a lot more useable. Because, as it stands, the generated configuration.nix has most of what we need commented out (see: the code snippet below for more details).

# Edit this configuration file to define what should be installed on
# your system. Help is available in the configuration.nix(5) man page, on
# https://search.nixos.org/options and in the NixOS manual (`nixos-help`).

{ config, lib, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

  # networking.hostName = "nixos"; # Define your hostname.
  # Pick only one of the below networking options.
  # networking.wireless.enable = true; # Enables wireless support via wpa_supplicant
  # networking.networkManager.enable = true; # Easiest to use and most distros use this by default.

  # Set your time zone.
  # time.timeZone = "America/New_York";

  # Configure network proxy if necessary
  # networking.proxy.default = "http://user:password@proxy:port/";
  # networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";

  # Select internationalisation properties.
  # i18n.defaultLocale = "en_US.UTF-8";
  # console = {
  # font = "Lat2-Terminus16";
  #  keyMap = "us";
  #  useXkbConfig = true; # use xkb.options in tty.
  # }

  # Enable the X11 windowing system.
  # services.xserver.enable = true;

  # Configure keymap in X11
  # services.xserver.xkb.layout = "us";
  # services.xserver.xkb.options = "eurosign:e,caps:escape";

  # Enable CUPS to print documents.
  # services.printing.enable = true;

  # Enable sound.
  # hardware.pulseaudio.enable = true;
  # OR
  # services.pipewire = {
  #  enable = true;
  #  pulse.enable = true;
  # }

  # Enable touchpad support (enabled default in most desktopManager).
  # services.libinput.enable = true;

  # Define a user account. Don't forget to set a password with ■passwd■.
  # users.users.name = {
  #  isNormalUser = true;
  #  extraGroups = [ "wheel" ]; # enable ■sudo■ for the user.
  # }

  # List packages installed in system profile. To search, run:
  # $ nix search wget
  # environment.systemPackages = with pkgs; [
  #  vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
  #  wget
  # ];

  # Some programs need SUID wrappers, can be configured further or are
  # started in user sessions.
  # programs.mtr.enable = true;
  # programs.gnupg.agent = {
  #   enable = true;
  #   enableSSHSupport = true;
  # };

  # List services that you want to enable:

  # Enable the OpenSSH daemon.
  # services.openssh.enable = true;

  # Open ports in the firewall.
  # networking.firewall.allowedTCPPorts = [ ... ];
  # networking.firewall.allowedUDPPorts = [ ... ];
  # Or disable the firewall altogether.
  # networking.firewall.enable = false;

  # Copy the NixOS configuration file and link it from the resulting system
  # (/run/current-system/configuration.nix). This is useful in cage you
  # accidentally delete configuration.nix.
  # system.copySystemConfiguration = true;

  # The option defines the first version of NixOS you have installed on this particular machine.
  # and is used to maintain compatibillity with application data (e.g. databases) created on older NixOS versions.
  #
  # Most users should NEVER change this value after the initial install, for any reason,
  # even if you've upgraded your system to a new NixOS release.
  #
  # This value does NOT affect the Nixpkgs version your pacakges and OS are pulled from,
  # so changing it will NOT upgrade your system - see https://nixos.org/manual/nixos/stable/#sec-upgrading for how to actually do that.
  #
  # This value being lower than the current NixOS release does NOT mean your system is
  # out of date, out of support, or vulnerable.
  #
  # Do NOT change this value unless you have manually inspected all the changes it would make to your configuration,
  # and migrated your data accordingly.
  #
  #For more information, see `man configuration.nix` or https://nixos.org/manual/nixos/stable/options#opt-system.stateVersion .
  system.stateVersion = "24.05"; # Did you read the comment?
}

To make our changes, we can use vim or nano again to load configuration.nix into an editing buffer:

# vim /mnt/etc/nixos/configuration.nix

From there we'll, uncomment out network manager, we'll enable flakes, we'll update our time zone, add an account, and anything else we might need.

Core configuration

The following configures/enables what I believe to be the most important features/options. You'll be able to enable most of them simply by uncommenting them out (removing the # before them) and or modifying their default values.

Add a hostname

networking.hostName = "my-nixos-machine"; # Define your hostname.

Your hostname will correlate with the host system defined in your nixosConfiguration. If you plan on sharing a single flake.nix between multiple NixOS machines, it's probably for the best you change this to something unique—well, unique amongst the machine's that have differing configurations.

Enable network manager

networking.networkManager.enable = true;

Network Manager is both the easiest and (unsurprisingly) the default networking tool for most Distros. It's also what I prefer using, but you might prefer wpa_supplicant instead.

Enable flakes

nix.settings.experimental-features = ["nix-command" "flakes"];

Flakes are one of my most favorite things about Nix. They're incredibly versatile from setting up modular, multi-machine configurations from a single flake.nix²⁵ to creating reproducible development environments²⁶. In short, they're quite handy.

Update the time zone

time.timeZone = "America/Los_Angeles"

I believe it's also possible to update your time zone imperatively as well, if you're Desktop Environment (i.e., Gnome, Plasma, etc.) allows it. As such, setting it from configuration.nix will thus establish it as the "default" global/system time zone.

Add a user account with superuser privileges, that's also in the networkmanager group

users.users.mycoolusername = {
  isNormalUser = true;
  extraGroups = ["wheel" "networkmanager"];
};

superuser privileges via the "wheel" group, gives your user (in this case a user named mycoolusername) access to sudo. If you tried to use sudo without being in the wheel group or being given explicit access somewhere in the sudoers file, well it becomes an "incident", and gets reported ... somewhere²⁷.

Adding your user to the networkmanager group, while possibly optional, helps to ensure your user has the right permissions to use/configure networkmanager.

Enable unfree packages (optional)

nixpkgs.config.allowUnfree = true;

Allowing "unfree" packages, will give you access to closed-source/proprietary binaries/software, such as Nvidia's proprietary Linux drivers. It's really up to you whether you want to enable this feature or not.

Helpful Packages/Software

environment.systemPackages = with pkgs; [
  vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
  wget
  firefox
  chromium # swap for ungoogled-chromium if you prefer
];

At minimum, you'll want to add a secondary terminal based editor like vim, and a browser like firefox or alternatively, chromium and or ungoogled-chromium.

Anything Else You Might Want

Beyond the Core configuration above, I'll leave it up to you to determine how you want to configure the rest of your machine. For example, you'll probably want to choose a Desktop Environment (DE) or alternatively, configure a Window Manager (WM). I'll leave the following suggestions to give you some ideas:

Add a Desktop Environment or, alternatively, a Window Manager

In 2024, the two most popular DEs remain to be Gnome and Plasma. You can enable Gnome like this (see: wiki.nixos.org/wiki/GNOME²⁸):

services.xserver.enable = true;
services.xserver.displayManager.gdm.enable = true;
services.xserver.desktopManager.gnome.enable = true;

Or KDE Plasma like this (see: wiki.nixos.org/wiki/KDE²⁹)

services.xserver.enable = true; # optional
services.displayManager.sddm.enable = true;
services.displayManager.sddm.wayland.enable = true;
services.desktopManager.plasma6.enable = true;

Alternatively you can configure a Window Manager, such as Sway, like this (see: wiki.nixos.org/wiki/Sway³⁰)

# A very minimal Sway configuration
environment.systemPackages = with pkgs; [
  grim # screenshot functionality
  slurp # screenshot functionality
  wl-clipboard # wl-copy and wl-paste for copy/paste from stdin / stdout
  mako # notification system developed by swaywm maintainer
];

# Enable the gnome-keyring secrets vault.
# Will be exposed through DBus to programs willing to store secrets.
services.gnome.gnome-keyring.enable = true;

# enable sway window manager
programs.sway = {
  enable = true;
  wrapperFeatures.gtk = true;
};

services.xserver = {
  displayManager.gdm = {
    enable = true;
    wayland = true;
  };
};

Enable PipeWire (sound)

# rtkit is optional but recommended
security.rtkit.enable = true;
services.pipewire = {
  enable = true; # if not already enabled
  alsa.enable = true;
  alsa.support32Bit = true;
  pulse.enable = true;
  # If you want to use JACK applications, uncomment this
  #jack.enable = true;
};

PipeWire is a modern replacement for Pulse Audio, and if you're using Ardour or some other DAW, you'll want to enable JACK as well (see: wiki.nixos.org/wiki/PipeWire³¹). For configuring Bluetooth audio you'll want to see: wiki.nixos.org/wiki/PipeWire#Bluetooth_Configuration³².

Enable Bluetooth

hardware.bluetooth.enable = true; # enables support for Bluetooth
hardware.bluetooth.powerOnBoot = true; # powers up the default Bluetooth controller on boot

The above should enable Bluetooth, and if your DE doesn't come with a Bluetooth manager GUI, you can additionally enable the blueman applet

services.blueman.enable = true;

For more see: wiki.nixos.org/wiki/Bluetooth³³.

Enable CUPS + Avahi

# Enable CUPS to print documents.
services.printing.enable = true;
services.avahi = {
  enable = true;
  nssmdns4 = true;
  openFirewall = true;
};

Enabling CUPS will allow you to print things. Enabling Avahi will allow you to find printers on your network. For more see: wiki.nixos.org/wiki/Printing³⁴.

Use an Alternative Shell (e.g., zsh)

programs.zsh.enable = true;
users.defaultUserShell = pkgs.zsh;

If you've been hacking away at the command line on a Mac in the last decade, chances are you've become attached to using zsh instead of bash. In that case, the above will change the default shell (bash) to something that feels much more at home (zsh). For more see: wiki.nixos.org/wiki/Zsh³⁵.

Alternatively, you can use fish³⁶ or even nushell³⁷.

Example configuration.nix

{ config, lib, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

  networking.hostName = "my-nixos-machine"; # Define your hostname.
  networking.networkManager.enable = true; # Easiest to use and most distros use this by default.

  # Set your time zone.
  time.timeZone = "America/Los_Angeles";

  # Enable Flakes
  nix.settings.experimental-features = ["nix-command" "flakes"];

  # Enable the Gnome Desktop Environment
  services.xserver.enable = true;
  services.xserver.displayManager.gdm.enable = true;
  services.xserver.desktopManager.gnome.enable = true;

  # Enable CUPS to print documents.
  services.printing.enable = true;
  services.avahi = {
    enable = true;
    nssmdns4 = true;
    openFirewall = true;
  };

  # Enable sound.
  security.rtkit.enable = true;
  services.pipewire = {
    enable = true; # if not already enabled
    alsa.enable = true;
    alsa.support32Bit = true;
    pulse.enable = true;
    # If you want to use JACK applications, uncomment this
    #jack.enable = true;
  };

  # Bluetooth
  hardware.bluetooth.enable = true; # enables support for Bluetooth
  hardware.bluetooth.powerOnBoot = true; # powers up the default Bluetooth controller on boot

  # Define a user account. Don't forget to set a password with ■passwd■.
  users.users.mycoolusername = {
    isNormalUser = true;
    extraGroups = ["wheel" "networkmanager"];
  };

  # Default shell => ZSH
  programs.zsh.enable = true;
  users.defaultUserShell = pkgs.zsh;

  # Allow Unfree packages
  nixpkgs.config.allowUnfree = true;

  # List packages installed in system profile. To search, run:
  # $ nix search wget
  environment.systemPackages = with pkgs; [
    vim
    wget
    firefox
    chromium
  ];

  system.stateVersion = "24.05"; # Did you read the comment?
}

If you're happy with the above, we can move on to the best part, the flake.nix.

flake.nix

Alright, we're nearly at the finish line. The final piece of the (installation) puzzle is to create a flake.nix at /mnt/etc/nixos, in the same place where you've created the hardware-configuration.nix and configuration.nix, to bring everything together.

The flake.nix is where you'll be able to add additional modules as inputs, such as home-manager and lanzaboote.

We can define a minimal flake.nix like so:

{
  description = "A minimal flake.nix for a NixOS machine";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };
  outputs = {self, nixpkgs, ... }@inputs: {
    nixosConfigurations = {
      my-nixos-machine = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux"; # Assumes a standard x86 CPU
        modules = [./configuration.nix];
      };
    };
  };
}

And with that, we can finally proceed to the very much anticipated install command.

nixos-install

This is it, moment of truth time. To begin, we’ll change directories to our nixos directory:

cd /mnt/etc/nixos

Then we’ll update our flake. However, because the minimal NixOS install doesn't enable the nix-command nor flakes, we'll need to append these features as flags to actually perform this step:

# nix --extra-experimental-features nix-command --extra-experimental-features flakes flake update

If that presented no errors, your output should look something like this:

[nixos@nixos:/mnt/etc/nixos]$ sudo nix --extra-experimental-features nix-command --extra-experimental-features flakes flake update
warning: creating lock file '/mnt/etc/nixos/flake.lock'

[nixos@nixos:/mnt/etc/nixos]$

{
  inputs = {
-    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+    nixpkgs.url = "github:NixOS/nixpkgs/5633bcff0c61";
  };
}

With a clean flake update, that's the go-ahead to perform the magical install command.

# nixos-install --root /mnt --no-root-passwd --flake /mnt/etc/nixos#my-nixos-machine

Be warned though, this step could take a while (even assuming no compilation errors). As such, now's probably a good time to stretch your legs and get some fresh air. You've probably been sitting here reading this guide for a while now, so, maybe relax your eyes, and go enjoy a refreshment of some kind (a cup of tea, coffee, etc.). That way, if there are any unexpected errors (likely syntax, like a forgotten semi-colon), you'll at least be better prepared to deal with it...

...Alright, if everything did go according to plan you should see output like the following:

[nixos@nixos:/mnt/etc/nixos]$ sudo nixos-install --root /mnt --no-root-passwd --flake /mnt/etc/nixos#my-nixos-machine
copying channel...
building the flake in path:/mnt/etc/nixos?lastModified=1729820879@narHash=sha256-xRGYbMOgJiuxxVKLtkxb0yi01srhw/NL652Uq/ghXXY%3D...
Installing the boot loader...
setting up /etc...
Initializing machine ID from random generator.
Created "/boot/EFI".
Created "/boot/EFI/systemd".
Created "/boot/EFI/BOOT".
Created "/boot/loader".
Created "/boot/EFI/Linux".
Copied "/nix/store/xg6f0c5pchmc2jq84s4np19jirnn90mn-systemd-256.6/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/systemd/systemd-bootx64.efi".
Copied "/nix/store/xg6f0c5pchmc2jq84s4np19jirnn90mn-systemd-256.6/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/BOOT/BOOTX64.EFI".
Created EFI boot entry "Linux Boot Manager".
Installation finished!

[nixos@nixos:/mnt/etc/nixos]$

Final Steps

VERY IMPORTANT: We still need to setup a password for our user account. We can do that like so:

# nixos-enter --root /mnt -c “passwd mycoolusername”

Once that's all set, it's finally time to reboot. You can do that safely³ like so:

# umount -R /mnt
# sudo swapoff -L NIXSWAP
# sudo vgchange -a n lvmroot
# sudo cryptsetup close /dev/mapper/cryptroot
# reboot

Or dangerously via a simple: $ reboot

If everything went well, you’ll be greeted by LUKS asking for your passphrase.

<<< NixOS Stage 1 >>>

loading module dm-snapshot...
loading module cryptd...
loading module dm_mod...
running udev...
Starting sytemd-udevd version 256.6
Passphrase for /dev/disk/by-label/NIXLUKS: _

If it went really well, it'll accept your passphrase, and you'll arrive at your Display Manager of choice (or a TTY if that's what you configured), and you'll be able to login to your user with the password you just set.

At this point you can either continue configuring/tweaking your new NixOS machine to your liking before continuing to the next section, or you can just keep pushing through, the choice is up to you.

Part 02: Secure Boot with Lanzaboote

Lanzaboote is a wonderful project that I’ve been happily using on my NixOS machines for about a year now without a single issue. Still yet, it’s always a good idea to back up any important data before setting it up (like, your Windows BitLocker Recovery Keys, in case you're securely dual booting), and especially before updating Lanzaboote to the latest release.

In any event, we’ll be following along with the quick start guide lanzaboote/docs/QUICK_START⁸ for this section.

PLEASE BE ADVISED: THE QUICK START GUIDE LINKED ABOVE WILL PROVIDE YOU WITH FAR MORE ACCURATE/CURRENT INSTRUCTIONS THAN WHAT'S SHOWN HERE FOR THIS ARTICLE. THE FOLLOWING SECTIONS ABOUT LANZABOOTE SHOULD BE REGARDED AS SUGGESTIONS/COMMENTARY FOR EDUCATIONAL PURPOSES ONLY, RATHER THAN AS A TRUE INSTRUCTION MANUAL.

Confirm NixOS is installed in UEFI mode

Assuming you met the prerequisites and you successfully installed NixOS into the UEFI, the output of bootctl status should look like this:

❯ bootctl status
System:
     Firmware: UEFI 2.60 (INSYDE Corp. 22532)
Firmware Arch: x64
  Secure Boot: disabled (disabled)
 TPM2 Support: yes
 Measured UKI: no
 Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 256.6
...

If the firmware is UEFI and the current boot loader is systemd-boot we can continue.

Generate the Keys

We’ll need to use sbctl for this. You could add it to your system packages like so:

{pkgs, ...}:
{
  environment.systemPackages = with; pkgs [
    # sbctl needed to generate keys
    sbctl
  ];
}

However, we want to just run it from an ephemeral nix shell via $ nix-shell -p sbctl (we're going to declare this package in a module, as we'll see).

Either way, once you have sbctl just run # sbctl create-keys like so:

❯ sudo sbctl create-keys
[sudo] password for mycoolusername:
Created Owner UUID fcdc9ade-1757-4fd6-8376-bc21c7f4d093
Creating secure boot keys...✓
Secure boot keys created!

Configure the Lanzaboote Module (Flake)

The following is how I personally like to modularize my own NixOS flake based configuration. What we'll do is amend our minimal flake from earlier, adding lanzaboote as an input, and we'll enable it by adding it to our hosts module array. We're also going to create a directory in our /etc/nixos folder, called modules that will contain a module I'm calling lanza.nix. This is what the flake should look like:

{
  description = "A minimal flake.nix for a SecureBoot-enabled NixOS machine";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    lanzaboote = {
      url = "github:nix-community/lanzaboote/v0.4.1";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs = {self, nixpkgs, lanzaboote, ... }@inputs: {
    nixosConfigurations = {
      my-nixos-machine = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux"; # Assumes a standard x86 CPU
        modules = [
         ./configuration.nix
         lanzaboote.nixosModules.lanzaboote
         ./modules/lanza.nix
        ];
      };
    };
  };
}

Of course, the ./modules/lanza.nix doesn't exist yet, so let's create it and bring it up in our editor.

sudo mkdir /etc/nixos/modules
sudo touch /etc/nixos/modules/lanza.nix
sudo vim /etc/nixos/modules/lanza.nix

Then we can create something like this:

{pkgs, lib, ... }:

{
  environment.systemPackages = with pkgs; [
    # For debugging and troubleshooting Secure Boot.
    sbctl
  ];

  # Lanzaboote currently replaces the systemd-boot module.
  # This setting is usually set to true in configuration.nix
  # generated at installation time. So we force it to false
  # for now.
  boot.loader.systemd-boot.enable = lib.mkForce false;

  boot.lanzaboote = {
    enable = true;
    pkiBundle = "/var/lib/sbctl";
  };
}

Enable Lanzaboote

With the module configured, you should be cleared to run # nixos-rebuild switch, which will result in Lanzaboote being installed/enabled. You can check everything went well with the output from # sbctl verify

❯ sudo sbctl verify
Verifying file database and EFI images in /boot...
✓ /boot/EFI/BOOT/BOOTX64.EFI is signed
✓ /boot/EFI/Linux/nixos-generation-1.efi is signed
✓ /boot/EFI/Linux/nixos-generation-2.efi is signed
✗ /boot/EFI/nixos/kernel-linux-6.11.5.efi is not signed
✓ /boot/EFI/systemd/systemd-bootx64.efi is signed

If the output looks clean, we can enable secure boot.

"It is expected that the files ending with bzImage.efi are not signed." - The QUICK START guide for Lanzaboote

Enabling Secure Boot

if you followed the beginning of this guide, secure boot is currently disabled. What we’re going to do now is turn it back on, but with setup mode enabled.

Enable Setup Mode

My Lenovo laptop has this setup mode, and if you happen to have it as well, simply turn it on and proceed to the next step.

Enroll the Keys

Now that secure boot’s enabled (sorta), we can enroll the keys we generated earlier with vendor keys from Microsoft.

$ sudo sbctl enroll-keys --microsoft
[sudo] password for lani:
Enrolling keys to EFI variables...
With vendor keys from microsoft...✓
Enrolled keys to the EFI variables!

And now, just reboot! Once you’re back in, you can verify secure boot is activated (user mode).

$ bootctl status
System:
      Firmware: UEFI 2.60 (INSYDE Corp. 225332)
 Firmware Arch: x64
   Secure Boot: enabled (user)
  TPM2 Support: yes
  Measured UKI: yes
  Boot into FW: supported

Part 2.5: Unlocking LUKS with TPM2 using systemd-cryptenroll

At the very beginning of this guide, I promised I’d show you how to auto decrypt your LUKS device with TPM2. So, to do that, we’re going to amend the lanza.nix module we created earlier, to both install tpm2-tss and to create a tiny script (luksCryptenroller) to leverage a handy tool included in systemd: systemd-cryptenroll. The command our script calls on our LUKS device (NIXLUKS) looks something like this:

# systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=0+7 /dev/disk/by-label/NIXLUKS

According to the man pages for systemd-cryptenroll⁴², the flags on the command do the following:

• --wipe-slot=tpm2 clears out any previous keys bound to the slot.
• --tpm2-device=auto automatically grabs the default TPM2 chip (usually found on your motherboard).
• --tpm2-pcrs=0+7 binds the keys to Platform Configuration Registers 0 (platform-code, i.e., system firmware) and 7 (secure-boot-policy, i.e., the secure boot state; changes when enabled/disabled, or firmware certificates changes).

Then, we can implement said command into our lanza.nix module like this:

{pkgs, lib, ... }:
let
  luksCryptenroller = pkgs.writeTextFile {
    name = "luksCryptenroller";
    destination = "/bin/luksCryptenroller";
    executable = true;

    # Note: You can hardcode additional LUKS devices like so:
    # text = let
    #   ...
    #   luksDevice02 = "BEEGLUKS01";
    #   luksDevice03 = "BEEGLUKS02";
    # in ''
    #   ...
    #   sudo systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=0+7 /dev/disk/by-label/${luksDevice02}
    #   sudo systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=0+7 /dev/disk/by-label/${luksDevice03}
    # '';

    text = let
      luksDevice01 = "NIXLUKS";
    in ''
      sudo systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=0+7 /dev/disk/by-label/${luksDevice01}
    '';
  };
in
{
  environment.systemPackages = [
    luksCryptenroller
    # For debugging and troubleshooting Secure Boot.
    pkgs.sbctl
    # Needed to use the TPM2 chip with `systemd-cryptenroll`
    pkgs.tpm2-tss
  ];

  # Lanzaboote currently replaces the systemd-boot module.
  # This setting is usually set to true in configuration.nix
  # generated at installation time. So we force it to false
  # for now.
  boot.loader.systemd-boot.enable = lib.mkForce false;

  boot.lanzaboote = {
    enable = true;
    pkiBundle = "/var/lib/sbctl";
  };
}

I'll admit, not the prettiest (or the DRYest) bash script in the world, but hey, it gets the job done. A far more sophisticated approach would probably store the LUKS devices in an array, and use a for loop to run through it, calling systemd-cryptenroll with the current/ith device in the array. I'll leave that up to you to implement.

Enrolling the keys

This is rather straight forward just run $ luksCryptenroller (you'll immediately be prompted for your superuser password). Then just enter your passphrase for each LUKS device, in the order you declared them in.

Part 03: Configuring Home Manager (Abridged)

home-manager is a very extensive utility for configuring the user (rather than the global/system) environment on a Nix/NixOS machine. It's far too much to talk about in detail here, so I'll leave you with the official Home Manager Manual⁴³, a very handy tool: Home Manager Option Search⁴⁴, and finally the nix-community/home-manager/issues⁴⁵ page for when things go awry (It doesn't happen that often, but I'll admit weird bugs do happen on occasion).

As such, I'm going to very briefly walk you through configuring home-manager via further amending our flake.nix, and setting up the most important features I use it for (shell configuration, git/gh configuration, etc.).

Step 0 - fonts.nix

Shells look best with monospaced fonts, especially monospaced nerd-fonts, so before we get ahead of ourselves, let's create a fonts.nix in the systemwide space.

# touch /etc/nixos/modules/fonts.nix

And we'll bring it into an editor with:

# vim /etc/nixos/modules/fonts.nix

To create the following:

# modules/fonts.nix
{pkgs, ...}:
{
  fonts = {
    fontconfig = {
      enable = true;
    };
    packages = with pkgs; [
      noto-fonts
      noto-fonts-emoji-blob-bin
      noto-fonts-cjk-sans
      nerd-fonts._0xproto # personal fav monospaced font. However, you can use whatever monospaced nerd font you'd like.
      nerd-fonts.symbols-only
    ];
  };
}

As you'll notice, this module enables a system wide fontconfig (you could also use home-manager to enable a user space one), and several common font packages, along with my favorite monospaced font (0xProto) packaged as a nerd-font.

We can then import it into configuration.nix like so:

# configuration.nix
{ config, lib, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
      # system modules
      ./modules/fonts.nix
    ];

  ...
}

We can then build the new configuration like so:

# nixos-rebuild switch

And with single # nixos-rebuild switch we now have all the fonts we need to continue.

Configuring Home Manager via flake.nix

This is rather straight forward (it's one of the neat things about flakes!), just like we added lanzaboote, we'll add home-manager as an input, and configure it as a module⁴⁶.

{
  description = "A minimal flake.nix for a SecureBoot-enabled NixOS machine, with Home Manager";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    lanzaboote = {
      url = "github:nix-community/lanzaboote/v0.4.1";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs = {self, nixpkgs, lanzaboote, home-manager, ... }@inputs: {
    nixosConfigurations = {
      my-nixos-machine = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux"; # Assumes a standard x86 CPU
        specialArgs = {
          inherit inputs; # this passes down the inputs
        };
        modules = [
          ./configuration.nix
          lanzaboote.nixosModules.lanzaboote
          ./modules/lanza.nix
          home-manager.nixosModules.home-manager
          {
            home-manager.useGlobalPkgs = true;
            home-manager.useUserPackages = true;
            home-manager.users.mycoolusername = import ./home;
            home-manager.extraSpecialArgs = inputs; # from the passed down input, we can pass these as args to `home.nix`
          }
        ];
      };
    };
  };
}

Then we'll create our home directory, along with the module's we'll need:

❯ sudo mkdir /etc/nixos/home
❯ sudo mkdir /etc/nixos/home/modules
❯ sudo mkdir /etc/nixos/home/modules/shell
❯ sudo touch /etc/nixos/home/default.nix
❯ sudo touch /etc/nixos/home/home-configuration.nix
❯ sudo touch /etc/nixos/home/modules/shell/default.nix
❯ sudo touch /etc/nixos/home/modules/shell/zsh.nix
❯ sudo touch /etc/nixos/home/modules/shell/starship.nix
❯ sudo touch /etc/nixos/home/modules/shell/git-gh.nix

The above is pretty verbose, so let's walk through each module.

{
   home-manager.users.mycoolusername = import ./home;
}

home/default.nix

Continuing, let's start with home/default.nix. Using a superuser privileged editor, you'll want to assemble home/default.nix like this:

{
  imports = [
    ./home-configuration.nix
    ./modules/shell
  ];
}

Inline with the spirit of default.nix, this simply imports a file called ./home-configuration.nix (which we'll configure next), and another default.nix which declares the default imports for the shell module.

home/home-configuration.nix

And we can assemble home-configuration.nix like so:

{
  home.username = "mycoolusername";
  home.homeDirectory = "/home/mycoolusername";
  home.stateVersion = "24.05";
  programs.home-manager.enable = true;
}

The above configures Home Manager to act on mycoolusername's home directory, and enables itself.

home/modules/shell/default.nix

Another default.nix module, it'll import modules used primarily within the terminal shell.

{
  imports = [
    ./zsh.nix
    ./starship.nix
    ./git-gh.nix
  ];
}

home/modules/shell/zsh.nix

{config, pkgs, ...}: {
  home.packages = with pkgs; [
    pfetch-rs
  ];

  programs.zsh = {
    enable = true;
    autosuggestion.enable = true;
    syntaxHighlighting.enable = true;
    history = {
      size = 10000;
      path = "${config.xdg.dataHome}/zsh/history";
    };
    initExtra = ''
    case $(tty) in
      (/dev/tty[1-9]);;
      (*)
        eval pfetch;;
    esac
    '';
  };
}

The above configures zsh (which we declared and enabled as the default shell from configuration.nix), with some common plugins, a decent history size, and even runs pfetch every time you open a fresh shell. The little if/case statement in initExtra ensures pfetch doesn't run in a bare tty (which, I'm sure you're very familiar with if you followed Part 01: From Zero to NixOS all the way through).

home/modules/shell/starship.nix

{
  programs.starship = {
    enable = true;
    settings = {
      add_newline = false;

      nix_shell = {
        symbol = " ";
      };

      git_status = {
        format = "([\$all_status$ahead_behind\]($style) )";
        modified = "󰇂 ";
        ahead = " $\{count}";
        conflicted = "󱚝 ";
        behind = " $\{count}";
        diverged = "󰹺   $\{ahead_count}  $\{behind_count}";
        up_to_date = " ";
        deleted = "󰮉 ";
        untracked = "󱚠 ";
        stashed = " ";
        staged = " ";
        style = "bold blue";
      };
    };
  };
}

The above is my personal Starship Prompt configuration, and I hope it serves you well in your version control adventures (if you choose to implement this module).

home/modules/shell/git-gh.nix

{
  programs.git = {
    enable = true;
    userName = "mycoolgithubusername";
    userEmail = "mycoolgithubacctemail@xample.com";
    extraConfig = {
      safe.directory = "/etc/nixos";
      core.editor = "nvim"; # We didn't cover this, but I trust you can setup neovim on your own, perhaps via nixvim ;3 (https://nix-community.github.io/nixvim/)
    };
  };
  programs.gh = {
    enable = true;
  };
}

The Final Step: nixos-rebuild switch

With all the modules configured, it's time to run # nixos-rebuild switch, or # nixos-rebuild boot if you want to cut down on the misc Nix profile generations (only drawback is you have to reboot). If everything compiled successfully, it's now time to bask in the triumph of having accomplished a LOT today, complete with a base Home Manager configuration as icing on the cake.

Discussion

Welcome to the end of a very long blog post guide. If you completed the above, you have earned a very well deserved break and celebration of your accomplishments today (or however long it took to get through this guide).

What you've done is no small feat. In reaching the end of this guide, you've installed NixOS with Full Disk Encryption, enabled secure boot with lanzaboote, enabled your devices TPM2 chip to decrypt your LUKS devices during boot with systemd-cryptenroller, and finally configured a slick zsh shell with Starship Prompt, some plugins, and configured git and gh, ready to turn your /etc/nixos folder into a repo⁴⁸. Pretty rad!

If you knew nothing about Nix when you first got here, well, I hope you were able to get the gist of it after reaching the end here, and picked up a few things along the way.

I also hope that in being your guide on this journey through the major circles of Nix, that you were able to overcome some of the steep learning curve Nix/NixOS is infamously known for. If however, I failed to help with that... Welp, I guess I'll just have to update this post in accordance to your feedback, won't I?

In any event, I hope you learned something, and if you were ever a chronic distrohopper¹ like I was, I hope NixOS brings you the same peace of mind it brought me, serving as the final distro following a very long series of hops.

Finally, It's just me writing/editing these articles, and they're quite long. As such, it's entirely possible I've accidentally a word or semi-colon or two. Should you find any errors in this guide, I'd really appreciate it if you either left a comment below explaining what's wrong, opened an issue on this site's repo, or submitted a pull request to correct this article directly. Thank you.

PS, I'd love to hear your thoughts! Feel free to drop a comment or question below, or reach out to me on social media to let me know what you thought of this article!

Since I finally managed to break the CI/CD pipeline for this site, I've been thinking a lot about that infamous quote from the Senator for Alaska. Not for its accuracy (or lack thereof), but because in this one instance, I did, in fact, break (or jam) a critical tube that gets this site out to the internet. Though, I should admit that's not what took this site offline for 4 hours (we'll get to that part soon).

The Situation

At around 0600 Zulu time on February 21st, I rebased this site's production branch with main, and pushed two of my latest commits. One updated my flake.nix inputs (updated flake.lock), the other was a simple fix to the cards on the landing page. About 30 minutes later I received the following email.

What followed was what I can only describe as an adventure.

That skeet now serves as both a historical record of what happened that Thursday night, and was the inspiration behind this very blog post.

A Preliminary Investigation

Debugging is an Art. While I'm not the world's greatest detective, I am pretty good at sniffing out the cause of the issue, and assembling a solution thereafter. So, let's go through what I did.

POI #01: Timing Out & Freezing

The first thing I did was look into the workflow logs, where I discovered our first Point of Interest (POI), the timeout error.

Given my experience with the beta version of SSTv3 (Ion), an occasional timeout on sst deploy wasn't entirely unexpected. It's why I set timeout-minutes: 30 on the Deploy step in the first place. So, I decided to give it another go.

This gave a Locked state error, since the previous deploy attempt didn't get to gracefully cancel deployment to AWS. So, following the error message's guidance, I ran sst unlock from my terminal and tried again.

Huh. I was audibly stunned. Watching this attempt go by, I was confounded why it was stuck at the same place as the first attempt. At about a minute and 23 seconds in, I pulled the plug.

While that might seem pre-emptive, compared to a successful production run up to this point, the process would be about half way done by now, so freezing up at line 39 was incredibly odd.

Troubleshooter's Step 01: Unplug it and Plug it Back in

I was feeling puzzled, until I was suddenly struck with genius. I would simply perform the equivalent of unplugging and plugging it back in again via running sst remove and sst deploy, magically solving my problems! What could go wrong?

While I was grateful that attempt 04 displayed some signs of life, at about 4 minutes into the Deploy however, it was clear something had gone wrong. So, I pulled the plug on this attempt too.

To make matters worse, because I did run sst remove --stage=production, my production site was now ripped from the internet. While this would be fine if I had either a backup to divert traffic to, or realized I could've just re-run the last passing production Build Action, I unfortunately did not. This was cause for some light panic.

Troubleshooter's Step 02: Update Deps

As I was reeling from terror, I decided to try an emergency dependencies update of Node.js and Turbo. I didn't have any evidence that they weren't the cause of the problems, but a cheap hail-mary attempt seemed worth a shot.

No dice. As a follow-up, I wondered if re-creating bun.lock might help, so I tried that too.

As you can imagine, that didn't work either.

Troubleshooter's step 03: Pin Deps

If updating dependencies wasn't working, perhaps pinning them to an older version might help. I decided sst might be playing a role here, so I rolled it back, and ran the CI again.

Oops. I correct the Lock once more, and sent through my final hail mary attempt for the evening.

Sigh. I let out a pained sigh. This is gonna require thinking, isn't it?, I whined to myself. So, I took a break for a couple of hours, and contemplated.

Approaching a Solution

After my basic troubleshooting session brought me to somewhere worse than square one (no website), I started mulling over what I learned.

What I found peculiar, was that consecutive runs would hang around the same place. For whatever reason, attempting to build @website/web (this site), would lockup the Deploy step either right away (if the other functions don't exist yet), or 75 lines in if my metrics monitoring functions need to be created. Either way, this pointed to something going wrong with how @website/web was being built.

New POIs

This realization, narrowed things down to a few potential causes:

• @website/web is being built, but something is failing to report that step finished (sigterm), freezing the Deploy step as a result.
• @website/web is failing to build in the CI runner for reasons.
  • Perhaps the sst CLI has a bug in reading the flag syntax?
  • Perhaps Ubuntu/Debian is being difficult with SST again (I think this was an issue during the beta).
• The Turborepo tasks for @website/web aren't being executed properly.
  • A task that occurs before build, like test, might be failing to send a sigterm, or bun/sst/turbo is failing to pick it up.
  • In any event, the build task for @website/web gets stuck waiting to be executed, causing the hang.

So, I decided to go down this list from ez to difficult in trying to solve the problem.

Too EZ: Check your syntax

While I can't find the relevant issue now, I do remember coming across a closed issue in the sst repo, detailing that running a command like sst COMMAND --FLAG=VAR was causing issues but sst COMMAND --FLAG VAR wasn't. I believe I tried to run this locally, but that didn't fix it, so I moved on.

Redefining Turborepo's Tasks

Armed with the possibility the tasks weren't executing properly, I decided to remove the prebuild task as dependency of test from the turbo.json in both @website/web and @website/showcase. I also made build depend on other build tasks first (suggested by the turborepo docs)

    "test": {
-      "dependsOn": ["prebuild"]
+      "dependsOn": []
    },
    "build": {
-      "dependsOn": ["test"],
+      "dependsOn": ["test", "^build"],

I also changed the buildCommand (for both web and showcase) in my sst.aws.Nextjs function to specify bun specifically.

export const web = new sst.aws.Nextjs("Web", {
  path: "apps/web",
-  buildCommand: "turbo build:open-next",
+  buildCommand: "bun run turbo build:open-next;",
  server: {
    runtime: "nodejs22.x"
  },

Then (locally) I held my breath and ran sst deploy --stage production --verbose. Thank fuck! I was overwhelmed with relief, when I saw things had gone smoothly, and everything was back online.

For a bit I even contemplated calling it here, but I knew that deploying locally and deploying in the CI were two very different things. However, having tasted this victory, I was confident I could coax a second from the CI runner.

So, with a foolish grin, I pulled the website down with sst remove, because I knew an even greater victory awaited me in the CI runner.

Overcoming the CI runner

Since everything worked locally, I figured either it would just work in the CI runner, or if it didn't, I could just further reduce the barriers to the build step like I did earlier. So, I rebased main onto production and pushed things through.

Unsurprisingly, things did not just work. However, I did have an inkling of what I was doing this time around, so I got to work.

Experiment 01: Isolating the Tasks

From earlier, I altered the build commands back to what they were, in case that was causing a problem.

export const web = new sst.aws.Nextjs("Web", {
  path: "apps/web",
-  buildCommand: "bun run turbo build:open-next;",
+  buildCommand: "turbo build:open-next",
  server: {
    runtime: "nodejs22.x"
  },

Then I altered the tasks array somewhat back to what it was (which failed locally, but this is science now). I set prebuild to depend on nothing, and made test depend on prebuild (as it was).

  "extends": ["//"],
  "tasks": {
    "prebuild": {
+      "dependsOn": [],
      "inputs": ["$TURBO_DEFAULT$", "content/**"],
      "outputs": [".contentlayer", ".contentlayermini", ".versionvault"]
    },
    "test": {
-      "dependsOn": []
+      "dependsOn": ["prebuild"]
    },
    "build": {
      "dependsOn": ["test", "^build"],

This gave an interesting result.

While I wasn't enthusiastic about what I saw, it was an interesting result. In hindsight, this might've worked, I think I just got nervous around the two-minute mark and assumed the worse. Still, I pressed on.

Experiment 02: Simplifying the Task Dependencies

I decided to do two things. I continued to tweak the build command, and then I did what I believed was the solution (simplifying the task deps) to start at the build task immediately.

export const web = new sst.aws.Nextjs("Web", {
  path: "apps/web",
-  buildCommand: "turbo build:open-next",
+  buildCommand: "turbo build:open-next;",
  server: {
    runtime: "nodejs22.x"
  }

    "test": {
-      "dependsOn": ["prebuild"]
+      "dependsOn": []
    },
    "build": {
-      "dependsOn": ["test", "^build"],
+      "dependsOn": ["^build"],
      "outputs": [".next/**", "public/dist/**", "public/sw.js"],
      "inputs": [
        "$TURBO_DEFAULT$",

This gave me the result I was working so hard towards.

I did it! I did the thing! I was ecstatic for the rest of the evening, basking in victory. It's just unfortunate this did not last into the next day when I setup the dev workflow. My attempt at integrating the lesson I had just learned.

Revenge of the CI Runner

In setting up the dev Action Workflow (about two weeks ago at the time of writing), I hit a painfully familiar snag—after hitting a few unrelated snags, of course.

Let me preface with: this adventure wasn't as bad as it looks. In order:

• The first five build attempts failed due to a misconfiguration of an AWS role.
• The next two failed due to my handle failed deploy job on failure, failed to unlock the state (ironic).
• The eighth run I canceled due to seeing a very familiar deploy printout
• Runs nine and ten timed-out (just like before)
• Run eleven (Dev Build Action #5 attempt 2) failed due to state being locked again
• Run twelve (Dev Build Action #6) I canceled after it seemed like it was going to just hang.
• Run thirteen (what a lucky number) succeeded as Dev Build Action #7.

While I am going to explain what I changed, to make run thirteen a success, I do need to preface that my changes are apparently no longer necessary given the success of the most recent build action (#19). I'll explain that towards the end.

Nuclear Option: SST buildCommand set to "exit 0;"

After banging my head against the keyboard in frustration, I decided upon the nuclear option. Instead of letting sst deploy run the build command, it's possible to just let turbo do it instead. This is very similar to the Nx Monorepo Configuration guide in the OpenNext Docs.

In my root turbo.json I created a deploy task that depends on build.

{
  "$schema": "https://turbo.build/schema.json",
  "tasks": {
    "build": {
      "dependsOn": ["^build"]
    },
+    "deploy": {
+      "dependsOn": ["build"]
+    },
  }
}

In my root package.json I added deploy and deploy:dev to the scripts object.

{
  "scripts": {
    "build": "turbo run build",
    "dev": "turbo run dev",
    "lint": "turbo run lint",
    "test": "turbo run test",
    "test:watch": "turbo run test:watch",
+    "deploy": "turbo run deploy",
+    "deploy:dev": "turbo run deploy && sst deploy --stage=dev --verbose"
  },
}

In my sst.aws.Nextjs function I set the buildCommand to exit 0; (thus SST won't handle the build process).

export const web = new sst.aws.Nextjs("Web", {
  path: "apps/web",
-  buildCommand: "bun run build:open-next",
+  buildCommand: "exit 0;",

Finally In the nested turbo.json of each application (@website/web, @website/showcase), I created a task called deploy, and set it dependent on the build:open-next task.

{
  "extends": ["//"],
  "tasks": {
    "build": {
      outputs": [".next/**", "public/dist/**", "public/sw.js"],
      "inputs": ["$TURBO_DEFAULT$", "next.config.*"]
    },
    "build:open-next": {
      "dependsOn": ["build"],
      "env": ["OPEN_NEXT_VERSION", "NEXT_PUBLIC_DEPLOYED_URL"],
      "outputs": [".open-next/**"],
      "inputs": ["$TURBO_DEFAULT$", ".open-next.config.ts"]
    },
+    "deploy": {
+      "dependsOn": ["build:open-next"]
+    }
    }
  }

I also altered the Dev workflow to run bun run deploy:dev (the script in the root package.json) instead of sst directly.

      # ROTFB
      - name: Deploy
-        run: bun sst deploy --stage dev --verbose
+        run: bun run deploy:dev
        timeout-minutes: 30

This worked perfectly.

This worked so perfectly, I've had consistent deployments with it no problem over the last week and a half or so. There is one caveat, which leads us to our next section.

Dude, where's my variables?

Bypassing SST's buildCommand, like I did, the rub is that you lose out on having resources passed down to the application's build environment. While the resources part can be solved with wrapping turbo deploy with sst shell i.e. sst shell bun run turbo deploy, that apparently does nothing for the environmental variables that you can define in a config.

export const web = new sst.aws.Nextjs('Web', {
  path: 'apps/web',
  buildCommand: 'exit 0;',
  server: {
    runtime: 'nodejs22.x',
  },
  environment: {
    NEXT_PUBLIC_DEPLOYED_URL:
      $app.stage === 'production' ? 'https://laniakita.com' : `https://${$app.stage}.laniakita.com`,
  },
  domain: {
    name: $app.stage === 'production' ? 'laniakita.com' : `${$app.stage}.laniakita.com`,
    dns: sst.cloudflare.dns(),
  },
});

So, in the example above, a variable defined between lines 7-12 just doesn't get passed down. While I've thought about some clever workarounds importing specific .env files based on a task, I decided to try something.

Going Full Circle

Last night, I was brought full circle, when I tested the waters on returning to what I was doing weeks ago, before all these shenanigans took place. I used the buildCommand.

export const web = new sst.aws.Nextjs("Web", {
  path: "apps/web",
-  buildCommand: "exit 0;",
+  buildCommand: "turbo run build:open-next"

Likewise, I edited the scripts in the root package.json once more.

{
  "scripts": {
    "build": "turbo run build",
    "dev": "turbo run dev",
    "lint": "turbo run lint",
    "test": "turbo run test",
    "test:watch": "turbo run test:watch",
    "deploy": "turbo run deploy",
-   "deploy:dev": "turbo run deploy && sst deploy --stage=dev --verbose"
-   "deploy:production": "turbo run deploy && sst deploy --stage=production --verbose"
+   "deploy:dev": "sst deploy --stage=dev --verbose",
+   "deploy:production": "sst deploy --stage=production --verbose"

  },
}

I pushed everything from my local terminal, and was surprised to see a little green check mark against my hesitant commit message.

I then sent several more changes through, to test whether that was a fluke. It was not.

Given everything just works, like it should. Like it did before, for months, leading up to this. I'll admit to feeling a little vexed that evening, but more so just relieved.

Discussion

This was an adventure. This was too much adventure. I want off Mr. Bone's Wild Adventure.

If the internet is like a series of tubes, then the wild ride I just endured can be blamed on a jammed tube, a criss-crossed tube, and a return tube.

In other words, this website failed to deploy to production, a jam in the CI tube, which culminated in the site going offline. Making some changes to how the site is processed through the tube, let it go through. Yet, those same changes weren't sufficient for a different tube, and it jammed again. A criss-crossing if you will from failure-to-success-to-failure-to-success. Then, everything fell down the return tube (for reasons), and we went full circle with the website being able to go through the dev tube with the original build instructions.

Metaphorical tubes aside, I had originally intended this article to be somewhat instructional. I had assumed that this was clearly a configuration problem on my part, because a poor craftsman blames their tools. So, I wanted to figure it out, write it down, and paste my opinionated configuration guide to the internet for anyone else looking to configure Turborepo, Bun, Next.js and SST with a Github Action to handle deployments.

However, given my experiment last night, which demonstrated sequential, successful deployments were possible using the original, wrapped build commands... I've been given some pause by this whole thing, I'll admit.

In the end, I'm left only more confused, than when I began.

This wasn't the post I planned to write. Initially, I thought this post was going to be dedicated exclusively to my bespoke content management system. Where, I thought I’d wax poetic for a while about how sometimes the best solutions, are tailor made. However, I’ve since come back to my senses to realize that the best solution is the one with the smallest number of compromises, hand crafted be damned.

So, that’s where contentlayer2¹ (maintained fork of contentlayer) came in to the equation. it's a library I stumbled upon that did exactly what I’d come up with, but in a better, more robust, and much more practical manner. In short, it's beautiful.

In honesty, I was awestruck by contentlayer’s pitch video². Especially so, because I’d just gone through the whole process of pouring blood, sweat, and tears into my own content management system for this site. It really wasn't long after that, that I swapped my whole custom solution out for it.

But, swallowed pride aside, creating my own content backend was quite a learning experience for me, so I wanted to talk about it. As such, I present to you my technical summary on my custom solution.

Introduction

I started my life in web working with traditional CMSes like WordPress³ and then briefly, Ghost⁴. Later, I took the JAMstack⁵ hype train as far as it could go, fetching data from headless CMSes⁶ like Strapi⁷ and Contentful⁸, and integrating it into Gatsby⁹ frontends. Then, sometime before building this site, I caught myself up on the latest headless CMSes people have been working with. I experimented with Payload CMS¹⁰ and Directus¹¹, before diving into off-the-shelf backends like Supabase¹², and PocketBase¹³ (my personal favorite).

However, something about all the options I weighed kept disappointing me in one way or another. So, after a day of feeling the frustration of yet another imperfect solution, I found myself one late evening going "Yeah, well, I'll build my own CMS, with Blackjack, and Hookers! In fact, forget the CMS"¹⁴. So, that's what I did.

My idea was pretty straightforward. Instead of a traditional user-friendly CMS, I based it on Astro's¹⁵ model of content management (Content Collections¹⁶) —leveraging markdown¹⁷, yaml¹⁸, and jsx¹⁹ in .mdx²⁰ files to store and organize content—with my own twist, a database I can cache all that data i want into. The latter meant I could fetch and generate content live in production, in taking advantage of Next.js’ SSR features²¹.

As such, the following was the strategy I'd come up with as I poured over documentation²² and the various Next.js markdown blog examples^{23, 24} Vercel²⁵ has available:

1. Scan the posts.
2. Process the data.
3. Store data into a SQLite database
4. Fetch data from the database.
5. Render the data.

Sounds easy enough, right? Well, sorta. The largest hurdle to overcome was in the processing phase, when I decided simply fetching the featured Image src²⁶ wasn’t good enough. That's because I wanted srcset²⁷ blurs and that’s when things got complicated. A smaller hurdle, was realizing I wasn’t fetching the image transformations properly. Investigating why is how I wound up learning about Next’s loaderFile²⁸ configuration option, but I digress as that’s somewhat specific to the fact I use a CDN²⁹ (i.e., AWS CloudFront³⁰) to distribute this site (& it’s assets).

Now, while my custom solution worked, it was, admittedly, flawed. First, because I really only implemented the first three letters of the CRUD³¹ acronym for this. Second, by the time I came up with a way to auto delete content I’d removed, well, that's when I stumbled upon a rad little library called contentlayer2, as I'd mentioned before. That was the turning point which pretty much signaled the end of any further development on my custom solution.

Before that happened, I should mention why I wound up using Turso³², a SQLite³³ (libSQL³⁴) DBaaS³⁵ for this project, since SQLite's whole purpose is that it's embeddable.

Now, I actually did try that; embedding the SQLite database right into my Next.js application, and leveraging Bun’s native SQLite driver³⁶ in the process. However, when I was testing development builds of this site on Vercel, I learned there were some limitations^{37, 38, 39}.

The first issue was that, ephemeral storage in static functions makes SQLite impractical³⁷. Second, while a read-only implementation is possible³⁸, you are limited by the serverless function's storage capacity³⁹. Finally, I couldn't get read-only SQLite to work on dynamically rendered routes. In my testing, reading from the embedded SQLite database was only possible on routes generated using generateStaticParams⁴⁰.

So, in thinking about the future, and not wanting to limit my project pre-emptively, a DBaaS seemed my only viable option. However, by the time I'd actually realized that, I'd already written my Drizzle ORM⁴¹ statements for SQLite, and I wasn't in the mood for converting them into a different SQL format. So, of Drizzle's SQLite compatible drivers⁴², I wound up using Turso.

Setup

Of the ORMs, I had narrowed it down to two options, Prisma⁴⁵ or Drizzle ORM⁴¹. I settled on Drizzle because I liked the focus on TypeScript, and I liked that you could write raw(ish) SQL as an escape hatch using its magic SQL⁴⁶ operator. I also really liked the idea that the knowledge gained in working with it, would be transferable, since one of their taglines is if you know SQL, you know Drizzle. I'm also just a sucker for companies that know how to execute self-deprecating advertising strategies^{47, 48}, so my heart was settled on using it.

Drizzle + Turso

Anyways, since I did end up using Drizzle, there was some configuration required before I could flesh out the rest of my solution. Aside from registering for Turso and generating my API keys, the first thing I had to do was setup a drizzle.config.ts⁴⁹ file to use with Turso.

// drizzle.config.ts
import type { Config } from 'drizzle-kit';
import { Resource } from 'sst';

dotenv.config();

export default {
  schema: './src/lib/db/schema/*',
  out: './drizzle/migrations',
  dialect: 'sqlite',
  driver: 'turso',
  dbCredentials: {
    url: Resource.TursoUrl.value,
    authToken: Resource.TursoAuth.value,
  },
} satisfies Config;

I loosely followed Turso's guide on setting it up with Drizzle⁵⁰, then came up with the resultant code above. In sum:

• I'm importing multiple schemas using the wildcard glob⁵¹ pattern (*).
• SQL migrations get written into the root drizzle/ directory.
• I'm using the SQLite flavor of Drizzle, in tandem with the Turso driver.
• I load in my credentials for Turso using the { dbCredentials } object.
• There's a type check which ensures that config object is in agreement with the Config type imported from drizzle-kit.

Something I should note, is that instead of injecting the Turso credentials from a .env file using something like process.env.TURSO_AUTH⁵², I took advantage of SST's⁵³ secret⁵⁴ component instead. You'll notice I tapped into the { Resource }⁵⁵ object from sst using some simple dot notation⁵⁶. This really just avoids the hassle of passing around a .env.local⁵⁷ file between work machines / dev environments.

Schemas

Once I had that, then I just had to define some schemas/tables⁵⁸ for my posts. This resulted in creating the following tables:

• authors
• tagSlugs
• featured_images
• posts

The idea was that the authors, featured_images, and tagSlugs tables would respectively create one-many⁵⁹, and many-many⁵⁹ relations to the posts table. Something like this:

In keeping with SQLites Datatypes⁶⁰ and what's available in Drizzle⁶¹ Some commonalities between these tables (as we'll see), is that I'm using a string for each rows id and primary key column, and a integer set to timestamp for the date column. That's because the id is a UUIDv4⁶² generated using Bun's crypto.randomUUID()⁶³, and the date is a utc string. The rawStr column, is simply the raw utf8 string generated from reading the .mdx file into memory.

Of note, in the below schemas, I'm manually exporting their types. Unfortunately, at the time, I simply wasn't aware of drizzle's $inferInsert⁶⁴, $inferSelect⁶⁴ functions to automatically generate types. So, that's why I'm doing it artisanally, in case you were wondering.

// authors.ts

import { sqliteTable, text, integer } from 'drizzle-orm/sqlite-core';

export interface Authors {
  id: string;
  slug: string;
  date: Date;
  name: string;
  mastodon?: string;
  mastodonURL?: string;
  localKey: string;
  rawStr: string;
}

export const authors = sqliteTable('authors', {
  id: text('id').primaryKey(),
  slug: text('slug').unique().notNull(),
  date: integer('date', { mode: 'timestamp' }),
  name: text('name'),
  mastodon: text('mastodon'),
  mastodonURL: text('mastodon_url'),
  localKey: text('local_key'),
  rawStr: text('raw_str'),
});

// tagSlugs.ts

import { sqliteTable, text, integer } from 'drizzle-orm/sqlite-core';

export interface Tags {
  id: string;
  slug: string;
  date: Date;
  title: string;
  localKey: string;
  rawStr?: string;
}

export const tagSlugs = sqliteTable('tagSlugs', {
  id: text('id').primaryKey(),
  slug: text('slug').unique(),
  date: integer('date', { mode: 'timestamp' }),
  title: text('title').unique(),
  localKey: text('local_key'),
  rawStr: text('raw_str'),
});

//featured-images.ts

import { integer, sqliteTable, text } from 'drizzle-orm/sqlite-core';

export interface FeaturedImages {
  id: string;
  slug: string;
  date: Date;
  title?: string;
  fileLocation: string;
  caption?: string;
  credit?: string;
  creditUrlText?: string;
  creditUrl?: string;
  altText: string;
  localKey: string;
  blur: string;
  height: number;
  width: number;
  rawStr: string;
}

export const featuredImages = sqliteTable('featured_images', {
  id: text('id').primaryKey(),
  slug: text('slug').unique().notNull(),
  date: integer('date', { mode: 'timestamp' }),
  title: text('title'),
  fileLocation: text('file_location'),
  caption: text('caption'),
  credit: text('credit'),
  creditUrlText: text('credit_url_text'),
  creditUrl: text('credit_url'),
  localKey: text('local_key'),
  altText: text('alt_text'),
  blur: text('blur'),
  height: integer('height'),
  width: integer('width'),
  rawStr: text('raw_str'),
});

Since the posts.ts schema/table is the most complicated, being the one that integrates all the other tables, it's important to talk about what's happening. You'll notice in the below, I'm importing the previously defined schemas authors, tagSlugs, and featured_images, and consuming them both as foreign keys and in relational tables.

// posts.ts

import { sqliteTable, text, primaryKey, integer } from 'drizzle-orm/sqlite-core';
import { relations } from 'drizzle-orm';
import { authors } from './authors';
import { tagSlugs } from './tagSlugs';
import { featuredImages } from './featured-images';

export const authorsRelations = relations(authors, ({ many }) => ({
  posts: many(posts),
}));

export const featuredImagesRelations = relations(featuredImages, ({ many }) => ({
  posts: many(posts),
}));

export interface Posts {
  id: string;
  slug: string;
  date: Date;
  tagSlugs: string[];
  author: string;
  headline: string;
  subheadline?: string;
  featuredImage: string;
  altCaption?: string;
  localKey: string;
  rawStr: string;
}

export const posts = sqliteTable('posts', {
  id: text('id').primaryKey(),
  authorId: text('author_id')
    .references(() => authors.id, { onUpdate: 'cascade', onDelete: 'cascade' })
    .notNull(),
  date: integer('date', { mode: 'timestamp' }),
  slug: text('slug').unique().notNull(),
  headline: text('headline').unique().notNull(),
  subheadline: text('subheadline'),
  featuredImageId: text('featured_image_id').references(() => featuredImages.id, {
    onUpdate: 'cascade',
    onDelete: 'cascade',
  }),
  altCaption: text('alt_caption'),
  localKey: text('local_key'),
  rawStr: text('raw_str'),
});

export const postsRelations = relations(posts, ({ one, many }) => ({
  author: one(authors, {
    fields: [posts.authorId],
    references: [authors.id],
  }),
  featuredImage: one(featuredImages, {
    fields: [posts.featuredImageId],
    references: [featuredImages.id],
  }),
  postToTags: many(postsToTags),
}));

export const postsToTags = sqliteTable(
  'posts_to_tagSlugs',
  {
    postId: text('post_id')
      .notNull()
      .references(() => posts.id, { onUpdate: 'cascade', onDelete: 'cascade' }),
    tagId: text('tag_id')
      .notNull()
      .references(() => tagSlugs.id, { onUpdate: 'cascade', onDelete: 'cascade' }),
  },
  (t) => {
    return {
      pk: primaryKey({ columns: [t.postId, t.tagId] }),
    };
  },
);

export const postsToTagsRelations = relations(postsToTags, ({ one }) => ({
  tag: one(tagSlugs, {
    fields: [postsToTags.tagId],
    references: [tagSlugs.id],
  }),
  post: one(posts, {
    fields: [postsToTags.postId],
    references: [posts.id],
  }),
}));

In using them as foreign keys, I'm really just taking advantage of the cascade operations.

As far as relations go, I'm using drizzles relations⁶⁵ feature, and creating relational tables for both one-to-many (authors, featured_images) and many-to-many (tagSlugs) relations. In truth, this was somewhat extra, since I had foreign keys available to me on turso, but I was curious to know what drizzle was capable of, and this is one such addition that drizzle brings to the table (no pun intended >.<), beyond (most) vanilla SQL functions.

Database Config

With all the schemas fleshed out the last thing to setup, was a connection to the remote DB on Turso⁶⁶, where I then imported the schemas directly into the client connection.

// turso-db.ts

import { drizzle } from 'drizzle-orm/libsql';
import { Resource } from 'sst';
import { createClient } from '@libsql/client';
import * as authors from './schema/authors';
import * as tagSlugs from './schema/tagSlugs';
import * as featuredImages from './schema/featured-images';
import * as posts from './schema/posts';

dotenv.config();

const client = createClient({
  url: Resource.TursoUrl.value,
  authToken: Resource.TursoAuth.value,
});

export const maindb = drizzle(client, { schema: { ...authors, ...tagSlugs, ...featuredImages, ...posts } });

With that squared away, the final step was to push our schemas as tables to our database on turso, with a single command: drizzle-kit push⁶⁷. Though, since I was importing credentials via SST, that command was more like: sst shell bun run drizzle-kit push⁶⁸, but I digress.

Overview of my CMS

With the the database configured, the rest of the CMS could then be fleshed out. This process was in theory quite simple, following the outline from earlier. All I had to do was read content into memory, process them, and send it into the database via insert statements.

Scanning

In practice, this took the form of an extensive script I wrote that took advantage of Bun’s speedy File I/O APIs⁶⁹, to scan a folder that held all the blog content, which then read any Markdown and MDX files into memory as UTF-8 strings, sans any ignored ones.

From there I used gray-matter⁷⁰ to parse the stringified front matter, to then create an object to store both the front matter data and the raw string itself.

Processing

Then I checked that data for a UUID before generating a fresh UUIDv4 using Bun.crypto which gets injected into the original file’s front matter, where it can then be read back into memory to update the data object.

I also performed a similar step for the post slug as well, where I checked the front matter for the existence of a slug before generating one from the file name. I also checked if the slug and file name match, and if not, the file name overrides the slug. This is injected into the original file as well, and like the above, read back into memory to update the working data object.

Then I had a whole Image processing step, which, admittedly, is a little more complicated than the other two steps (as we’ll see). In short, a function scans the front matter data for an imageSrc, then it copies the (relatively defined) image from the content/assets folder, into Next’s /public folder¹⁴. Then it runs the image through plaiceholder⁷¹ to generate a blurry placeholder image (in base64⁷² format) and to also fetch the image dimensions (height, width). Finally, it appends all that data to the object I’d generated earlier.

Storing Data

Then, with another script, I just pushed all the data to the SQLite database on Turso using drizzle statements to insert/update the database.

Fetching Data

Once all the data was loaded, I simply fetched it using drizzle’s query builder (mostly, some calls were proper select statements), and I took advantage of using the React cache⁷³ hook built into Next.js to memoize those DB calls.

Rendering Data

Once I had the data, the only thing left to do was to use mdx-bundler⁷⁴ to process the raw MDX string, and I just loaded everything else into my front end.

Scanning the Posts

Alright, the very first thing to do, was to ingest the files that actually had the content, and bring everything into memory. The below script puts into action exactly that. It’s a little complicated, but most of the complexity stems from the added processing steps. We'll go through this, chunk by chunk, over the following sections, this is just to show you the final product, upfront.

The Big Script: fetch-mdx.ts

One thing to note, is that this is the raw script I was previously using, complete with eslint ignore directives, suggestions, and the comments I left for myself, to help explain to myself what’s going on.

Now, I did think about cleaning the script up, but I felt it was better to show you exactly what I was using right up until the switch to contentlayer2. Why? Perhaps for posterity, or perhaps I just find the code I wrote a while ago interesting due to its now foreign nature to myself.

// fetch-mdx.ts

#! /usr/bin/env bun
/* eslint-disable no-undef -- bun runtime will provide bun functions */
/* eslint-disable no-console -- doesn't run in the browser, so this is fine */

import path from 'node:path';
import { readdir, access } from 'node:fs/promises';
import matter from 'gray-matter';
import { getPlaiceholder } from 'plaiceholder';

const isNonEmptyArrayOfStrings = (value: unknown): value is string[] => {
  return Array.isArray(value) && value.length > 0 && value.every((item) => typeof item === 'string');
};

interface ConfigProps {
  // must be relative path from root project directory => './content'
  contentFolder: string;
  // array of relative paths INSIDE the content folder ['./assets']
  foldersToExclude?: string[];
  // array of literal file name/ext => ['README.md']
  filesToExclude?: string[];
  // debug
  debug?: boolean;
  suppressErr?: boolean;
}

export const defaultConfig: ConfigProps = {
  contentFolder: './content',
  foldersToExclude: ['./assets'],
  filesToExclude: ['LICENSE', 'README.md'],
};

/*
 * @example batchFetchMDXPaths({config})
 * () => ['./content/blog/post0.mdx', ..., './content/blog/postN.md']
 */
export const batchFetchMDXPaths = async ({
  contentFolder,
  foldersToExclude,
  filesToExclude,
  debug,
  suppressErr,
}: ConfigProps): Promise<string[] | undefined> => {
  try {
    const dir = await readdir(contentFolder, { recursive: true });

    const excludedFolders = foldersToExclude?.map((folder) => {
      const cleanedFolderPath = folder.replace('./', '');

      return cleanedFolderPath;
    });

    //const absPath = path.resolve(path.join(process.cwd(), contentFolder))

    const fileArr = dir.map((item): string | undefined => {
      debug && console.log('logging raw path:', item);

      if (excludedFolders?.some((folder) => item.startsWith(folder))) {
        debug && console.log('skipping ', item);
        return;
      }

      if (filesToExclude?.some((file) => item.endsWith(file))) {
        debug && console.log('ommitting ', item);
        return;
      }

      if (item.endsWith('.mdx') || item.endsWith('md')) {
        return `${contentFolder}/${item}`;
      }
      return undefined;
    });
    const filter1 = fileArr.filter((el) => el);
    // validate found paths
    const cwd = process.cwd();
    const validatedArr = await Promise.all(
      filter1.map(async (pathStr) => {
        try {
          await access(path.resolve(path.join(cwd, pathStr!)));
          return pathStr;
        } catch (err) {
          console.error(err);
        }
      }),
    );

    const finalArr = validatedArr.filter((el) => el);

    debug && console.log(finalArr);

    if (isNonEmptyArrayOfStrings(finalArr)) return finalArr;
  } catch (err) {
    if (err instanceof Error && 'code' in err && err.code === 'ENOENT') {
      !suppressErr &&
        console.error(
          "Ooops! Couldn't open that directory! Are you sure that folder is relative to root your project directory, i.e. './src/content/posts'? ",
          err,
        );
      return;
    }
    !suppressErr && console.error(err);
  }
};

interface ImageUtilProps {
  fmatter: Record<string, unknown>;
  mdxPath: string;
  publicPath?: string;
  imageKey: string;
  debug?: boolean;
}

const newimageEmbedPath = async ({
  fmatter,
  mdxPath,
  publicPath,
  imageKey,
  debug,
}: ImageUtilProps): Promise<string | undefined> => {
  if (imageKey && typeof imageKey === 'string' && imageKey in fmatter) {
    debug && console.log('found hero image with key', imageKey);

    //const imageType = 'type' in fmatter && (fmatter.type as string);

    const currentImagePath = fmatter[imageKey];

    if (typeof currentImagePath !== 'string') return;

    //debug && console.log(currentImagePath)

    const splitStr = mdxPath.split('/');
    // need to get parent path before mutating array with pop.
    const parentPath = splitStr.slice(0, splitStr.length - 1).join('/');
    //debug && console.log(parentPath)

    // now we can get the mdx file name to use as the folder later.
    const mdxFile = splitStr.pop();
    if (!mdxFile) return;
    //const mdxFileSlug = mdxFile.split('.')[0];
    //debug && console.log(mdxFileSlug)

    const imgToCopyFilePath = path.resolve(parentPath, currentImagePath);
    //debug && console.log(imgToCopyFilePath)

    const publicCopyPath = `/public/${publicPath}/${currentImagePath.split('/').pop()}`;

    //debug && console.log(publicCopyPath)

    const embedPublicCopyPath = `/${publicPath}/${currentImagePath.split('/').pop()}`;
    //debug && console.log(embedPublicCopyPath)

    const pathToCheck = path.join(process.cwd(), publicCopyPath);
    //debug && console.log(pathToCheck)
    const constPublicImgFile = Bun.file(pathToCheck);
    const imgFile = Bun.file(imgToCopyFilePath);

    /*
     * If the file isn't in the public folder, then copy it.
     * If an image already exists in the public folder,
     * but the declared frontmatter image is
     * different (diff in size), then replace it.
     */
    const checkImg = await constPublicImgFile.exists();

    if (!checkImg) {
      debug && console.log('image not in public folder, copying ...');
      await Bun.write(`${process.cwd()}${publicCopyPath}`, imgFile);
    } else if (constPublicImgFile.size !== imgFile.size) {
      debug && console.log('found image is different from public folder, copying ...');
      await Bun.write(`${process.cwd()}${publicCopyPath}`, imgFile);
    } else {
      debug && console.log('image is the same, not copying');
    }

    return embedPublicCopyPath;
  }
};

const imgMetaPlusBlurryPlaiceHolders = async ({ fmatter, mdxPath, imageKey, debug }: ImageUtilProps) => {
  if (imageKey && imageKey in fmatter) {
    debug && console.log(`using ${fmatter[imageKey] as string} to generate img data + blurs`);
    const currentImagePath = fmatter[imageKey];
    //console.log(currentImagePath)
    const splitStr = mdxPath.split('/');
    const parentPath = splitStr.slice(0, splitStr.length - 1).join('/');
    const imgToCopyFilePath = path.resolve(parentPath, currentImagePath as string);

    const imgFile = Bun.file(imgToCopyFilePath);
    const arrayBuf = await imgFile.arrayBuffer();
    const buf = Buffer.from(arrayBuf);

    const {
      base64,
      metadata: { height, width },
    } = await getPlaiceholder(buf);

    return { base64, height, width };
  }
};

const comboImageProcessing = async ({ fmatter, mdxPath, imageKey, publicPath, debug }: ImageUtilProps) => {
  if (imageKey && imageKey in fmatter) {
    // need to do this first (the other function seems to mutate something)
    const imgBlurPlusMetaRes = await imgMetaPlusBlurryPlaiceHolders({ fmatter, mdxPath, imageKey, debug });
    const imgBlurData = imgBlurPlusMetaRes?.base64;
    const imgHeight = imgBlurPlusMetaRes?.height;
    const imgWidth = imgBlurPlusMetaRes?.width;

    const newImgPath = await newimageEmbedPath({ fmatter, mdxPath, imageKey, publicPath, debug });
    fmatter[imageKey] = newImgPath;
    return { ...fmatter, blur: imgBlurData, height: imgHeight, width: imgWidth };
  }
};

interface InjectionPointProps {
  fileArr: string[];
  strOfInterest: string;
  precisionPoint: number;
  debug?: boolean;
}

const getInjectionPoint = ({ fileArr, strOfInterest, precisionPoint, debug }: InjectionPointProps) => {
  const getPoint = fileArr.map((strLine, index): number | undefined => {
    // we're going to look for the first "---" of the front matter
    // then inject. We can test that we're not adding at the end by
    // checking if the key following injection is valid
    const keyGuess = fileArr[index + precisionPoint]?.split(':')[0];
    //console.log(keyGuess)
    //console.log(strLine.split(':')[0])
    const point = index + precisionPoint;
    if (strLine === strOfInterest && keyGuess) {
      debug && console.log('inject at ', index + precisionPoint, 'before ', keyGuess);

      return point;
    } else if (strLine.split(':')[0] === strOfInterest) {
      debug && console.log('found', strOfInterest, 'at', index, 'Injecting at', point, 'before', keyGuess);
      return point;
    }
    return undefined;
  });
  const injectionPoint = getPoint.filter((el) => el)[0];
  return injectionPoint;
};

interface InjectionProps {
  rawFile: string;
  absFilePath: string;
  debug?: boolean;
}

const injectUUID = async ({ rawFile, absFilePath, debug }: InjectionProps) => {
  const uuid = crypto.randomUUID();
  const fileArr = rawFile.split('\n');

  // we need to search the file string to find out where
  // we can safely inject the uuid.

  const injectionPoint = getInjectionPoint({ fileArr, strOfInterest: '---', precisionPoint: 1, debug });

  if (typeof injectionPoint !== 'number') return;
  fileArr.splice(injectionPoint, 0, `id: ${uuid}`);

  const finalString = fileArr.join('\n');

  const fileWritePath = absFilePath;

  debug && console.log(`saving updated markdown file to `, fileWritePath);

  await Bun.write(fileWritePath, finalString);
  const newMatter = matter(finalString).data;

  // we'll need to update the image path in memory, if it exists

  return newMatter;
};

const injectSlug = async ({ rawFile, absFilePath, debug }: InjectionProps) => {
  const fileArr = rawFile.split('\n');

  const fileWritePath = absFilePath;
  console.log(fileWritePath);
  const fileName = fileWritePath.split('/').pop();
  console.log(fileName);
  if (!fileName) return;
  const slug = fileName.split('.')[0];

  // assuming we have the id, we'll inject it right after
  const injectionPoint = getInjectionPoint({ fileArr, strOfInterest: 'id', precisionPoint: 1, debug });

  if (typeof injectionPoint !== 'number') return;
  fileArr.splice(injectionPoint, 0, `slug: ${slug}`);

  const finalString = fileArr.join('\n');

  debug && console.log(`saving updated markdown file to `, fileWritePath);

  await Bun.write(fileWritePath, finalString);
  const newMatter = matter(finalString).data;

  // we'll need to update the image path in memory, if it exists

  return newMatter;
};


const comboInject = async ({ rawFile, absFilePath, debug }: InjectionProps) => {
  const uuid = crypto.randomUUID();
  const fileArr = rawFile.split('\n');

  const fileWritePath = absFilePath;
  //console.log(fileWritePath);
  const fileName = fileWritePath.split('/').pop();
  //console.log(fileName);
  if (!fileName) return;
  const slug = fileName.split('.')[0];

  // we need to search the file string to find out where
  // we can safely inject the uuid.

  const injectionPointId = getInjectionPoint({ fileArr, strOfInterest: '---', precisionPoint: 1, debug });

  if (typeof injectionPointId !== 'number') return;
  fileArr.splice(injectionPointId, 0, `id: ${uuid}`);

  // i think we can just feed the current filarArr, can't we?
  // assuming we have the id, we'll inject it right after
  const injectionPointSlug = getInjectionPoint({ fileArr, strOfInterest: 'id', precisionPoint: 1, debug });

  if (typeof injectionPointSlug !== 'number') return;
  fileArr.splice(injectionPointSlug, 0, `slug: ${slug}`);

  debug && console.log(`saving updated markdown file to `, fileWritePath);
  const finalString = fileArr.join('\n');

  await Bun.write(fileWritePath, finalString);
  const newMatter = matter(finalString).data;

  return newMatter;
};


const updateSlug = async ({ rawFile, absFilePath, debug }: InjectionProps) => {
  const fileArr = rawFile.split('\n');

  const fileWritePath = absFilePath;
  //console.log(fileWritePath);
  const fileName = fileWritePath.split('/').pop();
  //console.log(fileName);
  if (!fileName) return;
  const slug = fileName.split('.')[0];

  // assuming we have the id, we'll inject it right after
  const injectionPoint = getInjectionPoint({ fileArr, strOfInterest: 'slug', precisionPoint: 0, debug });

  if (typeof injectionPoint !== 'number') return;
  fileArr.splice(injectionPoint, 1, `slug: ${slug}`);

  const finalString = fileArr.join('\n');

  debug && console.log(`saving updated markdown file to `, fileWritePath);

  await Bun.write(fileWritePath, finalString);
  const newMatter = matter(finalString).data;

  // we'll need to update the image path in memory, if it exists

  return newMatter;
};

interface MatterProcessorProps {
  frontMatter: Record<string, unknown>;
  absFilePath: string;
  mdxPath: string;
  rawFile: string;
  imageKey?: string;
  publicPath?: string;
  priorityConfig?: Record<string, unknown>;
  debug?: boolean;
}


const matterProcessor = async ({
  frontMatter,
  absFilePath,
  mdxPath,
  rawFile,
  imageKey,
  publicPath,
  priorityConfig,
  debug,
}: MatterProcessorProps): Promise<Record<string, unknown> | undefined> => {
  // I've made the DRY Principle sadge. I'm sorry.
  const fileNameWithExt = mdxPath.split('/').pop();
  const fileNameOnlyRaw = fileNameWithExt!.split('.');
  const fileNameOnly = fileNameOnlyRaw[0];
  const commonConfig = {
    absFilePath,
    mdxPath,
    rawFile,
    imageKey,
    publicPath,
    debug,
  };
  debug && console.log(frontMatter);
  let priority;
  if ('type' in frontMatter && priorityConfig && (frontMatter.type as string) in priorityConfig) {
    debug && console.log('assigning', frontMatter.type, 'with priority', priorityConfig[frontMatter.type as string]);
    priority = priorityConfig[frontMatter.type as string];
    debug && console.log(priority);
  }

  if (!('id' in frontMatter) && !('slug' in frontMatter)) {
    debug && console.log('no uuid or slug found, injecting...');
    const newMatter = await comboInject({ absFilePath, rawFile, debug });

    if (!newMatter) return;

    const finalPass = await matterProcessor({
      ...commonConfig,
      frontMatter: newMatter,
    });

    const newFileRead = Bun.file(absFilePath);
    const newFileStr = await newFileRead.text();

    return { ...finalPass, priority, rawStr: newFileStr };
  } else if (!('id' in frontMatter)) {
    debug && console.log('no uuid found, injecting...');
    const newMatter = await injectUUID({ absFilePath, rawFile, debug });

    if (!newMatter) return;

    const finalPass = await matterProcessor({
      ...commonConfig,
      frontMatter: newMatter,
    });

    const newFileRead = Bun.file(absFilePath);
    const newFileStr = await newFileRead.text();

    return { ...finalPass, priority, rawStr: newFileStr };
  } else if (!('slug' in frontMatter)) {
    debug && console.log('no slug found, injecting...');
    const newMatter = await injectSlug({ absFilePath, rawFile });

    if (!newMatter) return;

    const finalPass = await matterProcessor({
      ...commonConfig,
      frontMatter: newMatter,
    });
    const newFileRead = Bun.file(absFilePath);
    const newFileStr = await newFileRead.text();

    return { ...finalPass, priority, rawStr: newFileStr };
  } else if ('slug' in frontMatter && frontMatter.slug !== fileNameOnly) {
    console.log('filename !== slug in frontmatter. fixing...');
    const newMatter = await updateSlug({ absFilePath, rawFile });

    if (!newMatter) return;

    const finalPass = await matterProcessor({
      ...commonConfig,
      frontMatter: newMatter,
    });
    const newFileRead = Bun.file(absFilePath);
    const newFileStr = await newFileRead.text();

    return { ...finalPass, priority, rawStr: newFileStr };
  } else if (
    imageKey &&
    imageKey in frontMatter &&
    'id' in frontMatter &&
    'slug' in frontMatter &&
    frontMatter.slug === fileNameOnly
  ) {
    debug && console.log(`uuid found in front matter with ${frontMatter.id as string}, not injecting`);
    debug && console.log(`slug found in front matter with ${frontMatter.slug as string}, not injecting`);
    debug && console.log(`found ${frontMatter[imageKey] as string}, processing image...`);
    const newMatter = await comboImageProcessing({ fmatter: frontMatter, mdxPath, imageKey, publicPath, debug });
    return { ...newMatter, priority, localKey: mdxPath, rawStr: rawFile };
  }

  return { ...frontMatter, priority, localKey: mdxPath, rawStr: rawFile };
};

interface BatchFetchFrontMatterProps extends ConfigProps {
  pathsArr?: string[];
  imageKey?: string;
  publicPath?: string; // i.e. 'assets/images/blog/heros'
  priorityConfig?: Record<string, number>;
}


/*
 * @example batchFetchFrontMatter([pathsArr])
 * () => [{front matter post0}, ..., {front matter postN}]
 */
const batchFetchFrontMatter = async ({
  pathsArr,
  imageKey,
  publicPath,
  debug,
  suppressErr,
  priorityConfig,
}: BatchFetchFrontMatterProps) => {
  if (!pathsArr) return;
  const cwd = process.cwd();
  try {
    const metaArr = await Promise.all(
      pathsArr.map(async (mdxPath: string) => {
        const absFilePath = path.resolve(path.join(cwd, mdxPath));
        const readIntoMem = Bun.file(absFilePath);
        const rawFile = await readIntoMem.text();
        const frontMatter = matter(rawFile).data;
        const res = await matterProcessor({
          frontMatter,
          absFilePath,
          mdxPath,
          rawFile,
          imageKey,
          publicPath,
          priorityConfig,
          debug,
        });
        return res;
      }),
    );
    metaArr.sort((a, b) => {
      return a && 'priority' in a && b && 'priority' in b ? (a.priority as number) - (b.priority as number) : 0;
    });
    return metaArr;
  } catch (err) {
    suppressErr && console.error(err);
  }
};

export interface BatchFetchMain extends ConfigProps, BatchFetchFrontMatterProps {}

// main function
export const batchFetchMain = async (fetchConfig: BatchFetchMain) => {
  const validMdxPaths = await batchFetchMDXPaths(fetchConfig);
  const frontMatterArr = await batchFetchFrontMatter({
    ...fetchConfig,
    pathsArr: validMdxPaths!,
  });
  fetchConfig.debug && console.log(frontMatterArr);
  return frontMatterArr;
};

Alright, there’s a lot going on here. Let's start with the main function, and work our way through it, shall we?

The Main Function: batchFetchMain()

// main function
export const batchFetchMain = async (fetchConfig: BatchFetchMain) => {
  const validMdxPaths = await batchFetchMDXPaths(fetchConfig);
  const frontMatterArr = await batchFetchFrontMatter({
    ...fetchConfig,
    pathsArr: validMdxPaths!,
  });
  fetchConfig.debug && console.log(frontMatterArr);
  return frontMatterArr;
};

Looking at batchFetchMain(), you’ll see it takes a configuration object of type BatchFetchMain, which extends types ConfigProps and BatchFetchFrontMatterProps. This object defines where the content is and what to exclude. It resembles something like this:

// ConfigProps + BatchFetchFrontMatterProps = BatchFetchMain
interface BatchFetchMain {
  // must be relative path from root project directory => './content'
  contentFolder: string;
  // array of relative paths INSIDE the content folder ['./assets']
  foldersToExclude?: string[];
  // array of literal file name/ext => ['README.md']
  filesToExclude?: string[];
  // debug
  debug?: boolean;
  suppressErr?: boolean;
  pathsArr?: string[];
  imageKey?: string;
  publicPath?: string; // i.e. 'assets/images/blog/heros'
  priorityConfig?: Record<string, number>;
}

From there, it generates an array of valid .mdx paths using batchFetchMDXPaths, as defined by the configuration object.

Fetching file paths: batchFetchMDXPaths()

batchFetchMDXPaths imports the following File I/O APIs:

import path from 'node:path';
import { readdir, access } from 'node:fs/promises';

And relies on the following function I found loving on stackoverflow ... somewhere:

const isNonEmptyArrayOfStrings = (value: unknown): value is string[] => {
  return Array.isArray(value) && value.length > 0 && value.every((item) => typeof item === 'string');
};

The resultant batchFetchMDXPaths() function could then be created:

/*
 * @example batchFetchMDXPaths({config})
 * () => ['./content/blog/post0.mdx', ..., './content/blog/postN.md']
 */
export const batchFetchMDXPaths = async ({
  contentFolder,
  foldersToExclude,
  filesToExclude,
  debug,
  suppressErr,
}: ConfigProps): Promise<string[] | undefined> => {
  try {
    const dir = await readdir(contentFolder, { recursive: true });

    const excludedFolders = foldersToExclude?.map((folder) => {
      const cleanedFolderPath = folder.replace('./', '');

      return cleanedFolderPath;
    });

    //const absPath = path.resolve(path.join(process.cwd(), contentFolder))

    const fileArr = dir.map((item): string | undefined => {
      debug && console.log('logging raw path:', item);

      if (excludedFolders?.some((folder) => item.startsWith(folder))) {
        debug && console.log('skipping ', item);
        return;
      }

      if (filesToExclude?.some((file) => item.endsWith(file))) {
        debug && console.log('ommitting ', item);
        return;
      }

      if (item.endsWith('.mdx') || item.endsWith('md')) {
        return `${contentFolder}/${item}`;
      }
      return undefined;
    });
    const filter1 = fileArr.filter((el) => el);
    // validate found paths
    const cwd = process.cwd();
    const validatedArr = await Promise.all(
      filter1.map(async (pathStr) => {
        try {
          await access(path.resolve(path.join(cwd, pathStr!)));
          return pathStr;
        } catch (err) {
          console.error(err);
        }
      }),
    );

    const finalArr = validatedArr.filter((el) => el);

    debug && console.log(finalArr);

    if (isNonEmptyArrayOfStrings(finalArr)) return finalArr;
  } catch (err) {
    if (err instanceof Error && 'code' in err && err.code === 'ENOENT') {
      !suppressErr &&
        console.error(
          "Ooops! Couldn't open that directory! Are you sure that folder is relative to root your project directory, i.e. './src/content/posts'? ",
          err,
        );
      return;
    }
    !suppressErr && console.error(err);
  }
};

Going through it, it recursively reads the contents of the given contentFolder using readdir and saves the result into an array of fs.Dirent objects I lovingly called dir:

...
  try {
    const dir = await readdir(contentFolder, { recursive: true });
    ...
  }
  catch (err) {
    ...
  }
...

Then it cleans up the paths of the foldersToExclude array from the config:

...
  try {
    ...
 const excludedFolders = foldersToExclude?.map((folder) => {
      const cleanedFolderPath = folder.replace('./', '');

      return cleanedFolderPath;
    });
    ...
  }
  catch (err) {
    ...
  }
...

Then it assembles the actual array of paths:

...
  try {
 ...
    const fileArr = dir.map((item): string | undefined => {
      debug && console.log('logging raw path:', item);

      if (excludedFolders?.some((folder) => item.startsWith(folder))) {
        debug && console.log('skipping ', item);
        return;
      }

      if (filesToExclude?.some((file) => item.endsWith(file))) {
        debug && console.log('ommitting ', item);
        return;
      }

      if (item.endsWith('.mdx') || item.endsWith('md')) {
        return `${contentFolder}/${item}`;
      }
      return undefined;
    });
    ...
  }
  catch (err) {
    ...
  }
...

Once it has those paths from the fileArr array, it filters any undefined paths:

...
  try {
    ...
    const filter1 = fileArr.filter((el) => el);
    ...
  }
  catch (err) {
    ...
  }
...

Then it validates the paths, by mapping through the filter1 array, checking if the path is accessible using access, and returning the path if it is, and an error if it isn't.

...
  try {
    ...
    // validate found paths
    const cwd = process.cwd();
    const validatedArr = await Promise.all(
      filter1.map(async (pathStr) => {
        try {
          await access(path.resolve(path.join(cwd, pathStr!)));
          return pathStr;
        } catch (err) {
          console.error(err);
        }
      }),
    );
    ...
  }
  catch (err) {
    ...
  }
...

Then it does one more filter pass for any undefined strings and finally returns the resultant array (finalArr) only if the array isn't empty:

...
  try {
    ...
    const finalArr = validatedArr.filter((el) => el);

    debug && console.log(finalArr);

    if (isNonEmptyArrayOfStrings(finalArr)) return finalArr;
  }
  catch (err) {
    ...
  }
...

If at any point during this whole process a catastrophic failure occurred, it would catch the error like so:

...
  try {
    ...
  }
  catch (err) {
    if (err instanceof Error && 'code' in err && err.code === 'ENOENT') {
      !suppressErr &&
        console.error(
          "Ooops! Couldn't open that directory! Are you sure that folder is relative to root your project directory, i.e. './src/content/posts'? ",
          err,
        );
      return;
    }
    !suppressErr && console.error(err);
  }
...

Once the finalArr is generated from batchFetchMDXPaths, the posts can then be read into memory, and move on to processing via batchFetchFrontMatter().

Post Processing & Content Data Generation

With all the data in memory, it became possible to actually do something with it. That's where the pre-processing step (of the posts) came into play. Just like before, we'll start with the main function (batchFetchFrontMatter()) and work our way through it.

Batch processing of front matter: batchFetchFrontMatter()

This function is responsible for performing all the processing of the post data, and returns a resultant array of assembled post objects. They get consumed in the push-mdx.ts script (which we'll see later), that does the actual database insertions.

/*
 * @example batchFetchFrontMatter([pathsArr])
 * () => [{front matter post0}, ..., {front matter postN}]
 */
const batchFetchFrontMatter = async ({
  pathsArr,
  imageKey,
  publicPath,
  debug,
  suppressErr,
  priorityConfig,
}: BatchFetchFrontMatterProps) => {
  if (!pathsArr) return;
  const cwd = process.cwd();
  try {
    const metaArr = await Promise.all(
      pathsArr.map(async (mdxPath: string) => {
        const absFilePath = path.resolve(path.join(cwd, mdxPath));
        const readIntoMem = Bun.file(absFilePath);
        const rawFile = await readIntoMem.text();
        const frontMatter = matter(rawFile).data;
        const res = await matterProcessor({
          frontMatter,
          absFilePath,
          mdxPath,
          rawFile,
          imageKey,
          publicPath,
          priorityConfig,
          debug,
        });
        return res;
      }),
    );
    metaArr.sort((a, b) => {
      return a && 'priority' in a && b && 'priority' in b ? (a.priority as number) - (b.priority as number) : 0;
    });
    return metaArr;
  } catch (err) {
    suppressErr && console.error(err);
  }
};

Alright, with this deceptively simple overview, it's time to work through it. So, the first thing to do is to map through the validated pathsArr, and get the absolute path to the current given path, using path.resolve with the cwd joined to the given mdxPath using path.join:

const batchFetchFrontMatter = async ({
  pathsArr,
  ...
}: BatchFetchFrontMatterProps) => {
  if (!pathsArr) return;
  const cwd = process.cwd();
  try {
    const metaArr = await Promise.all(
      pathsArr.map(async (mdxPath: string) => {
        const absFilePath = path.resolve(path.join(cwd, mdxPath));
        ...
      }),
    );
    ...
  } catch (err) {
 ...
  }
};

Then we can finally take advantage of Bun's unique File I/O APIs (Bun.file) API, in reading the .mdx file into memory as a UTF-8 string:

const batchFetchFrontMatter = async ({
  pathsArr,
  ...
}: BatchFetchFrontMatterProps) => {
  if (!pathsArr) return;
  const cwd = process.cwd();
  try {
      const metaArr = await Promise.all(
        pathsArr.map(async (mdxPath: string) => {
          const absFilePath = path.resolve(path.join(cwd, mdxPath));
          const readIntoMem = Bun.file(absFilePath);
          const rawFile = await readIntoMem.text();
        ...
        }),
      );
    ...
  } catch (err) {
 ...
  }
};

Then we can grab just the front matter using the matter function from gray-matter:

...
import matter from 'gray-matter';

const batchFetchFrontMatter = async ({
  pathsArr,
  ...
}: BatchFetchFrontMatterProps) => {
  if (!pathsArr) return;
  const cwd = process.cwd();
  try {
    const metaArr = await Promise.all(
      pathsArr.map(async (mdxPath: string) => {
        ...
        const rawFile = await readIntoMem.text();
        const frontMatter = matter(rawFile).data;
        ...
      }),
    );
    ...
  } catch (err) {
 ...
  }
};

Then it gets to the heart of the function, the matterProcessor, which is, "Ha, quite a doozy" (we'll break it down next), and returns the result:

/*
 * @example batchFetchFrontMatter([pathsArr])
 * () => [{front matter post0}, ..., {front matter postN}]
 */
const batchFetchFrontMatter = async ({
  pathsArr,
  ...
}: BatchFetchFrontMatterProps) => {
  if (!pathsArr) return;
  const cwd = process.cwd();
  try {
    const metaArr = await Promise.all(
      pathsArr.map(async (mdxPath: string) => {
        ...
        const res = await matterProcessor({
          frontMatter,
          absFilePath,
          mdxPath,
          rawFile,
          imageKey,
          publicPath,
          priorityConfig,
          debug,
        });
        return res;
      }),
    );
    ...
  } catch (err) {
    ...
  }
};

Zooming in: Processing with matterProcessor()

In brief, the posts get assigned a UUID and a slug as needed. Then, if they have a featured image, they get copied over to the public folder and a blurry srcset image gets generated and stored as a base64 string into the in-memory data object, along with the image dimensions. Finally, all that data generated during this step, gets returned as an array of objects.

interface MatterProcessorProps {
  frontMatter: Record<string, unknown>;
  absFilePath: string;
  mdxPath: string;
  rawFile: string;
  imageKey?: string;
  publicPath?: string;
  priorityConfig?: Record<string, unknown>;
  debug?: boolean;
}

const matterProcessor = async ({
  frontMatter,
  absFilePath,
  mdxPath,
  rawFile,
  imageKey,
  publicPath,
  priorityConfig,
  debug,
}: MatterProcessorProps): Promise<Record<string, unknown> | undefined> => {
  // I've made the DRY Principle sadge. I'm sorry.
  const fileNameWithExt = mdxPath.split('/').pop();
  const fileNameOnlyRaw = fileNameWithExt!.split('.');
  const fileNameOnly = fileNameOnlyRaw[0];
  const commonConfig = {
    absFilePath,
    mdxPath,
    rawFile,
    imageKey,
    publicPath,
    debug,
  };
  debug && console.log(frontMatter);
  let priority;
  if ('type' in frontMatter && priorityConfig && (frontMatter.type as string) in priorityConfig) {
    debug && console.log('assigning', frontMatter.type, 'with priority', priorityConfig[frontMatter.type as string]);
    priority = priorityConfig[frontMatter.type as string];
    debug && console.log(priority);
  }

  if (!('id' in frontMatter) && !('slug' in frontMatter)) {
    debug && console.log('no uuid or slug found, injecting...');
    const newMatter = await comboInject({ absFilePath, rawFile, debug });

    if (!newMatter) return;

    const finalPass = await matterProcessor({
      ...commonConfig,
      frontMatter: newMatter,
    });

    const newFileRead = Bun.file(absFilePath);
    const newFileStr = await newFileRead.text();

    return { ...finalPass, priority, rawStr: newFileStr };
  } else if (!('id' in frontMatter)) {
    debug && console.log('no uuid found, injecting...');
    const newMatter = await injectUUID({ absFilePath, rawFile, debug });

    if (!newMatter) return;

    const finalPass = await matterProcessor({
      ...commonConfig,
      frontMatter: newMatter,
    });

    const newFileRead = Bun.file(absFilePath);
    const newFileStr = await newFileRead.text();

    return { ...finalPass, priority, rawStr: newFileStr };
  } else if (!('slug' in frontMatter)) {
    debug && console.log('no slug found, injecting...');
    const newMatter = await injectSlug({ absFilePath, rawFile });

    if (!newMatter) return;

    const finalPass = await matterProcessor({
      ...commonConfig,
      frontMatter: newMatter,
    });
    const newFileRead = Bun.file(absFilePath);
    const newFileStr = await newFileRead.text();

    return { ...finalPass, priority, rawStr: newFileStr };
  } else if ('slug' in frontMatter && frontMatter.slug !== fileNameOnly) {
    console.log('filename !== slug in frontmatter. fixing...');
    const newMatter = await updateSlug({ absFilePath, rawFile });

    if (!newMatter) return;

    const finalPass = await matterProcessor({
      ...commonConfig,
      frontMatter: newMatter,
    });
    const newFileRead = Bun.file(absFilePath);
    const newFileStr = await newFileRead.text();

    return { ...finalPass, priority, rawStr: newFileStr };
  } else if (
    imageKey &&
    imageKey in frontMatter &&
    'id' in frontMatter &&
    'slug' in frontMatter &&
    frontMatter.slug === fileNameOnly
  ) {
    debug && console.log(`uuid found in front matter with ${frontMatter.id as string}, not injecting`);
    debug && console.log(`slug found in front matter with ${frontMatter.slug as string}, not injecting`);
    debug && console.log(`found ${frontMatter[imageKey] as string}, processing image...`);
    const newMatter = await comboImageProcessing({ fmatter: frontMatter, mdxPath, imageKey, publicPath, debug });
    return { ...newMatter, priority, localKey: mdxPath, rawStr: rawFile };
  }

  return { ...frontMatter, priority, localKey: mdxPath, rawStr: rawFile };
};

Let's chew through this one as well, shall we? First, we're defining a configuration type for this that looks like this:

interface MatterProcessorProps {
  frontMatter: Record<string, unknown>;
  absFilePath: string;
  mdxPath: string;
  rawFile: string;
  imageKey?: string;
  publicPath?: string;
  priorityConfig?: Record<string, unknown>;
  debug?: boolean;
}

Then we're consuming in the function props, which creates a local configuration object:

const matterProcessor = async ({
  frontMatter,
  absFilePath,
  mdxPath,
  rawFile,
  imageKey,
  publicPath,
  priorityConfig,
  debug,
}: MatterProcessorProps): Promise<Record<string, unknown> | undefined> => {
  // I've made the DRY Principle sadge. I'm sorry.
  const fileNameWithExt = mdxPath.split('/').pop();
  const fileNameOnlyRaw = fileNameWithExt!.split('.');
  const fileNameOnly = fileNameOnlyRaw[0];
  const commonConfig = {
    absFilePath,
    mdxPath,
    rawFile,
    imageKey,
    publicPath,
    debug,
  };
  debug && console.log(frontMatter);
  ...
};

Then we're assigning a priority to the current object's type (correlates to the respective schemas/tables we defined earlier), if it exists in the frontMatter and the configuration:

const matterProcessor = async ({
  frontMatter,
  priorityConfig,
  debug,
  ...
}: MatterProcessorProps): Promise<Record<string, unknown> | undefined> => {
  ...
  let priority;
  if ('type' in frontMatter && priorityConfig && (frontMatter.type as string) in priorityConfig) {
    debug && console.log('assigning', frontMatter.type, 'with priority', priorityConfig[frontMatter.type as string]);
    priority = priorityConfig[frontMatter.type as string];
    debug && console.log(priority);
  }
  ...
};

Then we go through the rest of the function. Essentially, we're walking through several different cases, and injecting/retrieving/copying data in a recursive fashion.

case_0: If there's no id or slug in the frontmatter, it injects them using comboInject() and as long as the result isn't undefined, it recursively calls the matterProcessor() function, using the commonConfig object and the freshly created front matter, aptly labled newMatter. Once that's done, it uses Bun.file to read the updated .mdx file, and assigns it to the returned rawStr key.

...
  if (!('id' in frontMatter) && !('slug' in frontMatter)) {
    debug && console.log('no uuid or slug found, injecting...');
    const newMatter = await comboInject({ absFilePath, rawFile, debug });

    if (!newMatter) return;

    const finalPass = await matterProcessor({
      ...commonConfig,
      frontMatter: newMatter,
    });

    const newFileRead = Bun.file(absFilePath);
    const newFileStr = await newFileRead.text();

    return { ...finalPass, priority, rawStr: newFileStr };
  }
...

Assigning UUIDs & Slugs

No ID or Slug => comboInject()

comboInject is the function that handles this case, it looks like this:

const comboInject = async ({ rawFile, absFilePath, debug }: InjectionProps) => {
  const uuid = crypto.randomUUID();
  const fileArr = rawFile.split('\n');

  const fileWritePath = absFilePath;
  //console.log(fileWritePath);
  const fileName = fileWritePath.split('/').pop();
  //console.log(fileName);
  if (!fileName) return;
  const slug = fileName.split('.')[0];

  // we need to search the file string to find out where
  // we can safely inject the uuid.

  const injectionPointId = getInjectionPoint({ fileArr, strOfInterest: '---', precisionPoint: 1, debug });

  if (typeof injectionPointId !== 'number') return;
  fileArr.splice(injectionPointId, 0, `id: ${uuid}`);

  // i think we can just feed the current filarArr, can't we?
  // assuming we have the id, we'll inject it right after
  const injectionPointSlug = getInjectionPoint({ fileArr, strOfInterest: 'id', precisionPoint: 1, debug });

  if (typeof injectionPointSlug !== 'number') return;
  fileArr.splice(injectionPointSlug, 0, `slug: ${slug}`);

  debug && console.log(`saving updated markdown file to `, fileWritePath);
  const finalString = fileArr.join('\n');

  await Bun.write(fileWritePath, finalString);
  const newMatter = matter(finalString).data;

  return newMatter;
};

You'll notice it generates the uuid (v4) with Bun's crypto.randomUUID(), then it generates the slug by splitting and popping the absolute file path (absFilePath). This essentially means the file name, is the ultimate source of truth for the slug, and if the filename changes, the slug changes with it.

Then it uses getInjectionPoint() to find out where to inject the uuid and slug. That function looks like this:

interface InjectionPointProps {
  fileArr: string[];
  strOfInterest: string;
  precisionPoint: number;
  debug?: boolean;
}

const getInjectionPoint = ({ fileArr, strOfInterest, precisionPoint, debug }: InjectionPointProps) => {
  const getPoint = fileArr.map((strLine, index): number | undefined => {
    // we're going to look for the first "---" of the front matter
    // then inject. We can test that we're not adding at the end by
    // checking if the key following injection is valid
    const keyGuess = fileArr[index + precisionPoint]?.split(':')[0];
    //console.log(keyGuess)
    //console.log(strLine.split(':')[0])
    const point = index + precisionPoint;
    if (strLine === strOfInterest && keyGuess) {
      debug && console.log('inject at ', index + precisionPoint, 'before ', keyGuess);

      return point;
    } else if (strLine.split(':')[0] === strOfInterest) {
      debug && console.log('found', strOfInterest, 'at', index, 'Injecting at', point, 'before', keyGuess);
      return point;
    }
    return undefined;
  });
  const injectionPoint = getPoint.filter((el) => el)[0];
  return injectionPoint;
};

Once it's injected into the raw file string array (fileArr), it can be condensed back into a string, saved into the .mdx file using Bun.file, and the fresh front matter data newMatter can be returned.

The other two cases, case_1: no slug in frontMatter and case_2: no id in frontMatter, repeat the respective comboInject processes, but for their respective needs.

No ID? =>injectUUID()

Both injectUUID() and injectSlug(), rely on the below interface InjectionProps.

interface InjectionProps {
  rawFile: string;
  absFilePath: string;
  debug?: boolean;
}

Then, like it's name, injectUUID(), takes the inputs from with the given props above, finds the place in the MDX string to inject it, and updates the file with the injected UUID.

const injectUUID = async ({ rawFile, absFilePath, debug }: InjectionProps) => {
  const uuid = crypto.randomUUID();
  const fileArr = rawFile.split('\n');

  // we need to search the file string to find out where
  // we can safely inject the uuid.

  const injectionPoint = getInjectionPoint({ fileArr, strOfInterest: '---', precisionPoint: 1, debug });

  if (typeof injectionPoint !== 'number') return;
  fileArr.splice(injectionPoint, 0, `id: ${uuid}`);

  const finalString = fileArr.join('\n');

  const fileWritePath = absFilePath;

  debug && console.log(`saving updated markdown file to `, fileWritePath);

  await Bun.write(fileWritePath, finalString);
  const newMatter = matter(finalString).data;

  // we'll need to update the image path in memory, if it exists

  return newMatter;
};

No Slug? => injectSlug()

As for handling just the slug, this does something similar, except it doesn't generate a UUID to inject, it simply takes the file pathname, and creates a slug from that.

const injectSlug = async ({ rawFile, absFilePath, debug }: InjectionProps) => {
  const fileArr = rawFile.split('\n');

  const fileWritePath = absFilePath;
  console.log(fileWritePath);
  const fileName = fileWritePath.split('/').pop();
  console.log(fileName);
  if (!fileName) return;
  const slug = fileName.split('.')[0];

  // assuming we have the id, we'll inject it right after
  const injectionPoint = getInjectionPoint({ fileArr, strOfInterest: 'id', precisionPoint: 1, debug });

  if (typeof injectionPoint !== 'number') return;
  fileArr.splice(injectionPoint, 0, `slug: ${slug}`);

  const finalString = fileArr.join('\n');

  debug && console.log(`saving updated markdown file to `, fileWritePath);

  await Bun.write(fileWritePath, finalString);
  const newMatter = matter(finalString).data;

  // we'll need to update the image path in memory, if it exists

  return newMatter;
};

Slug !== filename? => updateSlug()

and in case_03: slug in frontMatter !== to fileNameOnly the updateSlug function is used. It's extremely similar to the above function.

const updateSlug = async ({ rawFile, absFilePath, debug }: InjectionProps) => {
  const fileArr = rawFile.split('\n');

  const fileWritePath = absFilePath;
  //console.log(fileWritePath);
  const fileName = fileWritePath.split('/').pop();
  //console.log(fileName);
  if (!fileName) return;
  const slug = fileName.split('.')[0];

  // assuming we have the id, we'll inject it right after
  const injectionPoint = getInjectionPoint({ fileArr, strOfInterest: 'slug', precisionPoint: 0, debug });

  if (typeof injectionPoint !== 'number') return;
  fileArr.splice(injectionPoint, 1, `slug: ${slug}`);

  const finalString = fileArr.join('\n');

  debug && console.log(`saving updated markdown file to `, fileWritePath);

  await Bun.write(fileWritePath, finalString);
  const newMatter = matter(finalString).data;

  // we'll need to update the image path in memory, if it exists

  return newMatter;
};

In our final case, case_04: image (imageKey) declared in frontMatter, id and slug declared in frontMatter, and the slug matches fileNameOnly.

  else if (
    imageKey &&
    imageKey in frontMatter &&
    'id' in frontMatter &&
    'slug' in frontMatter &&
    frontMatter.slug === fileNameOnly
  ) {
    debug && console.log(`uuid found in front matter with ${frontMatter.id as string}, not injecting`);
    debug && console.log(`slug found in front matter with ${frontMatter.slug as string}, not injecting`);
    debug && console.log(`found ${frontMatter[imageKey] as string}, processing image...`);
    const newMatter = await comboImageProcessing({ fmatter: frontMatter, mdxPath, imageKey, publicPath, debug });
    return { ...newMatter, priority, localKey: mdxPath, rawStr: rawFile };
  }

This relies on a function called comboImageProcessing(), which brings us to our next section.

Generating Image Data

This step is somewhat intensive. In this step I'm generating the blurry srcset image using plaiceholder and grabbing the resultant image dimensions, as well as copying over the featured image into the public folder.

The latter strategy ensures that featured images can be served directly from Next.js itself. As well. it also keeps things consistent by replicating the functionality of a rehype plugin I currently use (rehype-mdx-import-media by Remco Haszing), which does the same thing, but for images contained in the content body of the mdx file, rather than the front matter.

Aside, we'll begin our look at this process by understanding the main function here, comboImageProcessing().

comboImageProcessing()

Similar to before, we have a interface for a generic configuration object.

interface ImageUtilProps {
  fmatter: Record<string, unknown>;
  mdxPath: string;
  publicPath?: string;
  imageKey: string;
  debug?: boolean;
}

Then we have the actual function comboImageProcessing(), that leverage inner functions to both generate the srcset and dimension data, and performs the copying step to the public folder.

const comboImageProcessing = async ({ fmatter, mdxPath, imageKey, publicPath, debug }: ImageUtilProps) => {
  if (imageKey && imageKey in fmatter) {
    // need to do this first (the other function seems to mutate something)
    const imgBlurPlusMetaRes = await imgMetaPlusBlurryPlaiceHolders({ fmatter, mdxPath, imageKey, debug });
    const imgBlurData = imgBlurPlusMetaRes?.base64;
    const imgHeight = imgBlurPlusMetaRes?.height;
    const imgWidth = imgBlurPlusMetaRes?.width;

    const newImgPath = await newimageEmbedPath({ fmatter, mdxPath, imageKey, publicPath, debug });
    fmatter[imageKey] = newImgPath;
    return { ...fmatter, blur: imgBlurData, height: imgHeight, width: imgWidth };
  }
};

We can see how the blurry srcset and the image dimensions are obtained below.

imgMetaPlusBlurryPlaiceHolders()

This function finds the absolute image path from the relative path given in the front matter, and then reads it into memory as an arrayBuffer, leveraging Bun's File I/O API. Then it transforms it from an arrayBuffer to simply a Buffer, which can be used as an input for getPlaiceHolder(). The latter is a function from plaiceholder, and it relies on sharp under the hood. It's also the actual function that generates the blurry srcset and provides the image dimensions.

import { getPlaiceholder } from 'plaiceholder';

const imgMetaPlusBlurryPlaiceHolders = async ({ fmatter, mdxPath, imageKey, debug }: ImageUtilProps) => {
  if (imageKey && imageKey in fmatter) {
    debug && console.log(`using ${fmatter[imageKey] as string} to generate img data + blurs`);
    const currentImagePath = fmatter[imageKey];
    //console.log(currentImagePath)
    const splitStr = mdxPath.split('/');
    const parentPath = splitStr.slice(0, splitStr.length - 1).join('/');
    const imgToCopyFilePath = path.resolve(parentPath, currentImagePath as string);

    const imgFile = Bun.file(imgToCopyFilePath);
    const arrayBuf = await imgFile.arrayBuffer();
    const buf = Buffer.from(arrayBuf);

    const {
      base64,
      metadata: { height, width },
    } = await getPlaiceholder(buf);

    return { base64, height, width };
  }
};

newImageEmbedPath()

With that out of the way, the final step is to copy the image from the content folder, to the public folder, which Next.js can use to serve the image. The below function both performs that step, as well as generates the new relative path (to the public folder), which will be saved into the in-memory post object.

const newimageEmbedPath = async ({
  fmatter,
  mdxPath,
  publicPath,
  imageKey,
  debug,
}: ImageUtilProps): Promise<string | undefined> => {
  if (imageKey && typeof imageKey === 'string' && imageKey in fmatter) {
    debug && console.log('found hero image with key', imageKey);

    //const imageType = 'type' in fmatter && (fmatter.type as string);

    const currentImagePath = fmatter[imageKey];

    if (typeof currentImagePath !== 'string') return;

    //debug && console.log(currentImagePath)

    const splitStr = mdxPath.split('/');
    // need to get parent path before mutating array with pop.
    const parentPath = splitStr.slice(0, splitStr.length - 1).join('/');
    //debug && console.log(parentPath)

    // now we can get the mdx file name to use as the folder later.
    const mdxFile = splitStr.pop();
    if (!mdxFile) return;
    //const mdxFileSlug = mdxFile.split('.')[0];
    //debug && console.log(mdxFileSlug)

    const imgToCopyFilePath = path.resolve(parentPath, currentImagePath);
    //debug && console.log(imgToCopyFilePath)

    const publicCopyPath = `/public/${publicPath}/${currentImagePath.split('/').pop()}`;

    //debug && console.log(publicCopyPath)

    const embedPublicCopyPath = `/${publicPath}/${currentImagePath.split('/').pop()}`;
    //debug && console.log(embedPublicCopyPath)

    const pathToCheck = path.join(process.cwd(), publicCopyPath);
    //debug && console.log(pathToCheck)
    const constPublicImgFile = Bun.file(pathToCheck);
    const imgFile = Bun.file(imgToCopyFilePath);

    /*
     * If the file isn't in the public folder, then copy it.
     * If an image already exists in the public folder,
     * but the declared frontmatter image is
     * different (diff in size), then replace it.
     */
    const checkImg = await constPublicImgFile.exists();

    if (!checkImg) {
      debug && console.log('image not in public folder, copying ...');
      await Bun.write(`${process.cwd()}${publicCopyPath}`, imgFile);
    } else if (constPublicImgFile.size !== imgFile.size) {
      debug && console.log('found image is different from public folder, copying ...');
      await Bun.write(`${process.cwd()}${publicCopyPath}`, imgFile);
    } else {
      debug && console.log('image is the same, not copying');
    }

    return embedPublicCopyPath;
  }
};

Zooming out

Once it has all of that, it returns the assembled processed object:

const matterProcessor = async ({
  frontMatter,
  absFilePath,
  mdxPath,
  rawFile,
  imageKey,
  publicPath,
  priorityConfig,
  debug,
}: MatterProcessorProps): Promise<Record<string, unknown> | undefined> => {
  ...
  return { ...frontMatter, priority, localKey: mdxPath, rawStr: rawFile };
};

From there, we can zoom further back out to the parent function to notice we return the assembled processed object as the res, to generate the metaArr array, which is then sorted and returned according to the priority we defined earlier:

/*
 * @example batchFetchFrontMatter([pathsArr])
 * () => [{front matter post0}, ..., {front matter postN}]
 */
const batchFetchFrontMatter = async ({
  pathsArr,
  imageKey,
  publicPath,
  debug,
  suppressErr,
  priorityConfig,
}: BatchFetchFrontMatterProps) => {
  if (!pathsArr) return;
  const cwd = process.cwd();
  try {
    const metaArr = await Promise.all(
      pathsArr.map(async (mdxPath: string) => {
        const absFilePath = path.resolve(path.join(cwd, mdxPath));
        const readIntoMem = Bun.file(absFilePath);
        const rawFile = await readIntoMem.text();
        const frontMatter = matter(rawFile).data;
        const res = await matterProcessor({
          frontMatter,
          absFilePath,
          mdxPath,
          rawFile,
          imageKey,
          publicPath,
          priorityConfig,
          debug,
        });
        return res;
      }),
    );
    metaArr.sort((a, b) => {
      return a && 'priority' in a && b && 'priority' in b ? (a.priority as number) - (b.priority as number) : 0;
    });
    return metaArr;
  } catch (err) {
    suppressErr && console.error(err);
  }
};

With all that out of the way, we can finally zoom even further back to our main function:

export const batchFetchMain = async (fetchConfig: BatchFetchMain) => {
  const validMdxPaths = await batchFetchMDXPaths(fetchConfig);
  const frontMatterArr = await batchFetchFrontMatter({
    ...fetchConfig,
    pathsArr: validMdxPaths!,
  });
  fetchConfig.debug && console.log(frontMatterArr);
  return frontMatterArr;
};

To notice we then return the frontMatterArr generated from the batchFetchFrontMatter(). With that, our walk through of the first part of the CMS (Scanning and Processing) is complete!

Inserting Content Data into the DB

Okay, so I've talked a lot about generating data, but where does it all go? Well, the remote database on Turso, of course! You can check out the below scripts to witness it in action! The first is a collection of insertion functions (bun-db-funcs.ts), the second is a helper function that imports the previous functions dynamically from the configuration object (push-mdx.ts), the final script integrates all of this into db-gen.ts, which is called by the runner.ts script which is called during prebuild (runs before next build).

bun-db-funcs.ts

The below isn't that complicated (thankfully). You'll notice that I'm primarily integrating all the schemas/tables defined all the way at the beginning of this article, and leveraging drizzle's statement builder to insert the data we generated into the database.

// bun-db-funcs.ts

/* eslint-disable no-console -- we're not in the browser, so this is fine. */
import { eq, and } from 'drizzle-orm';
//import { maindb } from '@/lib/db/bun-db';
import { maindb } from '@/lib/db/turso-db';
import { type Authors, authors } from '@/lib/db/schema/authors';
import { type Tags, tagSlugs } from '@/lib/db/schema/tagSlugs';
import { type Posts, posts, postsToTags } from '@/lib/db/schema/posts';
import { type FeaturedImages, featuredImages } from '@/lib/db/schema/featured-images';

export const insertAuthors = async (data: Authors): Promise<void> => {
  if (!data.id) {
    console.error('no author id! did you forget something?');
    return;
  }
  try {
    const authorData = data;
    console.log(authorData);

    // perform check should update
    const inserted = await maindb.query.authors.findFirst({
      where: eq(authors.id, authorData.id),
      columns: {
        id: true,
        slug: true,
        date: true,
        name: true,
        mastodon: true,
        mastodonURL: true,
        localKey: true,
        rawStr: true,
      },
    });

    const assembledData = {
      id: authorData.id,
      slug: authorData.slug,
      date: authorData.date,
      name: authorData.name,
      mastodon: authorData.mastodon,
      mastodonURL: authorData.mastodonURL,
      localKey: authorData.localKey,
      rawStr: authorData.rawStr,
    };

    if (JSON.stringify(assembledData) !== JSON.stringify(inserted)) {
      await maindb
        .insert(authors)
        .values(assembledData)
        .onConflictDoUpdate({
          target: authors.id,
          set: {
            slug: authorData.slug,
            date: authorData.date,
            name: authorData.name,
            mastodon: authorData.mastodon,
            mastodonURL: authorData.mastodonURL,
            localKey: authorData.localKey,
            rawStr: authorData.rawStr,
          },
        });
      console.log('inserted', authorData.name, 'into db');
    } else {
      console.log('author', authorData.name, 'already exists');
    }
  } catch (err) {
    console.error("Couldn't insert author:", err);
  }
};

export const insertTags = async (data: Tags): Promise<void> => {
  if (!data.id) {
    console.error('no tag id! did you forget something?');
    return;
  }
  try {
    const tagData = data;

    const inserted = await maindb.query.tagSlugs.findFirst({
      where: eq(tagSlugs.id, tagData.id),
      columns: {
        id: true,
        slug: true,
        date: true,
        title: true,
        localKey: true,
        rawStr: true,
      },
    });

    const assembledData = {
      id: tagData.id,
      slug: tagData.slug,
      date: tagData.date,
      title: tagData.title,
      localKey: tagData.localKey,
      rawStr: tagData.rawStr,
    };

    if (JSON.stringify(assembledData) !== JSON.stringify(inserted)) {
      await maindb
        .insert(tagSlugs)
        .values(assembledData)
        .onConflictDoUpdate({
          target: tagSlugs.id,
          set: {
            slug: tagData.slug,
            date: tagData.date,
            title: tagData.title,
            localKey: tagData.localKey,
            rawStr: tagData.rawStr,
          },
        });

      console.log('inserted', tagData.title, 'into db');
    } else {
      console.log('tag', tagData.title, 'already exists');
    }
  } catch (err) {
    console.error("Couldn't insert tagSlugs", err);
  }
};

export const insertFeaturedImages = async (data: FeaturedImages): Promise<void> => {
  if (!data.id) {
    console.error('no image id! did you forget something?');
    return;
  }
  try {
    const imgData = data;

    const inserted = await maindb.query.featuredImages.findFirst({
      where: eq(featuredImages.id, imgData.id),
      columns: {
        id: true,
        slug: true,
        date: true,
        fileLocation: true,
        caption: true,
        credit: true,
        creditUrlText: true,
        creditUrl: true,
        altText: true,
        localKey: true,
        blur: true,
        height: true,
        width: true,
        rawStr: true,
      },
    });

    const assembledData = {
      id: imgData.id,
      slug: imgData.slug,
      date: imgData.date,
      fileLocation: imgData.fileLocation,
      caption: imgData.caption,
      credit: imgData.credit ? imgData.credit : null,
      creditUrlText: imgData.creditUrlText ? imgData.credit : null,
      creditUrl: imgData.creditUrl ? imgData.creditUrl : null,
      altText: imgData.altText,
      localKey: imgData.localKey,
      blur: imgData.blur,
      height: imgData.height,
      width: imgData.width,
      rawStr: imgData.rawStr,
    };

    if (JSON.stringify(assembledData) !== JSON.stringify(inserted)) {
      await maindb
        .insert(featuredImages)
        .values(assembledData)
        .onConflictDoUpdate({
          target: featuredImages.id,
          set: {
            slug: imgData.slug,
            date: imgData.date,
            fileLocation: imgData.fileLocation,
            caption: imgData.caption,
            credit: imgData.credit,
            creditUrlText: imgData.creditUrlText,
            creditUrl: imgData.creditUrl,
            altText: imgData.altText,
            localKey: imgData.localKey,
            blur: imgData.blur,
            height: imgData.height,
            width: imgData.width,
            rawStr: imgData.rawStr,
          },
        });

      console.log('inserted', imgData.slug, 'into db');
    } else {
      console.log('img', imgData.slug, 'already exists');
    }
  } catch (err) {
    console.error("Couldn't insert images:", err);
  }
};

export const insertPosts = async (data: Posts): Promise<void> => {
  if (!data.id) {
    console.error('no post id! Did you forget something?');
    return;
  }
  try {
    const postData = data;

    // sleep a second
    console.log('let data load');
    const sleep = (ms: number) =>
      new Promise((r) => {
        setTimeout(r, ms);
      });
    await sleep(500);
    console.log("okay! let's continue");

    /*
  if (!(featuredImageIdRes && 'id' in featuredImageIdRes)) {
    console.error('Could not retrieve image id from slug! Did you forget something?');
    return;
  }


  if (!(authorIdRes && 'id' in authorIdRes)) {
    console.error('Could not retrieve author id from slug! Did you forget something?');
    return;
  }
*/
    const getAuthorID = async (slugStr: string) => {
      const authorIdRes = await maindb.query.authors.findFirst({
        where: eq(authors.slug, slugStr),
        columns: {
          id: true,
        },
      });
      return authorIdRes;
    };
    const getImgId = async (slugStr: string) => {
      const featuredImageIdRes = await maindb.query.featuredImages.findFirst({
        where: eq(featuredImages.slug, slugStr),
        columns: {
          id: true,
        },
      });
      return featuredImageIdRes;
    };

    const inserted = await maindb.query.posts.findFirst({
      where: eq(posts.id, postData.id),
      columns: {
        id: true,
        authorId: true,
        slug: true,
        date: true,
        headline: true,
        subheadline: true,
        featuredImageId: true,
        altCaption: true,
        localKey: true,
        rawStr: true,
      },
    });

    const authorIdfuncRes = await getAuthorID(postData.author);
    const imgIdfuncRes = await getImgId(postData.featuredImage);

    const assembledData = {
      id: postData.id,
      authorId: authorIdfuncRes!.id,
      slug: postData.slug,
      date: postData.date,
      headline: postData.headline,
      subheadline: postData.subheadline,
      featuredImageId: imgIdfuncRes!.id,
      altCaption: postData.altCaption ? postData.altCaption : null,
      localKey: postData.localKey,
      rawStr: postData.rawStr,
    };

    if (JSON.stringify(assembledData) !== JSON.stringify(inserted)) {
      await maindb
        .insert(posts)
        .values(assembledData)
        .onConflictDoUpdate({
          target: posts.id,
          set: {
            authorId: authorIdfuncRes?.id,
            slug: postData.slug,
            date: postData.date,
            headline: postData.headline,
            subheadline: postData.subheadline,
            featuredImageId: imgIdfuncRes?.id,
            altCaption: postData.altCaption,
            localKey: postData.localKey,
            rawStr: postData.rawStr,
          },
        });
      console.log('inserted', postData.slug, 'into db');
    } else {
      console.log('post', postData.slug, 'already exists');
    }

    await Promise.all(
      postData.tagSlugs.map(async (tagSlug) => {
        const res = await maindb.query.tagSlugs.findFirst({
          where: eq(tagSlugs.slug, tagSlug),
          columns: {
            id: true,
          },
        });
        if (!(res && 'id' in res)) return;

        const insertedPostToTags = await maindb.query.postsToTags.findFirst({
          where: and(eq(postsToTags.tagId, res.id), eq(postsToTags.postId, postData.id)),
          columns: {
            tagId: true,
            postId: true,
          },
        });

        const assembledDataPostToTags = {
          tagId: res.id,
          postId: postData.id,
        };

        if (JSON.stringify(assembledDataPostToTags) !== JSON.stringify(insertedPostToTags)) {
          await maindb
            .insert(postsToTags)
            .values(assembledDataPostToTags)
            .onConflictDoUpdate({
              target: [postsToTags.postId, postsToTags.tagId],
              set: { postId: postData.id, tagId: res.id },
            });
          console.log('associated', tagSlug, 'with', postData.slug, 'in db');
        } else {
          console.log(
            tagSlug,
            'with id:',
            res.id,
            'is already associated with \npost:',
            postData.slug,
            'with id',
            postData.id,
          );
        }
      }),
    );
    console.log('all done!');
  } catch (err) {
    console.error("Couldn't insert posts", err);
  }
};

One thing you'll probably notice, is that I'm assembling a temporary object within each insertion function (respective of the given table, e.g., authors), from what already exists in the database. By running this comparison, I'm ensuring that only new content is inserted into the database, or content that needs to be updated. As such, this drastically reduces the number of rows that need to be written.

Another thing you might notice is that I've commented out the maindb that's imported from @/lib/db/bun-db. That database, is the local one that relied on Bun's native SQLite driver. But, for the reasons I went over earlier, I wound up using Turso instead.

Finally, I wanted to point out the sleep() functions. Those were put in place, because I was having issues performing the does exist tests, as part of my attempt to reduce the numbers of row additions. My best guess, is that latency between the database, and my little program, was introducing false negatives.

For example, when I asked that database if there was content for the other tables that posts needed, it would say no, despite having just inserted that data. This would stop any posts from being inserted into posts. The sleep(), was a band-aid solution that gave the database enough time to reflect the newly inserted data, and successfully allowed for the rest of the script to execute, so posts could make their way to the posts table.

With that, you can then see how these functions might be integrated in the below push-mdx.ts script.

push-mdx.ts

The below is more or less a helper function, that integrates closely with a configuration object (which defines the functions to be imported, such as from the script above). This was essentially my attempt at creating a generic function, which others could use if they created their own functions to be imported and used.

// push-mdx.ts

#! /usr/bin/env bun
/* eslint-disable no-console -- bun is bun */
import { type BatchFetchMain, batchFetchMain } from './fetch-mdx';

interface DbFunctionsProps {
  dbFunctionModules: {
    insert: Record<string, unknown>;
  };
}

export const batchPushMain = async (fetchConfig: BatchFetchMain & DbFunctionsProps): Promise<void> => {
  // get processed front matter array
  const matterRes = await batchFetchMain(fetchConfig);

  if (!matterRes) {
    fetchConfig.debug && console.log('Ooops, no data found!');
    return;
  }

  // arr is sorted by priority so this should work:
  await Promise.all(
    matterRes.map(async (processedMDX): Promise<void> => {
      if (processedMDX && 'type' in processedMDX && (processedMDX.type as string)) {
        const funcType = processedMDX.type as string;
        if (funcType in fetchConfig.dbFunctionModules.insert) {
          const insModRaw = fetchConfig.dbFunctionModules.insert[funcType] as Record<string, string>;
          const insModKeys = Object.keys(insModRaw);
          const insModStr = insModKeys[0]!;
          const insModPath = insModRaw[insModStr]!;
          if (insModStr && insModPath) {
            /* eslint-disable-next-line @typescript-eslint/no-unsafe-assignment -- importing types would be a lot to ask for */
            const dbFuncs = await import(insModPath);
            /* eslint-disable-next-line @typescript-eslint/no-unsafe-call, @typescript-eslint/no-unsafe-member-access -- importing types would be a lot to ask for */
            await dbFuncs[insModStr](processedMDX);
          }
        }
      }
    }),
  );
};

What I'm most proud of here, is that I figured out how to dynamically import modules from the given string from the configuration object. It's admittedly not pretty, as I import the functions based on their given declared type, but the defined functions were successfully imported by leveraging Object.keys(), and React's dynamic import syntax. However, you'll probably notice the eslint directives to disable typechecking on those functions. That's simply because I didn't know of a easy way to type them.

While the latter problem could've been solved by simply creating/importing those function types, this script was written at a point where I thought I could release this thing to the public as ready to use software. Thus, demanding people provide types with their functions seemed like a big ask, in my opinion. Though, I suppose expecting users to create their own database functions in the first place was probably a much greater ask. By that point, why even use something like this? But, I digress.

As well, you'll notice I'm importing the main fetch function (batchFetchMain()), which again both fetches the posts to read, and performs various processing steps. If you'll recall, many of those steps are admittedly quite rigid, or inflexible, which would create further problems if I ever did wind up generalizing this code based content management system.

Inherent problems aside, you can see how my generic script was integrated with the configuration object in the next section.

db-gen.ts

This is where the two scripts above are integrated into one main function that does everything. batchPushMain() is called with the configuration object, shamelessly called laniConfig.

// db-gen.ts

import { batchPushMain } from './mdx-db/push-mdx';

const laniConfig = {
  contentFolder: './content',
  foldersToExclude: ['./assets'],
  filesToExclude: ['README.md', 'LICENSE'],
  imageKey: 'fileLocation',
  publicPath: 'assets/images/featured',
  priorityConfig: {
    authors: 1,
    tagSlugs: 2,
    featuredImages: 3,
    posts: 4,
  },
  dbFunctionModules: {
    insert: {
      authors: { insertAuthors: '@/lib/bun-db-funcs' },
      tagSlugs: { insertTags: '@/lib/bun-db-funcs' },
      featuredImages: { insertFeaturedImages: '@/lib/bun-db-funcs' },
      posts: { insertPosts: '@/lib/bun-db-funcs' },
    },
  },
  debug: false,
};

const dbGen = async (): Promise<void> => {
  try {
    console.log('trying to create db');
    await batchPushMain(laniConfig);
    console.log('db-gen successful');
  } catch (err) {
    console.error(err);
  }
};

export default dbGen;

The point of this script, was to create a simple function that could be integrated into the runner.ts prebuild script—which I'll cover a bit later—along with setting up a integrated configuration file for my CMS.

Fetching Data from the DB

What good would a CMS be if you couldn't access any of the data you stored on to it? Pretty awful I'm betting. So, that's why it was important to write the script below to get that data.

node-db-funcs.ts

One thing to point out, is there's a switch over from bun to node functions, and that's because the production build of a Next.js application runs on node. While it is possible to run the Next.js development server from bun, the production run time environment is strictly node, for good reason: Next.js often relies on the latest Node.js APIs that Bun simply hasn't had time to implement, yet.

// node-db-funcs.ts

/* eslint-disable-next-line import/named -- monorepo issues */
import { cache } from 'react';
import 'server-only';
import { eq, desc, like, or, and } from 'drizzle-orm';
import { maindb } from '@/lib/db/turso-db';
import { posts, postsToTags } from '@/lib/db/schema/posts';
import { tagSlugs } from '@/lib/db/schema/tagSlugs';

//import { maindb } from '@/lib/db/drizzle';

export interface PostsToTagsItem {
  tag: {
    slug: string;
    title: string;
    id: string;
  };
}

export interface QueryPostMetaItem {
  id: string;
  date: Date;
  slug: string;
  headline: string;
  subheadline: string;
  author: {
    name: string;
  };
  featuredImage?: {
    fileLocation: string;
    altText: string;
    blur: string;
    height: number;
    width: number;
  };
  tagSlugs: {
    slug: string;
    title: string;
    id: string;
  }[];
  localKey: string;
}

export interface QueryPost extends QueryPostMetaItem {
  rawStr: string;
}

export const queryPostMetas = cache(async () => {
  const postRes = await maindb.query.posts.findMany({
    orderBy: [desc(posts.date)],
    columns: {
      authorId: false,
      featuredImageId: false,
      rawStr: false,
    },
    with: {
      author: {
        columns: {
          name: true,
        },
      },
      postToTags: {
        columns: {
          tagId: false,
          postId: false,
        },
        with: {
          tag: {
            columns: {
              slug: true,
              title: true,
              id: true,
            },
          },
        },
      },
      featuredImage: {
        columns: {
          fileLocation: true,
          altText: true,
          blur: true,
          height: true,
          width: true,
        },
      },
    },
  });
  const finalRes = postRes.map((post) => {
    const tagSlugsOne = post.postToTags.map((tagSlugsObj) => {
      const slug = tagSlugsObj.tag.slug;
      const title = tagSlugsObj.tag.title;
      const id = tagSlugsObj.tag.id;

      return { slug, title, id };
    });
    delete (post as unknown as { postToTags: Record<string, unknown> | undefined }).postToTags;
    return { ...post, tagSlugs: tagSlugsOne };
  });
  //console.dir(finalRes, { depth: null });
  return finalRes;
});

export const queryPostByIdForJustRawStr = cache(async (idStr: string) => {
  const resOne = await maindb.query.posts.findFirst({
    where: eq(posts.id, idStr),
    columns: {
      rawStr: true,
    },
  });
  return resOne;
});

export const queryPostByIdandSlugOrJustIdForJustRawStr = cache(
  async ({ idStr, slugStr }: { idStr: string; slugStr: string }) => {
    const postRes = await maindb.query.posts.findFirst({
      where: or(and(like(posts.id, `${idStr}%`), eq(posts.slug, slugStr)), like(posts.id, `${idStr}%`)),
      columns: {
        rawStr: true,
      },
    });
    return postRes;
  },
);

export interface PostQ extends QueryPostMetaItem {
  featuredImage: {
    fileLocation: string;
    altText: string;
    credit?: string;
    creditUrl?: string;
    creditUrlText?: string;
    caption?: string;
    height: number;
    width: number;
    blur: string;
  };
  altCaption?: string;
  rawStr: string;
}

export const queryPostMetaByIdandSlugOrJustId = cache(
  async ({ idStr, slugStr }: { idStr: string; slugStr: string }) => {
    const postRes = await maindb.query.posts.findFirst({
      where: or(
        and(like(posts.id, `${idStr}%`), eq(posts.slug, slugStr)),
        or(like(posts.id, `${idStr}%`), eq(posts.id, idStr)),
      ),
      columns: {
        authorId: false,
        featuredImageId: false,
      },
      with: {
        author: {
          columns: {
            name: true,
          },
        },
        postToTags: {
          columns: {
            tagId: false,
            postId: false,
          },
          with: {
            tag: {
              columns: {
                slug: true,
                title: true,
                id: true,
              },
            },
          },
        },
        featuredImage: {
          columns: {
            fileLocation: true,
            altText: true,
            credit: true,
            creditUrl: true,
            creditUrlText: true,
            caption: true,
            blur: true,
            height: true,
            width: true,
          },
        },
      },
    });

    if (!postRes) return;

    const tagSlugsMap = postRes.postToTags.map((tagSlugsObj) => {
      const slug = tagSlugsObj.tag.slug;
      const title = tagSlugsObj.tag.title;
      const id = tagSlugsObj.tag.id;
      return { slug, title, id };
    });

    delete (postRes as unknown as { postToTags: Record<string, unknown> | undefined }).postToTags;

    return { ...postRes, tagSlugs: tagSlugsMap };
  },
);

/*
    where: or(
      or(
        and(like(posts.id, `${idStr}%`), eq(posts.slug, slugStr)),
        or(like(posts.id, `${idStr}%`), eq(posts.id, idStr)),
      ),
      and(eq(posts.id, idStr), eq(posts.slug, slugStr)),
    ),
*/

export interface TagQ {
  id: string;
  slug: string;
  date: string;
  title: string;
  localKey: string;
  rawStr: string;
}

export const getAllTags = cache(async () => {
  const res = await maindb.select().from(tagSlugs);
  return res;
});

export const getTag = cache(async ({ idStr, slugStr }: { idStr: string; slugStr: string }) => {
  const idRes2 = await maindb.query.tagSlugs.findFirst({
    where: or(and(like(tagSlugs.id, `${idStr}%`), eq(tagSlugs.slug, slugStr)), like(tagSlugs.id, `${idStr}%`)),
  });
  return idRes2;
});

// type of metaItem arr
export const getPostsWithTagID = cache(async (tagIdStr: string) => {
  const idRes = await maindb.query.tagSlugs.findFirst({
    where: like(tagSlugs.id, `${tagIdStr}%`),
    columns: {
      id: true,
    },
  });

  if (!idRes?.id) return;

  const queryRes = await maindb.query.postsToTags.findMany({
    where: eq(postsToTags.tagId, idRes.id),
    columns: {
      postId: false,
      tagId: false,
    },
    with: {
      post: {
        columns: {
          id: true,
          slug: true,
        },
      },
    },
  });

  const postRes = await Promise.all(
    queryRes.map(async (post) => {
      const innerRes = await queryPostMetaByIdandSlugOrJustId({ idStr: post.post.id, slugStr: post.post.slug });
      return innerRes;
    }),
  );

  return postRes;
});

In this script, I'm primarily just using the Drizzle query builder to create functions to select data from the database, to be returned as assembled chunks (with types), for use around the website. The latter can be seen in the next section.

As well, I'm also making use of the cache function, that Next.js integrates from React. In theory, this meant that each call to the database was memoized, so I'd reduce the number of row reads to the database. In practice, all these posts were statically generated, so there was ultimately no point to doing things like this, other than for science, I suppose—which is quite important!

Putting it Together

The last step was to ensure the dbGen() function ran before next build, which was accomplished simply by creating a script runner, and calling it in our package.json.

runner.ts

// runner.ts

/* eslint-disable no-console -- bun is bun */
import 'dotenv/config';
import dbGen from './db-gen';
import atomGen from './atom-gen';

const runScripts = async (): Promise<void> => {
  try {
    console.log('trying to create database');
    await dbGen();
    console.log('dbGen success');
    await atomGen();
    console.log('atomGen success');
  } catch (err) {
    console.error(err);
  }
};

try {
  await runScripts();
} catch (err) {
  console.error(err);
}

In the above, you'll notice I'm calling the dbGen() function from earlier, as well as a function I didn't cover called atomGen(). The latter is a script which generates the Atom Web Feed, and it relies on a library called jstoxml. I'll likely cover how that all works in a later blog post.

Aside, in the below you can see how the runner script is called with package.json. One thing to notice is that I had to directly declare my NODE_ENV as production, otherwise I'd run into issues with Drizzle being unable to work properly. I'm unsure if it was a bug in bun or if this just how things are supposed to work.

// package.json

{
  ...,
  "name": "laniakita-web",
  "scripts": {
 ...,
    "prebuild": "NODE_ENV=production bun ./src/scripts/runner.ts",
  },
}

With everything now in place, every time I ran bun run build, content would be scanned, processed, and fresh content would be inserted into the database, while old content would remain the same. I was then able to retrieve it using the functions I wrote in the previous section.

Rendering Content

So, now that we've seen how content get's transformed into data, stored, then retrieved from a remote database on Turso, the final piece of the puzzle was to load it into the frontend. The below is the old source code for these posts, which demonstrates exactly that.

import type { Metadata, ResolvingMetadata } from 'next';
import { useMemo } from 'react';
import { getMDXComponent } from 'mdx-bundler/client';
import { redirect } from 'next/navigation';
import { PostHeader } from '@/app/blog/post-header';
import {
  type PostQ,
  type QueryPostMetaItem,
  queryPostByIdandSlugOrJustIdForJustRawStr,
  queryPostMetaByIdandSlugOrJustId,
  queryPostMetas,
} from '@/lib/node-db-funcs';
import { resMdxV3 } from '@/utils/mdxbundler-main';
import BlogImageBlurServer from '@/app/blog/blog-image-blur-wrapper';
import descriptionHelper from '@/utils/description-helper';

export async function generateStaticParams() {
  const postMetas = (await queryPostMetas()) as unknown as QueryPostMetaItem[];
  return postMetas.map((meta) => ({
    id: meta.id.split('-').shift(),
    slug: meta.slug,
  }));
}

export async function generateMetadata(
  { params }: { params: { id: string; slug: string } },
  parent: ResolvingMetadata,
): Promise<Metadata> {
  const metaRes = (await queryPostMetaByIdandSlugOrJustId({
    idStr: params.id,
    slugStr: params.slug,
  })) as unknown as QueryPostMetaItem;
  const rawRes = await queryPostByIdandSlugOrJustIdForJustRawStr({ idStr: params.id, slugStr: params.slug });

  const findDescr = rawRes?.rawStr ? descriptionHelper(rawRes.rawStr) : ["Lani's Blog"];
  const descr = findDescr ? findDescr.filter((el) => el) : '';
  const previousImages = (await parent).openGraph?.images ?? [];
  const featuredImg = metaRes.featuredImage?.fileLocation;
  return {
    title: metaRes.headline,
    authors: [{ name: metaRes.author.name }],
    description: descr[0],
    openGraph: {
      title: metaRes.headline,
      description: descr[0],
      images: [featuredImg ? featuredImg : '', ...previousImages],
    },
    twitter: {
      card: 'summary',
      title: metaRes.headline,
      description: descr[0],
      images: [featuredImg ? featuredImg : '', ...previousImages],
    },
  };
}

export default async function BlogPostPage({ params }: { params: { id: string; slug: string } }) {
  const postQ = (await queryPostMetaByIdandSlugOrJustId({
    idStr: params.id,
    slugStr: params.slug,
  })) as unknown as PostQ;
  if (postQ.slug !== params.slug) {
    redirect(`/blog/posts/${params.id}/${postQ.slug}`);
  }
  const cwdFolderStrPre = postQ.localKey.split('/');
  const cwdFolderStr = cwdFolderStrPre.slice(0, cwdFolderStrPre.length - 1).join('/');
  const rawMDX = postQ.rawStr;
  if (!rawMDX) return;
  if (!cwdFolderStr) return;
  const resMdx = await resMdxV3(rawMDX, cwdFolderStr, params.id, 'blog');
  return (
    <main className='motion-safe:simple-color-trans -mb-0.5 min-h-full max-w-full bg-ctp-base dark:bg-ctp-midnight'>
      {(resMdx as unknown) !== undefined && (
        <article id='content' className='flex size-full flex-col items-center justify-center'>
          <PostHeader dataObject={postQ} />

          <div className='flex min-h-full items-center justify-center px-10 py-6'>
            <div className='prose-protocol-omega'>
              <MdxJsx code={resMdx.code} />
            </div>
          </div>
        </article>
      )}
    </main>
  );
}

export function MdxJsx({ code }: { code: string }) {
  const Component = useMemo(() => getMDXComponent(code), [code]);
  return <Component components={{ img: BlogImageBlurServer }} />;
}

The above is sorta complicated on first look, but the important thing to notice is that I'm taking the retrieved raw mdx string, and feeding it into my mdx-bundler function. That function simultaneously configures mdx-bundler with all the plugins I use for these posts.

Et voila! A CMS I built all by myself, with data flowing from a database, directly into this Next.js application, to generate a blog on the internet.

Discussion / Moving to Contentlayer2

Alright, so after pouring all this time and energy into creating the above system, what went so wrong, that I felt the need to switch to timlrx/contentlayer2? Well, nothing major per se, it was just that by the time I'd written everything out, it felt overly complex, and a little half baked. I suppose once all the pieces were in place, and the high of finishing something I'd worked months on wore off, I quickly began to realize all it's flaws.

For example, if you look over the bun-db-funcs.ts, you'll notice I put little sleep() functions in between insertion steps. That was put in place, because bun is either too fast, or again, there's too much latency for the operations to complete properly, or I've introduced a bug somehow with sloppy code (which honestly is more likely). In hindsight I could've modified those functions to be transactions, but by that point the novelty of the thing I'd just created had well worn off, and the slog of future maintenance, was beginning to creep in.

Likewise, I'd originally planned on creating a full blown CRUD application, which would've tied a admin dashboard directly into the Next.js application. That way I could edit and delete posts, simply through an admin panel, but this never fully materialized, beyond a little prototype I'd built in Svelte, before moving on to creating this site in Next.js. Why? I suppose I decided to prioritize getting content to a blog first, before creating any fancy GUIs that would've tied in an authentication framework, like Lucia, into the frontend application.

As such, once I'd come across contentlayer2, I was smitten. It was doing everything I was trying to do, but in a clean little package. There were no databases to configure. There was no worry about little pieces here and there throwing a wrench into the whole system. On the surface, it seemed like it just worked.

So, I dove in. I swallowed my pride, and I scrapped months of effort building my own CMS for something I can honestly say is better. However, because I did, maintenance on this site is now significantly less burdensome, and I'm far less worried about dependency updates breaking it.

Though, I will say that the time and energy I spent creating my own solution wasn't all for naught. In fact, I still use the image processing functions I'd created earlier, inside of my contentlayer configuration. You can see how I've done this with the below code blocks. The first is my contentlayer.config.ts and the second is my recycled image processing functions (image-process.ts).

// contentlayer.config.ts

import { defineDocumentType, makeSource } from 'contentlayer2/source-files';
import remarkGfm from 'remark-gfm';
import rehypeSlug from 'rehype-slug';
import rehypeShiki from '@shikijs/rehype';
import { rendererRich, transformerTwoslash } from '@shikijs/twoslash';
import rehypeMdxImportMedia from 'rehype-mdx-import-media';
import { imageProcessor, FeaturedImageR1 } from './src/lib/image-process';
import jsxToHtml from './src/lib/mdx-html';

const CONTENT_DIR = 'content';

export const Project = defineDocumentType(() => ({
  name: 'Project',
  filePathPattern: 'projects/**/*.yaml',
  contentType: 'data',
  fields: {
    id: { type: 'string', required: true },
    date: { type: 'date', required: true },
    updated: { type: 'date', required: false },
    title: { type: 'string', required: true },
    tech: {
      type: 'list',
      of: { type: 'string' },
    },
    imageSrc: { type: 'string', required: false },
    altText: { type: 'string', required: false },
    caption: { type: 'string', required: false },
    description: { type: 'string', required: true },
    blogPost: { type: 'string', required: false },
    link: { type: 'string', required: false },
  },
  computedFields: {
    url: {
      type: 'string',
      resolve: (project) => `/${project._raw.flattenedPath}`,
    },
    featured_image: {
      type: 'json',
      resolve: async (project): Promise<FeaturedImageR1> => {
        if (!project.imageSrc) return new FeaturedImageR1(false, '', '', 0, 0, '', '', '', null);
        const data = await imageProcessor({
          contentDir: CONTENT_DIR,
          prefix: `${CONTENT_DIR}/${project._raw.flattenedPath}`,
          imgPath: project.imageSrc,
          debug: false,
        });

        const res = new FeaturedImageR1(
          true,
          data.src,
          data.base64,
          data.height,
          data.width,
          data.resized,
          project.altText ?? '',
          project.caption ?? '',
          data._debug ?? null,
        );

        return res;
      },
    },
  },
}));

export const Author = defineDocumentType(() => ({
  name: 'Author',
  filePathPattern: 'authors/**/*.mdx',
  contentType: 'mdx',
  fields: {
    name: { type: 'string', required: true },
    mastodon: { type: 'string', required: false },
    github: { type: 'string', required: false },
  },
  computedFields: {
    url: {
      type: 'string',
      resolve: (author) => `/${author._raw.flattenedPath}`,
    },
  },
}));

export const Page = defineDocumentType(() => ({
  name: 'Page',
  filePathPattern: 'pages/**/*.mdx',
  contentType: 'mdx',
  fields: {
    title: { type: 'string', required: true },
    description: { type: 'string', required: false },
    date: { type: 'date', required: false },
  },
  computedFields: {
    url: {
      type: 'string',
      resolve: (page) =>
        `/${page._raw.flattenedPath.split('/').slice(1, page._raw.flattenedPath.split('/').length).join('/')}`,
    },
  },
}));

const Tag = defineDocumentType(() => ({
  name: 'Tag',
  filePathPattern: 'tagSlugs/**/*.mdx',
  contentType: 'mdx',
  fields: {
    id: { type: 'string', required: false },
    title: { type: 'string', required: true },
    slug: { type: 'string', required: false },
    date: { type: 'date', required: false },
  },
  computedFields: {
    url: {
      type: 'string',
      resolve: (tag) => `/${tag._raw.flattenedPath}`,
    },
  },
}));

const Category = defineDocumentType(() => ({
  name: 'Category',
  filePathPattern: 'catSlugs/**/*.mdx',
  contentType: 'mdx',
  fields: {
    id: { type: 'string', required: false },
    title: { type: 'string', required: true },
    slug: { type: 'string', required: false },
    date: { type: 'date', required: false },
  },
  computedFields: {
    url: {
      type: 'string',
      resolve: (category) => `/${category._raw.flattenedPath}`,
    },
  },
}));

export const Post = defineDocumentType(() => ({
  name: 'Post',
  filePathPattern: `posts/**/*.mdx`,
  contentType: 'mdx',
  fields: {
    id: { type: 'string', required: true },
    headline: { type: 'string', required: true },
    subheadline: { type: 'string', required: false },
    slug: { type: 'string', required: false },
    date: { type: 'date', required: true },
    updated: { type: 'date', required: false },
    author: { type: 'string', required: false },
    catSlugs: {
      type: 'list',
      of: Category,
    },
    tagSlugs: {
      type: 'list',
      of: Tag,
    },
    keywords: {
      type: 'list',
      of: { type: 'string' },
    },
    imageSrc: { type: 'string', required: false },
    altText: { type: 'string', required: false },
    caption: { type: 'string', required: false },
  },
  computedFields: {
    html: {
      type: 'string',
      resolve: (post) => {
        const renderedMdx = jsxToHtml(post.body.code);
        return renderedMdx;
      },
    },
    url: {
      type: 'string',
      resolve: (post) => `/blog/${post.id.split('-').shift()}/${post._raw.flattenedPath.split('/').pop()}`,
    },
    featured_image: {
      type: 'json',
      resolve: async (post): Promise<FeaturedImageR1> => {
        if (!post.imageSrc) return new FeaturedImageR1(false, '', '', 0, 0, '', '', '', null);
        const data = await imageProcessor({
          contentDir: CONTENT_DIR,
          prefix: `${CONTENT_DIR}/${post._raw.flattenedPath}`,
          imgPath: post.imageSrc,
          debug: false,
        });

        const res = new FeaturedImageR1(
          true,
          data.src,
          data.base64,
          data.height,
          data.width,
          data.resized,
          post.altText ?? '',
          post.caption ?? '',
          data._debug ?? null,
        );
        return res;
      },
    },
  },
}));

export default makeSource({
  contentDirPath: CONTENT_DIR,
  documentTypes: [Post, Category, Tag, Page, Project, Author],
  mdx: {
    remarkPlugins: [remarkGfm],
    rehypePlugins: [
      [
        rehypeShiki,
        {
          themes: {
            light: 'catppuccin-latte',
            dark: 'catppuccin-mocha',
          },

          transformers: [
            transformerTwoslash({
              explicitTrigger: true,
              renderer: rendererRich(),
            }),
          ],
        },
      ],
      rehypeMdxImportMedia,
      rehypeSlug,
    ],
    resolveCwd: 'relative',
    esbuildOptions(options) {
      options.outdir = `${process.cwd()}/public/assets/images/blog`;
      options.loader = {
        ...options.loader,
        '.png': 'file',
        '.jpg': 'file',
      };
      options.publicPath = `/assets/images/blog`;
      options.write = true;
      return options;
    },
  },
});

// image-process.ts

import { existsSync, copyFileSync, mkdirSync } from 'node:fs';
import { readFile, lstat } from 'node:fs/promises';
import path from 'node:path';
import { getPlaiceholder } from 'plaiceholder';
import sharp from 'sharp';

export interface DebugR1 {
  destination: string;
  status: {
    exists: boolean;
    existsInPublic: boolean;
  };
  didCopy: string;
  reason: string;
}

export class FeaturedImageR1 {
  hasImage: boolean;
  src: string;
  base64: string;
  height: number;
  width: number;
  resized: string;
  altText: string;
  caption: string;
  _debug: DebugR1 | null;

  constructor(
    hasImage: boolean,
    src: string,
    base64: string,
    height: number,
    width: number,
    resized: string,
    altText: string,
    caption: string,
    _debug: DebugR1 | null,
  ) {
    this.hasImage = hasImage;
    this.src = src;
    this.base64 = base64;
    this.height = height;
    this.width = width;
    this.resized = resized;
    this.altText = altText;
    this.caption = caption;
    this._debug = _debug;
  }
}

/**
 * A typeguarded version of `instanceof Error` for NodeJS.
 * author: Joseph JDBar Barron
 * {@link https://dev.to/jdbar}
 */

export function instanceOfNodeError<T extends new (...args: unknown[]) => Error>(
  value: Error,
  errorType: T,
): value is InstanceType<T> & NodeJS.ErrnoException {
  return value instanceof errorType;
}

// check in relative assets folder & public folder
const checkImgExists = (imgPath: string) => {
  let exists = false;

  // read image
  // 1. check if path exists
  if (existsSync(imgPath)) {
    exists = true;
  }

  return { exists };
};

const checkDuplicate = async (imgPathOne: string, imgPathTwo: string) => {
  let isDupe = 0;
  // WARNING: assumes both images exist
  const imgOne = await lstat(imgPathOne);
  const imgTwo = await lstat(imgPathTwo);
  if (imgOne.size === imgTwo.size) {
    isDupe = 1;
  }

  return isDupe;
};

interface Debug {
  destination: string;
  status: {
    exists: boolean;
    existsInPublic: boolean;
  };
  didCopy: string;
  reason: string;
}

interface ImageMoverRes {
  url: string;
  local: string;
  _meta: null | Debug;
}

const imageMover = async ({
  contentDir,
  prefix,
  imgPath,
  debug,
}: {
  contentDir: string;
  prefix: string;
  imgPath: string;
  debug?: boolean;
}): Promise<ImageMoverRes> => {
  if (debug) {
    console.debug('image:', imgPath);
    console.debug('url:', prefix);
  }
  /*
   * we essentially need to work backwards from
   * where the post is located (postParent), and the
   * location of the relative asset (imgPath).
   */
  const postParentRaw = prefix.split('/');
  postParentRaw.shift(); // removes CONTENT_DIR
  postParentRaw.pop(); // removes post filename
  const postParent = postParentRaw.join('/');
  const urlPath = path.join(postParent, imgPath); // location of the image in the public folder.
  const rawPath = path.resolve(path.join(contentDir, postParent, imgPath));
  const publicPath = path.resolve(path.join('./public', postParent, imgPath));

  if (debug) {
    console.debug('rawPath:', rawPath);
    console.debug('publicPath:', publicPath);
  }

  const statusOne = checkImgExists(rawPath);
  const statusTwo = checkImgExists(publicPath);
  const status = {
    exists: statusOne.exists,
    existsInPublic: statusTwo.exists,
  };

  let copied = 0;
  let isDupe = 0;

  // using sync functions, it's faster for our purposes.
  const copier = (from: string, to: string) => {
    try {
      // important to make the directory, it won't copy otherwise (ENOENT).
      mkdirSync(path.dirname(to), { recursive: true });
      copyFileSync(from, to);
      copied = 1;
      console.debug('copied successfully to:', to);
    } catch (err) {
      console.error(err);
    }
  };

  // if valid & in public folder, check if new image is the same
  if (status.exists && status.existsInPublic) {
    isDupe = await checkDuplicate(rawPath, publicPath);
    if (isDupe === 0) {
      copier(rawPath, publicPath);
    }
  }
  // if valid, but not in public folder, copy it
  if (status.exists && !status.existsInPublic) {
    copier(rawPath, publicPath);
  }

  const result = {
    url: urlPath,
    local: rawPath,
    _meta: debug
      ? {
          destination: publicPath,
          status,
          didCopy: copied === 1 ? 'copy' : 'no copy',
          reason: isDupe === 0 ? 'original' : 'duplicate',
        }
      : null,
  };

  return result;
};

interface BlurRes {
  base64: string;
  height: number;
  width: number;
}

const imageBlurBase64 = async (imgPath: string): Promise<BlurRes> => {
  const imgFile = await readFile(imgPath);
  const {
    base64,
    metadata: { height, width },
  } = await getPlaiceholder(imgFile);
  return { base64, height, width };
};

const imageResize = async (imgPath: string) => {
  const imgFile = await readFile(imgPath);
  const { data } = await sharp(imgFile)
    .resize(1600, 900, { kernel: 'lanczos3' })
    .toFormat('jpeg', { mozjpeg: true })
    .toBuffer({ resolveWithObject: true });

  const baseSixtyFour = `data:image/jpeg;base64,${data.toString('base64')}`;
  return baseSixtyFour;
};

export interface FeaturedImageRes extends BlurRes {
  src: string;
  resized: string;
  _debug: null | Debug;
}

export const imageProcessor = async ({
  contentDir,
  prefix,
  imgPath,
  debug,
}: {
  contentDir: string;
  prefix: string;
  imgPath: string;
  debug?: boolean;
}): Promise<FeaturedImageRes> => {
  try {
    const imgCopyRes = await imageMover({ contentDir, prefix, imgPath, debug });
    const blurRes = await imageBlurBase64(imgCopyRes.local);
    const resize64 = await imageResize(imgCopyRes.local);
    return { src: `/${imgCopyRes.url}`, ...blurRes, resized: resize64, _debug: debug ? imgCopyRes._meta : null };
  } catch (err) {
    console.error(err);
  }
  return { src: '', base64: '', height: 0, width: 0, resized: '', _debug: null };
};

Granted, they're a little modified—there's an additional image down scaler too—but the base of it is there. Likewise, if I hadn't sunk all this energy into my own solution in the first place, I'd never have been able to truly appreciate all the work that goes into CMSes and projects like contentlayer, nor would I be able to know their limitations so closely either.

Which, speaking of, contentlayer isn't perfect, it's far from it. Because it relies on json to store content, it creates significantly inflated application bundles. It's sort of a common complaint⁷⁵. Now, is it a solution that works for something small like a blog, yes, and damn well too. But, for something more data intensive, like a news aggregator? Forget it. You'd be much better served creating either the monstrosity I did earlier, or using an off the shelf CMS, because at least those can scale.

With that said, contentlayer for a site like mine, just seems like the right tool for the job. At this scale, it's as close to perfect as I'm going to get, and I'm incredibly appreciative of all the work behind it too.

This article was a long time coming, it's also quite long. In some respects, this might feel like an article created from a few separate articles I stitched together, and perhaps it is. However, I felt it was important to talk about the primary technologies I used in developing/deploying this site, because they're all pretty tightly integrated.

To elaborate, Next.js is the application framework I went with, and I deployed it with Ion SST. The latter is important because it uses OpenNext as a serverless adapter, which takes the output from next build, and splits it up into bundles that can deploy on specific AWS Lambda functions. This effectively modifies how a production Next.js application is compiled runs to ensure compatibility with AWS Lambda / Lambda@edge. In other words, modifying how it runs, modifies the behavior of the production site itself (a bit). So, I really wanted to talk about it.

As for the WebGL stuff, admittedly, that section could've gone into a separate article. However, learning how to create with those specific technologies was such a significant motivator for myself in rebuilding this site, that it was worth including.

Abstract

This article catalogs some of my thoughts on:

• Rebuilding this site
• Developing with Next.js
• Deploying with Ion/SST
• Working with WebGL wrappers/libraries like Three.js & React Three Fiber (a Three.js renderer for React).

As well, I break down any last thoughts in the discussion section.

Introduction

Next.js is to React, like a glass slipper is to Cinderella; they fit perfectly together (usually). The same could be said of Vercel’s Cloud Platform and Next.js too, and yet, I used a tool called Ion to deploy this site on AWS instead. While I think Vercel is a fantastic choice for most use cases, especially for Next.js applications, something about Ion just caught my eye.

If you haven’t heard of it, Ion is the upcoming v3 of SST, and it’s quite the Swiss Army knife of deployment tools. This is especially so with the switch from AWS CDK/CFN to Pulumi & Terraform providers to handle your Infrastructure as Code (IaC). In my use case, I used Ion’s Nextjs component which does a few very handy things in addition to some Next.js specific IaC which we’ll get to later.

Now, the main reason for rebuilding my personal site was simply because it no longer reflected my current skill set. Unlike the site you’re reading this on, my old site—nothing fancy, just a Gatsby frontend on Amplify—from a few years back was much less refined. So, I decided to wipe the slate clean, but I had some goals in mind. I wanted this website to be modern, yet ambitious, with an accompanying codebase that was flexible (read: organized) enough for me to feed my lust for cutting-edge technologies for years to come.

While the latter condition was solved by simply setting this site up in a monorepo (I used Turborepo for that), everything else meant becoming quite adventurous with this project’s stack. I specifically chose technologies that would push me well beyond my comfort zone to both learn and to integrate. That ultimately meant leaving the ease and familiarity of Svelte and SvelteKit to build this site with tools I’d once found overcomplicated and confusing—React and Next.js.

Likewise, it also meant learning a subfield of programming that I was once wholly intimidated by, graphics. In effect, I used Three.js, React Three Fiber, and even some GLSL to add some fun WebGL-flare. This can be seen in the shader on the landing page, and also this mini-game that almost became such a page. I also happened to stumble upon Zustand, a lightweight state management system for React, which integrated with the latter very well.

Furthermore, I even abandoned the various headless CMSes I’ve grown comfortable with over the years, in favor of something I can call my own. In effect, I wrote a prebuild script to assemble my data layer which takes advantage of Bun’s speedy File I/O API to find and process MDX files with gray-matter—and any contained featured images with plaiceholder—saving the resultant array of objects into a SQLite-ish (libSQL) database on Turso via Drizzle ORM statements. I then fetch that data inside a server component to render it out in various ways, such as with mdx-bundler.

I also wrote a prebuild route handler for the blog’s web feed. That one It fetches the blog posts from contentlayer, and integrates them into a purpose built object which is turned into an XML string with jstoxml. While I’d like to talk about this data layer portion further, it is, admittedly, a bit extensive. So, you’ll have to wait to read about in a later blog post.

Beyond that, I’m overall really happy with how everything turned out. While Next.js and Ion/SST aren’t without their flaws—something not too unexpected of bleeding-edge tools—the pairing still came together quite nicely (in tandem with everything else) to create something I’m quite proud of. Even if there’s still bugs (read: mistakes) I’d like to fix in the codebase, the end result works like it should for the most part. Page load times are quite fast. Its Lighthouse scores are decent enough (admittedly, I’m sure there’s still more I could do to improve accessibility). There’s still plenty of room for me to grow and build on top of it. As such, I don’t regret developing this site with Next.js or shipping it with Ion/SST at all!

What I Like About Next.js

Next.js is a batteries-included full stack framework that does much more than just offer bleeding-edge features from React’s canary branch. Its expansive range of APIs and configuration options is simply unparalleled as far as full stack frameworks for React go.

What I like most about Next.js, is its offering of convenient metadata generation tools, and the deep level of control you have over its various rendering, routing, and caching strategies. Not to mention the customizations you can make to its new compiler+bundler, SWC, too.

With features like the above, is it really a huge wonder as to why Next.js might be the most popular full stack framework in existence?

App Router (& Server Components)

The App router and React Server Components are probably old news at this point for most React devs, but it was new to me. So, on the off chance you’re unfamiliar with it, lemme give you a quick recap.

The App router is more than just a refreshed Pages router that includes the ability to co-locate components with a corresponding page/directory. It’s a major rewrite that enables functionality with React Server Components, an upcoming API in React 19 that has already caused a monumental shift in the React landscape. The very nature of server components even solves the majority of security risks involved with React based applications (see: How to Think About Security in Next.js). In short, server components are kind of a big deal.

Admittedly, I’m still pretty enthusiastic about server components, even after all this time fussing with them in building this site. In fact, their addition was a bit of a factor in luring me back over to the React side of things in the first place.

While they do create some frustrating scenarios, such as working out how to delicately interleave client and server components, there’s just something neat about them. Perhaps there’s something to creating data heavy UI components server side, then shipping that to the client browser as pure markup, that just makes my brain happy.

Now, I’d like to shift our focus back to the App router, away from all this gushing over server components, however, that’s not exactly possible. One of the more prominent features enabled by this tight integration of server components with the App router is something called streaming.

Streaming is a pretty big benefit of server components, and I believe it can improve the user experience by quite a bit. The largest benefit to streaming, in my opinion, is summarized nicely in the Next.js docs:

Streaming is particularly beneficial when you want to prevent long data requests from blocking the page from rendering as it can reduce the Time To First Byte (TTFB) and First Contentful Paint (FCP). It also helps improve Time to Interactive (TTI), especially on slower devices.

I should mention however, that streaming isn’t a configuration constant, or a special component like the App router’s instant fallback loading UI feature. Streaming is instead, a server-side rendering pattern that can be implemented on SSR pages only.

When implemented properly, streaming results in progressively sending components over to the client browser from fastest to slowest fetch request payload. This is done by simply wrapping each component found on such an SSR page with a <Suspense /> boundary.

In effect, components that don’t fetch any data will load first, while components that do, will stream-in to the client browser in the order it takes to complete their respective fetch requests. As such, streaming creates a better browsing experience as the page won’t feel stuck as it loads in. Well, given you can ship enough of the UI for it to feel that way, but I digress.

While these blog posts aren’t SSR yet—ergo no streaming—I’ve prepared these post pages to do exactly that in using mdx-bundler. It relies on esbuild to render mdx strings live in production. As such, it really won’t be difficult to implement the streaming rendering pattern when I do. Especially since I already invested in the App router.

Overall, I'm quite satisfied in my decision to use the App router as my page routing model. Beyond the App routers integration of server components, it’s forward thinking-features like streaming that improve the user experience, gives me hope for the next-generation of web applications (is that why it’s called Next.js?), and that makes me very happy.

Metadata & Metadata Accessories

With Meta having developed and maintained React, and React having not-so-subtly championed Next.js as the premier framework for React (at least for the bleeding edge branch), then, you might deduce that Next.js would be great at generating and working with metadata. But is that true? Yes, yes it is.

To demonstrate, there’s quite a lot of metadata APIs included in Next.js by default. To give you an idea of how much, I put together the following list. It's an overview of the metadata APIs I’m currently using for this site.

• From the generateMetadata API
  • To generate meta tagSlugs on simple static pages, I’m using the metadata Object.
  • To generate meta tagSlugs dynamically on dynamic routes I’m using the namesake generateMetadata function.
• From the Metadata files API
  • To auto-generate the various icon meta tagSlugs, I’m using favicon, icon, apple-icon file conventions.
  • To generate the robots.txt, I’m using a special robots.ts file.
  • To generate the sitemap.xml, I’m using the generateSitemaps function from its namesake API.

The generateMetadata API is quite handy, especially it’s namesake function. Even if the static {metadata} object is somewhat tedious to fill out, it’s highly preferred over the tedious nature of adding (and updating) each little meta tag in JSX. Likewise, the namesake generateMetadata function that gets used on dynamic routes, saves both my fingers and myself an even larger amount of time given the dynamic nature of the metadata.

The generateMetadata function can even be paired with the generateStaticParams function API on SSG dynamic routes as well. Funnily enough, it even works similarly to the generateStaticParams function too. The main difference of course between it and the latter, is that the returned data to map over is plugged into a {metadata} object, instead of the {params} object.

Aside, the Metadata files API has quite a lot of time saving utilities too. Beyond the special files that allow you to generate things like the robots.txt or sitemap.xml with functions similar to those in the generateMetadata API, the file-based metadata generation for icons is a feature that I really love.

Like the generateMetadata API, it too has saved me quite a lot of the mind-numbingly tedious grunt work in hard-coding the different icons into their respective meta tagSlugs. The difference (and best part) of course, is that it does this automatically without even an object! All you need to do is simply give your icon a corresponding filename to its file format, in accordance with the icon file conventions under the Metadata files API, et voilà! Essential icons automatically defined in your site’s header!

While these APIs go more in depth, metadata utilities are one of the crazy-cool sleeper features that Next.js provides. Features that just seem to be absent in other frameworks. I suppose this might be because metadata & metadata tools don’t seem all that important in the Proof-of-Concept stage. However, given the ubiquity of metadata and its uses, metatagSlugs can add A LOT of polish to the production build of a web application, in my opinion.

Regardless of how you feel about SEO, metadata shows up everywhere. The little tagSlugs are used in way more places than just search engine algorithms. In the web browser alone, favicon tagSlugs show a site’s icon, title tagSlugs name the current tab, and minor tagSlugs like the author and a date show up in a browser’s reader view. Likewise, metadata and thus meta tagSlugs, are pretty much the sole determining factor in how a website will preview itself on social media.

For those reasons, I’m quite pleased with the suite of metadata tools Next.js provides OOTB. Especially because tools like these set Next.js worlds apart from other frameworks. While it’s not so complicated to just write your own in-house versions of these tools, having these batteries make Next.js an attractive option for myself and I imagine other lazy time efficient devs as well.

Server Actions (Form Actions)

Admittedly, I’ve not had a chance to play with the Server Actions API too much aside from experiments with handling a user’s theme preference via cookies.

Aside, I should state my surprise to learn that native server side form handling didn’t really exist in Next.js until relatively recently with React 19. I say that because SvelteKit’s had this feature for a long while now (see: form actions). Granted, Next.js has had the option of creating a custom API endpoint for forms submitted client side for a long while too (see: Next.js 14 announcement post that talks about this), which I suppose gives you the same functionality. What’s really changed is that React, and by extension Next.js, offers a native API to handle this, which simplifies the process; a much welcome addition.

Regardless, server actions now being stable in Next.js 14 are a much-needed feature in my opinion, and I’m really grateful to have a Next.js equivalent to SvelteKit’s form actions. Likewise, I imagine that server actions likely make implementing an Authentication framework like Lucia a little less complicated, which is quite a nice side effect to boot.

Have Rendering Your Way

The level of control you have over how you want to render routes/pages is really impressive. If you felt like it, you could statically generate (SSG) one route, and dynamically render (SSR) pages for another route. You could even do SSR on the edge for that route (or another one) if you felt like it. To do that, you just need to export a constant in your page.tsx or layout.tsx:

export const runtime = ‘edge’ // ‘nodejs’ (default) | ‘edge’

Now that I think about it, Next.js might be the only framework I’ve worked with that offers such a deep level of control over page rendering. I think the reason for this is that other frameworks (SvelteKit, Remix, Astro, etc.) have made SSR a first class (& sometimes only) citizen. Granted adapters exist for SvelteKit (adapter-static), but it’s sorta an all or nothing decision, isn’t it?

Aside, I find this quite a nifty feature of Next.js that is also pretty much unparalleled in other frameworks. I really appreciate the fact that I can render posts like this statically, and if I added it, I could render a user dashboard dynamically on request on the edge runtime, served at the edge. I could do all of that without it being an all or nothing decision. That’s just awesome!

As such, Next.js is in a league of its own in the rendering department. The level of control it offers is unmatched. If you’ve got a variety of content with different rendering needs, Next.js might just be the perfect framework for you.

What I Found Confusing Interesting in Next.js

I won’t lie, the App router’s caching mechanisms are a bit complicated. You can learn more about how it works via this GitHub discussion thread in the main Next.js repo.

However, without reviewing it under a microscope, I don’t feel it's possible for me to offer a nuanced opinion on it. I suppose I can point out that on SSG pages the React cache API has been quite a wonderful feature. I use it to memoize calls to my DB, so that’s nice.

With that said, instead of griping about my ignorance, I’ll keep this section to something that’s a little odd for a mainstream framework like Next.js, undocumented features. Such mysterious quirks aren’t a bad thing per se, I imagine most software’s got a few, but Next.js likely has more than most.

Undoubtedly, these little mysteries are a result of Next’s bleeding-edge nature, since features are often added faster than the docs can be written. However, this does lead to some interesting scenarios. The most intriguing involving the configuration of the Next.js middleware.

Middleware & minimalMode

In brief, the Next.js middleware performs its namesake function, as a control layer that sits between client browser and the server. Typically, you’d extend the middleware by integrating it with the NextResponse API to write logic that determines whether your site should produce a modified response based on an incoming request—e.g., when to allow a request for CORS.

Beyond that, The middleware set to an undocumented configuration called minimalMode is the secret sauce which enables Next.js to deploy correctly onto Vercel’s serverless platform. The OpenNext FAQs discuss this secret middleware configuration further. They explain a bit about how Vercel builds a Next.js application, how the middleware gets separated out in the minimalMode configuration to be deployed at the edge, and how that process relies on Vercel’s own proprietary infra.

While most developers working with Next.js (and by extension Vercel) have no need to spare a thought to this undocumented configuration, I found it an intriguing feature of Next.js nonetheless. Admittedly, that might have something to do with how I decided to deploy this Next.js application, but I digress.

Debug Flags

I’ll admit, secret debug flags aren’t as exciting as speculating on minimalMode and the mysteries surrounding Vercel’s proprietary infrastructure. However, this hidden flag may still come in handy during a late-night debugging session. So, let’s talk about it.

A while ago I’d found this post by Martin Capodici on their blog that describes how they had uncovered an undocumented debug flag used to print out helpful diagnostics for the router cache in Next.js. All you need to do is set is NEXT_PRIVATE_DEBUG_CACHE=1, and the caching diagnostics are yours.

Like I said, it’s not the coolest secret flag in the world, but I did find something cool when I was reading up on it. Interestingly, I stumbled upon a rad little package called @neshca/cache-handler that just so happens to describe the behavior of NEXT_PRIVATE_DEBUG_CACHE in a little more detail via their troubleshooting guide.

Granted, I don’t have a need for such a utility, but if you’re hosting a Next.js application on a distributed system like a k8s cluster, this might be the package for you. @neshca/cache-handler solves the cache validation issues borne from multiple instances of the same Next.js application, by letting you replace the default Next.js cache with a shared cache via special cache handlers. Cache Handlers for redis seem to be provided OOTB.

Fun caching tools aside, extra debugging information is always handy. So, it’s nice that Next.js has these flags built-in, even if they’re frustratingly lacking in documentation.

What I Like about SST/Ion

The Ion flavor of SST is rad. I am enamored with its endless utility, Ion may as well be a Swiss Army Knife in my DevOps toolbox. In integrating the Pulumi engine into Ion, there’s a wide range of Pulumi (& even Terraform) providers available OOTB in the form of components, letting you configure an application’s Infrastructure as Code (IaC) with ease.

Likewise, Ion provides a CLI in the form of sst, which does much more than just deploy your infra. Depending on your sst.config.ts, the sst CLI provides a wrapper around your application’s dev and build scripts to inject resources defined and linked in your IaC, right into your application. This occurs in both your local dev environment and of course on your production infra upon sst deploy; it’s quite wonderful.

IaC is Rad

I’m a huge nerd, I get that, but something about declaring infra from a source controlled config file, and spinning it up from the command line just makes my brain happy. It probably scratches the same itch that Nix/NixOS does.

If I had to reason why I’m so delighted by IaC, I imagine the reproducibility aspect is likely the largest contributing factor. There’s something inherently comforting about the fact that I can just copy my IaC files over to a new project (with fresh API keys). Equally, it’s nice I could even roll back my infra in the same project (based on the config from a previous commit), instantly creating the same backend infra as before.

While of course infra requirements differ between projects, it’s really nice to have a base configuration that I can use as a boilerplate in my other projects. If only because it saves me the step of rewriting many of the same infra declarations.

Likewise, the fact I can save even more time by skipping (imperative) setup hell—endlessly clicking through menus, double-checking everything looks good, then getting reset because I accidentally skipped a required field—is a literal godsend for me. So, the time savings factor (a direct benefit of IaC) gets a huge plus from me too.

Pulumi (& it’s Terraform Bridge)

The Ion flavor of SST uses Pulumi providers (& by extension Terraform providers) as components, that allow you to define your applications IaC. What’s nice is that you don’t even need a Pulumi account for this, as the SST team integrated Pulumi’s engine right into Ion’s CLI.

Linked Resources

It’s important to reiterate that Ion isn’t simply a different flavor of Terraform, as SST themselves pointed out in that same blog post. While Ion gives you the provider components, it goes quite a many steps further. One such step is in allowing you to integrate your defined infra/resources directly into your application through a concept called linking.

All you need to do is link your defined resources back to your application in its sst.config.ts file. Once that’s done, you can access them anywhere in your application! No other IaC tool to my knowledge does that, making Ion quite unique in that regard.

Open Next

The Nextjs component relies on OpenNext in compiling a Next.js application. It’s an open source serverless adapter that makes deploying a Next.js application via serverless functions outside Vercel even possible with Ion/SST. OpenNext even attempts to achieve feature-parity with a Next.js app deployed on Vercel’s serverless platform, via an architecture based on a combination of AWS services and performing slight modifications to how the Next.js middleware is bundled and run.

In deploying to AWS, I found OpenNext to be a lovely adapter for my Next.js application. Everything just worked (mostly, we'll get into that). The best part was that I didn’t need to make any tough decisions regarding runtimes either.

The latter is because Node.js is thankfully available in a Lambda function, unlike in a Cloudflare worker. While Workers offer partial support for Node.js APIs, due to their nature of being Edge functions, it’s unlikely Next.js will ever work 100% the same as it would on Vercel or even on AWS.

Runtimes aside, the only thing I got snagged on with OpenNext was the fact that sharp (needed for plaiceholder) isn’t included in the final build.

Despite the fact I included it as a dependency, plaiceholder just couldn’t find sharp.

As a workaround, I used a Lambda Layer with sharp installed to it.

Then I just matched the sharp version on the layer, with the one in my package.json, using the SHARP_VERSION env. I injected that right before running open-next build.

Minor gripe aside, I’m extremely grateful to the people who’ve created OpenNext, and all the wonderful people who continue to maintain it. While the little adapter might not give you the complete Vercel experience 100% of the time (how could it? Vercel uses their own proprietary infra) it’s pretty damn close, and that’s incredible. Overall, I’m quite satisfied using OpenNext, and I won’t hesitate to use it in deploying future Next.js projects.

SST Console

In honesty, I feel deploying things the hard way has meant forgoing some of the luxuries offered by managed serverless-deployment services like Vercel. That’s why it’s really wonderful the SST team found a way to offer the most important luxury of all, the console.

Granted, the SST console is a little slow (though that’s probably more an AWS issue), it was incredibly helpful in debugging errors in production. Without it, I’d be digging through the logs of the various Lambda Functions, trying to get a glimpse of what was causing my 500 internal server errors. So, I’m very, very, thankful to the SST team for not only creating their own console, but offering it with a very generous free tier to boot.

Things I'd Appreciate in the Upcoming Stable Release of Ion/SST

Ion is an utterly amazing tool in my full stack toolbox. It’s also in beta, so There’s also expectedly some sharp edges. As such, the following sections run through some of the things I felt could use a bit of polish. As well, there are notes about some things that might help you/your team if you’re looking to work with this super rad, but undoubtedly bleeding-edge tool, too.

Note: Ion is still in beta (as of 7/21/24), and isn’t stable yet. By the time it is, it’ll just be called SST v3. So, if you’re reading this from the future, it’s entirely likely that everything I’m about to be salty about has been fixed. Reader discretion is advised.

Docs

The Ion docs are still a work in progress (as is Ion itself), so I really can’t fault the SST team too much here. However, it would be nice if some of the examples/guides from the old V2 docs could be migrated/converted over to the Ion docs site. For example, the configuring AWS section from the old docs could probably be dropped into the Ion docs without too much editing.

Beyond simple migrations, There’s also a few features/behaviors that seem to be missing documentation. Notably, the guide on using Cloudflare as a custom domain provider fails to mention that in addition to the CLOUDFLARE_API_TOKEN, it’s really important to set the CLOUDFLARE_DEFAULT_ACCOUNT_ID variable too. Without the latter var set, you’ll likely find sst deploy breaks, resulting in a vague set of error messages from the Go compiler.

Transforms (Lambda Layer)

/*
 * The bun shebang probably isn't necessary (no Bun specific APIs).
 * But, it ensures I get Bun's versions of the Node filesystem APIs.
 */
#! /usr/bin/env bun

import { cp, mkdir } from 'node:fs/promises';
import { join } from 'node:path';

const cwd = process.cwd();
const monoRepoCorrection = '../../';
const node_modules = join(cwd, monoRepoCorrection, './node_modules');
const openNextServerDefault = join(cwd, '.open-next/server-functions/default');

const sharpInstall = `sharp`;

const sharpDeps = ['color', 'detect-libc', 'semver'];
const colorDeps = ['color-convert', 'color-string'];
const colorStringDeps = ['color-name', 'simple-swizzle'];
// color-convert depends on color-name

const pkgsToCopy = [sharpInstall, ...sharpDeps, ...colorDeps, ...colorStringDeps];

const t0 = performance.now();

export default async function copySharp() {
  try {
    console.info('copying', sharpInstall, 'to:', openNextServerDefault);
    const pkgPaths = pkgsToCopy.map((pkg) => {
      const from = `${node_modules}/${pkg}`;
      const to = `${openNextServerDefault}/node_modules/${pkg}`;
      return {
        source: from,
        dest: to,
      };
    });
    for await (const pkg of pkgPaths) {
      console.info('creating dirs from', pkg.source, 'to:', pkg.dest);
      await mkdir(pkg.dest, { recursive: true });
    }
    for await (const copied of pkgPaths) {
      console.info('copying', copied.source, 'to:', copied.dest);
      await cp(copied.source, copied.dest, { recursive: true });
    }

    console.info('finished in ', performance.now() - t0, `ms`);
  } catch (err) {
    console.error(err);
  }
}

await copySharp();

{
  "extends": ["//"],
  "tasks": {
    "build:open-next": {
      "env": ["OPEN_NEXT_VERSION", "NEXT_PUBLIC_DEPLOYED_URL"],
      "outputs": [
        ".open-next/**",
        "!.open-next/cache/**",
        "public/dist/**",
        "public/sw.js",
        ".contentlayer",
        ".contentlayermini"
      ],
      "inputs": ["$TURBO_DEFAULT$", ".env", ".env.local", ".env.development", ".env.production"]
    },
    "copy-sharp": {
      "inputs": ["$TURBO_DEFAULT$", ".env", ".env.local", ".env.development", ".env.production", ".open-next/**"],
      "outputs": [".open-next/server-functions/default/**"],
      "dependsOn": ["build:open-next"]
    }
  }
}

For whatever reasons, I just couldn’t replicate the functionality of the old SST v2 transforms (a couple of months ago), to declaratively define and create a Lambda Layer from my sst.config.ts / project repo. What I could do is link an existing (imperatively created) Lambda layer, but I couldn’t figure out how to create it from the config itself.

The reason I even looked into this in the first place, was because OpenNext doesn’t include the sharp module in the production build. So, the only way to have those blurry placeholders you see generate dynamically, was by creating a Lambda Layer running Node.js with the sharp module loaded into it (there’s more to this story, but that will be another article).

However, in trying to replicate this guide from the old docs, (whilst converting things as best I could to account for Pulumi’s aws-classic provider, inline with their docs), I found things would deploy, but I never received any sort of error message, nor did it create the Lambda Layer. This was very confusing to me.

 transform: {
    server: (args) => {
      args.nodejs = {
        esbuild: {
          external: [“sharp”],
        },
      };
      args.layers = [
       new aws.lambda.LayerVersion(‘MySharp’, {
         layerName: “lambdaSharp”,
         code: new $util.asset.FileArchive(‘./layers/sharp’),
       })
      ];
    },
  },

> IIRC, In addition to defining the layer like this, I’m fairly certain I tried a dot notation access of the output ARN string, but that didn’t work either.

I later came across this probably relevant issue, (which thankfully appears to be fixed now), but I eventually gave up and just defined it manually in the end, setting the transform.server.layers to the generated ARN.

 transform: {
    server: (args) => {
      args.nodejs = {
        esbuild: {
          external: [“sharp”],
        },
      };
      args.layers = [‘arn:aws:lambda:us-west-1:555555555:layer:WebSharp:1’];
    },
  },

AWS SSO Integration

This I think is still an open issue, that will probably get fixed eventually, but it’s something I should point out anyway (at least if you’re running into issues). Because, theoretically, you should be able to configure the SSO profile directly in the sst.config.ts, like so.

However, as it stands, I have to preface my sst commands with an environmental variable set to my SSO profile of choice with AWS_PROFILE=$MY_SSO_PROFILE, to get things working/deploying properly. If I don’t, I run into the errors pointed out in that issue thread I linked earlier. While it’s not a horrible workaround, it is a little annoying. Granted I could likely export this from my .zshrc config, or even a .env file, but having it just work in the sst.config.ts feels like it would be ideal.

Nix Compatibility

I’ll admit, I’m a fairly niche user, but Nix compatibility would be hella cool. While there’s actually an open PR in the nixpkgs repo to add sst to nixpkgs, due to the nature of the sst CLI, compatibility with Nix is a little clunky. However, there’s an open issue in the Ion repo discussing a possible solution that could be implemented on SST’s end to make it just work with Nix, so perhaps sst will be fully compatible with Nix one day.

In any event, your best bet when working with SST applications on NixOS for now, is likely going to be to run the sst CLI from a distrobox container. However, you could probably get the binaries working via steam-run if I’m honest, but I haven’t tried it yet.

Working with Three.js, & React Three Fiber

Three.js is a WebGL wrapper written in JavaScript, and React Three Fiber is a React renderer for three.js. Also, it is absolutely the coolest thing I’ve ever managed to learn in all my years spent developing for the web.

Initially, I was going to try to create most of this site in it. This would mean the majority of elements you’d interact with, would be contained entirely in the three.js canvas! Sorta like how Google renders Google Docs. The only thing that stopped me, was when I realized what this would mean in reality: long load times, no server components, no SSR.

Now, for Google Docs or anything that falls firmly in the software category, those supposed drawbacks, are just the baseline expectation. No one expects something so heavy to load instantly. However, for something much closer to a traditional website, like a blog? Oh, those are drawbacks.

As such, I made the tough decision to be much more selective about where and how I use the Three.js canvas. The landing page I felt was an important place to demonstrate my knowledge of both it & GLSL shaders, so that’s what I put there. However, because it is the landing page, I did my best to create as simple a scene as possible; shaving bandwidth down as much as I could. Because originally I was going to do something much heavier, but shifted gears once I’d realized how heavy.

However, because this site renders its content much more traditionally, I was able to make heavy use of server components, experimenting with their weirdness to my hearts content.

Aside, it’s important to state it’s possible to mix vanilla Three.js with React Three Fiber. Of course, doing so defeats the purpose of the latter a tad bit, but occasionally I’ve found it useful to use a primitive object every now and then.

Integrating with Next.js

When I first began this project, I took some inspiration from the pmndrs/react-three-next boilerplate, even basing my next.config.mjs off it. It’s actually how I discovered you can chain plugins with an accumulator function.

The most interesting thing about the boilerplate is it uses pmndrs/tunnel-rat to create an alternative <View /> component from pmndrs/drei. In testing, I couldn’t really figure out a benefit for doing this. My hunch is that the tunnel-rat method predates some changes to the <View /> component which might’ve complicated things in Next.js.

Regardless, the <View /> component from either method works with the Next.js App router just fine. The main benefit of cutting up the canvas like this, is that you don’t need to wait for it to load in again between pages, granted you’ve wrapped those pages with a component that provides a canvas. The only slowdowns you’d see, would result from loading models. Ideally, those models should be lazy loaded/dynamically imported with next/dynamic.

As well, if you’re working with WebGL in a Next.js application, you’re going to want to put use client at the top of the pages/components that make use of it. WebGL relies heavily on a client’s hardware, especially WebGPU. While you can SSR pages that import client components featuring these elements, Next.js/React won’t compile if you try to use these APIs directly in a server component for obvious reasons.

Paper Cuts with Safari

Originally, I was going to make heavy use of the <View /> component. The only thing that stopped me from doing so, was when I realized how ungraceful it looked in Safari. You see, if the <View /> doesn’t take up the entire page, the <View /> component starts to jitter as you scroll up or down. Here's an open issue demonstrating this behavior.

According to this comment, this behavior occurs because Safari doesn’t sync scroll events with the window.requestAnimationFrame() API. This would help to explain why only Safari seems to experience this issue.

One solution to this is to use a virtual-scroller like Lenis. It's just that, I've got some reservations about that solution.

While, I think virtual-scrollers like Lenis look and feel great on desktop browsers, especially with a mouse scroll wheel, touch based Safari interactions with Lenis are another story. While I can't comment on how something like Lenis feels on a touch-based Android device running Chrome (Chromium) or Firefox (Gecko), I can talk about how I'm not enthusiastic about it on Safari (WebKit) for iOS.

Due to the nature of Safari, there are some limitations to using Lenis as a solution. While yes, it does fix the jitters, the fact that fps is suddenly capped to 60, and then an abysmal 30 fps if power saving mode is engaged, doesn't thrill me.

Sure, a virtual-scroller is better than jitters, I won't argue that fact. However, on mobile Safari at least, virtual-scrollers result in a scrolling experience that I honestly find irritating. When I tested Lenis, I found I couldn't flick up or down a page readily without it stopping in its tracks, nor could I scroll in either direction at a consistent speed. You can even replicate this behavior on the Lenis website itself.

My eventual solution, was to just do away with the <View /> component, and embed the canvas directly. I did this after investigating Sketchfab's website, where I realized they solved the jitter problem by not using <View /> components at all, as they simply embed the canvas into whatever model-preview you're currently hovering over.

Although, that solution is quite traditional, it worked for my purposes however, so that's what I went with. Nevertheless, If you are creating a complete Three.js experience, then I'd suggest using Lenis. I say this despite my reservations, because Lenis or another virtual-scroller are really your only options.

My Favorite React State-Management Solution: Zustand

Created by the Poimandres dev collective (Pmndrs)—the same collective behind React Three Fiber & the React Three ecosystem—Zustand is a featherweight alternative to React Redux, and I love the little state-management solution dearly.

It’s incredibly minimal (feeling more like an extension of React’s Context API), which makes learning it, and integrating it rather painless.

In addition, because it comes from Poimandres, it integrates superbly into a react-three/fiber scene, with minimal performance impact. You can see it in action on their demo site, or you can see how I used it for my toy-clicker game, bot-clicker, on its project page.

However, integrating Zustand with React Three isn’t the only way to use it, of course. The theme toggle in the navbar actually relies on it. While I’m persisting the theme state via the localStorage API, rather than with Zustand’s Persist middleware, it’s kept in memory via a zustand store.

To pass this state around, a provider component wraps the navbar & theme toggle, as well as any other component that needs this context. In effect, state can be sent from the sliding switch, to any of the wrapped components.

While, the primary function of the sliding switch adds or removes the dark class on the root HTML element—this is used by tailwind CSS to change colors from light to dark and vice-versa—it’s utility in being used in a Zustand store allows me to modify non-css elements too, like a Three.js scene to reflect the updated theme state.

Overall, I’m thrilled by the existence of Zustand, especially having gotten used to the ease of Svelte’s store API, that makes state management a breeze. So, it was really nice to have found this minimalist, but really powerful, alternative for React as well.

Discussion

Overall Thoughts on Next.js

Next.js 14 is the current major version of Next.js, having released close to a year ago (10/26/23), with the latest minor update (non-canary) just this past week (14.2.5). Now, having last used Next.js when it was at version 11?, developing with Next 14 has admittedly been quite the learning experience. Even so, I’m overall quite satisfied with how everything turned out.

Granted, the most common complaint you’ll find about Next.js is usually in reference to the cognitive load it puts on developers—a result of its breadth of features, customizability, and the level of control you have over its various APIs (caching, routing, rendering, etc.)—and I don’t disagree with that assessment. Next.js has a lot of moving parts, ergo it’s a lot to learn. Additionally, it really didn’t help that googling the various Next.js APIs and clicking on what pops up will typically take you to the old Pages router version instead of the newer App router alternative/equivalent API. The latter is a little better nowadays, but still not perfect (guess Vercel’s SEO game was just too good).

Furthermore, I’ll even admit to knowing that I don’t know everything about Next.js. For example, I’ve got a rough idea of its caching model, but it’s not as complete as I’d like. However, this isn’t because I think the caching model is overcomplicated, it’s more so just been a lack of need to dive so deeply into it (perhaps exploring it in-depth would make a good blog post?).

Ignorance aside, once I managed to get a good enough mental-model of Next.js in my head, working with it was pretty smooth. At this point, I find most of the added thinking is spent in the optimization and testing departments.

In sum, it’s my opinion that Next’s key strengths more than make up for its drawbacks. What I like the most about Next.js is just how, well, next it is. It integrates unreleased features from React (which, albeit annoying sometimes for compatibility, but I understand why), it’s also incredibly forward-thinking in the features it provides, like streaming. Not to mention the level of control you have over even minute details like the precise runtime for a page route. Additionally, it’s one of the most batteries-included frameworks I’ve used as far as JavaScript/TypeScript frameworks go, and that’s a huge plus too. Oh, and it makes TypeScript a first-class citizen to top it all off, which makes my brain happy.

Overall Thoughts on Ion/SST

Ion might still be in beta, but once I was able to get things going, it's been a really solid deployment tool. Ion’s major defining difference from its predecessors, and leading reason why I chose beta software over the stable SST v2, is that it uses Pulumi and Terraform providers to provision your IaC, instead of Amazon’s CDK/CFN. The fact that your app’s architecture can be made up from services provided by a whole host of different cloud service providers instead of just the services offered by AWS, was a major selling point to me.

The only real drawback to Ion/SST that I can think of, is you’ll need to set up your own CI/CD pipeline with something like GitHub Actions or Circle CI. Well, unless of course you want to run sst deploy manually after every commit, then don't let me stop you. If configuring that sounds like a drag, SST conveniently provides a CI/CD service called seed, which should just work with an Ion/SST codebase OOTB.

Overall, I’m incredibly pleased with using SST/Ion to deploy Next.js applications to AWS (CloudFront, Lambda, S3, etc.). Even though Ion isn’t quite stable yet (it’ll be renamed SST v3 by then), I’m still quite satisfied with it. While it can’t replace all the tools in my DevOps toolbox (nor should it), it’s definitely going to get some heavy usage.

Conclusion

This site is a Next.js 14 application, and with the help of Ion/SST + OpenNext, it gets deployed via serverless AWS Lambda functions, with integrated S3 buckets (& other AWS Services), where it’s then distributed onto CloudFront’s CDN for your enjoyment.

While both Next.js and Ion/SST have their flaws, they’re both fantastic tools in my full stack toolbox. Their strengths more than make up for their perceived weaknesses, and I'll happily use this combination again going forward.

In addition, there's a whole slew of neat tools and technologies contained in this sites stack that I really enjoyed learning and building with. It's quite likely I'll find a use for these tools and libraries in later projects as well.

Overall, I couldn't be happier with how this site came out, and I'm excited to see how both it and I, will evolve over time.

Finally, if you have any questions, or want to share your thoughts on this piece, or would like to submit corrections, please either reach out to me on social media, leave a comment below, open an issue, or submit a PR to the repo directly. Whichever way, I’d love to hear your thoughts, and I appreciate (constructive) feedback, thank you.