How to make all/most filesystems read-only in arch linux (arm)

minimum

/etc/fstab

Add ro to options of desired file systems (e.g. /boot).

kernel parameters

Add ro in /boot/cmdline.txt or /boot/grub/grub.cfg, etc. to the kernel command line.

nice-to-haves

pam_tally

Problem: Without further steps, one cannot log in from the ttys (while remote login via ssh still works).
Solution: Disable (comment out) pam_tally.so in /etc/pam.d/system-login or use below fix for log directory (recommended, as otherwise remounting / as rw is not easily reversible).

/var/log

Mount tmpfs on /var/log (who needs logs anyways?), put

none /var/log tmpfs defaults 0 0

into /etc/fstab.
Optionally save old logs on shutdown, put

[Unit]
Description=save /var/log on shutdown

[Service]
Type=oneshot
RemainAfterExit=true
ExecStop=/usr/bin/save-var-log

[Install]
WantedBy=multi-user.target

into /etc/systemd/system/save-var-log.service and

#!/bin/bash

if [ "$(whoami)" != 'root' ]; then
>&2 echo 'must be run as root'
exit 1
fi

cd /var/log || exit $?

tar -czf - * \
| ssh target-host '
cat > "logs/'"$(hostname)"'-var-log-$(date --iso-8601=seconds).tgz"
'

into /usr/bin/save-var-log. Enable by running systemctl enable --now save-var-log.service.

/var/tmp

Problem:
ntpdate.service: Failed to run 'start' task: Read-only file system
ntpd.service: Failed to run 'start' task: Read-only file system
haveged.service: Failed to run 'start-pre' task: Read-only file system
systemd-resolved.service: Failed to run 'start' task: Read-only file system
Solution: Mount a tmpfs also on /var/tmp.

logrotate

Problem: logrotate.service: Failed to run 'start' task: Read-only file system
Solution: Disable logrotate.timer, we don't need it (seems to be non-sufficient).

archbuild

Problem: archbuild, pacman (and more) need a writable /var/cache
Solution: Put it on a separate partition (mounted rw).
However, because resize2fs: On-line shrinking not supported, one needs to do the repartitioning offline.

nginx

Problem: nginx expects /var/lib/nginx to be writable.
Solution: Nothing permanent seems to be stored there, so we opt for a tmpfs and put

none /var/lib/nginx tmpfs defaults 0 0

into /etc/fstab.
Problem: nginx also expects /var/log/nginx to be existent (and writable).
Solution: Generate this directory on startup of the unit by putting

[Service] ExecStartPre=/usr/bin/mkdir -p /var/log/nginx

into /etc/systemd/system/nginx.service.d/override.conf.

troubleshooting

remount after update fails

Problem: mount -o remount,ro / fails with mount: /: mount point is busy.
Cause: Some executables use libraries which were deleted during update. The used inodes must be freed before remounting.
Solution: Find offending processes with

lsof +L1

Restart offending processes

# for pid 1:
systemctl daemon-reexec
# for other pids:
systemctl status $pid
systemctl restart $found_daemon

Problem: Sometimes, this does not report any problems, but the remount still fails.
Solution: It seems, lsof +L1 does not show files which were deleted and now exist with different content. Check in /var/log/pacman.log, which packages were updated, and check with pacman -Ql package, which files they own. Then use lsof | grep /usr/lib/libofupdatedpackage.so to find pids which use those files (most probably only libraries are relevant) and restart them the syme way as above. One-command-does-it-all-solution:

lsof / | grep -F <(pacman -Qql $(sed 's@^\['"$(date +%F)"' \S\+ \[ALPM] upgraded \(\S\+\) .*$@\1@;t;d' /var/log/pacman.log) | grep '[^/]') | awk '{print $2}' | while read -r pid; do if [ ${pid} -eq 1 ]; then systemctl daemon-reexec; continue; fi; systemctl status ${pid} 2>/dev/null | head -n1 | grep -v '^Failed to get' | awk '{print $2}'; done | sort -u | xargs -r systemctl restart; mount -o remount,ro /