ro
to options
of desired file systems (e.g. /boot
).
ro
in /boot/cmdline.txt
or /boot/grub/grub.cfg
, etc. to the kernel command line.
pam_tally.so
in /etc/pam.d/system-login
or use below fix for log directory (recommended, as otherwise remounting / as rw
is not easily reversible).
tmpfs
on /var/log
(who needs logs anyways?), put
none /var/log tmpfs defaults 0 0
/etc/fstab
.
[Unit]
Description=save /var/log on shutdown
[Service]
Type=oneshot
RemainAfterExit=true
ExecStop=/usr/bin/save-var-log
[Install]
WantedBy=multi-user.target
/etc/systemd/system/save-var-log.service
and
#!/bin/bash
if [ "$(whoami)" != 'root' ]; then
>&2 echo 'must be run as root'
exit 1
fi
cd /var/log || exit $?
tar -czf - * \
| ssh target-host '
cat > "logs/'"$(hostname)"'-var-log-$(date --iso-8601=seconds).tgz"
'
/usr/bin/save-var-log
. Enable by running systemctl enable --now save-var-log.service
.ntpdate.service: Failed to run 'start' task: Read-only file system
ntpd.service: Failed to run 'start' task: Read-only file system
haveged.service: Failed to run 'start-pre' task: Read-only file system
systemd-resolved.service: Failed to run 'start' task: Read-only file system
tmpfs
also on /var/tmp
.
logrotate.service: Failed to run 'start' task: Read-only file system
logrotate.timer
, we don't need it (seems to be non-sufficient).
archbuild
, pacman
(and more) need a writable /var/cache
rw
).resize2fs: On-line shrinking not supported
, one needs to do the repartitioning offline.nginx
expects /var/lib/nginx
to be writable.
none /var/lib/nginx tmpfs defaults 0 0
/etc/fstab
.nginx
also expects /var/log/nginx
to be existent (and writable).
[Service]
ExecStartPre=/usr/bin/mkdir -p /var/log/nginx
/etc/systemd/system/nginx.service.d/override.conf
.mount -o remount,ro /
fails with mount: /: mount point is busy.
lsof +L1
# for pid 1:
systemctl daemon-reexec
# for other pids:
systemctl status $pid
systemctl restart $found_daemon
lsof +L1
does not show files which were deleted and now exist with different content.
Check in /var/log/pacman.log
, which packages were updated, and check with pacman -Ql package
, which files they own.
Then use lsof | grep /usr/lib/libofupdatedpackage.so
to find pids which use those files (most probably only libraries are relevant) and restart them the syme way as above.
One-command-does-it-all-solution:
lsof / | grep -F <(pacman -Qql $(sed 's@^\['"$(date +%F)"' \S\+ \[ALPM] upgraded \(\S\+\) .*$@\1@;t;d' /var/log/pacman.log) | grep '[^/]') | awk '{print $2}' | while read -r pid; do if [ ${pid} -eq 1 ]; then systemctl daemon-reexec; continue; fi; systemctl status ${pid} 2>/dev/null | head -n1 | grep -v '^Failed to get' | awk '{print $2}'; done | sort -u | xargs -r systemctl restart; mount -o remount,ro /