smbd.service fails to start on readonly file system

Question

I have a Pi setup where the root file system has been made read-only to avoid corruption of the OS due to abrupt power loss/improper shutdown, using the following script (original here):

#!/bin/bash

# CREDIT TO THESE TUTORIALS:
# petr.io/en/blog/2015/11/09/read-only-raspberry-pi-with-jessie
# hallard.me/raspberry-pi-read-only
# k3a.me/how-to-make-raspberrypi-truly-read-only-reliable-and-trouble-free

if [ $(id -u) -ne 0 ]; then
    echo "Installer must be run as root."
    echo "Try 'sudo bash $0'"
    exit 1
fi

clear

echo "This script configures a Raspberry Pi"
echo "SD card to boot into read-only mode,"
echo "obviating need for clean shutdown."
echo "NO FILES ON THE CARD CAN BE CHANGED"
echo "WHEN PI IS BOOTED IN THIS STATE. Either"
echo "the filesystems must be remounted in"
echo "read/write mode, card must be mounted"
echo "R/W on another system, or an optional"
echo "jumper can be used to enable read/write"
echo "on boot."
echo
echo "Links to original tutorials are in"
echo "script source. THIS IS A ONE-WAY"
echo "OPERATION. THERE IS NO SCRIPT TO"
echo "REVERSE THIS SETUP! ALL other system"
echo "config should be complete before using"
echo "this script. MAKE A BACKUP FIRST."
echo
echo "Run time ~5 minutes. Reboot required."
echo
echo -n "CONTINUE? [y/N] "
read
if [[ ! "$REPLY" =~ ^(yes|y|Y)$ ]]; then
    echo "Canceled."
    exit 0
fi

# FEATURE PROMPTS ----------------------------------------------------------
# Installation doesn't begin until after all user input is taken.

INSTALL_RW_JUMPER=0
INSTALL_HALT=0
INSTALL_WATCHDOG=0

# Given a list of strings representing options, display each option
# preceded by a number (1 to N), display a prompt, check input until
# a valid number within the selection range is entered.
selectN() {
    for ((i=1; i<=$#; i++)); do
        echo $i. ${!i}
    done
    echo
    REPLY=""
    while :
    do
        echo -n "SELECT 1-$#: "
        read
        if [[ $REPLY -ge 1 ]] && [[ $REPLY -le $# ]]; then
            return $REPLY
        fi
    done
}

SYS_TYPES=(Pi\ 3\ /\ Pi\ Zero\ W All\ other\ models)
WATCHDOG_MODULES=(bcm2835_wdog bcm2708_wdog)
OPTION_NAMES=(NO YES)

echo -n "Enable boot-time read/write jumper? [y/N] "
read
if [[ "$REPLY" =~ (yes|y|Y)$ ]]; then
    INSTALL_RW_JUMPER=1
    echo -n "GPIO pin for R/W jumper: "
    read
    RW_PIN=$REPLY
fi

echo -n "Install GPIO-halt utility? [y/N] "
read
if [[ "$REPLY" =~ (yes|y|Y)$ ]]; then
    INSTALL_HALT=1
    echo -n "GPIO pin for halt button: "
    read
    HALT_PIN=$REPLY
fi

echo -n "Enable kernel panic watchdog? [y/N] "
read
if [[ "$REPLY" =~ (yes|y|Y)$ ]]; then
    INSTALL_WATCHDOG=1
    echo "Target system type:"
    selectN "${SYS_TYPES[0]}" \
        "${SYS_TYPES[1]}"
    WD_TARGET=$?
fi

# VERIFY SELECTIONS BEFORE CONTINUING --------------------------------------

echo
if [ $INSTALL_RW_JUMPER -eq 1 ]; then
    echo "Boot-time R/W jumper: YES (GPIO$RW_PIN)"
else
    echo "Boot-time R/W jumper: NO"
fi
if [ $INSTALL_HALT -eq 1 ]; then
    echo "Install GPIO-halt: YES (GPIO$HALT_PIN)"
else
    echo "Install GPIO-halt: NO"
fi
if [ $INSTALL_WATCHDOG -eq 1 ]; then
    echo "Enable watchdog: YES (${SYS_TYPES[WD_TARGET-1]})"
else
    echo "Enable watchdog: NO"
fi
echo
echo -n "CONTINUE? [y/N] "
read
if [[ ! "$REPLY" =~ ^(yes|y|Y)$ ]]; then
    echo "Canceled."
    exit 0
fi

# START INSTALL ------------------------------------------------------------
# All selections have been validated at this point...

# Given a filename, a regex pattern to match and a replacement string:
# Replace string if found, else no change.
# (# $1 = filename, $2 = pattern to match, $3 = replacement)
replace() {
    grep $2 $1 >/dev/null
    if [ $? -eq 0 ]; then
        # Pattern found; replace in file
        sed -i "s/$2/$3/g" $1 >/dev/null
    fi
}

# Given a filename, a regex pattern to match and a replacement string:
# If found, perform replacement, else append file w/replacement on new line.
replaceAppend() {
    grep $2 $1 >/dev/null
    if [ $? -eq 0 ]; then
        # Pattern found; replace in file
        sed -i "s/$2/$3/g" $1 >/dev/null
    else
        # Not found; append on new line (silently)
        echo $3 | sudo tee -a $1 >/dev/null
    fi
}

# Given a filename, a regex pattern to match and a string:
# If found, no change, else append file with string on new line.
append1() {
    grep $2 $1 >/dev/null
    if [ $? -ne 0 ]; then
        # Not found; append on new line (silently)
        echo $3 | sudo tee -a $1 >/dev/null
    fi
}

# Given a filename, a regex pattern to match and a string:
# If found, no change, else append space + string to last line --
# this is used for the single-line /boot/cmdline.txt file.
append2() {
    grep $2 $1 >/dev/null
    if [ $? -ne 0 ]; then
        # Not found; insert in file before EOF
        sed -i "s/\'/ $3/g" $1 >/dev/null
    fi
}

echo
echo "Starting installation..."
echo "Updating package index files..."
apt-get update

echo "Removing unwanted packages..."
# Let's keep dbus...that includes avahi-daemon, a la 'raspberrypi.local',
# also keeping xserver & lightdm for GUI login (WIP, not working yet)
apt-get remove -y --force-yes --purge triggerhappy logrotate \
 dphys-swapfile fake-hwclock
apt-get -y --force-yes autoremove --purge

# Replace log management with busybox (use logread if needed)
echo "Installing busybox-syslogd..."
apt-get -y --force-yes install busybox-syslogd; dpkg --purge rsyslog

echo "Configuring system..."

# Install boot-time R/W jumper test if requested
GPIOTEST="gpio -g mode $RW_PIN up\n\
if [ \`gpio -g read $RW_PIN\` -eq 0 ] ; then\n\
\tmount -o remount,rw \/\n\
\tmount -o remount,rw \/boot\n\
fi\n"
if [ $INSTALL_RW_JUMPER -ne 0 ]; then
    apt-get install -y --force-yes wiringpi
    # Check if already present in rc.local:
    grep "gpio -g read" /etc/rc.local >/dev/null
    if [ $? -eq 0 ]; then
        # Already there, but make sure pin is correct:
        sed -i "s/^.*gpio\ -g\ read.*$/$GPIOTEST/g" /etc/rc.local >/dev/null

    else
        # Not there, insert before final 'exit 0'
        sed -i "s/^exit 0/$GPIOTEST\\nexit 0/g" /etc/rc.local >/dev/null
    fi
fi

# Install watchdog if requested
if [ $INSTALL_WATCHDOG -ne 0 ]; then
    apt-get install -y --force-yes watchdog
    # $MODULE is specific watchdog module name
    MODULE=${WATCHDOG_MODULES[($WD_TARGET-1)]}
    # Add to /etc/modules, update watchdog config file
    append1 /etc/modules $MODULE $MODULE
    replace /etc/watchdog.conf "#watchdog-device" "watchdog-device"
    replace /etc/watchdog.conf "#max-load-1" "max-load-1"
    # Start watchdog at system start and start right away
    # Raspbian Stretch needs this package installed first
    apt-get install -y --force-yes insserv
    insserv watchdog; /etc/init.d/watchdog start
    # Additional settings needed on Jessie
    append1 /lib/systemd/system/watchdog.service "WantedBy" "WantedBy=multi-user.target"
    systemctl enable watchdog
    # Set up automatic reboot in sysctl.conf
    replaceAppend /etc/sysctl.conf "^.*kernel.panic.*$" "kernel.panic = 10"
fi

# Install gpio-halt if requested
if [ $INSTALL_HALT -ne 0 ]; then
    apt-get install -y --force-yes wiringpi
    echo "Installing gpio-halt in /usr/local/bin..."
    cd /tmp
    curl -LO https://github.com/adafruit/Adafruit-GPIO-Halt/archive/master.zip
    unzip master.zip
    cd Adafruit-GPIO-Halt-master
    make
    mv gpio-halt /usr/local/bin
    cd ..
    rm -rf Adafruit-GPIO-Halt-master

    # Add gpio-halt to /rc.local:
    grep gpio-halt /etc/rc.local >/dev/null
    if [ $? -eq 0 ]; then
        # gpio-halt already in rc.local, but make sure correct:
        sed -i "s/^.*gpio-halt.*$/\/usr\/local\/bin\/gpio-halt $HALT_PIN \&/g" /etc/rc.local >/dev/null
    else
        # Insert gpio-halt into rc.local before final 'exit 0'
        sed -i "s/^exit 0/\/usr\/local\/bin\/gpio-halt $HALT_PIN \&\\nexit 0/g" /etc/rc.local >/dev/null
    fi
fi

# Add fastboot, noswap and/or ro to end of /boot/cmdline.txt
append2 /boot/cmdline.txt fastboot fastboot
append2 /boot/cmdline.txt noswap noswap
append2 /boot/cmdline.txt ro^o^t ro

# Move /var/spool to /tmp
rm -rf /var/spool
ln -s /tmp /var/spool

# Move /var/lib/lightdm and /var/cache/lightdm to /tmp
rm -rf /var/lib/lightdm
rm -rf /var/cache/lightdm
ln -s /tmp /var/lib/lightdm
ln -s /tmp /var/cache/lightdm

# Make SSH work
replaceAppend /etc/ssh/sshd_config "^.*UsePrivilegeSeparation.*$" "UsePrivilegeSeparation no"
# bbro method (not working in Jessie?):
#rmdir /var/run/sshd
#ln -s /tmp /var/run/sshd

# Change spool permissions in var.conf (rondie/Margaret fix)
replace /usr/lib/tmpfiles.d/var.conf "spool\s*0755" "spool 1777"

# Move dhcpd.resolv.conf to tmpfs
touch /tmp/dhcpcd.resolv.conf
rm /etc/resolv.conf
ln -s /tmp/dhcpcd.resolv.conf /etc/resolv.conf

# Make edits to fstab
# make / ro
# tmpfs /var/log tmpfs nodev,nosuid 0 0
# tmpfs /var/tmp tmpfs nodev,nosuid 0 0
# tmpfs /tmp     tmpfs nodev,nosuid 0 0
replace /etc/fstab "vfat\s*defaults\s" "vfat    defaults,ro "
replace /etc/fstab "ext4\s*defaults,noatime\s" "ext4    defaults,noatime,ro "
append1 /etc/fstab "/var/log" "tmpfs /var/log tmpfs nodev,nosuid 0 0"
append1 /etc/fstab "/var/tmp" "tmpfs /var/tmp tmpfs nodev,nosuid 0 0"
append1 /etc/fstab "\s/tmp"   "tmpfs /tmp    tmpfs nodev,nosuid 0 0"

# PROMPT FOR REBOOT --------------------------------------------------------

echo "Done."
echo
echo "Settings take effect on next boot."
echo
echo -n "REBOOT NOW? [y/N] "
read
if [[ ! "$REPLY" =~ ^(yes|y|Y)$ ]]; then
    echo "Exiting without reboot."
    exit 0
fi
echo "Reboot started..."
reboot
exit 0

After running the above script (i.e. making the root file system read-only) and restarting the Pi, smbd.service fails to start:

$ service smbd status 
● smbd.service - Samba SMB Daemon
     Loaded: loaded (/lib/systemd/system/smbd.service; enabled; vendor preset: ena
     Active: failed (Result: exit-code) since Thu 2016-11-03 22:51:05 IST; 3min 24
       Docs: man:smbd(8)
             man:samba(7)
             man:smb.conf(5)
    Process: 678 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited, status=1/FAIL
 Main PID: 678 (code=exited, status=1/FAILURE)

Nov 03 22:51:04 sanjaya systemd[1]: Starting Samba SMB Daemon...
Nov 03 22:51:05 sanjaya systemd[1]: smbd.service: Main process exited, code=exit
Nov 03 22:51:05 sanjaya systemd[1]: Failed to start Samba SMB Daemon.
Nov 03 22:51:05 sanjaya systemd[1]: smbd.service: Unit entered failed state.
Nov 03 22:51:05 sanjaya systemd[1]: smbd.service: Failed with result 'exit-code'

...and...

$ journalctl -xe
-- Unit smbd.service has begun starting up.
Nov 03 22:51:05 sanjaya systemd[1]: smbd.service: Main process exited, code=exit
Nov 03 22:51:05 sanjaya systemd[1]: Failed to start Samba SMB Daemon.
-- Subject: Unit smbd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit smbd.service has failed.
-- 
-- The result is failed.
Nov 03 22:51:05 sanjaya systemd[1]: smbd.service: Unit entered failed state.
Nov 03 22:51:05 sanjaya systemd[1]: smbd.service: Failed with result 'exit-code'

I suspected that this could be a result of file system being read-only. The script gives an option to boot with writable mode using GPIO pins. Sure enough, doing so has smbd.service working normally.

So I next created tmpfs based mount points for all samba directories with following additional entries in /etc/fstab:

tmpfs  /run/samba        tmpfs  nodev,nosuid  0  0
tmpfs  /var/cache/samba  tmpfs  nodev,nosuid  0  0
tmpfs  /var/spool/samba  tmpfs  nodev,nosuid  0  0

/tmp/, /var/log/ and /var/tmp/ are already tmpfs mount points.

However, even after this smbd.service fails to start and gives the same error messages as earlier.

Booting with writable mode using the GPIO pins again has smbd.service working normally, so the above mount points don't seem to create any new issue(s).

How to get smbd.service/Samba working with a read-only file system?

Ingo · Accepted Answer · 2018-08-03T21:18:09.050

I do not understand what this script do. Nowadays read only filesystems are made with union filesystems. As far as I know this also has no limitations to be used only with Raspbian Stretch Light. I searched the script for union and overlay and filesystem and could not find any proper hit.

With a union filesystem you will overlay your read only filesystem with a read/write filesystem that only exists in the ram. This is direct supported by the kernel. For the operating system it acts like a normal read/write filesystem but changes are only stored in the ram. So on reboot all changes are lost. It is also easy to make the underlaying read only filesystem read/write for maintenance without rebooting it and monitor a GPIO pin.

How to use this you can look at How do I make the OS reset itself every time it boots up?.

Then the smbd.service/Samba should also work because it sees a normal read/write filesystem.

OK! Let me try this... Sounds much less hacky and is what I require... — Kanchu, Aug 04 '18 at 10:47
@Kanchu Glad to help you :-) To finish your question: can you please accept my answer? — Ingo, Aug 04 '18 at 13:03

smbd.service fails to start on readonly file system

How to get smbd.service/Samba working with a read-only file system?

1 Answers1