Cannot SSH into periphery RPis from master RPi using python script automated from crontab: Host key verification failed

Question

Project Overview

I have a series of RPi 3B+ which collect T/RH data using connected sensors. These "periphery" units collect data every 5 minutes, append the data to a CSV file that holds a day's worth of data, and stores the data locally in a /data/ directory.

I have a "master" RPi 3B+ whose job is to periodically (approximately every 15 minutes) connect to these periphery units via SSH and copy the available data from the /data/ directory, process it, generate figures, and push these figures to GitHub.

The master RPi has been set up to connect without a password to the periphery units via ssh-copy-id. I was able to ssh into the periphery units from the master and, through that, stored the periphery units in the /home/pi/.ssh/known_hosts directory.

The Problem

I created a script to automate this process and it works perfectly when I manually run it. The code used to connect to each device and download the data is shown below:

os.system(f'scp -r -o ConnectTimeout=3 pi@{ip_address}:{path_to_raw} {self.path_to_data}/interim/')  

I have run the script successfully both within and outside a virtual environment. The problem happens when I try to schedule the script via /etc/crontab. When I do this, the master can no longer connect to the periphery units with the error Host key verification failed. The /etc/crontab file is below:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
Example of job definition:
.---------------- minute (0 - 59)
|  .------------- hour (0 - 23)
|  |  .---------- day of month (1 - 31)
|  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
|  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
|  |  |  |  |
*  *  *  *  * user-name command to be executed
#/5  * * * root sh /home/pi/bleed-orange-measure-iaq/update_figures.sh
/5  * * * root /usr/bin/python3 /home/pi/bleed-orange-measure-iaq/src/data/make_dashboard.py

The first command I have commented out was my attempt to run the script by wrapping it in a shell script but that was also unsuccessful so I figured I would just execute the python command.

The Question

My question is why can I not connect to the periphery units when I place the script within crontab or in a service file? I understand now that the cron and login environments are different, but I have supplied both a PATH environment variable and used full path names.

Answer 1

You've edited your question, and several things in my answer are now passé. You've also accepted an answer now, and I will take that to mean you've got your objective system/software working. I'll just take this opportunity to address a few points:

1. user crontabs

Every user (by default) gets the ability to create his own crontab, which means the user can schedule and run cron jobs. These jobs are "limited" by the user's permissions, group memberships, and all of the usual security protections that are built into Linux.

2. if a cron job requires root privileges:

Wrt cron jobs run by user pi on a Raspberry Pi, you should know a bit of history, and understand its implications:

• Under previous defaults in RPi distros, user pi was already added to the sudo group, and also configured with the NOPASSWD option. Those defaults were changed earlier this year (2022). The NOPASSWD option was significant for cron users because it meant that they could successfully use sudo in user pi's crontab. This was an aberration; I know of no other Linux distro that does this by default. The unfortunate result of this NOPASSWD default for sudo is that some Pi users didn't understand that sudo shouldn't be used in a crontab. So - if you see sudo used in questions or answers here, you should know that this is not the way to run cron jobs with root privileges - such jobs should be run from root's crontab:

  sudo crontab -e
  

  Also know that running a job from root's crontab means that sudo is not required - everything runs with root privileges. Use appropriate caution.

• ICYI, this NOPASSWD option was configured (and can be re-configured) from the file /etc/sudoers.d/010_pi-nopasswd. This is done as follows:

  sudo visudo /etc/sudoers.d/010_pi-nopasswd
  # once your editor is opened, enter one line:
  pi ALL=(ALL) NOPASSWD: ALL 
  

  This edit will work under The Organization's new default configuration for any userid you choose providing two things are true:

  1. the user is a member of the sudo group
  2. the file /etc/sudoers has the following line at the end of the file:

  #includedir /etc/sudoers.d
  note that the # is not a comment, but a #include

3. With respect to your particular issues:

• The file /etc/crontab is not intended to serve as the root crontab!

  /etc/crontab is a system wide crontab with defaults that apply to all crontabs. You should not use it as the root crontab!. Instead, the root crontab should be edited using the sudo crontab -e as shown above.

  You should also know that the crontab at /etc/crontab is also used to run the cron jobs under the /etc/cron.hourly, .daily, .weekly & .monthly folders. Some of these are important for your system's "good health". You should take a look at them - there are several under /etc/cron.daily. Another reason not to edit the /etc/crontab file!

• Your other issue regarding your inability to make SSH connections as the root user seems to have been explained correctly in your accepted answer :). As clarification for others who may read this, the problem seems to have been that your SSH client knew its public key, and known-hosts files were supposed to be in /home/pi/.ssh, but was mis-directed because the command was run by the root user. Your comment indicated that your issue was resolved by using pi's crontab instead of root's crontab (or /etc/crontab which also runs w/root privileges). However, some readers may be wondering: How would this problem be solved if root privileges were needed in the script?*. I think there are (at least) three possible answers:

  1. Copy your /home/pi/.ssh folder to /root.

  2. Create a /root/.ssh folder from scratch.

  3. This may be an exception to Rule 2 above; i.e. you do need sudo commands in a user crontab.

     It may be an interesting experiment?

---

Former answer:

As Milliways mentioned in his answer, your question is a bit short on details, but perhaps this will help:

1. Wrt cron, you must realize first that cron runs under a completely different environment than you do from your login shell. You can verify this for yourself, and get a record of your cron user's environment by running the printenv command; run it from your login shell (CLI) as follows:

$ printenv > myshellenv.txt

Now create a cron job by adding this line to your crontab:

* * * * * printenv > /home/pi/mycronenv.txt

Afterward, compare these two files & note the substantial differences!

I suspect that your error message Host key verification failed. is due to the fact that cron has no idea where your ~/.ssh folder is. In general, there are two ways to handle this:

1. Use full path directives in your crontab, or
2. Define the PATH environment var in your crontab

This will work. If you still can't fiddle it, post some details & we'll respond.

Answer 2

the Problem

The issue is where ssh (and scp) looks for the host key. The host key is stored in the known_hosts file. The known_hosts file is stored in /home/pi/.ssh/ or /home/fritz/.ssh/ folder when you use ssh as the normal user.

In the cronttab you use the script as root. In that case ssh looks for known_hosts in the folder /root/.ssh/. It does not find that file or the particular entry in the file known_hosts for root, as the host is "unknown" to the root.

crontab as normal user

It seems you have edited crontab with sudo crontab -e. This created a crontab for the user root. Since root is not the usual user who uses ssh and scp The host_key verification is not stored in this user's "home" folder.

When you prefix any command with sudo you temporarily become the user root for that command. With the sudo crontab -e you have created a cronjob for the user root.

Files (Do not edit directly)

When you use the command sudo crontab -e you indirectly create/edit the file /var/spool/cron/crontabs/root

When you use the command crontab -e you indirectly create/edit the file /var/spool/cron/crontabs/pi

Do not edit these files directly using a text editor. Always use the crontab -e or sudo crontab -e.

System services and applications place system-wide cron jobs in the file /etc/crontab.

Here is more on using crontab -e and editing the /etc/crontab file: https://superuser.com/questions/290093/difference-between-etc-crontab-and-crontab-e

An Analogy

It is like becoming the acting manager for a day when the manager is sick and logging in with your manager's userID and password when you are the acting manager.

Now the manager's home folder does not have all the folders and files your home folder has. So if you look for a file in the manager's home folder, that you saved in your home folder yesterday (such as known_hosts) you will not find it there, because today you are logged in as the manager, yesterday you were logged in as yourself.

Think you using sudo prefix as logging in as the acting manager. The sudo crontab -e is like editing the manager's calendar and add a recurring meeting for the manager every Friday when you meant to put that meeting in your own calendar.

Solution

You can revert the change and use crontab -e as the normal user pi or fritz or whatever, and put the line there, then the scp will run as the normal user, and not root.

Don't ssh as root

Alternately you can try ssh into the remote server as root manually first with this command:

sudo ssh pi@<ip address of the other pi>

This may not work as ssh as the root with the `rot's password is disabled by default because it is a security risk. The root user may not have a password assigned for the same security reasons.

hope this helps

Answer 3

Your Question(s) are not answerable because you have provided insufficient detail and no diagnostics. In fact the Question is not Pi specific - it is a general Linux question and Questions should be restricted to a single issue.

You appear to be trying to run a couple of python programs then by some (unspecified) process using ssh access it.

Most of us doing remote logging would use a socket interface, but there is a common tool mqtt for this kind of task which does the job.

See https://raspberrypi.stackexchange.com/a/105549/8697 for an example I use.

This code is intended to run from crontab and runs every 15 minutes.

The following is an extract from my crontab.

PYTHONPATH=/home/pi/bin/python
# This will publish into Broker on localhost
# Should Publish nothing if sensor missing
1,16,31,46 * * * * /home/pi/mqtt-dht11s.py

I have a second task to aggregate the data - this is run on the same Pi using crontab but it can be on any other computer by substituting an IP for localhost

6,21,36,51 * * * * /usr/bin/mosquitto_sub -h localhost -C 1 -t weather >> weather.log

There are very many tutorials. See https://mosquitto.org