20

For a project I'm working on I'd like to create a "stock" image based on Raspbian that I can install on many Pis.

Ideally I'd like to be able to (on my computer) take the Raspbian image, apt-get some packages, edit some config and then create and SD card image from that to put on my Pis.

Is there a simple or recommended way to do this? Any pitfalls I may encounter?

I am keen to avoid manually tinkering with a pi and then cloning that image. Presumably Raspbian is made in some way that isn't someone manually clicking buttons on a Pi. I'd like to do it that way, if I can.

Ideally I want to create something that can be automatically built on an continuous integration server - I don't want a build process that involves "write an SD card, put it in a Pi" because I want to be ale to create images without any physical access to a Raspberry Pi.

techraf
  • 4,319
  • 10
  • 31
  • 42
Andy Smith
  • 432
  • 1
  • 5
  • 19
  • 3
    May come in handy: How can I mount a Raspberry Pi Linux distro image? -- note the emphasized line in the accepted answer, "If you do change anything, those changes will be included in the .img file." Unfortunately, this doesn't provide the opportunity to apply apt-get; the easiest way to do that would be to use a VM and rsync with a mounted image identical to the one used to start the VM (since VM images themselves aren't good for this) but I think the only VM option here is QEMU, which AFAICT is a PITA. – goldilocks May 26 '16 at 11:39
  • Another approach would be net-booting. – lexu Jul 18 '16 at 07:03
  • The Win32DiskImager have a Read function, have you tried that? – Kong Chun Ho Jul 18 '16 at 17:38

4 Answers4

6

The Raspberry Pi foundation uses a set of bash scripts automating QEMU to generate their images. You should still look into the "industry standards" buildroot and Yocto too.

https://github.com/RPi-Distro/pi-gen

http://cellux.github.io/articles/diy-linux-with-buildroot-part-1/

flakeshake
  • 6,235
  • 1
  • 14
  • 33
  • If the OP is looking to create an industry standard release image.. yes thats the way to go but its a massive learning curve and I wish no hobbyist down that route. – Piotr Kula Jul 19 '16 at 14:36
  • 1
    This sounds like what I'm looking for! Are there any guides about for using this? build.sh seems to assume a bunch of environment variables are set. – Andy Smith Jul 19 '16 at 17:44
  • Probably covered by #4 under TODO ;) Hasn't been touched in a few months so either the people involved are busy with other things or have lost interest in it. – goldilocks Jul 21 '16 at 15:06
  • @goldilocks so this isn't where the base rasbian image comes from? – Andy Smith Jul 21 '16 at 18:21
  • Maybe; I don't have any reason to contradict flakeshake on that. If so then the "too busy" or "uninterested" applies to some of the things on the TODO list (actually, one of them looks to have been attended to, but the TODO was not updated at the same time). If it works for its intended purpose then it works -- figuring out how to use it if you've never done so is another issue, lol. Glancing through it looks reasonably well documented in comments. – goldilocks Jul 21 '16 at 18:33
  • 3
    I'll be honest and say that my impression is you are pursuing the Y side of an XY problem with too much vigour -- it's implied you think the simpler method (just set up an image on a pi, which can be done from a computer via remote login, and the image itself tweaked by mounting, which can be done on a PC, and all of this can be automated with scripting) is too much hassle, and so you have put your hopes in a method that is actually much more hassle. – goldilocks Jul 21 '16 at 18:33
  • For future googlers - This does indeed appear to be where raspbian comes form, so it can probably be considered to be up to date. – Andy Smith Jul 23 '16 at 08:48
  • I do this both ways for creating Ubuntu (not Raspbian) images. For x86 images, I use QEMU running on RHEL8. For RPi/ARM images, I just create a base SD card and modify it, and ship the cloned card. I can't actually see any advantage to using the QEMU approach in the RPi world. It also introduces a lot of headaches in QEMU cross-architecture support (because, rationally, you'd be running QEMU on x86, not ARM). – EML Aug 15 '22 at 15:58
5

My project Nard SDK has many of the features you want. It's not Raspbian but it has Raspbian binary emulation and are thus very close. With Nard you build everything on a PC and the process is very fast and reproducible. Everything is scriptable and can be done by e.g. a nightly continuous integration.
http://www.arbetsmyra.dyndns.org/nard/

Ronny Nilsson
  • 898
  • 5
  • 13
0

I think what you are looking for is some kind of (software) configuration management. With this you would be able to boot up a stock raspbian image and then push over your own configuration (including new packages and the like). There are plenty of SCM systems out there. The downside is that most of these systems are aimed to support large scale networks and so are maybe a bit overkill for your purpose. You also need some kind of provisioning server. Most popular SCM's seem to be Puppet, Chef, Ansible. Articles to get you started:

This is kind of a generic answer but SCM is a wide field. Maybe this is worth digging into depending on what your plans are.

Community
  • 1
duenni
  • 111
  • 3
  • Ansible seems promising because it's agentless and you don't need a full blown server to run it. – duenni Jul 21 '16 at 06:26
  • 1
    Not really, I want the image to be already built, not something that I tack stuff on to. This suggestion doesn't work if my Pi doesn't have a network connection. – Andy Smith Jul 21 '16 at 18:17
-1

In my opinion, doing it in the way you suggested (mounting, doing workarounds for updating and configuring, etc.) presents more challenges than actual solutions. Sure, it may be fun to do it your way if you're interested, but the best and easiest way may be the one you're avoiding.

I recommend you get a Pi and do what you have to do on it, then make an image of its SD card. Then, use that image on your other Pis since that image is already configured.

Note: I'll assume you'll use Raspbian. I'll also assume you're just an average person and not some person who prepares images on a frequent basis (e.g. Raspberry Pi Foundation).

Steps if you do tinker with your Pi:

  1. Download and install fresh OS image of your choice
  2. Plug in Ethernet cable/connect to WiFi
    • Possibly do a LAN scan if you don't know your Pi's IP addres
  3. Do stuff via SSH
    • apt-get update -y && apt-get upgrade to update packages
    • Your personal configuration needs
  4. Make an image, then use that image for other Pis.

Steps if you don't tinker with your Pi (see note above):

  1. Download fresh OS image
  2. Mount image using instructions from https://raspberrypi.stackexchange.com/a/13138/24224 (you also have to consider what OS/image you have since some have different FS layouts)
    • Have a calculator ready.
  3. Find the deb file for each and every package you want to update, plus their dependencies, plus those dependencies' dependencies, plus those dependencies' dependencies' dependencies. apt-offline is a hassle. Refer to this link
    • Then find a way to install all of those packages without using/tinkering with your Pi. apt-get won't work offline. deb file details here. Good luck.
  4. Update the Pi kernels and low-level stuff using Hexxeh's tool
  5. Configure to your requirements
  6. Write modified image to your Pis' SD cards.

I don't see any benefit in not tinkering with your Pi.

Cut yourself some slack and tinker with your Pi. You only have to do it once.

Community
  • 1
Aloha
  • 7,136
  • 1
  • 28
  • 52
  • Its almost impposible to get programs manually by getting, dependencies, and there dependencies, and there dependencies, etc. But if you use apt-get and say you want VLC you would do, apt-get install vlc --yes --print-uris. That will give a list of links; once downloaded, put them in /var/cache/apt/archives/partial and run dpkg -i partial/*.deb to install. Even better, if you get Synaptics, it has a option to generate package download script. Run that script on a Linux Computer and you got your packages. Then you can install them with synaptics again. – NULL Jul 19 '16 at 13:32
  • @NULL Then find a way to install all of those packages without using/tinkering with your Pi. – Aloha Jul 19 '16 at 13:45
  • @NULL As far as I know, you can only use dpkg on the system itself, not on a mounted image. Also, OP is planning on using his/her PC, so the *.deb files you have is for a wrong architecture in the first place (PC is x86/x64, Pi is ARMv6/v7) – Aloha Jul 19 '16 at 13:47
  • As I specify in my question, I don't want to do 1. As for 2, that sounds nuts :). What makes you think @flakeshakes suggestion is any more difficult from 2? – Andy Smith Jul 21 '16 at 18:19
  • @AndySmith I'll quote goldilock's comment: I'll be honest and say that my impression is you are pursuing the Y side of an XY problem with too much vigour -- it's implied you think the simpler method (just set up an image on a pi, which can be done from a computer via remote login, and the image itself tweaked by mounting, which can be done on a PC, and all of this can be automated with scripting) is too much hassle, and so you have put your hopes in a method that is actually much more hassle. – Aloha Jul 22 '16 at 09:55
  • @AndySmith With that said, flakeshake's method is, in my opinion, intended for mass-production type of stuff. You're free to tinker with it, but given that compilation/image preparation/etc. takes a long while if your PC is not beefy enough, it's not worth it in my opinion. It's not the best approach. As you've asked Any pitfalls I may encounter?, yes, there are a lot. – Aloha Jul 22 '16 at 09:59
  • @AndySmith The only pitfall I can imagine with 1 is that you just have to use a Pi. That's it. No fuss, no compiler installs, no configuration mishaps, no hours wasted because "it won't boot", no time wasted compiling. Just get up and use a Pi. Using a Pi is, by far, the easiest solution, and flakeshake's approach is usually reserved for experts. – Aloha Jul 22 '16 at 10:02
  • @AndySmith I really hope you get my point. – Aloha Jul 22 '16 at 10:21
  • @PandaLion98 But that's precisely what I want - an image that I can mass produce and release versions of (ala raspbian). I already know how to install software on a pi, that's not what I'm looking to do. I'm familiar with the "XY problem" but, frankly, sometimes the person asking is asking for a particular thing for a decent reason that they've thought about. flakeshake's link is exactly what I'm looking for - the raspbian base that I can extend. Thanks for your input. – Andy Smith Jul 23 '16 at 08:46
  • @AndySmith an image that I can mass produce Can you explain this a bit more? – Aloha Jul 23 '16 at 09:05
  • @PandaLion98 Shall we discuss in "The Bakery" so as not to mess up these comments any further? – Andy Smith Jul 23 '16 at 09:12
  • 1
    Note that following #1 the OP will send his WIFI password to all his customers, together with the history of commands which he used to set up the system and a lot of other bits which will take hours to hunt down and remove. – Dmitry Grigoryev Jul 06 '17 at 09:51
  • @DmitryGrigoryev Since the OP looks like they're technically capable, I see this as a non-issue. – Aloha Jul 07 '17 at 04:22