Image of a 16Gb card containing unpartitioned space at the end: Truncating possible?

Question

I wanted to prepare a distro for some friends.
As I didn't have anything else lying around, I used a 16Gb SD card.

I started with the default Debian Weezy 2Gb image and did not do any resizing.
Now, as I am done, I wanted to create an image of what I did, but I get a 16Gb file, containing 14Gb of unpartitioned (not partitioned and free) space at the end.

[I used Windows // Win32DiskImager as I have nothing free to stick into the rPI right now]

Can I just truncate the image file right behind the partitioned space and transfer the leading part onto a smaller card, then?

If you have access to a linux computer, run dd if=/dev/path/to/SD/card of=~/SpecialImage.img, then install GParted and run gparted ~/SpecialImage.img. Once GParted opens up your .img file, simply resize the partitions to your liking! (Both commands must be run as root, sudo su should get you what you need. When the $ switches to a #, you're Root. Be careful, this is the Linux equivalent of God.) GParted is basically a GUI frontend for the arcane partiton management tools you'll need. Using GParted just makes everything so much simpler and likelier to succeed. — JamesTheAwesomeDude, Apr 26 '13 at 05:56
Yes, I read your question, and I can tell you that just truncating a disk image is dangerous. Sure, you can just clip off the last bytes of an image, but there's no gaurantee that you won't accidentally shear off any files, especially since Linux intentionally spaces its files out. Plus, there might be some important metadata at the end of the filesystem or something like that. It's a safer bet overall to use the proper filesystem resizing utilities than to try to just lop off the end of a file yourself. — JamesTheAwesomeDude, Apr 27 '13 at 03:53
Well, but I am talking about unpartitioned space. Even Linux won't put any data on Disk Space that is marked unpartitioned but stay within the block as written in the Partition Table. My concern was caused by the image file format: It may be, that the image file has metadata lying at the end of the file, so I was worried cutting off file information. (Hope my previous comment didn't sound too hard, sry ;) — Nippey, Apr 27 '13 at 10:59
Ohh, I get what you're saying now. If you're worried about any corrupted trailing free space, then run GParted on the image after you've truncated it. It'll probably list the last area as either "unallocated" or "corrupted" - create a new "unformatted" partition in that space. Then, you won't have any issues. :) — JamesTheAwesomeDude, Apr 27 '13 at 18:32

Nippey · Accepted Answer · 2015-05-29T07:27:02.117

I finally found a resource that is explaining my question.

http://softwarebakery.com/shrinking-images-on-linux

Short:

Yes, truncating is possible!

Summary of the process:

Extracting the partition information from the image using fdisk:

$ fdisk -lu image.img
Disk image.img: 4096 MB, 4096000000 bytes, 8000000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a1bc7

      Device Boot      Start         End      Blocks   Id  System
image.img1              2048     5872026     5869978    b  W95 FAT32

We see, that the partition has a size of about 2,8Gb (5872026 * 512), the rest is unpartitioned.

So, everything after the end of the partition can be removed.
This will be done with the tool truncate. Don't forget to add 1 to the number of sectors, as block-numbers start at 0.

$ truncate --size=$[(5872026+1)*512] image.img

Edit:

For those being too lazy to switch to Linux:
Also works on Windows with Cygwin's fdisk.exe and truncate.exe!

What is Cygwin?

I may cite: "Cygwin: Get that Linux feeling - on Windows" (https://www.cygwin.com/).
It is a set of executeables that can run under Windows, but provide all the command line programs you usually know from Linux. Cygwin can take very long to install if you select every package during install, but to follow this example, simply make sure, that besides the default configuration, also the packages util-linux (fdisk) and coreutils (truncate) are selected.
Depending on your environment, you man need to add /usr/bin and /usr/sbin to your $PATH.

What package do I need to install to get fdisk.exe and truncate.exe? — PythonNut, May 28 '15 at 17:54
Hi @PythonNut, I updated my answer above and added the required packages :o) — Nippey, May 29 '15 at 07:28
What if the partition has the size of the disk, even if less is used? — piegames, Feb 03 '17 at 18:21

score 5 · Answer 2 · edited Mar 25 '17 at 07:44

I know this is an old question, but I would like to show how to do this process on Mac, because it is not as easy: fdisk doesn't have the -l option, and truncate is not installed by default:

1. Step 1: Install truncate on Mac OS X:

You need MacPorts or Homebrew for this. I use MacPorts. If you don't have one of these, go ahead and install them first. Link to MacPorts

Now, we can install truncate. Open up your Terminal and type:

sudo port install truncate

For Brew installation:

brew install truncate

This should do it.

2. Use Disk Utilities to mount our IMG so that it is visible to the diskutil terminal command.

You will see in a moment why we need this step. Open the Disk Utility app. Click on File (on the top bar) -> Open Disk Image and select your IMG file.

3. Check the partition size of the IMG and where it is mounted.

On a Terminal, type:

diskutil list

And it should show something like this, somewhere in the end:

/dev/disk3 (disk image):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        +16.0 GB    disk3
   1:             Windows_FAT_32 boot                    62.9 MB    disk3s1
   2:                      Linux                         3.9 GB     disk3s2

So, we see that about 12GB are unpartitioned. We need to truncate those.

The reason we need this command is to check out where the disk image is mounted. In my case, it is under: /dev/disk3

4. Find out the actual partition size.

3.9 GB and 62.9 MB are values that won't work with truncate. We need to find the partition sizes in bytes.

5. Run fdisk.

In your Terminal, run this command:

   fdisk /dev/diskX

Where X is the number that you found out in the previous step. This should result in something like this:

         Starting       Ending
 #: id  cyl  hd sec -  cyl  hd sec [     start -       size]
------------------------------------------------------------------------
 1: 0C    0 130   3 -    8  40  32 [      8192 -     122880] Win95 FAT32L
 2: 83    8  40  33 -  478  79  49 [    131072 -    7553024] Linux files*
 3: 00    0   0   0 -    0   0   0 [         0 -          0] unused      
 4: 00    0   0   0 -    0   0   0 [         0 -          0] unused

6. Time to truncate!

On your terminal, cd to the directory that your image is. Then, write:

truncate FILE SIZE

FILE is your file of course.

SIZE is the size in bytes. What I did, is to add to the size column of the fdisk command, the start column and multiply by 512. So, in my case SIZE will be: 512 * (7553024 + 131072) = 3934257152, which is approximately 3.9 GB.

You will need the start and size of the last partition that is shown by fdisk. (Not necessarily the biggest, but the one in the end of the IMG file)

I experimented a bit, and whenever I entered 7553024 * 512 or (7553024 + 1) * 512 bytes, the IMG file was corrupt. So, just to be sure, do it as I indicate above. It might add more than actually needed, but it is a safe option.

7. (Optional) Easily test if the IMG is not corrupt.

Go once more in the Disk Utility, and try to open the new IMG file as before. If it mounts, you can also see the new (smaller) size. If it doesn't mount, something went wrong. (Maybe try to increase the size in the truncate command)

This is not the best test, but it is a sure way to check if the new IMG is corrupt or not. So, don't really count on it, but it is worth the try...

I hope someone finds this helpful!

Nice one! Thanks for expanding knowledge :) Good that you pointed out that 512*(Size+1) does not work. As you can see, I used End and not Size, as you have to keep both partitions if there are more than one. So, using 7553024 + 131072 (+1) is 100% correct! — Nippey, Jun 13 '16 at 06:55

score 1 · Answer 3 · answered Dec 12 '18 at 10:08

1

Recent Win32DiskImager actually has an option to only include allocated partitions in the disk image. This will do the trimming in one click.

answered Dec 12 '18 at 10:08

Dmitry Grigoryev

27,928
6
53
144

Excellent, makes it much easier on Windows! – Nippey Dec 12 '18 at 14:20

score 0 · Answer 4 · answered Apr 26 '19 at 20:16

Just something I have recently noticed as I do a lot of this Win32diskimg operations for backing up my RPI images for deployment. I recently starting testing SD "trimming" fstrim to clean parts of the SD card not in use for theoretical wear leveling. Not sure about the effect it will have but one side effect was when I imaged the 16GB SD partition and then used 7zip to compress it. I went from the usual 16GB to 9.5GB to now a 2.5GB file. I am testing now to make sure nothing is wrong but it appears that using TRIM to mark the unused space as 0's makes the image compression far more efficient (which does make sense). Just wanted to report at this was super easy to accomplish and no multiple steps.

While not wrong it doesn't really address the question. – RalfFriedl Apr 27 '19 at 07:02 — RalfFriedl, Apr 27 '19 at 07:02