Hard drive encryption (and to a lesser extent, secure hard drive erasing) has been in the news regarding child pornography crossing the US border, government officials around the world erasing incriminating emails and other nefarious headlines.
The one that caught my eye was of a person crossing the border from Canada into the United States.
His laptop was searched in customs. Border Agents found all kinds of pornography on the computer (including child pornography) and arrested the owner of the laptop. Later on, when the computer-savvy investigators went to collect evidence on the computer, they were presented with an interesting prompt when they attempted to access the Z: drive.
Enter your PGP passphrase:
PGP is short for “pretty good privacy.” PGP is so good, in fact even the government can’t break into the files on the computer. The investigators couldn’t collect any evidence.
When a grand jury instructed the man to provide the passphrase, the judge cited the 5th Amendment and agreed with the defense that he cannot be compelled to divulge the passphrase. According to the 5th Amendment, he cannot be obligated to testify to his own detriment. In court, the man said he “wasn’t sure” if child porn was on the computer. He admitted to downloading lots of pornography, and if he found kiddie porn, he deleted it.
Sensational stories like these tend to cast a negative light on tools like encryption and permanently erasing files. If you’ve got nothing to hide, you wouldn’t use these tools… right?
Wrong!
The more paranoid folks among us would immediately be able to name a dozen or more reasons. I’ll just name a few real-life senarios that happen every day to people all over the world, often times due to unfortunate events such as a company or government laptop going missing, hell… even a 1.44MB floppy disc contained a hundred thousand records of some government employees in the UK.
Here are the bigger ones…
1. Identity theft.
2. That “home video” you promised to destroy.
3. Selling a hard drive or other rewritable storage device.
That said, GNU shred which is part of coreutils is a great tool to delete those sorts of files we all probably use at one time or another.
Here’s a snippet of the manual:
NAME
shred – overwrite a file to hide its contents, and optionally delete itSYNOPSIS
shred [OPTIONS] FILE […]DESCRIPTION
Overwrite the specified FILE(s) repeatedly, in order to make it harder
for even very expensive hardware probing to recover the data.Mandatory arguments to long options are mandatory for short options
too.-f, –force
change permissions to allow writing if necessary-n, –iterations=N
Overwrite N times instead of the default (25)-s, –size=N
shred this many bytes (suffixes like K, M, G accepted)-u, –remove
truncate and remove file after overwriting-v, –verbose
show progress-x, –exact
do not round file sizes up to the next full block;this is the default for non-regular files
-z, –zero
add a final overwrite with zeros to hide shredding
And the comments from shred.c, written by Colin Plumb:
* Do a more secure overwrite of given files or devices, to make it harder
* for even very expensive hardware probing to recover the data.
*
* Although this process is also known as "wiping", I prefer the longer
* name both because I think it is more evocative of what is happening and
* because a longer name conveys a more appropriate sense of deliberateness.
*
* For the theory behind this, see "Secure Deletion of Data from Magnetic
* and Solid-State Memory", on line at
* http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html
*
* Just for the record, reversing one or two passes of disk overwrite
* is not terribly difficult with hardware help. Hook up a good-quality
* digitizing oscilloscope to the output of the head preamplifier and copy
* the high-res digitized data to a computer for some off-line analysis.
* Read the "current" data and average all the pulses together to get an
* "average" pulse on the disk. Subtract this average pulse from all of
* the actual pulses and you can clearly see the "echo" of the previous
* data on the disk.
*
* Real hard drives have to balance the cost of the media, the head,
* and the read circuitry. They use better-quality media than absolutely
* necessary to limit the cost of the read circuitry. By throwing that
* assumption out, and the assumption that you want the data processed
* as fast as the hard drive can spin, you can do better.
Oh, the things you can find in source code header comments…. that’s fantastic commentary from someone who knows about the topic.
I ran a 6-pass random wipe (followed by a round of zeros) on a 160GB hard drive as well as a 2GB flash drive that were laying around the house. I’m sure there was all sorts of personal information on those. The 160GB drive took about 7 hours to finish and the 2GB thumb drive took about 26 minutes to complete.
I’ll be writing an article on using PGP to encrypt a hard drive in Linux in the near future, but in the mean time, enjoy the video of my 2GB flash drive loosing another 7 write cycles off it’s rewrite capacity. 🙂
21 replies on “Using Shred to Wipe Hard Drives – DoD Uses It – You Should Too!”
“I’ll be writing an article on using PGP to encrypt a hard drive in Linux in the near future”
Please do so. I just got a laptop, which I’ve already installed Kubuntu Gutsy on (no GNOME here 😀 ). I’ve been thinking about encrypting the filesystem, since I have a much higher chance of losing this than the desktop I was using before.
crypt-manager works ecellent in Ubuntu to encrytp directories, I suggest to give it a try:
http://code.google.com/p/crypt-manager/
I always read your excellent blog, from Santiago, Chile. The post about mounting an iphone in linux and with rythmbox, was very very useful for me, thank you and congratulations
Sorry for my english
Another good Hard Drive wiper is “DBAN” or Darik’s Boot And Nuke http://dban.sourceforge.net/
It supports windows and Unix file systems. FAT, VFAT, NTFS, ReiserFS, EXT, and UFS.
For those of you using Gutsy, you might want to check out the alternative installer disc from Ubuntu. This disc allows for you to do “/” (root) encryption from the installer; exception being /boot. This is useful for those of us who want to start out encrypted rather than after the fact.
For the extra paranoid, you could even put the /boot partition on a thumb drive to keep someone from tampering with your kernel to swipe your pass-phrase. 🙂
More information from Ubuntu:
https://wiki.ubuntu.com/EncryptedFilesystemsInstaller
Hey guys, I actually e-mailed this to Wayne, but I suppose I’ll post it here too:
The Setup:
Two Virtual Machines, both running Ubuntu 7.10 installs from the alternate CD. The only difference in the setup is one uses the install-time encryption method and the other does not. On the desktop of the user account on each VM I put a file called CANYOUFINDME. I shutdown each VM and did a simple test on each:
The Test:
Unencrypted:
enjoi:~/Documents/Virtual Machines.localized enjoi$ cd Unencrypted.vmwarevm/
enjoi:~/Documents/Virtual Machines.localized/Unencrypted.vmwarevm enjoi$ strings Unencrypted.vmdk | grep -i canyou
CANYOUFINDME
^C
Encrypted:
enjoi:~/Documents/Virtual Machines.localized enjoi$ cd Encrypted.vmwarevm/
enjoi:~/Documents/Virtual Machines.localized/Encrypted.vmwarevm enjoi$ strings Encrypted.vmdk | grep -i canyou
strings: can’t map file: Encrypted.vmdk ((os/kern) no space available)
enjoi:~/Documents/Virtual Machines.localized/Encrypted.vmwarevm enjoi$
It’s always nice to see something actually in action.
Cheers,
– E
I always get a giggle at comp-sci types arguing over which bit patterns to write to the disk to do the best erasing when we build hardware “randomizers” into the write path circuitry to help spread out the spectrum written to disk.
As far as the number of passes, I wouldn’t do fewer than 15 if I really cared about wiping data. Each time you servo a disk track you have some off track performance, so to get very good erasure you do need to do multiple writes to handle off-track effects well. If you put the disk on a spin stand you can pull stuff off even better than usual. But you’ve got to be really motivated to do that.
But I’m looking forward to the post on encrypting filesystems, especially if you’ve got good data on the relative performance before and after. Seagate’s got their FDE line with Momentus portables the new 1TB Baracuda coming down the pike which are the ultimate with AES and no speed penalty, but you pay a bit more for those drives.
I prefer truecrypt (www.truecrypt.com) as opposed to PGP.
First off, nice blog.
Secondly, some nice comments here.
I feel I must add (and someone should correct me if wrong), that hard drive management stores extra blocks in case of bad data, and no matter of OS finangling will access this data to delete it with the same level of zeal that something like shred uses.
So in order to have your data completely secure, encryption, from the get go, as soon as you buy the drive, is the answer. Hard disk never sees plaintext.
Debian Etch and Lenny installers have had this option for a while now, I just got around to re-doing my entire install (very little reason to reformat ever with a good Linux distro), and am now fully encrypted.
A caveat that I don’t see mentioned often. It will slow things down. Sometimes even unbearably so. Hard drives are a few orders of magnitude slower than ram, and avoiding their use is good practice. But when all HD reads and writes are encrypted, you effectively burden the CPU upon every read and write, unlike good DMA read writes previously. So running a lot of things at once, your system may crawl when you tax the drive. This is why I suspect that FDE has not become an issue of discussion until this Core 2 Duo era we live in… 🙂
My Celeron M 1.5 is okay now that I have tweaked my setup somewhat. A custom kernel, and perhaps a Pentium M 2.0GHz would be dandy for everything but video editing, I suspect. But now I know that I will not be fully encyrpting drives on some of my retro hardware laptops like I had planned. This would crush my P3 600MHz Thinkpad.
@JST
Drives have a fair number of spare tracks. When you hit a TA (Thermal Asperity) or similar event the drive goes into recover mode and uses some increasingly severe recovery methods. If it can recover the sector (and generally it can) the sector is remapped to one on one of those spare sectors, data copied over, and you never know about it. So if you ever see a sector fail in a modern drive you know it’s time to go get a new one since you’re running on margin.
Since a sector is only 512 bytes these days (going to 4K but slowly), your data exposure risk to one of these remappings is relatively low, especially as things like TAs tend to grow and effectively erase more data in that sector.
The Seagate FDE drives use a special faster controller with AES hardware support inside the drive so that they can do full-speed encryption without burdening the CPU. They’re completely transparent except to your wallet. But if I were a company and had anyone dealing with SSNs or credit card numbers they’d be mandatory in their computers.
nerdbert,
Very informative. I have not used a drive with hardware encryption yet, and was curious as to where/how the passphrase was entered. Does it need OS support to prompt during boot, or is it related to the same BIOS-based password protected HDs that I have seen and used?
Also, if you know of one off the top of your head, would you be able to point to a good whitepaper or book to read up on how modern hard disks actually work?
@ JST
*cough* Here’s an educational vid on Perpendicular Magnetic Recording…
http://www.hitachigst.com/hdd/research/recording_head/pr/PerpendicularAnimation.html
Wayne,
You do know that ext3 is a journaling file system a using meta data to keep track of everything on your hard drive.
That is the only way it can recover information in a power blink, system crash, etc.
So it is still possible to recover the data through the meta data on the file system…
-Shane
PredatoryFern,
A few years ago, I fell in love with that animation.
They had this form where you could order a free shirt that had the… _|_ mathematical symbol for perpendicular on it and the text “Get Perpendicular”, with Hitachi on the front. I missed the opportunity, and want it so badly.
@JST,
No, there’s nothing that’s popularly available on disk drives as currently done. The stuff that’s out there is pretty generic, but you’ll never see discussion of modern encoding schemes, ECC schemes, error recovery and management, or the like. That’s all vendor proprietary and secret. There are even quite a few secret ATA commands that are vendor proprietary that are usually disabled and hidden after manufacture.
As to the Seagate FDE drives, they have to have a password entered at boot time since it’s full disk encryption and unless you give the right passphrase you don’t get access to the disk at all. It’s an enhanced version of the BIOS password since it’s actually the key to the AES engine.
There were older variations of this scheme that weren’t all that secure. The old version used by the IBM laptop drives was a far simpler encryption scheme that could be broken by brute force if needed (there were several companies that specialized in this).
[…] fscking /w linux videot välkmälupulga […]
7 passes is generally fine and is what the DOD recommends.
@Shane
PGP’s Shred Free Space includes the option to shred metadata as well as file space on NTFS volumes. I have no idea how it works, but I presume that they were able to get enough info from M$ on the innards of NTFS to know how to do that safely. My guess is, considering how hard it’s been to get even enough info to create the Linux NTFS drivers, it may be a trick to find out what has to be found out to accomplish that.
However, what would be the degree of difficulty finding and shredding metadata for such as Ext3 and Reiser? At least the technical details are freely available. Anyone know enough about them (which means, a lot more than I do) to estimate the practicality of getting shred to handle those cases?
What about doing multiple shreds versus one shred with the Guttman method?
For example, I do one shred with the 35-pass Guttman method once. Or, I can do the 7-pass DoD method 5 times. Which would be more effective for bypassing any forms of data forensics?
[…] Here is a good tutorial show you how to Use Shred to Wipe Hard Drives in Ubuntu: Hard drive encryption (and to a lesser extent, secure hard drive erasing) has been in the news regarding child pornography crossing the US border, government officials around the world erasing incriminating emails and other nefarious headlines. […]
I find that many times articles written this way do not keep the attention of the reader, this was not the case in this article. Excellent read, thanks for the post and looking forward to more!
To erase data permanently on disks with bad sectors, you need WBD(Wipe Bad Disk).