Linux Mac Windows

Using Mozilla Prism To Run Web Apps Seamlessly

Prism is a new side project by the Mozilla team. It’s a nifty way to run specific web applications in an embedded browser of sorts.

If you haven’t seen it yet, imagine a letterboxed web browser – all you see is the content. There is nothing else showing in the window – back/forward navigation buttons, location bar, status bar, all gone.

I’ve found a dozen uses for it, but I think my two favorites are embedded videos (like Chris Pirillo) and Google Reader.

Prism is available for all major platforms – Linux, Mac and Windows. It’s really simple, just install it (or unzip it) and open up the Prism progam, and fill out the information it asks for, like URL or Name.

Prism does not share cookies or any other settings with Firefox, Internet Explorer, or Safari. This is nice if you have separate accounts and don’t want to mix things up.

In Ubuntu 7.10, I setup a Widget Layer rule in Advanced Desktop Effects Settings (ccsm) to match all windows set to be “Always On Top” using the following rule: state=above This setting lets me easily toggle windows to be set as a widget (or not) by simply setting the window as “Always On Top” after right clicking the title bar. Really simple to setup, then just F9 by default to view the widget layer.

Here’s a screenshot of my current setup:

Do you use Prism or Widget Layer rules in a creative way? Let me know in the comments.


Is It Just Me, Or Does Sun Comes Off As A Creepy Uncle?

I got an email from a professor today asking for some input on Java and Solaris… 15-20 minutes of my time is not asking much… but it seems like Sun is the type of guy who is offering some candy from the sliding door of a panel van…

We are inviting you to provide feedback and input by participating in this survey. We are offering a chance to win one of 4 American Express gift cards, each in the amount of US$50. We anticipate you allocating fifteen to twenty minutes to complete the relevant sections. The deadline for completion is February 5th.

Our objective is to better understand the following:

* What is your experience so far with Java and Solaris courses?
* What is your opinion of the books and other courseware available?
* Do you plan to work in a field using Java or Solaris after graduation?
* What do you know about professional certification?



What Do Torvalds, Gates and Jobs Have In Common?

In a shocking discovery, I’ve found that all three (Linus Torvalds, Bill Gates, and Steve Jobs) shop at Banana Republic. Steve Jobs stands out (although not much) by ditching the under-sweater collar, and going with a mock turtle.


C’mon guys – ditch the sweater-look-a-like bullshit already!


HowTo: Help Prevent RSI (The Silent Killer) – With Workrave

When most people think of repetitive stress injuries (RSI for short) they probably think about ergonomic keyboards, comfortable chairs, good posture and workspace positioning. In addition to those common cures (which are sometimes really expensive) there is a tool called Workrave that assists you in avoiding RSI by reminding you to just take a break.

It’s available in the Ubuntu and Debian repositories by default, and can be found pre-packaged for other distributions as well. For those of you forced to run Windows at work (You do run Linux at home, don’t you?) there is a version for that as well, available on the Workrave website.

Even if we have a perfectly ergonomic environment (that undoubtedly cost hundreds or thousands of dollars to setup) using a computer for extended periods of time can wreak havok on our body. According to Wikipedia, the following symtoms indicate an increased risk for repetitive stress injuries.

Recurring pain or soreness in neck, shoulders, upper back, wrists or hands.
Tingling, numbness, coldness or loss of sensation.
Loss of grip strength, lack of endurance, weakness.
Muscles in the arms and shoulders feel hard and wiry when palpated.
Pain or numbness while lying in bed. Often early stage RSI sufferers mistakenly think they are lying on their arms in an awkward position cutting off circulation.

I’ve had a few of those symptoms which prompted me to write this article, as well as investigate various options outfitting my home office with ergonomic products. Plus, Workrave has the added bonus of being completely free, whereas a new task chair I’m looking at is far from it.

Let’s take a look at the program.

We can easily install it on an apt-based distribution with this one liner:
apt-get install workrave

After we’ve got it installed, the next step should be to add it into the startup programs, so you’ll always have it running to be reminded to take a break.

Gnome users would point to System > Preferences > Sessions. Then on the startup tab, click the add button. Simply type the following into the Edit Startup Program dialog box:

Name: Workrave
Command: /usr/bin/workrave
Comment: Workrave is a program that assists in the recovery and prevention of Repetitive Strain Injury (RSI). Just in case you forget.

Then we can start it up by pressing Alt+F2 and type in /usr/bin/workrave and bang on the return key.

Boom there we have it, sitting pretty in the Gnome menu.


There are three timers showing by default, here’s what each of the icons stand for:

Hand: Microbreak
Cup: Rest Break
Door: Self-Suggested Time Limit for the Day

Microbreaks are just that – a very short break (typically 30 seconds or less) that encourages you to let off the keyboard, sit back and relax, and refocus your eyes at an object other than your monitor.

Rest Breaks are intended to be a 5 minute-ish break, where you can hopefully get up and stretch your muscles and if you’re still at the computer, helpful 3D renderings of a woman (who may or may not need a breast reduction to avoid future back trouble) showing you how to stretch your fingers, arms, neck and back muscles, etc.


If you’ve never had any problems, the Workrave website suggests the following schedule: “10 seconds micropause every 10 minutes, and a 5 minute restbreak every hour.” To me, that sounds about right – approximately 6 minutes of break time scheduled throughout the hour.

They’ve added a few nice touches to the program that make the application just about perfect:

1. Skipping or postponing a break is an option if you’re busy, or right click on the timer status bar to suspend the program entirely.

workrave10.png workrave11.png

2. When you stop activity on the computer for about 30 seconds, the program identifies the inactivity and will reset microbreak timers – it assumes you’re not using the computer.

3. Ability to run the program on multiple computers while synchronizing times on each system – if you take a rest break and then come back to sit down at another computer, it doesn’t prompt you for another rest break until the time has run out again.

4. Crazy stats… going back on a calendar you can see how much time you spent moving the mouse, how far you moved it, how many times you clicked, as well as total number of keystrokes – and more!

workrave13.png workrave14.png

5. And finally, if you really want to enforce a “get tough” attitude on yourself, you can set the program to block the entire screen when a break is scheduled to occur.


I really like this program, and if you’ve had any of the symptoms above, I urge you to try it. What’s the worst that can happen?


Don’t answer that just yet, I’m taking a break.

Interview Linux

Interview with Sean Moss-Pultz, CEO of OpenMoko – Ninjas *ARE* Better Than Pirates

Sean Moss-Pultz was kind enough to answer a few questions with fsckin w/ linux. He’s got some pretty interesting answers to questions plaguing geeks and non-geeks alike, such as the eternal debate on which is better – Ninjas or Pirates? Keep reading for some candid responses from the CEO of the company behind the Dash Express GPS device and the in-development OpenMoko phone. Dash was recently named one of the top 10 startup companies to watch in 2008 by Wired Magazine.

0. How did the name “OpenMoko” come about, and does it have a meaning behind it similar to the Neo1973?

“Moko” is short for Mobile Kommunikations. Think of the “K”’s as a tribute to all hackers around the world that build software that drives our platform forward.

“Open” is obvious. But what we mean by “open” perhaps is not. We’re not just talking about software. It’s really the entire business model of our company. For example, this coming week we will release the full mechanical CAD drawings of our Neo’s case. We want people to remix our work. We want to be open from the iron to the eyeballs.

OpenMoko started as a project inside FIC (Taiwan) and has recently been turned into it’s own company. So, we’ve had to build a open software stack, build a team, build a product, plan a future, and build a company, while everyone gets to watch.

Most of the challenges, I would say, are philosophical in nature. Not technical.

In an closed company you go through various stages of hardware development. These stages are hidden from the general public. Prototype hardware is built and passed out to a few select internal developers. Later, more hardware is built. In a pilot run. This is distributed internally to more engineers in the company. Finally, it’s presented to the public like it just came out of the oven.

OpenMoko is inside out. Our prototype are shared with developers around the world. Why? because our engineers are outside the company as well as inside.

In some ways we are like a reality TV show. Showing how one builds a gadget of the future. It’s like a cross of Survivor, Dirty Jobs, and Ice Road Truckers.

1. What is your favorite brain-food? Favorite beverage?

Not sure if nachos are brain-food…but I love them! As for drinks, I would say nothing is more refreshing than water. I like things that can blend in to any environment.

2. Awhile back you said “failure is not an option.” Would you expand on that statement and explain who ultimately looses if you fail?

Hehe…perhaps I was caught up in the passion of my own thoughts. Working every breathing second of the day on something for more than year will do that to you 😉 The point that I was trying to make is that we (collectively) need the phone to be freed. It’s the only way major change will occur in this mobile industry.

3. In a philosophical sense, which is better: Ninjas or Pirates?

Oh ninjas all the way. They were artists using their art to ensure their survival in a time of violent political turmoil. Their strategies were cultural opposites of the conventional ruling Japanese Samurai class. But they lived in harmony with nature. And always kept an eye to where the future was heading.

OpenMoko has an internal ninja. You’re hear more from him later this year.

4. Due to the touch screen and lack of buttons, comparisons are inevitably made between your product to the iPhone. Two questions in one: Why is being compared to the iPhone a good thing? Why is it bad?

It’s a good thing because it brought us lots of press. It’s a bad thing because most of the press talked about us as an iPhone killer. That’s not at all what we’re trying to do.

End user freedom is our passion. Apple gives you an incredibly polished experience. Exactly how you they want you to have it. But the end user really has no freedom. They cannot change the device if they don’t like the way Apple choose to make things.

All these comparisons, unfortunately, caused a lot of confusion. But in the end it was all good. We were forced to refine our message.

5. How much can you bench press?

Not sure anymore…but I can still bench my weight. And I’m gaining weight here in Taiwan (too much good food) 😉

6. Your device seems to pull the opposite direction in where carriers seem to be headed. Revenue streams such as ringtone purchases and other paid-for customizations would obviously be impacted by your project. Have carriers in general expressed any interest in OpenMoko despite the drawbacks?

Carriers are extremely interested. But their reasons vary as much as their responses. So I think it will be a while before see us packaged with 2-year contracts.
Right now we’re focusing on being fast and flexible. So we can react to what our customers want. This is a huge challenge because of the scope of our efforts. Most companies choose either software, hardware, or manufacturing. We’re doing all three. I’m convinced this is the only way it’s possible to making a completely open phone.

7. Is OpenMoko affected by the Year 2038 Bug?

Probably. But we’ve got a heck of a lot more serious Year 2008 bugs.

Anybody want to help? 🙂

8. (As far as I know) OpenMoko hasn’t adopted GPLv3, instead opting to utilize the GPLv2 and LGPL. What are the reasons for using licenses other than GPLv3, and is the GPLv3 compatable with “doing business” in 2008?

I am not a lawyer so I only feel comfortable replying on a personal level. I really the message behind GPLv3. The loopholes it tries to close are real and destructive to the spirit of the original message. In short we will use it. But I cannot make a sweeping statement and say we will use it for everything. Besides, technically that’s just not possible. We make products and a software distribution. This forces us to work with many different licenses.

To deal with these kind of problems we have come up with something we (informally) call our “Software Freedom Requirements”. These are still being refined, but let me just state what we have now:

1) Libraries cannot be GPL — The idea is that libraries are what you use to build your house, your private space that you can control in any way you like. Our preference is LGPL, BSD/MIT less so, but still acceptable. GPL is not acceptable.
2) Everything but libraries should be GPL — The idea is that this is the shared public good, democracy. LGPL and even less so BSD/MIT would be acceptable, but not be loved. We think that those licenses defend the rights of developers, but not the rights of the end user.
3) Contributions must be welcomed back — A company that wants to work with us has to accept valuable contributions back into their mainline, without transfer of copyright, based on technical rather than political merit. This makes dual-licensing impossible. If the company has a program of buying back contributions including transfer of copyright that is also acceptable, as long as valuable contributions will indeed get bought back.

We also have “Hardware Freedom Requirements”. Equally as important, because, as Wolfgang Spraul once said, “Software is what makes hardware dangerous.” 😉

Listed in order of priority:

1) Fully and publicly disclose datasheets, [or]
2) Fully disclose datasheets to OpenMoko and allow us to use that documentation to write a documented GPL driver, [or]
3) Fully release an existing driver under the GPL.

We’ve worked with those for almost a year now. So things are extremely concise. Hopefully we can simplify our message in the Software Freedom Requirements over time.

OpenMoko is, and will always try to be, the most progressive mobile distribution available. Like Moglen, we want to “Resist the resistance!”

9. What’s the progress on your AE86 restoration? Got any pictures yet?

Since OpenMoko started just about everything else in my life has stopped — my AE86 is no exception. Here’s a picture — about 75% done.
Sean Moss-Pultz’s AE86

10. The Dash Express was recently announced and uses almost the same hardware that the OpenMoko does. I would assume that there was some significant collaboration with their teams. How much work was able to be directly integrated back into the OpenMoko project?

Yeah lots of collaboration — we’re all the same team 😉

This was our first OpenMoko-based device that will ship in high-volume to the consumer. So naturally we’re thrilled with results.

Look for more great stuff in 2008. We’re just getting warmed up.

I’d like to personally thank Sean for spending the time to answer these questions. I for one am eagerly looking forward to progress OpenMoko makes towards launching a mass-market open source phone. You can count on at least one person (me) to ditch their locked down, proprietary iPhone when OpenMoko’s device is ready for prime-time.


HowTo: Setup and Benchmark Encrypted Partitions in Ubuntu

In a previous article, I talked about using shred to securely delete files. Now we’ll delve into using encrypted volumes in Linux to secure our data in the first place, so that we don’t need to use programs like shred. Along the way, we’ll benchmark the raw performance of an encrypted volume and compare the results to an unencrypted volume and see just what kind of real world compromises we see.

To start out we need free space on a drive that isn’t partitioned, or enough patience to resize an existing one. Just about everything here needs root privileges, since we’re working with filesystems. It would be easiest to start a root terminal withsudo su, then enter your password.

First, we install the tools to get the encrypted partition going:
apt-get install cryptsetup hashalot gparted

Next, we use gparted to create a 20GB partition at the end of my disk. It’s a dead simple drag n’ drop application similar in function to Partition Magic or other GUI partition editors… hopefully you don’t need instructions. Make sure to record the name of the new partition! Everything here that says /dev/sda2 is going to change based on your hardware and partitioning scheme.

After that completes (which can take some time if any resizing or moving of an existing partition happens), we need to set a password.
cryptsetup --verbose --verify-passphrase luksFormat /dev/sda2

This command will create a device called /dev/mapper/sda2 and give us access to the encrypted volume after verifying the password:
cryptsetup luksOpen /dev/sda2 sda2

By now we’re knee deep in waist-high water. I’m not quite sure what that means… I just made it up. Say it out loud… rolls off the tongue. Sorry… where was I? Ah right. I’ll try to explain where we’re at right now, for my benefit as well as yours.

At this moment, we have a partition called /dev/sda2. That raw partition now has an encrypted container inside, located at /dev/mapper/sda2. The last step is to actually format the encrypted volume so we can actually put some files on there. This can also be done in gparted if you want to split things up into multiple partitions, use the drive dropdown box to find the mapper.
/sbin/mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/sda2

Next, we’ll make a directory to mount the encrypted volume and then actually mount it:
mkdir /mnt/test
mount /dev/mapper/sda2 /mnt/test

Now we can copy files into /mnt/test and every file located there will be encrypted. Sweet!
To unmount the volume, use the following commands:
umount /mnt/test
cryptsetup luksClose sda2

I bet you’re asking the question we all are… How fast is it? Good question. The answer is a pain in the ass to be honest. This almost ended up being two separate articles because the benchmarking was not going very well… but here we go… how to benchmark hard drives in Linux with FOUR different tools:

The first plan I had was to perform two separate clean installs on an entire disk, run several benchmarks and quote some articles on how hard it would be to crack into the encrypted disk. Those results followed the expected trend of a significant (approximately 10%) degradation in read, write, and seek times on the encrypted volume. However, those results could be tainted, because they were run in a graphical environment that had lots going on in the background. I decided to throw those out. More accurate results tied directly to actual performance can be achieved by installing a text-mode only system, and using a separate partition at the end of the disk. By using failsafe text mode, we’ll limit the number of extra services/daemons/etc running.

Processor: Intel E6400 Core2Duo 3.2 GHz
Hard Drive: Western Digital 150GB RaptorX 10,000 RPM
RAM: 4GB, no swap partition used.

Benchmarks Used:
Bonnie++ 1.03
Options: bonnie -s 14176 -d /mnt/test
Bonnie++ is a benchmark suite that is aimed at performing a number of simple tests of hard drive and file system performance. The -s 14176 option sets the program to use is four times the amount of memory available on our testbed, which is the recommended setting. This is to make sure the OS is not doing any sort of caching in RAM to skew results. -d /mnt/test sets the program to use /mnt/test as the location to save the temporary file. Bonnie is a nice benchmark, but it’s got a problem. The results are nearly indecipherable to read by someone unfamiliar with the output, and this page helped me read them.


Options: set size 10000 10000000 (10KB - 10MB pseudo-randomly sized files)
set number 2000 (2000 generated files)
set transactions 2500 (2500 read/write/etc actions made on those files)

PostMark is a benchmark for servers. We can use it and gain some additional insight into how a server would function if it were working on 10KB to 10MB sized files… it’s a benchmark that would need to be customized for the application desired to gain any direct correlations from the results.


IOZone3 3.279
Options used:
sudo iozone -a -R -g 10g -R -f /mnt/test/iozone
Iozone is useful for determining a filesystem performance. The benchmark tests file I/O performance for the following operations: Read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read/write, pread/pwrite variants. The options setup the following variables: Auto Mode, Excel/CSV formatted results, create a 10GB test file on /mnt/test called iozone.

IOZone outputs a TON of data, and they have lots of pretty graphs on their website, but there’s more than enough analysis already for this article. Here are the two Excel (???) formatted files for your pleasure. No graphs by are included in the output by default – shame really, since their graphs look great.

Easy-Bake Tar-Gzip-Gunzip-Untar Oven Benchmark Test
For the last test, we’ll use a home-grown benchmark using tar to archive 3.5GB of highly compressed HD videos, gzip the archive, then unzip and untar the compressed tar.gz file into the current directory, forcibly overwriting the existing files. This constitutes a “real-world” scenario more than the above benchmarks with reads, writes, re-writes and plenty of seeking. We can easily count how long each operation takes to complete and get a really quick, dirty and simple comparison with time counted for each operation. The fact that we’re working with the same data four times… do the math – at least 3.5GB of data traversing to and fro across the subsystems of the testbed 8 times (input/output cycles for each action) for around 28GB of data flying about. The unencrypted data manipulation test completed 42 seconds faster, for a 6% lead over the encrypted filesystem.

time tar -cf archive.tar *.MTS && time gzip archive.tar && time gunzip archive.tar.gz -f && time tar xvf archive.tar --overwrite

Not shabby at all for something I pulled out of my rear Easy Bake Oven, eh?

Here’s graphs of the results:

When attempting to benchmark a filesystem, there’s so much choice out there. We know there should be a performance hit when running an encrypted filesystem, and we can look at all the graphs we want, but in the end, you’ll see a 5-10% degradation in speed when running an encrypted drive.

That’s it for now, I’ve got an interview lined up with Sean Moss-Pultz, CEO of OpenMoko later this week… Did I mention now is a good time to subscribe to my RSS feed? Let’s get that ol’ counter on the side a notch over 1k, shall we?


State of Affairs: The Linux Girlfriend Project – Two Months In The Trenches

About 2 months ago, I convinced my girlfriend to try out Linux for a month after a really nasty bit of spyware infected her computer. This isn’t a bash on Microsoft, but it happened twice in about a month.

Unfortunately, a roommate who pays a portion of the internet bill (and thus welcome to use the computer, which is located in the living room) likes to browse nefarious websites, I’ll let you speculate what type of websites he visits. Browsing those kind of sites by itself is completely fine by me, as long as it doesn’t fsck up the computer. The spyware/adware/etcware attracting behaviors of the roommate got some nasty sh!t on the computer – TWICE in a month. I tried to get everyone in the house to use Firefox, and that went over well.

Even if the shady websites are surfed upon using Firefox, if you install software from said websites, it’s all over anyways. I can’t help stupid user syndrome. Quite simply, I had enough at that point, I’m sure as hell not going to backup, fdisk, format and reinstall Windows every two weeks. Plus, I really don’t want to have that uncomfortable conversation with the roommate about his online behaviors, to be honest.

Push came to shove, and my girlfriend let me install the operating system of my choosing, since I would be the one supporting it.

We tried out Puppy Linux because it has good performance on older computers. That didn’t go so well. Getting wireless to work with ndiswrapper was a real pain in the neck in Puppy.

Eventually, I ended up settling on Linux Mint, and a few customizations later, the computer had an uncanny resemblance of the Windows XP operating system it replaced – “Start Menu,” Bliss background and everything. It was good enough to fool both roommates… they’ve NEVER mentioned a peep about any problems at all. As a test, I asked one of them (who were unaware of the switch) if they noticed any problems with the computer since I had been working on it.

“No problems at all, it’s been great. Ever since you changed it to Firefox it runs faster once it’s loaded up.”

I don’t even think they know they’re using Linux, to be honest.

There have been a few issues that presented themselves over the past two months after the switch, but they were minor cases of vendor lock in – both Blackberry and the iPhone don’t happily sync or upgrade firmware easily on Linux. Technically, it is possible to get them working. Are those righteous hacks usable by a mere mortal? No way in hell! So we ended up settling on a dual-boot scenario.

There have been a few issues other than the phone synchronizing, such as an Adobe Flash or “IE Only Website” render problem here and there, but other than those, nothing notable has come up in quite some time.

Overall, I’d say it’s been a success. These .exe files keep popping up on the desktop every once in awhile… but as the saying goes… No harm, no foul.

Addendum: She loves The GIMP, and has spent many hours playing klickety, which is an addictive bejeweled-ish color matching to clear the board game. Also, there seems to be a graphical driver issue where the computer seems a little bit sluggish. I think a video card change would fix that up nicely.


Using Shred to Wipe Hard Drives – DoD Uses It – You Should Too!

Hard drive encryption (and to a lesser extent, secure hard drive erasing) has been in the news regarding child pornography crossing the US border, government officials around the world erasing incriminating emails and other nefarious headlines.

The one that caught my eye was of a person crossing the border from Canada into the United States.

His laptop was searched in customs. Border Agents found all kinds of pornography on the computer (including child pornography) and arrested the owner of the laptop. Later on, when the computer-savvy investigators went to collect evidence on the computer, they were presented with an interesting prompt when they attempted to access the Z: drive.

Enter your PGP passphrase:

PGP is short for “pretty good privacy.” PGP is so good, in fact even the government can’t break into the files on the computer. The investigators couldn’t collect any evidence.

When a grand jury instructed the man to provide the passphrase, the judge cited the 5th Amendment and agreed with the defense that he cannot be compelled to divulge the passphrase. According to the 5th Amendment, he cannot be obligated to testify to his own detriment. In court, the man said he “wasn’t sure” if child porn was on the computer. He admitted to downloading lots of pornography, and if he found kiddie porn, he deleted it.

Sensational stories like these tend to cast a negative light on tools like encryption and permanently erasing files. If you’ve got nothing to hide, you wouldn’t use these tools… right?


The more paranoid folks among us would immediately be able to name a dozen or more reasons. I’ll just name a few real-life senarios that happen every day to people all over the world, often times due to unfortunate events such as a company or government laptop going missing, hell… even a 1.44MB floppy disc contained a hundred thousand records of some government employees in the UK.

Here are the bigger ones…

1. Identity theft.
2. That “home video” you promised to destroy.
3. Selling a hard drive or other rewritable storage device.

That said, GNU shred which is part of coreutils is a great tool to delete those sorts of files we all probably use at one time or another.

Here’s a snippet of the manual:

shred – overwrite a file to hide its contents, and optionally delete it

shred [OPTIONS] FILE […]

Overwrite the specified FILE(s) repeatedly, in order to make it harder
for even very expensive hardware probing to recover the data.

Mandatory arguments to long options are mandatory for short options

-f, –force
change permissions to allow writing if necessary

-n, –iterations=N
Overwrite N times instead of the default (25)

-s, –size=N
shred this many bytes (suffixes like K, M, G accepted)

-u, –remove
truncate and remove file after overwriting

-v, –verbose
show progress

-x, –exact
do not round file sizes up to the next full block;

this is the default for non-regular files

-z, –zero
add a final overwrite with zeros to hide shredding

And the comments from shred.c, written by Colin Plumb:

* Do a more secure overwrite of given files or devices, to make it harder
* for even very expensive hardware probing to recover the data.
* Although this process is also known as "wiping", I prefer the longer
* name both because I think it is more evocative of what is happening and
* because a longer name conveys a more appropriate sense of deliberateness.
* For the theory behind this, see "Secure Deletion of Data from Magnetic
* and Solid-State Memory", on line at
* Just for the record, reversing one or two passes of disk overwrite
* is not terribly difficult with hardware help. Hook up a good-quality
* digitizing oscilloscope to the output of the head preamplifier and copy
* the high-res digitized data to a computer for some off-line analysis.
* Read the "current" data and average all the pulses together to get an
* "average" pulse on the disk. Subtract this average pulse from all of
* the actual pulses and you can clearly see the "echo" of the previous
* data on the disk.

* Real hard drives have to balance the cost of the media, the head,
* and the read circuitry. They use better-quality media than absolutely
* necessary to limit the cost of the read circuitry. By throwing that
* assumption out, and the assumption that you want the data processed
* as fast as the hard drive can spin, you can do better.

Oh, the things you can find in source code header comments…. that’s fantastic commentary from someone who knows about the topic.

I ran a 6-pass random wipe (followed by a round of zeros) on a 160GB hard drive as well as a 2GB flash drive that were laying around the house. I’m sure there was all sorts of personal information on those. The 160GB drive took about 7 hours to finish and the 2GB thumb drive took about 26 minutes to complete.

I’ll be writing an article on using PGP to encrypt a hard drive in Linux in the near future, but in the mean time, enjoy the video of my 2GB flash drive loosing another 7 write cycles off it’s rewrite capacity. 🙂


Transcoding MTS/M2TS AVCHD Video Into AVI Files with Free Software

dscn1233.JPGThis holiday season, I was an extremely lucky recipient of a Sony HDR-SR5 video camera with a whopping 40GB hard drive inside.

When I opened it, the first question in my head was not atypical of a Linux users’ train of thought:

“Is it compatible with Linux?”

The answer was a tiny bit more complicated than I imagined, and immersed me into the otherwise unknown world of video transcoding, which is something millions of people do every day but probably never think about it. The best example is on YouTube, where it happens behind the scenes in the Google computing cloud, and everything “just works.”

I’ve read a few things about the format the camera uses, called AVCHD. Even in Windows (or OSX for that matter) many people have headaches with the AVCHD format, so I figure that Linux users are probably going to be having a helluva time trying to get it to work.

The files the camera generates are compressed at an incredible 15:1 ratio. I’ve seen it mentioned that a Standard Definition compressed clip of is actually larger in file size than an HD clip in AVCHD format.

The compression is where a majority of the AVCHD format complaints come originate from, since there is not much out there in terms of being able to edit video stored in the format without uncompressing it first. It takes some rather respectable computing horsepower to actually convert the files into an uncompressed format that can be played with in a video editing software package. Using the method detailed here, it takes about 5-6 minutes to uncompress and transcode a minute of video, on an Intel Core2Duo running at 3.2GHz with 4GB of RAM and a 10,000RPM hard drive – OUCH!

When plugging in the Sony HDR-SR5, it functions as a USB Mass Storage Device. Copying files from the camera to the computer is a simple drag n’ drop affair. This would be a familiar process to anyone who has used a digital camera and imports pictures to their computer.

It would certainly be faster to playback the videos in real-time and capture them using the component outputs from the camera… but that wouldn’t be a challenge. I’ve never actually installed the software that came with the Sony Handycam, so I’m not sure what I’m missing out on, but it doesn’t seem like all that much, to be honest. I wouldn’t be surprised if it was the typical Sony software suite that is about as useless as the media it is distributed on.

Let’s get down to the meaty bits, shall we?

The first thing I searched for was “AVCHD Linux” and found a post over at AVS Forum. On a side note, my Google-Fu is really spot on lately.

That post is a gold mine. It literally contains everything we need to transcode the video into formats of our choosing that are usable in non-linear video editing programs such as Avidemux or Cinelerra.

m2tstoavi is the best information I have found (so far!) to uncompress the AVCHD video files on my camcorder and convert them into a .AVI file in Linux.

Installation is very simple, all it requires in Ubuntu is a quick apt-get install:

sudo apt-get install mplayer ffmpeg x264 faad2 faac a52dec mencoder faad libfaad2-0 libfaad2-dev subversion csh

Then edit the install script as noted in the README to grab the newest version of ldecod. Everything else is completely automated – nice job Axel!

I did need to modify the settings a tiny bit to actually get audio synchronized as well as fix a possible issue with the script itself. I had problems until I commented out the echo ffmpeg yadda yadda part. When modified, it runs the following commands:

xporthdmv -hn $file 1 1 1 && mv bits0001.mpa $audiofile
ldecod -i bits0001.mpv -o $videofifo &
ffmpeg -r 29.97 -s 1440x1080 -i $videofifo -i $audiofile -vcodec mpeg4 -sameq -acodec copy -aspect 16:9 -b 15000k

I’m using the m2tstoavi.fifo script that uses named pipes (denoted above with dollar signs) to stream the output of one command into another. It is efficient with disk space and does not fill up the drive as the regular m2tstoavi script does, not sure why the fifo version is not the default. Ah well, nothing is perfect.

Once installed, I just run this command to convert every MTS video in the current directory to .AVI files:

./m2tstoavi.fifo *

It turns this proprietary format mess:


Into precious memories:


All in all, I’m glad I don’t need proprietary software for this conversion. I really wish it were faster – but here’s the point of this exercise:

I never used Windows software, so I don’t know what I’m missing. I’m happy with the solution I have, and if 20 people comment on this post and say they can edit HD video in realtime, good for them.

If they do, then I would have a better reason to upgrade to a quad-core CPU.

Here’s a (VERY unpolished!) video talking about the process in a little bit more detail, and showing the actual encoding process and the finished result. I’m not responsible for anybody tripping out at the end on my extremely crazily bad camera skills in the rendered video.

Oh yeah, and for those of you who were wondering, I also received quite possibly one of the best books I’ve read in a long time… it’s all about how cool Eric S. Raymond is ninjas are.