Categories
Linux

HowTo: Setup and Benchmark Encrypted Partitions in Ubuntu

In a previous article, I talked about using shred to securely delete files. Now we’ll delve into using encrypted volumes in Linux to secure our data in the first place, so that we don’t need to use programs like shred. Along the way, we’ll benchmark the raw performance of an encrypted volume and compare the […]

In a previous article, I talked about using shred to securely delete files. Now we’ll delve into using encrypted volumes in Linux to secure our data in the first place, so that we don’t need to use programs like shred. Along the way, we’ll benchmark the raw performance of an encrypted volume and compare the results to an unencrypted volume and see just what kind of real world compromises we see.

To start out we need free space on a drive that isn’t partitioned, or enough patience to resize an existing one. Just about everything here needs root privileges, since we’re working with filesystems. It would be easiest to start a root terminal withsudo su, then enter your password.

First, we install the tools to get the encrypted partition going:
apt-get install cryptsetup hashalot gparted

Next, we use gparted to create a 20GB partition at the end of my disk. It’s a dead simple drag n’ drop application similar in function to Partition Magic or other GUI partition editors… hopefully you don’t need instructions. Make sure to record the name of the new partition! Everything here that says /dev/sda2 is going to change based on your hardware and partitioning scheme.

After that completes (which can take some time if any resizing or moving of an existing partition happens), we need to set a password.
cryptsetup --verbose --verify-passphrase luksFormat /dev/sda2

This command will create a device called /dev/mapper/sda2 and give us access to the encrypted volume after verifying the password:
cryptsetup luksOpen /dev/sda2 sda2

By now we’re knee deep in waist-high water. I’m not quite sure what that means… I just made it up. Say it out loud… rolls off the tongue. Sorry… where was I? Ah right. I’ll try to explain where we’re at right now, for my benefit as well as yours.

At this moment, we have a partition called /dev/sda2. That raw partition now has an encrypted container inside, located at /dev/mapper/sda2. The last step is to actually format the encrypted volume so we can actually put some files on there. This can also be done in gparted if you want to split things up into multiple partitions, use the drive dropdown box to find the mapper.
/sbin/mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/sda2

Next, we’ll make a directory to mount the encrypted volume and then actually mount it:
mkdir /mnt/test
mount /dev/mapper/sda2 /mnt/test

Now we can copy files into /mnt/test and every file located there will be encrypted. Sweet!
To unmount the volume, use the following commands:
umount /mnt/test
cryptsetup luksClose sda2

I bet you’re asking the question we all are… How fast is it? Good question. The answer is a pain in the ass to be honest. This almost ended up being two separate articles because the benchmarking was not going very well… but here we go… how to benchmark hard drives in Linux with FOUR different tools:

Method:
The first plan I had was to perform two separate clean installs on an entire disk, run several benchmarks and quote some articles on how hard it would be to crack into the encrypted disk. Those results followed the expected trend of a significant (approximately 10%) degradation in read, write, and seek times on the encrypted volume. However, those results could be tainted, because they were run in a graphical environment that had lots going on in the background. I decided to throw those out. More accurate results tied directly to actual performance can be achieved by installing a text-mode only system, and using a separate partition at the end of the disk. By using failsafe text mode, we’ll limit the number of extra services/daemons/etc running.

Testbed:
Processor: Intel E6400 Core2Duo 3.2 GHz
Hard Drive: Western Digital 150GB RaptorX 10,000 RPM
RAM: 4GB, no swap partition used.

Benchmarks Used:
Bonnie++ 1.03
Options: bonnie -s 14176 -d /mnt/test
Bonnie++ is a benchmark suite that is aimed at performing a number of simple tests of hard drive and file system performance. The -s 14176 option sets the program to use is four times the amount of memory available on our testbed, which is the recommended setting. This is to make sure the OS is not doing any sort of caching in RAM to skew results. -d /mnt/test sets the program to use /mnt/test as the location to save the temporary file. Bonnie is a nice benchmark, but it’s got a problem. The results are nearly indecipherable to read by someone unfamiliar with the output, and this page helped me read them.

bonnie.png

PostMark
Options: set size 10000 10000000 (10KB - 10MB pseudo-randomly sized files)
set number 2000 (2000 generated files)
set transactions 2500 (2500 read/write/etc actions made on those files)
run
quit

PostMark is a benchmark for servers. We can use it and gain some additional insight into how a server would function if it were working on 10KB to 10MB sized files… it’s a benchmark that would need to be customized for the application desired to gain any direct correlations from the results.

pm1.png
pm2.png

IOZone3 3.279
Options used:
sudo iozone -a -R -g 10g -R -f /mnt/test/iozone
Iozone is useful for determining a filesystem performance. The benchmark tests file I/O performance for the following operations: Read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read/write, pread/pwrite variants. The options setup the following variables: Auto Mode, Excel/CSV formatted results, create a 10GB test file on /mnt/test called iozone.

IOZone outputs a TON of data, and they have lots of pretty graphs on their website, but there’s more than enough analysis already for this article. Here are the two Excel (???) formatted files for your pleasure. No graphs by are included in the output by default – shame really, since their graphs look great. iozone.zip

Easy-Bake Tar-Gzip-Gunzip-Untar Oven Benchmark Test
For the last test, we’ll use a home-grown benchmark using tar to archive 3.5GB of highly compressed HD videos, gzip the archive, then unzip and untar the compressed tar.gz file into the current directory, forcibly overwriting the existing files. This constitutes a “real-world” scenario more than the above benchmarks with reads, writes, re-writes and plenty of seeking. We can easily count how long each operation takes to complete and get a really quick, dirty and simple comparison with time counted for each operation. The fact that we’re working with the same data four times… do the math – at least 3.5GB of data traversing to and fro across the subsystems of the testbed 8 times (input/output cycles for each action) for around 28GB of data flying about. The unencrypted data manipulation test completed 42 seconds faster, for a 6% lead over the encrypted filesystem.

time tar -cf archive.tar *.MTS && time gzip archive.tar && time gunzip archive.tar.gz -f && time tar xvf archive.tar --overwrite

Not shabby at all for something I pulled out of my rear Easy Bake Oven, eh?

Here’s graphs of the results:
snapshot2.jpg
ebo-resize.png

Conclusion:
When attempting to benchmark a filesystem, there’s so much choice out there. We know there should be a performance hit when running an encrypted filesystem, and we can look at all the graphs we want, but in the end, you’ll see a 5-10% degradation in speed when running an encrypted drive.

That’s it for now, I’ve got an interview lined up with Sean Moss-Pultz, CEO of OpenMoko later this week… Did I mention now is a good time to subscribe to my RSS feed? Let’s get that ol’ counter on the side a notch over 1k, shall we?

44 replies on “HowTo: Setup and Benchmark Encrypted Partitions in Ubuntu”

Very nice article Wayne. I was expecting to see a greater hit to performance than just 5-10%. I wonder about performance across a RAID…

Nice article πŸ™‚
I was using a whole encrypted system (except of /boot) a long time now and wondered why my firefox is so damn slow.
As my new notebook recieves in the next time, I decided to make a clean install on my desktop without encryption.

You really can feel a performance boost, when “unencrypting” the system.

Thats sad but it doesn’t matter. I will set up an encrypted system on my notebook, when it arrives.

Interesting, but could you try to show CPU usage when using the encrypted file system? perhaps the speed hit is not noticeable because you’ve got a fast processor.

Thanks everyone, you all rock!

Tavo – I’ll see if I can graph CPU usage (second by second) on my easy bake oven test and maybe Bonnie and post some graphs… I’ll turn off the 2nd core to see if the CPU usage maxes out… it was running all the way up to 75%ish during the easy bake test

The encrypted filesystem used like 2% more CPU on average.

lhr: aes-loop is what you are looking for, you can make an encrypted file that can be mounted as a filesystem. I dont know about converting an existing filesystem to an encrypted one.

Hmmmm, I think I’ll just reformat in that case and move the data back onto the hard drive. I just installed Kubuntu on there, so there shouldn’t be much junk. Thanks for your help!

Considering the scrutiny that “stolen laptops” are getting recently, coming up with a clean and efficient method of encrypting as much of the disk space as possible would seem to me a priority both in business, and for personal use. Not that anyone would want to put a backdoor or key logger on my machine, or plant kiddy-porn. Nooo, can’t imagine.

Maybe a boot image on a USB key, which then switches over to the HD, which is entirely encrypted, once the correct encryption key has been provided? Take out the USB key once the system has finished booting, and if the machine is stolen the thief gets _nothing_, not even the swap partition would be in the clear.

I’m very pleased that there was only a 5-15% performance hit, but not all of us have multi-core systems. That may very well be the source of your success, and certainly something to consider when buying the next laptop. If I didn’t enjoy following Debian Unstable with the latest kernel, I would have done something encrypted a long time ago.

As others have mentioned, RAID performance would be interesting, although I doubt RAID will make any difference. It’s not the reading/writing that is changing, just the introduction of one more transaction on the data being read/written.

If you made an unencrypted /boot and /usr you could have the rest of the disk encrypted, right? Just keep anything you care about out of those partitions and you’d be good to go.

I’m a little curious about the swap partition myself. Does the crypt stuff ever cache anything in swap? I would hope not, but clearing out the swap seems to be a crucial thing to do as well.

And speaking of swap, what about resume images? Do we just turn off sleep modes? This sounds like an issue for laptops.

John,
I’ve been using my encrypted laptop for a while now, and it works with suspend to disk just fine. Upon the next boot, kernel still prompts for the key, and then reads in the image and you are back to where you were.

Of course, depending on how paranoid you are, if you are looking to have a very secure system, you either have to be physically guarding it – or it has to be off. Resume from a s3 suspend need not re-prompt for the disk key, because the disk is never closed.

It is essential that swap be an encrypted partition, of course. Aside from your /home, it is the most important thing to encrypt. You could use a separate partition for that, or if you use an encrypted root, one of the LVM logical volumes inside should be your swap.

Regarding performance issues. I’m glad to see the 5-10% numbers here, that isn’t bad for what you are getting. As others have indicated, other than disk speeds, CPU is what seems to matter to me. On a Core2, you have boatloads of power and most of the CPU’s time is idly spent anyways. Thing is, with a less powerful system, cpu usage and disk speed are strongly tied. Run a program, disk access drops – copy a large file and your programs crawl. You don’t have the spare CPU in that case.

So I might add that for a modern system, luks full disk makes sense. But for older hardware it may be wiser to just encrypt your swap and maybe tmp, and use something like truecrypt for your important files, or encrypt /home. You still get rock solid encryption for important data, your browser cache, your settings, and the nice thing about encrypting just swap is that you need not use a key. Randomly generate one on boot and discard on shutdown. Entering a lengthy passphrase every boot is the small price to pay (in addition to disk performance) for security. Using a key can make this simpler however.

Ehi, IMHO we’re getting out of the track…

Can anyone tell me just ONE kind of big file we have to encrypt? Or can we guess that the only things that matter to us to encrypt are basically text files?

Because the only example I can make of large file to crypt, is some “home-made video” you want not to be seen, but this is real exception. I think for most system all we CARE to crypt can stay in a separate partition (let’s call it “personal”) of max 2-5GB, with very low I/O load…

Plus the swap, obviously – ehi Waine, nice trick the 4GB ram πŸ˜€ it’s the first time I see a REAL use for a so huge amount of it.

I don’t care if anyone sees how I tweeked my /etc/fstab, but I want to hear why you seem to, maybe I’m loosing something. Thanks!

Jaba: It’s not just ones pr0n stash or home made video, though most home users may see the need for that particular application. Encryption is more to keep your data safe from as many people as possible. You decided whether or not it’s important enough to protect your photos, journals, source codes, financial information, documents and so on safe.

Bob Roberson: I pretty sure you’re spot on with the RAID mostly affecting read/write rates. RAID would affect encryption performance only in that it could be a bottle neck if it wasn’t serving the data up as fast as your machine was crunching it.

Jaba,
It would be a bit of a mistake to assume that you only encrypt your data if there is something in some particular naughty or illegal data you have on it.

The old argument, “I don’t have anything to hide.” approaches this the wrong way. I forget who it was, but someone said that their counterargument to that was to simply ask for the other persons wallet. I don’t have anything to hide, but if my laptop were to get lost with all of the data on it unencrypted, someone can find a lot about me, and that is just creepy. I don’t want someone going through all my (PG) home movies, gigabytes of photos of family and friends, reading my journals, snooping to see what I do, what web sites I go to, any odd financial information.

You are right that, you probably could get by encrypting files one by one, ala pgp. You could also use a truecrypt to make a big hidden partition in a file to store certain things in there. You could use aes-loop to encrypt your home tmp and swap. But computers are still very bad at keeping information in one place. A hardened unix system can go a long way, but the caches, spools, buffers, temporary files, backups – information tends to “leak”. An attacker can use that information to get to everything, potentially. It is much easier to just know that the entire disk is encrypted, all that any one can see really is that it has one partition on it (or two in case of a /boot partition).

To elaborate a little on that last point. Grub cannot read encrypted partitions yet, and I don’t think it will be able to do so with stage 1. I don’t think they are even working on it for Grub 2. So if you put Grub in your MBR, and you want it to boot your kernel, you need that in a separate unencrypted partition, usually a 100MB little thing. The kernel once loaded is large and smart enough to prompt for the passphrase to get the system going.

There is a security risk in this, theoretically. Against a lost laptop, you are still fine mostly. Sure, someone knows you use linux and perhaps which version, and whatever other operating systems are on there – they make also know some of what you do based on how you have customized your kernel. If you prettyize grub with a splash and colors, then they get “information” about you, but essentially you are good here. The problem is against someone that actually wants to get to your info. It could be possible to steal the laptop, replace the kernel or grub or any of that with a compromised version – say, one that looks the same to you, but saves your passphrase into a hidden place on disk, or connects to a remote server when available and reports your phrase. So for a really secure installation, boot from a usb key, use two factor authentication both on the key and in your mind.

Of course, this would be too much to guard my pg movies, so I’m fine with /boot. But full disk encryption does NOT imply “something to hide”. I can’t tell you how much peace of mind I have now, with full disk encryption, a proper and sane backup policy, and my laptop on my homeowners insurance. A lost laptop for me is now an “ah shucks” affair and a call to my insurance agency to get some cash to buy a new one, where then I do a restore. As opposed to a “#@%&*%!&^&!!!!!!!!” situation that it was before.

@PredatoryFern:
Quote: “photos, journals, source codes, financial information, documents and so on”
Beside photos, that’s what I mean for “personal documents”. And photos can be merged with my “videos” thesis. Thanks for your intention, but it seems to me you didn’t answered me.

@JST:
I have to answer to you… ehm… “inline”? πŸ˜€

if there is something in some particular naughty or illegal
NEVER said that… I encrypt my hd work partition because I have things on it more valuable than the laptop itself. That’s all, I don’t know how did you come to say that…

You are right that, you probably could get by encrypting files one by one, ala pgp
That sounds crazy for me too, and never said that…

I don’t want someone going through all my (PG) home movies, gigabytes of photos of family and friends, reading my journals, snooping to see what I do, what web sites I go to, any odd financial information.
Well, I simply have it all well backupped and safely stored… And besides, it’s a little ODD that a home thief (or anyone storing a laptop) would care of anything ’bout your personal life… I’m no famous no-one, and thieves historically go for GOLD, if you can understand what I mean…

caches, spools, buffers, temporary files, backups – information tends to β€œleak”
Like before, it simply seems we have different ideas of “what to hide” and “why”. Beside this, I beg you pardon, are we talking of the same OS? Everyone keeps telling me that personal information are GRANTED to be stored in /home, temporary files on /tmp, and the only other treat would be swap… Do you know anything I don’t? Please feel free to illuminate me.

@anyone else wanting to answer me:
Please, I must re-iterate my question to be very clear:
Can anyone tell me just:
ONE kind
of big file
WE
HAVE
to encrypt?
Or why would you encrypt the
ENTIRE ROOT?
I mean, /usr, /bin, /lib, /boot, /sbin, /opt, /dev, etc.

I would really appreciate if anyone can find one kind of file that anyone less than paranoiac would reputate eligible for encryption. Something that anyone reading this simple and usefull guide would read and say: YES, I need to encrypt a large partition (or root) to be sure my personal data cannot be stolen from me Can we suppose a laptop user, to have high risk to be stolen and hopefully not the entire family album and video history? πŸ˜€
Or anyone that could really explain me why would anyone (me?) care about root encryption?

Thanks a lot, really

-Jaba

Jaba, it’s not about big files. It’s about little things. I prefer to encrypt as much as possible because it means that what I think is important is in there with everything else, especially the things I don’t know about.

At the least you _must_ encrypt swap if you are going to care about anything. Encrypt by partition unless you do not think you’re going to leave unencrypted file-fragments.

@Bob:

Thanks for your advice. My theory, by now, is that what I need (what everyone normally needs about encryption) is to encrypt a little partition (just some Gigs), in wich to store just the personal files. Plus, obviously, swap partition and eventually /tmp (this depends on what you want to keep safe, and through wich application will pass).

With Linux, one of the best features I found, is that there’s NOTHING you can’t know. By now, I am SURE everything I want to be safe is in my encrypted partition. But I may be wrong.
My question is posted here because I hope someone who really knows things me and you don’t know (Wayne himself, maybe) could tell me if I am wrong or not about that. I appreciate your two cents, but that doesn’t adds any value to anything also said.

Last thing: I really can’t understand what do you mean by “unencrypted file-fragments”. I can imagine something similar about swap, but we have said it must be encrypted, so it can’t be. I’m waiting for your explaination, thanks again.

@ Wayne: sorry this conversation isn’t being much constructive. I hope this isn’t bothering you, and I appreciate very much the space you are giving us. Thanks to you too.

Wow. Lots of comments on this article. TrueCrypt 5.0 just got released today. Linux GUI added and it’s not longer affected by upgrades to the kernel. For Windows users, they can now encrypt with pre-boot authentication.

This article was extremely useful but, being a noob and not too verbose with Unix/Linux command line statements yet, I’d like to ask this: Can you encrypt a folder within a file system?
If so, how do you decrypt the same folder at a later stage so no passwords are required?

@Jaba:

Hopefully I can answer your question about why / should be encrypted:

1) Leftover file fragments:
Lets say you’re working on a private document on your desktop. You finish for the day and copy this document to your secure partition, then unmount it. You delete the original document and your data is safe.

Or is it?

Deleting a file only removes the inode (the filesystem’s index to where the file is stored). The actual (private) data is still on the disk, and some trivial computer forensics (google photorec) will recover the deleted file.

If you know what you’re doing, you might securely wipe the file instead of just deleting it, because a secure wipe destroys the data on disk before removing the inode, but even that won’t guarantee your safety. Many programs write temporary files in the course of operation, and they don’t always put their files in /tmp. If the environment variable TMPDIR is set the program will often prefer that directory (an attacker could trick you into setting TMPDIR perhaps). vim will write a .swp file in the same directory as the file you are editing. gimp will store its tile cache in ~/.gimp*/ etc etc. You can bet these files are *not* securely wiped and fragments of your original file can still be recovered (possibly the whole file).

2) secure by default
You could possibly keep track of all these messy temp files and make sure you wipe them all the time, but it takes discipline, even for trivial cases. If you screw up once your data could be leaked. Encrypting the root reverses the trend: your data will be safe by default. If you want better performance you can always put your /usr directory in its own unencrypted partition, or only encrypt /home, /tmp and swap, but I don’t want people knowing what software I have installed anyway (why does he have john the ripper on there? he *must* be up to something!)

Finally, you asked for an example large files that have to be kept private: Right now I have about 45-50GB of videos and photos on my PC, most of which isn’t mine. I have permission to keep all of them for personal use, but if my laptop gets stolen and someone starts selling my associates content then I’ve failed to protect their commercial interests. Other examples: work-related database dumps or source code repositories, scientific data, the ever pervasive pr0n (including legally downloaded), the cache of your local http proxy, etc etc.

Sorry for the long post. I hope this gives you some idea of why we might want / encrypted πŸ˜‰

Cheers,

Darren

@Darren

GREAT – GREAT – GREAT

Thank you so much – that was a simply perfect answer, clear and complete. You’ve completely answered all my doubts – thanks!.
Now I now how to act to secure my work – best of all, I know WHY.

By now, my configuration is:
1. no swap
2. tmpfs for all temporary dirs
3. auto-clean firefox cache/cookies/history on shutdown
4. Desktop folder encrypted, work folder as subdirectory
5. work backups on encrypted secure digital

I feel enough safe for my purpose, and works like a charm on my eeepc – which anyway wasn’t designed for large files or photo/video editing πŸ˜‰ thanks a lot, you’ve been very clear about leftover fragments.

Thanks everybody for your support – and to Waine for the place
Hope this discussion will be useful to many

-Jaba

ps btw, the inode/data part isn’t really as simple as shown…
http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html
go down to Q: How can I recover (undelete) deleted files from my ext3 partition?
ext3 does enough the trick in the background… not safe if someone is stealing your pc looting for data, but safe enough for common use.

@Jaba

Glad I could be of assistance πŸ™‚ and thanks for the link, I didn’t know ext3 did this.

It’s interesting to note that *all* of the file remains on disk until it gets overwritten – what’s being deleted is the list of locations where the file is (or was) stored. While this makes undeletion much harder, a dedicated attacker can still recover the entire file with enough time, provided another file hasn’t overwritten it. The filesystem makes the attacker’s job easier by attempting to write the file to disk in consecutive blocks (for best performance). Smaller files especially will tend to be kept intact, making them especially vulnerable.

Cheers,

Darren

@Darren

I agree ^_^

But – I assure you – “he” must be a VERY dedicated attacker. I lost myself big chunks of data multiple times, and I was never able of recover it. Maybe for smaller files could really be a lot simpler.

Thanks again

-Jaba

@Jaba

Yes, big files would be a tough job (I feel your pain, I’ve lost my data several times before).

Small files are easier – I lost 6 hours of work once when I accidentally deleted a file (on an ext3 partition) and was able to recover it in about 60 minutes using nothing but grep and an old copy of the file.

Cheers,

Darren

Hey Wayne,

I was wondering what you used to generate your graphs. It looks a bit too polished to be gnuplot. Did whatever you used just slurp up, for example, the bonnie++ .csv results, or did you have to write a script to do the heavy lifting?

Way late here…

I would suggest that one good reason for encrypting the root filesystem on a laptop is to prevent installation of a trojan horse that could subsequently steal/hijack/whatever -anything- you do on the laptop.

Scenario
(1) You leave your laptop in a hotel room, office, whatever.
(2) An attacker gains access via social engineering, forcible entry, whatever.
(3) The attacker installs a trojan in the boot sequence of the laptop. He could do this even w/ BIOS passwords, login passwords, etc. All he has to do is remove the hard drive from the laptop, plug it into a USB device and then access it as an additional hard drive on another laptop. He wouldn’t even need to load your Linux system–he just needs to be able to read/write whatever type of filesystem your root is.
(4) He puts everything back and leaves.
(5) Now you load your laptop. Even w/ encrypted home, once you’ve mounted /home he has access to it all via his trojan, and obviously he could get whatever you do on the web.

You could halt this process at steps (1)/(2) by always maintaining physical security of your laptop. This is probably the cheapest/easiest solution in most cases, but might not be possible in your situation.

Or, you could halt this process at step (3) by encrypting your root filesystem. Arguably, in order for this to work, you’d probably need to keep your /boot on a USB key which would always be required for booting the computer. Otherwise, your unencrypted /boot could be hijacked.

My two cents.

With this config:

M/B SuperMicro X7SBE-О LGA775, i3210
CPU Intel(R) Xeon(R) X3370 @ 3.00GHz
RAID controller 3ware 9690SA-8I (OEM) PCI-E x8
HDD 73 Gb SAS Hitachi 15000rpm 16Mb 6 pcs. in RAID10
RAM 2Gb ECC CL6 x 4

I have about 330 – 350 MB/s secquental reading and writing, but with ecryption speed fall to ~120 MB/s writing and ~99MB/s reading.

By using this (http://www.holtznet.de/luks/) small test I saw that max enryption speed can be reached on my system with this settings: “-c aes-xts-plain -s 256”
write: 134 MB/s
read: 144 MB/s
but this values is terribly slow, its about 280% slower than without encryption.

Therefore when disk perfomance is above some speed, “bottleneck” is CPU perfomance and the fact that dm-crypt uses only one core.

Very useful information on the dm-crypt and related issues.
http://osdir.com/ml/linux.kernel.device-mapper.dm-crypt/2006-11/msg00018.html

By now I spend about a week, and still didn’t found any decision how increase encryption speed.

Any suggestion?

Leave a Reply

Your email address will not be published. Required fields are marked *