Comments on: More on alignment, ext2, and partitioning on SSDs

By: chris

chris — Sat, 28 Aug 2010 19:35:24 +0000

First off let me say I love your blog. I find myself asking the same questions everyday. Have you looked into the parallel capabilities of modern flash drives? I don’t have access to many modern SSD, but to get the r/w speed they claim there must be some parallelization going on. Chances are, however, that even if parallelization (r/w to multiple chips at the same time, like RAID) is happening the wear-leveling will nullify any intelligent decisions made.

By: Brian Smith

Brian Smith — Wed, 21 Oct 2009 14:45:50 +0000

I did some more research. It seems that 2.6.28 definitely doesn’t have support for 4K logical sectors. Patches were sent to LKML in the 2.6.29/2.6.30 timeframe but I don’t know if they were ever integrated in any release. In 2.6.31, Linux can interogate the disk to determine what its native sector size is, but that doesn’t mean 2.6.31 supports 4K sector sizes fully enough to affect performance. And, I couldn’t find anything in the 2.6.29, 2.6.30, or 2.6.31 changelogs indicating 4K sectors support at all.

I did find some discussions that were inconclusive about whether it was *safe* to use 512-byte logical sectors on a disk with 4096 byte physical sectors. A few people are saying that 512-on-4096 emulation is NOT safe for magnetic disks. Whether 512-on-4096 emulation is safe for an SSD depends on the SSD controller. It seems most SSD controllers are using a (mostly-)append-only log-structured filesystem internally; in that case, it seems like 512-on-4096 emulation is safe. I that safety-critical applications should always write using the physical sector size (4096 bytes for most SSDs today) for maximum atomicity/safety. I also heard someone mention that many SSDs–especially laptop/desktop class SSDs–can’t/won’t expose 4096-byte sector I/O operations to the OS, only doing 512-on-4096 emulation. I don’t if that is true.

It seems to me that, for most SSDs, a 4K write is always going to be non-destructive if it fails. IMO, ATA should have a mechanism for the disk to say “writes of XXXXk (e.g. 128k) or smaller are non-destructive if they fail” and/or expose a non-destructive write operation. That would enable *huge* performance optimizations for VMMs, filesystems, and databases. Maybe ATA already has one or both of those already; it is something that should be looked into.

I don’t have a machine that I can reformat appropriately to run those tests. That’s why I’m so interesting in running your tests.

By: uberVU - social comments

uberVU - social comments — Wed, 21 Oct 2009 06:43:22 +0000

Social comments and analytics for this post…

This post was mentioned on Twitter by patrichards: RT @rethinkdb We’ve released more benchmarks that test alignment, ext2, and partitioning on SSDs. Read more: http://bit.ly/1×92Zd…

By: Slava

Slava — Tue, 20 Oct 2009 18:59:50 +0000

Brian, I tried various memory alignments (from 512B to 128K) and it didn’t make a difference. It’s probably a good idea to align it to max(VM page size, block size tested) – I’ll push the code to do it in a bit.

In the meantime, if you have an SSD, would you mind trying the following:

# rebench /dev/sdb # this sets block size and stride to 512B
# rebench -b 4096 -s 4096 /dev/sdb

What results do you get (on what hardware and kernel)? When I move from 512B block size to 4K I see a performance drop on SuperTalent SSD, as well as Intel SLC. On the current test machine they’re connected via SATA directly to the motherboard (I’m not sure what board and SATA controller this machine has, I’ll find out tonight). I’m on Ubuntu 2.6.28-14-server.

By: Brian Smith

Brian Smith — Tue, 20 Oct 2009 16:52:18 +0000

Please include the exact command line you used to run rebench in your posts. Also, what kind of hardware configuration are you using?

I think it is really strange that 4096-byte reads are slower than 512-byte reads. I think it is so strange I dug into the code.

I notice you hard-code HARDWARE_BLOCK_SIZE to 512 and then use this for buffer alignment with posix_memalign. I recommend aligning on max(8192, max(VMM page size, block size being tested)) boundaries instead. I’m not sure it would make a difference but it may affect DMA transfers from the disk–especially if the OS/driver is trying to align all reads on memory page boundaries AND disk page boundaries. Imagine you have a 4K read aligned on a disk page but not aligned on a RAM page. I could easily see this requiring a DMA transfer into a kernel buffer, and then a memcpy into your buffer. If everything is aligned exactly right, you should be able to get a DMA directly into your buffer. However, historically there’ve been all kinds of issues in Linux that have prevented zero-copy I/O like this; I don’t know if all of them have been resolved.