ext4 slower than tmpfs

While you might expect ext4 to be slower than tmpfs, I found it is slower to have buffered writes to ext4 than tmpfs. i.e. The writes are not actually made to disk in either case.

I have a test which writes data/messages via a shared memory file as an IPC between two threads.  The data written is smaller than the dirty data cache size.  I have used these kernel parameters on a machine with 32 GB. The test only writes 8 - 12 GB so it should fit within space easily

    vm.dirty_background_ratio = 50
    vm.dirty_ratio = 80

I have mounted the file system with the following.  The disk is an PCI SSD, but this shouldn't make a difference.

     noatime,data=writeback,barrier=0,nobh

This is the results I get (using a beta version of Chronicle 2.0)


filesystemtiny
  4 bytes  
small
   16 byte   
 medium
64 byte
large
   256 byte  
 tmpfs 185 M
msg/sec
96 M
msg/sec
30.7 M
msg/sec
10.9 M
msg/sec
  ext4148 M
msg/sec
71 M
msg/sec
17.7 M
msg/sec
7.2 M
msg/sec

The units are millions or messages per second.  The median of three tests are listed.

I was wonder why "writeback" writing to ext4 is significantly slower than using tmpfs. I was wondering has any else seen this problem? Any suggestions welcome.

Comments

  1. Hi,

    (Third attempt to post this: Blogspot is convinced that I'm a robot.)

    Two feeble ideas for you to chuck away:

    1) ext4 may be doing lots of extra book-keeping, eg to make sure state is consistent through a crash.

    2) tmpfs may benefit from having smaller block-size internally (maybe 32 bytes rather than (say) 512 bytes or 4kB), so may do less copying of data in memory 'around' the stuff you actually care about.

    Rgds

    Damon

    ReplyDelete
  2. Hi,
    The main difference between tmpfs/ramfs and any regular filesystem is that tmpfs and ramfs do not have any backing store. It's just an interface to the kernels caching mechanism used also by any filesystem. For any regular filesystem kernel has basically more work to do to deal with the backing store related operations. See https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt
    I am not a kernel guru, but this may explain your numbers.

    Kind Regards,
    Michał Šrajer

    ReplyDelete
  3. I'm not surprised by this. The management of the tmpfs being entirely memory is trivial compared to ext4 which has to maintain various different superblock copies, journals, and the other complexities of a more mature file-system.

    ReplyDelete

Post a Comment

Popular posts from this blog

Low Latency Microservices, A Retrospective

Unusual Java: StackTrace Extends Throwable

System wide unique nanosecond timestamps