ext4 slower than tmpfs

By
While you might expect ext4 to be slower than tmpfs, I found it is slower to have buffered writes to ext4 than tmpfs. i.e. The writes are not actually made to disk in either case.

I have a test which writes data/messages via a shared memory file as an IPC between two threads.  The data written is smaller than the dirty data cache size.  I have used these kernel parameters on a machine with 32 GB. The test only writes 8 - 12 GB so it should fit within space easily

    vm.dirty_background_ratio = 50
    vm.dirty_ratio = 80

I have mounted the file system with the following.  The disk is an PCI SSD, but this shouldn't make a difference.

     noatime,data=writeback,barrier=0,nobh

This is the results I get (using a beta version of Chronicle 2.0)


filesystemtiny
  4 bytes  		small
   16 byte   		 medium
64 byte		large
   256 byte  
 tmpfs 185 M
msg/sec		96 M
msg/sec		30.7 M
msg/sec		10.9 M
msg/sec
  ext4148 M
msg/sec		71 M
msg/sec		17.7 M
msg/sec		7.2 M
msg/sec

The units are millions or messages per second.  The median of three tests are listed.

I was wonder why "writeback" writing to ext4 is significantly slower than using tmpfs. I was wondering has any else seen this problem? Any suggestions welcome.

Comments

  1. Unknown3 July 2013 at 10:05

    Hi,

    (Third attempt to post this: Blogspot is convinced that I'm a robot.)

    Two feeble ideas for you to chuck away:

    1) ext4 may be doing lots of extra book-keeping, eg to make sure state is consistent through a crash.

    2) tmpfs may benefit from having smaller block-size internally (maybe 32 bytes rather than (say) 512 bytes or 4kB), so may do less copying of data in memory 'around' the stuff you actually care about.

    Rgds

    Damon

    ReplyDelete
  2. trpaslik.pl4 July 2013 at 19:08

    Hi,
    The main difference between tmpfs/ramfs and any regular filesystem is that tmpfs and ramfs do not have any backing store. It's just an interface to the kernels caching mechanism used also by any filesystem. For any regular filesystem kernel has basically more work to do to deal with the backing store related operations. See https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt
    I am not a kernel guru, but this may explain your numbers.

    Kind Regards,
    Michał Šrajer

    ReplyDelete
  3. Anonymous11 July 2013 at 23:54

    I'm not surprised by this. The management of the tmpfs being entirely memory is trivial compared to ext4 which has to maintain various different superblock copies, journals, and the other complexities of a more mature file-system.

    ReplyDelete

Post a Comment

Popular posts from this blog

Java is Very Fast, If You Don’t Create Many Objects

By
Image
  You still have to watch how many objects you create. This article looks at a benchmark passing events over TCP/IP at 4 billion events per minute using the net.openhft.chronicle.wire.channel package in Chronicle Wire and why we still avoid object allocations..  One of the key optimisations is creating almost no garbage. Allocation is a very cheap operation and collection of very short-lived objects is also very cheap. Does this really make a difference? What difference does one small object per event (44 bytes) make to the performance in a throughput test where GC pauses are amortised? While allocation is as efficient as possible, it doesn’t avoid the memory pressure on the L1/L2 caches of your CPUs and when many cores are busy, they are contending for memory in the shared L3 cache.  Results Benchmark on a Ryzen 5950X with Ubuntu 22.10. JVM Vendor, Version No objects Throughput, Average Latency* One object per event Throughput, Average Latency* Azul Zulu 1.8.0_322 60.6 M event/s, 528
Read more

System wide unique nanosecond timestamps

By
A Unique Identifier can be very useful for tracing. Those ids are even more useful when they contain a high-resolution timestamp.  Not only do they record the time of an event, but if unique can help trace events as they pass through the system. Such unique timestamps however can be expensive depending on how they are implemented.   This post explores a lightweight means of producing a unique, monotonically increasing system-wide nano-second resolution timestamp available in our open-source library. Uses for Unique Identifiers Unique identifiers can be useful to associate with a piece of information so that information can be referred to later unambiguously. This could be an event, a request, an order id, or a customer id. They can naturally be used as a primary key in a database or key/value store to retrieve that information later. One of the challenges of generating these identifiers is avoiding creating duplicates while not having an increasing cost.  You could record every identi
Read more

Comparing Approaches to Durability in Low Latency Messaging Queues

By
Image
  A significant feature of Chronicle Queue Enterprise is support for TCP replication across multiple servers to ensure the high availability of application infrastructure. I generally believe that replicating data to a secondary system is faster than syncing to disk, assuming the round trip network delay wasn’t high due to quality networks and co-located redundant servers. This is the first time I have benchmarked it with a realistic example. Little’s Law and Why Latency Matters In many cases, the assumption is that the latency won't be a problem as long as throughput is high enough. However, latency is often a key factor in why the throughput isn’t high enough. Little’s law states, “ the long-term average number  L  of customers in a  stationary  system is equal to the long-term average effective arrival rate  λ  multiplied by the average time  W  that a customer spends in the system”. In computer terminology, the level of concurrency or parallelism a system has to support must be
Read more