Efficient Memory Mapping for Terabyte Sparse Files in Java
On Linux, you can create sparse files , where only the pages (of 4 KiB) that are touched utilise either memory or disk space. This allows you to memory map large virtual regions without worrying about wasted memory or disk In this program, you can see it reserves 8 TiB (8,192 GiB) Figure 1. Test 1: Sparse file Tip: x << y means x × 2 y therefore 1L << 10 = 1 KiB (1024 bytes), 1L << 20 = 1 MiB (1024 2 bytes), 1L << 30 = 1 GiB (1024 3 bytes), 1L << 40 = 1 TiB etc Using multiples of 10 for the shift makes them easier to read. 64L << 20 is 64 × 2 20 = 64 × 1024 2 = 64 MiB. The virtual memory size of the above process is just over 8192 GiB at 8200.7 GiB, but the RSS (Resident Set Size) is only 122,060 KB, or 122 MB. Figure 2. RES for Test 1 On disk, the extents reported are 8 TiB, however the amount of disk (and memory) actually used is just 20 KiB....