Memory-mapped file
A memory-mapped file is a segment of virtual memory[1] that has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource. This resource is typically a file that is physically present on disk, but can also be a device, shared memory object, or other resource that an operating system can reference through a file descriptor. Once present, this correlation between the file and the memory space permits applications to treat the mapped portion as if it were primary memory.
History
TOPS-20 PMAP
An early (c. 1969)
SunOS 4 mmap
SunOS 4[5] introduced Unix's mmap
, which permitted programs "to map files into memory."[1]
Windows Growable Memory-Mapped Files (GMMF)
Two decades after the release of TOPS-20's PMAP, Windows NT was given Growable Memory-Mapped Files (GMMF).
Since "CreateFileMapping
function requires a size to be passed to it" and altering
a file's size is not readily accommodated, a GMMF API was developed.[6][7] Use of GMMF requires
declaring the maximum to which the file size can grow, but no unused space is wasted.
Benefits
The benefit of memory mapping a file is increasing I/O performance, especially when used on large files. For small files, memory-mapped files can result in a waste of
Certain application-level memory-mapped file operations also perform better than their physical file counterparts. Applications can access and update data in the file directly and in-place, as opposed to seeking from the start of the file or rewriting the entire edited contents to a temporary location. Since the memory-mapped file is handled internally in pages, linear file access (as seen, for example, in
A possible benefit of memory-mapped files is a "lazy loading", thus using small amounts of RAM even for a very large file. Trying to load the entire contents of a file that is significantly larger than the amount of memory available can cause severe thrashing as the operating system reads from disk into memory and simultaneously writes pages from memory back to disk. Memory-mapping may not only bypass the page file completely, but also allow smaller page-sized sections to be loaded as data is being edited, similarly to demand paging used for programs.
The memory mapping process is handled by the
Types
There are two types of memory-mapped files:
Persisted
Persisted files are associated with a source file on a disk. The data is saved to the source file on the disk once the last process is finished. These memory-mapped files are suitable for working with extremely large source files.[10]
Non-persisted
Non-persisted files are not associated with a file on a disk. When the last process has finished working with the file, the data is lost. These files are suitable for creating shared memory for inter-process communications (IPC).[10]
Drawbacks
The major reason to choose memory mapped file I/O is performance. Nevertheless, there can be tradeoffs. The standard I/O approach is costly due to system call overhead and memory copying. The memory-mapped approach has its cost in minor page faults—when a block of data is loaded in page cache, but is not yet mapped into the process's virtual memory space. In some circumstances, memory mapped file I/O can be substantially slower than standard file I/O.[11]
Another drawback of memory-mapped files relates to a given architecture's
mmap also tends to be less scalable than standard means of file I/O, since many operating systems, including Linux, have a cap on the number of cores handling page faults. Extremely fast devices, such as modern NVM Express SSDs, are capable of making the overhead a real concern.[12]
I/O errors on the underlying file (e.g. its removable drive is unplugged or optical media is ejected, disk full when writing, etc.) while accessing its mapped memory are reported to the application as the SIGSEGV/SIGBUS signals on POSIX, and the EXECUTE_IN_PAGE_ERROR structured exception on Windows. All code accessing mapped memory must be prepared to handle these errors, which don't normally occur when accessing memory.
Only hardware architectures with an MMU can support memory-mapped files. On architectures without an MMU, the operating system can copy the entire file into memory when the request to map it is made, but this is extremely wasteful and slow if only a little bit of the file will be accessed, and can only work for files that will fit in available memory.
Common uses
Perhaps the most common use for a memory-mapped file is the process loader in most modern operating systems (including Microsoft Windows and Unix-like systems.) When a process is started, the operating system uses a memory mapped file to bring the executable file, along with any loadable modules, into memory for execution. Most memory-mapping systems use a technique called demand paging, where the file is loaded into physical memory in subsets (one page each), and only when that page is actually referenced.[13] In the specific case of executable files, this permits the OS to selectively load only those portions of a process image that actually need to execute.
Another common use for memory-mapped files is to share memory between multiple processes. In modern
Platform support
Most modern operating systems or runtime environments support some form of memory-mapped file access. The function
Some free portable implementations of memory-mapped files for Microsoft Windows and POSIX-compliant platforms are:
The Java programming language provides classes and methods to access memory mapped files, such as FileChannel
.
The D programming language supports memory mapped files in its standard library (std.mmfile module).[21]
Ruby has a gem (library) called Mmap, which implements memory-mapped file objects.
Since version 1.6,
For Perl there are several modules available for memory mapping files on the CPAN, such as Sys::Mmap[23] and File::Map.[24]
In the Microsoft .NET runtime,
PHP supported memory-mapping techniques in a number of native file access functions such as file_get_contents() but has removed this in 5.3 (see revision log).
For the
The
The Julia (programming language) has support for I/O of memory mapped binary files through the Mmap
module within the Standard Library.[26]
References
- ^ a b Chris Siebenmann (7 June 2018). "The history of Unix's confusing set of low-level ways to allocate memory".
- ^ Development began 1969, shipped 1976
- ^ "TOPS-20 Monitor Calls Reference Manual" (PDF).
- ^ "System 1022 Database System".
We had a PMAP cache for file I/O(like PA1050) in extended sections.
- ^ Dec. 1988
- ^ Jeffrey Richter (October 1995). "Add Growable Memory-Mapped Files to your App". Microsoft Systems Journal. pp. 17–28.
- ^ Creating a shared memory block that can grow in size
- ^ "Using mmap() for Advanced File I/O - BrainDump". Archived from the original on 7 August 2011. Retrieved 21 May 2011.
- ^ , "What Do Memory-Mapped Files Have to Offer?".
- ^ a b "Memory-Mapped Files". Microsoft Developer Network. Retrieved 4 January 2016.
- Matthew Dillon
- ^ Papagiannis, Anastasios; Xanthakis, Giorgos; Saloustros, Giorgos; Marazakis, Manolis; Bilas, Angelos (2020). Optimizing Memory-mapped I/O for Fast Storage Devices. USENIX ATC '20. pp. 813–827.
- ^ "Demand Paging"
- ^ Memory Mapped Files Archived 9 February 2007 at the Wayback Machine
- ^ Apple – Mac OS X Leopard – Technology – UNIX Archived 23 April 2009 at the Wayback Machine
- ^ CreateFileMapping Function (Windows)
- ^ "Sharing memory between processes: Memory Mapped Files". Boost.org.
- ^ "Memory-Mapped Files". Boost.org.
- ^ "Memory Mapped Files for Windows and POSIX systems". SourceForge.
- ^ "cpp-mmf". GitHub.
- ^ "std.mmfile - D Programming Language". Digital Mars. Retrieved 4 December 2011.
- ^ "New Modules in 1.6". Archived from the original on 30 December 2006. Retrieved 23 December 2008.
- ^ "Sys::Mmap Perl Module".
- ^ "File::Map Perl Module".
- ^ DotNet Archived 19 April 2010 at the Wayback Machine
- ^ "Memory-mapped I/O · The Julia Language". docs.julialang.org. Retrieved 3 September 2023.