sync (Unix)
sync is a standard
As a function in C, the sync()
call is typically declared as void sync(void)
in <unistd.h>
. The system call is also available via a command line utility also called sync, and similarly named functions in other languages such as Perl and Node.js (in the fs module).
The related system call fsync()
commits just the buffered data relating to a specified file descriptor.[1] fdatasync()
is also available to write out just the changes made to the data in the file, and not necessarily the file's related metadata.[2]
Some Unix systems run a kind of flush or update daemon, which calls the sync function on a regular basis. On some systems, the cron daemon does this, and on Linux it was handled by the pdflush daemon which was replaced by a new implementation and finally removed from the Linux kernel in 2012.[3] Buffers are also flushed when filesystems are unmounted or remounted read-only,[4] for example prior to system shutdown.
Some applications, such as LibreOffice, are also call sync function to save recovery information in an interval.
Database use
In order to provide proper
fsync()
and fdatasync()
,[5] in order for commits to be durable.[6] Unfortunately, for any single client writing a series of records, a rotating hard drive can only commit once per rotation, which makes for at best a few hundred such commits per second.[7]Databases also employ transaction log files (typically much smaller than the main data files) that have information about recent changes, such that changes can be reliably redone in case of crash; then the main data files can be synced less often.
Error reporting and checking
To avoid any data loss return values of fsync()
should be checked because when performing I/O operations that are buffered by the library or the kernel, errors may not be reported at the time of using the fsync()
, msync()
or close()
.[8] Prior to 2018, Linux's fsync()
behavior under certain circumstances failed to report error status,[9][10] change behavior was proposed on 23 April 2018.[11]
Performance controversies
Hard disks may default to using their own volatile write cache to buffer writes, which greatly improves performance while introducing a potential for lost writes.[12] Tools such as hdparm -F will instruct the HDD controller to flush the on-drive write cache buffer. The performance impact of turning caching off is so large that even the normally conservative FreeBSD community rejected disabling write caching by default in FreeBSD 4.3.[13]
In
Firefox 3.0, released in 2008, introduced fsync
system calls that were found to degrade its performance; the call was introduced in order to guarantee the integrity of the embedded SQLite database.[18]
fsync
.[19] He also concedes however (quoting Mike ShaverOn some rather common Linux configurations, especially using the ext3 filesystem in the "data=ordered" mode, calling fsync doesn't just flush out the data for the file it's called on, but rather on all the buffered data for that filesystem.[20]
See also
References
- ^ fsync specification
- ^ fdatasync specification
- ^ "R.I.P. Pdflush [LWN.net]".
- ^ "mount - Does umount calls sync to complete any pending writes". Unix & Linux Stack Exchange. Retrieved 2021-05-02.
- ^ Vondra, Tomas (2 February 2019). "PostgreSQL vs. fsync". Osuosl Org. Archived from the original (mp4) on 10 February 2019. Retrieved 10 February 2019.
- ^ PostgreSQL Reliability and the Write-Ahead Log
- ^ Tuning PostgreSQL WAL Synchronization Archived 2009-11-25 at the Wayback Machine
- ^ "Ensuring data reaches disk [LWN.net]".
- ^ "PostgreSQL's fsync() surprise [LWN.net]".
- ^ "Improved block-layer error handling [LWN.net]".
- ^ "Always report a writeback error once - Patchwork". Archived from the original on 2018-05-04. Retrieved 2018-05-03.
- ^ Write-Cache Enabled?
- ^ FreeBSD Handbook — Tuning Disks
- ^ Marshall Kirk McKusick. "Disks from the Perspective of a File System - ACM Queue". Queue.acm.org. Retrieved 2014-01-11.
- ISBN 978-1-84951-031-8.
- ^ "Enabling FUA for SATA drives (Was Re: [RFC][PATCH] libata: Enable SATA disk fua detection on default) (Linux SCSI)".
- ^ "Linux-Kernel Archive: [PATCH RFC] libata: FUA updates".
- ^ "Shaver » fsyncers and curveballs". Archived from the original on 2012-12-09. Retrieved 2009-10-15.
- ^ "Don't fear the fsync!".
- ^ "Delayed allocation and the zero-length file problem".