Newline
This article needs additional citations for verification. (February 2016) |
A newline (frequently called line ending, end of line (EOL), next line (NEL) or line break) is a control character or sequence of control characters in character encoding specifications such as ASCII, EBCDIC, Unicode, etc. This character, or a sequence of characters, is used to signify the end of a line of text and the start of a new one.[1]
History
In the mid-1800s, long before the advent of
Later, in the age of modern teleprinters, standardized character set control codes were developed to aid in white space text formatting. ASCII was developed simultaneously by the International Organization for Standardization (ISO) and the American Standards Association (ASA), the latter being the predecessor organization to American National Standards Institute (ANSI). During the period of 1963 to 1968, the ISO draft standards supported the use of either CR+LF or LF alone as a newline, while the ASA drafts supported only CR+LF.
The sequence CR+LF was commonly used on many early computer systems that had adopted Teletype machines—typically a Teletype Model 33 ASR—as a console device, because this sequence was required to position those printers at the start of a new line. The separation of newline into two functions concealed the fact that the print head could not return from the far right to the beginning of the next line in time to print the next character. Any character printed after a CR would often print as a smudge in the middle of the page while the print head was still moving the carriage back to the first position. "The solution was to make the newline two characters: CR to move the carriage to column one, and LF to move the paper up."[2] In fact, it was often necessary to send extra padding characters—extraneous CRs or NULs—which are ignored but give the print head time to move to the left margin. Many early video displays also required multiple character times to scroll the display.
On such systems, applications had to talk directly to the Teletype machine and follow its conventions since the concept of device drivers hiding such hardware details from the application was not yet well developed. Therefore, text was routinely composed to satisfy the needs of Teletype machines. Most minicomputer systems from DEC used this convention. CP/M also used it in order to print on the same terminals that minicomputers used. From there MS-DOS (1981) adopted CP/M's CR+LF in order to be compatible, and this convention was inherited by Microsoft's later Windows operating system.
The
).Representation
The concepts of carriage return (CR) and line feed (LF) are closely associated and can be considered either separately or together. In the physical media of typewriters and printers, two axes of motion, "down" and "across", are needed to create a new line on the page. Although the design of a machine (typewriter or printer) must consider them separately, the abstract logic of software can combine them together as one event. This is why a newline in character encoding can be defined as CR
and LF
combined into one (commonly called CR+LF
or CRLF
).
Some character sets provide a separate newline character code. EBCDIC, for example, provides an NL character code in addition to the CR and LF codes. Unicode, in addition to providing the ASCII CR and LF control codes, also provides a "next line" (NEL) control code, as well as control codes for "line separator" and "paragraph separator" markers.
Operating system | Character encoding | Abbreviation | hex value | dec value | Escape sequence |
---|---|---|---|---|---|
POSIX standard oriented systems: Unix and Unix-like systems (Linux, macOS, *BSD, AIX, Xenix, etc.), QNX 4+, Multics, BeOS, Amiga, RISC OS, and others[5] |
ASCII | LF | 0A | 10 | \n |
Symbian OS, Palm OS, Amstrad CPC , and most other early non-Unix and non-IBM operating systems
|
CR LF | 0D 0A | 13 10 | \r\n | |
Commodore 8-bit machines (C64, C128), Acorn BBC, ZX Spectrum, TRS-80, Apple II series, Oberon, the classic Mac OS, HP Series 80, MIT Lisp Machine, and OS-9 | CR | 0D | 13 | \r | |
Acorn BBC[6] and RISC OS spooled text output[7] | LF CR | 0A 0D | 10 13 | \n\r | |
QNX pre-POSIX implementation (version < 4) | RS
|
1E | 30 | \036 | |
Atari 8-bit machines
|
ATASCII | EOL | 9B | 155 | |
OS/400 )
|
EBCDIC | NL | 15 | 21 | \025 |
ZX80 and ZX81 (home computers from Sinclair Research Ltd) | ZX80/ZX81 proprietary encoding | 76 | 118 |
- OS/400)—use NL (New Line, 0x15)[8] as the character combining the functions of line feed and carriage return. The equivalent Unicode character (, which stores text files as one record per line. In most file formats, no line terminators are actually stored.
0x85
) is called NEL (Next Line). EBCDIC also has control characters called CR and LF, but the numerical value of LF (0x25) differs from the one used by ASCII (0x0A). Additionally, some EBCDIC variants also use NL but assign a different numeric code to the character. However, those operating systems use a record-based file system - Operating systems for the CDC 6000 series defined a newline as two or more zero-valued six-bit characters at the end of a 60-bit word. Some configurations also defined a zero-valued character as a colon character, with the result that multiple colons could be interpreted as a newline depending on position.
- RSX-11 and OpenVMS also use a record-based file system, which stores text files as one record per line. In most file formats, no line terminators are actually stored, but the Record Management Services facility can transparently add a terminator to each line when it is retrieved by an application. The records themselves can contain the same line terminator characters, which can either be considered a feature or a nuisance depending on the application. RMS not only stores records, but also stores metadata about the record separators in different bits for the file to complicate matters even more (since files can have fixed length records, records that are prefixed by a count or records that are terminated by a specific character). The bits are not generic, so while they can specify that CRLF or LF or even CR is the line terminator, they can not substitute some other code.
- Fixed line length was used by some early mainframe operating systems. In such a system, an implicit end-of-line was assumed every 72 or 80 characters, for example. No newline character was stored. If a file was imported from the outside world, lines shorter than the line length had to be padded with spaces, while lines longer than the line length had to be truncated. This mimicked the use of punched cards, on which each line was stored on a separate card, usually with 80 columns on each card, often with sequence numbers in columns 73–80. Many of these systems added a carriage control character to the start of the next record; this could indicate whether the next record was a continuation of the line started by the previous record, or a new line, or should overprint the previous line (similar to a CR). Often this was a normal printing character such as
#
that thus could not be used as the first character in a line. Some early line printers interpreted these characters directly in the records sent to them.
Communication protocols
Many communications protocols have some sort of new line convention. In particular, protocols published by the Internet Engineering Task Force (IETF) typically use the ASCII CRLF sequence.
In some older protocols, the new line may be followed by a checksum or parity character.
Unicode
The Unicode standard defines a number of characters that conforming applications should recognize as line terminators:[9]
- LF: Line Feed, U+000A
- Vertical Tab, U+000B
- Form Feed, U+000C
- CR: Carriage Return, U+000D
- CR+LF: CR (U+000D) followed by LF (U+000A)
- NEL: Next Line, U+0085
- LS: Line Separator, U+2028
- PS: Paragraph Separator, U+2029
While it may seem overly complicated compared to an approach such as converting all line terminators to a single character (e.g. LF), because Unicode is designed to preserve all information when converting a text file from any existing encoding to Unicode and back (round-trip integrity), Unicode needs to make the same distinctions between line breaks made by other encodings.
For example:
Recognizing and using the newline codes greater than 0x7F (NEL, LS and PS) is not often done. They are multiple bytes in UTF-8, and the code for NEL has been used as the ellipsis (…
) character in Windows-1252. For instance:
- ECMAScript accepts LS and PS as line breaks,[13] but considers U+0085 (NEL) whitespace instead of a line break.[14]
- JSON[15] allows LS and PS characters within strings, while ECMAScript prior to ES2019[16][17] treated them as newlines, and therefore illegal syntax.[18]
- YAML[19] no longer recognizes them as line breaks as of version 1.2, in order to be compatible with JSON.
- Windows Notepad, the default text editor of Microsoft Windows, does not treat any of NEL, LS, or PS as line breaks.
- gedit, the default text editor of the GNOME desktop environment, treats LS and PS, but not NEL, as line breaks.
The Unicode special characters U+2424 (SYMBOL FOR NEWLINE, 
), U+23CE (RETURN SYMBOL, ⏎
), U+240D (SYMBOL FOR CARRIAGE RETURN, ␍
) and U+240A (SYMBOL FOR LINE FEED, ␊
) are glyphs intended for presenting a user-visible character to the reader of the document, and are thus not recognized themselves as a newline.
In programming languages
To facilitate the creation of portable programs, programming languages provide some abstractions to deal with the different types of newline sequences used in different environments.
The C programming language provides the escape sequences '\n' (newline) and '\r' (carriage return). However, these are not required to be equivalent to the ASCII LF and CR control characters. The C standard only guarantees two things:
- Each of these escape sequences maps to a unique implementation-defined number that can be stored in a single char value.
- When writing to a file, device node, or socket/fifo in text mode, '\n' is transparently translated to the native newline sequence used by the system, which may be longer than one character. When reading in text mode, the native newline sequence is translated back to '\n'. In binary mode, no translation is performed, and the internal representation produced by '\n' is output directly.
On Unix platforms, where C originated, the native newline sequence is ASCII LF (0x0A), so '\n' was simply defined to be that value. With the internal and external representation being identical, the translation performed in text mode is a no-op, and Unix has no notion of text mode or binary mode. This has caused many programmers who developed their software on Unix systems simply to ignore the distinction completely, resulting in code that is not portable to different platforms.
The C library function
Another common problem is the use of '\n' when communicating using an Internet protocol that mandates the use of ASCII CR+LF for ending lines. Writing '\n' to a text mode stream works correctly on Windows systems, but produces only LF on Unix, and something completely different on more exotic systems. Using "\r\n" in binary mode is slightly better.
Many languages, such as
Java, PHP,[21] and Python[22] provide the '\r\n' sequence (for ASCII CR+LF). In contrast to C, these are guaranteed to represent the values U+000D and U+000A, respectively.
The Java I/O libraries do not transparently translate these into platform-dependent newline sequences on input or output. Instead, they provide functions for writing a full line that automatically add the native newline sequence, and functions for reading lines that accept any of CR, LF, or CR+LF as a line terminator (see BufferedReader.readLine()). The System.lineSeparator() method can be used to retrieve the underlying line separator.
Example:
String eol = System.lineSeparator();
String lineColor = "Color: Red" + eol;
Python permits "Universal Newline Support" when opening a file for reading, when importing modules, and when executing a file.[23]
Some languages have created special
Example in C#:
string eol = Environment.NewLine;
string lineColor = "Color: Red" + eol;
string eol2 = "\n";
string lineColor2 = "Color: Blue" + eol2;
Issues with different newline formats
The different newline conventions cause text files that have been transferred between systems of different types to be displayed incorrectly.
Text in files created with programs which are common on Unix-like or classic Mac OS, appear as a single long line on most programs common to MS-DOS and Microsoft Windows because these do not display a single line feed
or a single carriage return
as a line break.
Conversely, when viewing a file originating from a Windows computer on a Unix-like system, the extra CR may be displayed as a second line break, as ^M, or as <cr> at the end of each line.
Furthermore, programs other than text editors may not accept a file, e.g. some configuration file, encoded using the foreign newline convention, as a valid file.
The problem can be hard to spot because some programs handle the foreign newlines properly while others do not. For example, a
Even if a program supports different newline conventions, these features are often not sufficiently labeled, described, or documented. Typically a menu or combo-box enumerating different newline conventions will be displayed to users without an indication if the selection will re-interpret, temporarily convert, or permanently convert the newlines. Some programs will implicitly convert on open, copy, paste, or save—often inconsistently.
Most textual
The standard Internet Message Format[26] for email states: "CR and LF MUST only occur together as CRLF; they MUST NOT appear independently in the body". Differences between SMTP implementations in how they treat bare LF and/or bare CR characters have led to SMTP spoofing attacks referred to as "SMTP smuggling".[27]
The File Transfer Protocol can automatically convert newlines in files being transferred between systems with different newline representations when the transfer is done in "ASCII mode". However, transferring binary files in this mode usually has disastrous results: any occurrence of the newline byte sequence—which does not have line terminator semantics in this context, but is just part of a normal sequence of bytes—will be translated to whatever newline representation the other system uses, effectively corrupting the file. FTP clients often employ some heuristics (for example, inspection of filename extensions) to automatically select either binary or ASCII mode, but in the end it is up to users to make sure their files are transferred in the correct mode. If there is any doubt as to the correct mode, binary mode should be used, as then no files will be altered by FTP, though they may display incorrectly.[28]
Conversion between newline formats
Text editors are often used for converting a text file between different newline formats; most modern editors can read and write files using at least the different ASCII CR/LF conventions.
For example, the editor Vim can make a file compatible with the Windows Notepad text editor. Within vim
:set fileformat=dos
:wq
Editors can be unsuitable for converting larger files or bulk conversion of many files. For larger files (on Windows NT/2000/XP) the following command is often used:
D:\>TYPE unix_file | FIND /V "" > dos_file
Special purpose programs to convert files between different newline conventions include unix2dos and dos2unix, mac2unix and unix2mac, mac2dos and dos2mac, and flip.[29] The tr command is available on virtually every Unix-like system and can be used to perform arbitrary replacement operations on single characters. A DOS/Windows text file can be converted to Unix format by simply removing all ASCII CR characters with
$ tr -d '\r' < inputfile > outputfile
or, if the text has only CR newlines, by converting all CR newlines to LF with
$ tr '\r' '\n' < inputfile > outputfile
The same tasks are sometimes performed with
if the platform has a Perl interpreter:$ awk '{sub("$","\r\n"); printf("%s",$0);}' inputfile > outputfile # UNIX to DOS (adding CRs on Linux and BSD based OS that haven't GNU extensions)
$ awk '{gsub("\r",""); print;}' inputfile > outputfile # DOS to UNIX (removing CRs on Linux and BSD based OS that haven't GNU extensions)
$ sed -e 's/$/\r/' inputfile > outputfile # UNIX to DOS (adding CRs on Linux based OS that use GNU extensions)
$ sed -e 's/\r$//' inputfile > outputfile # DOS to UNIX (removing CRs on Linux based OS that use GNU extensions)
$ perl -pe 's/\r?\n|\r/\r\n/g' inputfile > outputfile # Convert to DOS
$ perl -pe 's/\r?\n|\r/\n/g' inputfile > outputfile # Convert to UNIX
$ perl -pe 's/\r?\n|\r/\r/g' inputfile > outputfile # Convert to old Mac
The file command can identify the type of line endings:
$ file myfile.txt
myfile.txt: ASCII English text, with CRLF line terminators
The Unix egrep (extended grep) command can be used to print filenames of Unix or DOS files (assuming Unix and DOS-style files only, no classic Mac OS-style files):
$ egrep -L '\r\n' myfile.txt # show UNIX style file (LF terminated)
$ egrep -l '\r\n' myfile.txt # show DOS style file (CRLF terminated)
Other tools permit the user to visualise the EOL characters:
$ od -a myfile.txt
$ cat -e myfile.txt
$ cat -v myfile.txt
$ hexdump -c myfile.txt
Interpretation
Two ways to view newlines, both of which are
In text intended primarily to be read by humans using software which implements the
Reverse and partial line feeds
RI (
Similarly, PLD (U+008B PARTIAL LINE FORWARD, decimal 139) and PLU (U+008C PARTIAL LINE BACKWARD, decimal 140) can be used to advance or reverse the text printing position by some fraction of the vertical line spacing (typically, half). These can be used in combination for subscripts (by advancing and then reversing) and superscripts (by reversing and then advancing), and may also be useful for printing diacritics.
See also
References
- ^ "What is a Newline?". www.computerhope.com. Retrieved 10 May 2021.
- ISBN 9780735710016. Archived from the original(PDF) on 8 April 2022. Retrieved 4 January 2023.
- ZDNet. Archived from the originalon 13 May 2018. Retrieved 4 January 2023.
[A]fter decades of frustration, and having to download a real text editor to change a single line in a config file from a Linux box, Microsoft has updated Notepad to be able to handle end of line characters used in Unix, Linux, and macOS environments.
- ^ Lopez, Michel (8 May 2018). "Introducing extended line endings support in Notepad". Windows Command Line. Archived from the original on 6 April 2019. Retrieved 4 January 2023.
As with any change to a long-established tool, there's a chance that this new behavior may not work for your scenarios, or you may prefer to disable this new behavior and return to Notepad's original behavior. To do this, you can change [...registry keys...] to tweak how Notepad handles pasting of text, and which EOL character to use when Enter/Return is hit
- ^ Kahn-Greene, Will Guaraldi. "ASCII chart". bluesock.org.
- ISBN 978-0946827008. Retrieved 30 January 2019.
- ^ "Character Output". RISC OS 3 Programmers' Reference Manual. 3QD Developments Ltd. 3 November 2015. Retrieved 18 July 2018.
- ^ IBM System/360 Reference Data Card, Publication GX20-1703, IBM Data Processing Division, White Plains, NY
- ^ Heninger, Andy (20 September 2013). "UAX #14: Unicode Line Breaking Algorithm". The Unicode Consortium.
- ^ "C1 Control Character Set of ISO 6429" (PDF). ITSCJ. IPSJ. 1 October 1983. Retrieved 3 March 2022.
- ^ Control Functions for Coded Character Sets (PDF) (Report). ECMA International. June 1991.
- ^ Character Code Structure and Extension Techniques (PDF) (Report) (6th ed.). ECMA International. December 1994.
- ^ "ECMAScript 2019 Language Specification". ECMA International. June 2019. 11.3 Line Terminators.
- ^ "ECMAScript 2019 Language Specification". ECMA International. June 2019. 11.2 White Space.
- .
- ^ "Subsume JSON (a.k.a. JSON ⊂ ECMAScript)". GitHub. 22 May 2018.
- ^ "ECMAScript 2019 Language Specification". ECMA International. June 2019. 11.8.4 String Literals.
- ^ "ECMAScript 2018 Language Specification". ECMA International. June 2018. 11.8.4 String Literals.
- ^ "5.4. Line Break Characters". YAML Ain't Markup Language revision 1.2.2. 1 October 2021.
- ^ "binmode". Perl documentation. Perl 5 Porters.
- ^ "PHP: Strings - Manual". PHP Manual. The PHP Group.
- ^ "2. Lexical analysis". The Python Language Reference. The Python Foundation.
- ^ "What's new in Python 2.3". Python Software Foundation.
- ^ "PHP: Predefined Constants - Manual". PHP Manual. The PHP Group.
- ^ Bernstein, D. J. "Bare LFs in SMTP".
- .
- ^ Longin, Timo (18 December 2023). "SMTP Smuggling - Spoofing E-Mails Worldwide". SEC Consult.
- ^ Zeil, Steven (19 January 2015). "File Transfer". Old Dominion University. Archived from the original on 14 May 2016.
When in doubt, transfer in binary mode.
- ^ Sapp, Craig Stuart. "ASCII text converstion between UNIX, Macintosh, MS-DOS". Center for Computer Research in Music and Acoustics. Archived from the original on 9 February 2009.
- ^ "C1 Controls and Latin-1 Supplement" (PDF). unicode.org. Retrieved 13 February 2016.
External links
- The Unicode reference; see paragraph 5.8 in Chapter 5 of the Unicode 4.0 standard (PDF)
- "The [NEL] Newline Character".
- The End of Line Puzzle
- Line counter based on Newline Character
- Understanding Newlines at the Wayback Machine (archived 20 August 2006)
- "The End-of-Line Story"