Pipeline (software)
In
Narrowly speaking, a pipeline is linear and one-directional, though sometimes the term is applied to more general flows. For example, a primarily one-directional pipeline may have some communication in the other direction, known as a
Implementation
Pipelines are often implemented in a multitasking OS, by launching all elements at the same time as processes, and automatically servicing the data read requests by each process with the data written by the upstream process. This can be called a multiprocessed pipeline. In this way, the scheduler will naturally switch the CPU among the processes so as to minimize its idle time. In other common models, elements are implemented as lightweight threads or as coroutines to reduce the OS overhead often involved with processes. Depending on the OS, threads may be scheduled directly by the OS or by a thread manager. Coroutines are always scheduled by a coroutine manager of some form.
Read and write requests are usually blocking operations. This means that the execution of the source process, upon writing, is suspended until all data can be written to the destination process. Likewise, the execution of the destination process, upon reading, is suspended until at least some of the requested data can be obtained from the source process. This cannot lead to a deadlock, where both processes would wait indefinitely for each other to respond, since at least one of the processes will soon have its request serviced by the operating system, and continue to run.
For performance, most operating systems implementing pipes use pipe
Such a buffer command can be implemented using
Some notable examples of pipeline software systems include:
- RaftLib – C/C++ Apache 2.0 License
VM/CMS and z/OS
Traditional application programs on IBM mainframe operating systems have no standard input and output streams to allow redirection or piping. Instead of spawning processes with external programs, CMS Pipelines features a lightweight dispatcher to concurrently execute instances of more than 200 built-in programs that implement typical UNIX utilities and interface to devices and operating system services. In addition to the built-in programs, CMS Pipelines defines a framework to allow user-written
Data on IBM mainframes typically resides in a record-oriented filesystem and connected I/O devices operate in record mode rather than stream mode. As a consequence, data in CMS Pipelines is handled in record mode. For text files, a record holds one line of text. In general, CMS Pipelines does not buffer the data but passes records of data in a lock-step fashion from one program to the next. This ensures a deterministic flow of data through a network of interconnected pipelines.
Object pipelines
Beside byte stream-based pipelines, there are also object pipelines. In an object pipeline, processing elements output objects instead of text.
Pipelines in GUIs
Graphical environments such as
For instance, a user browsing the
Conceptually, this method could be used with a conventional save dialog box, but this would require the user's programs to have an obvious and easily accessible location in the filesystem. As this is often not the case, GUI pipelines are rare.
Other considerations
The name "pipeline" comes from a rough analogy with physical plumbing in that a pipeline usually[1] allows information to flow in only one direction, like water often flows in a pipe.
Pipes and
The concept of pipeline is also central to the Cocoon web development framework or to any XProc (the W3C Standards) implementations, where it allows a source stream to be modified before eventual display.
This pattern encourages the use of text streams as the input and output of programs. This reliance on text has to be accounted when creating
See also
- Anonymous pipe
- Component-based software engineering
- Flow-based programming
- GStreamer for a multimedia framework built on plugin pipelines
- Graphics pipeline
- Iteratees
- Named pipe, an operating system construct intermediate to anonymous pipe and file.
- Pipeline (computing) for other computer-related versions of the concept.
- Kahn process networks to extend the pipeline concept to a more generic directed graph structure
- Pipeline (Unix) for details specific to Unix
- Plumber – "intelligent pipes" developed as part of Plan 9
- Producer–consumer problem – for implementation aspects of software pipelines
- Software design pattern
- Stream processing
- XML pipeline for processing of XML files
Notes
- ^ There are exceptions, such as "broken pipe" signals.
- ^ "Monadic I/O and UNIX shell programming".