Parallel computing
Parallel computing is a type of computation in which many calculations or processes are carried out simultaneously.[1] Large problems can often be divided into smaller ones, which can then be solved at the same time. There are several different forms of parallel computing: bit-level, instruction-level, data, and task parallelism. Parallelism has long been employed in high-performance computing, but has gained broader interest due to the physical constraints preventing frequency scaling.[2] As power consumption (and consequently heat generation) by computers has become a concern in recent years,[3] parallel computing has become the dominant paradigm in computer architecture, mainly in the form of multi-core processors.[4]
Parallel computing is closely related to concurrent computing—they are frequently used together, and often conflated, though the two are distinct: it is possible to have parallelism without concurrency, and concurrency without parallelism (such as multitasking by time-sharing on a single-core CPU).[5][6] In parallel computing, a computational task is typically broken down into several, often many, very similar sub-tasks that can be processed independently and whose results are combined afterwards, upon completion. In contrast, in concurrent computing, the various processes often do not address related tasks; when they do, as is typical in distributed computing, the separate tasks may have a varied nature and often require some inter-process communication during execution.
Parallel computers can be roughly classified according to the level at which the hardware supports parallelism, with
In some cases parallelism is transparent to the programmer, such as in bit-level or instruction-level parallelism, but explicitly
A theoretical
Background
Traditionally,
Parallel computing, on the other hand, uses multiple processing elements simultaneously to solve a problem. This is accomplished by breaking the problem into independent parts so that each processing element can execute its part of the algorithm simultaneously with the others. The processing elements can be diverse and include resources such as a single computer with multiple processors, several networked computers, specialized hardware, or any combination of the above.
To deal with the problem of power consumption and overheating the major
An operating system can ensure that different tasks and user programmes are run in parallel on the available cores. However, for a serial software programme to take full advantage of the multi-core architecture the programmer needs to restructure and parallelize the code. A speed-up of application software runtime will no longer be achieved through frequency scaling, instead programmers will need to parallelize their software code to take advantage of the increasing computing power of multicore architectures.[14]
Amdahl's law and Gustafson's law
Optimally, the speedup from parallelization would be linear—doubling the number of processing elements should halve the runtime, and doubling it a second time should again halve the runtime. However, very few parallel algorithms achieve optimal speedup. Most of them have a near-linear speedup for small numbers of processing elements, which flattens out into a constant value for large numbers of processing elements.
The potential speedup of an algorithm on a parallel computing platform is given by Amdahl's law[15]
where
- Slatency is the potential speedup in latency of the execution of the whole task;
- s is the speedup in latency of the execution of the parallelizable part of the task;
- p is the percentage of the execution time of the whole task concerning the parallelizable part of the task before parallelization.
Since Slatency < 1/(1 - p), it shows that a small part of the program which cannot be parallelized will limit the overall speedup available from parallelization. A program solving a large mathematical or engineering problem will typically consist of several parallelizable parts and several non-parallelizable (serial) parts. If the non-parallelizable part of a program accounts for 10% of the runtime (p = 0.9), we can get no more than a 10 times speedup, regardless of how many processors are added. This puts an upper limit on the usefulness of adding more parallel execution units. "When a task cannot be partitioned because of sequential constraints, the application of more effort has no effect on the schedule. The bearing of a child takes nine months, no matter how many women are assigned."[16]
Amdahl's law only applies to cases where the problem size is fixed. In practice, as more computing resources become available, they tend to get used on larger problems (larger datasets), and the time spent in the parallelizable part often grows much faster than the inherently serial work.[17] In this case, Gustafson's law gives a less pessimistic and more realistic assessment of parallel performance:[18]
Both Amdahl's law and Gustafson's law assume that the running time of the serial part of the program is independent of the number of processors. Amdahl's law assumes that the entire problem is of fixed size so that the total amount of work to be done in parallel is also independent of the number of processors, whereas Gustafson's law assumes that the total amount of work to be done in parallel varies linearly with the number of processors.
Dependencies
Understanding data dependencies is fundamental in implementing parallel algorithms. No program can run more quickly than the longest chain of dependent calculations (known as the critical path), since calculations that depend upon prior calculations in the chain must be executed in order. However, most algorithms do not consist of just a long chain of dependent calculations; there are usually opportunities to execute independent calculations in parallel.
Let Pi and Pj be two program segments. Bernstein's conditions[19] describe when the two are independent and can be executed in parallel. For Pi, let Ii be all of the input variables and Oi the output variables, and likewise for Pj. Pi and Pj are independent if they satisfy
Violation of the first condition introduces a flow dependency, corresponding to the first segment producing a result used by the second segment. The second condition represents an anti-dependency, when the second segment produces a variable needed by the first segment. The third and final condition represents an output dependency: when two segments write to the same location, the result comes from the logically last executed segment.[20]
Consider the following functions, which demonstrate several kinds of dependencies:
1: function Dep(a, b) 2: c := a * b 3: d := 3 * c 4: end function
In this example, instruction 3 cannot be executed before (or even in parallel with) instruction 2, because instruction 3 uses a result from instruction 2. It violates condition 1, and thus introduces a flow dependency.
1: function NoDep(a, b) 2: c := a * b 3: d := 3 * b 4: e := a + b 5: end function
In this example, there are no dependencies between the instructions, so they can all be run in parallel.
Bernstein's conditions do not allow memory to be shared between different processes. For that, some means of enforcing an ordering between accesses is necessary, such as semaphores, barriers or some other synchronization method.
Race conditions, mutual exclusion, synchronization, and parallel slowdown
Subtasks in a parallel program are often called
Thread A | Thread B |
1A: Read variable V | 1B: Read variable V |
2A: Add 1 to variable V | 2B: Add 1 to variable V |
3A: Write back to variable V | 3B: Write back to variable V |
If instruction 1B is executed between 1A and 3A, or if instruction 1A is executed between 1B and 3B, the program will produce incorrect data. This is known as a race condition. The programmer must use a lock to provide mutual exclusion. A lock is a programming language construct that allows one thread to take control of a variable and prevent other threads from reading or writing it, until that variable is unlocked. The thread holding the lock is free to execute its critical section (the section of a program that requires exclusive access to some variable), and to unlock the data when it is finished. Therefore, to guarantee correct program execution, the above program can be rewritten to use locks:
Thread A | Thread B |
1A: Lock variable V | 1B: Lock variable V |
2A: Read variable V | 2B: Read variable V |
3A: Add 1 to variable V | 3B: Add 1 to variable V |
4A: Write back to variable V | 4B: Write back to variable V |
5A: Unlock variable V | 5B: Unlock variable V |
One thread will successfully lock variable V, while the other thread will be locked out—unable to proceed until V is unlocked again. This guarantees correct execution of the program. Locks may be necessary to ensure correct program execution when threads must serialize access to resources, but their use can greatly slow a program and may affect its reliability.[22]
Locking multiple variables using
Many parallel programs require that their subtasks
Not all parallelization results in speed-up. Generally, as a task is split up into more and more threads, those threads spend an ever-increasing portion of their time communicating with each other or waiting on each other for access to resources.[26][27] Once the overhead from resource contention or communication dominates the time spent on other computation, further parallelization (that is, splitting the workload over even more threads) increases rather than decreases the amount of time required to finish. This problem, known as parallel slowdown,[28] can be improved in some cases by software analysis and redesign.[29]
Fine-grained, coarse-grained, and embarrassing parallelism
Applications are often classified according to how often their subtasks need to synchronize or communicate with each other. An application exhibits fine-grained parallelism if its subtasks must communicate many times per second; it exhibits coarse-grained parallelism if they do not communicate many times per second, and it exhibits embarrassing parallelism if they rarely or never have to communicate. Embarrassingly parallel applications are considered the easiest to parallelize.
Flynn's taxonomy
Michael J. Flynn created one of the earliest classification systems for parallel (and sequential) computers and programs, now known as Flynn's taxonomy. Flynn classified programs and computers by whether they were operating using a single set or multiple sets of instructions, and whether or not those instructions were using a single set or multiple sets of data.
Flynn's taxonomy |
---|
Single data stream |
Multiple data streams |
SIMD Subcategories[30] |
See also |
|
The single-instruction-single-data (SISD) classification is equivalent to an entirely sequential program. The single-instruction-multiple-data (SIMD) classification is analogous to doing the same operation repeatedly over a large data set. This is commonly done in signal processing applications. Multiple-instruction-single-data (MISD) is a rarely used classification. While computer architectures to deal with this were devised (such as systolic arrays), few applications that fit this class materialized. Multiple-instruction-multiple-data (MIMD) programs are by far the most common type of parallel programs.
According to
Granularity
Bit-level parallelism
From the advent of
Historically, 4-bit microprocessors were replaced with 8-bit, then 16-bit, then 32-bit microprocessors. This trend generally came to an end with the introduction of 32-bit processors, which has been a standard in general-purpose computing for two decades. Not until the early 2000s, with the advent of x86-64 architectures, did 64-bit processors become commonplace.
Instruction-level parallelism
A computer program is, in essence, a stream of instructions executed by a processor. Without instruction-level parallelism, a processor can only issue less than one instruction per clock cycle (IPC < 1). These processors are known as subscalar processors. These instructions can be re-ordered and combined into groups which are then executed in parallel without changing the result of the program. This is known as instruction-level parallelism. Advances in instruction-level parallelism dominated computer architecture from the mid-1980s until the mid-1990s.[33]
All modern processors have multi-stage
Most modern processors also have multiple
Task parallelism
Task parallelisms is the characteristic of a parallel program that "entirely different calculations can be performed on either the same or different sets of data".[35] This contrasts with data parallelism, where the same calculation is performed on the same or different sets of data. Task parallelism involves the decomposition of a task into sub-tasks and then allocating each sub-task to a processor for execution. The processors would then execute these sub-tasks concurrently and often cooperatively. Task parallelism does not usually scale with the size of a problem.[36]
Superword level parallelism
Superword level parallelism is a
Hardware
Memory and communication
Main memory in a parallel computer is either
Computer architectures in which each element of main memory can be accessed with equal
Computer systems make use of
Processor–processor and processor–memory communication can be implemented in hardware in several ways, including via shared (either multiported or multiplexed) memory, a crossbar switch, a shared bus or an interconnect network of a myriad of topologies including star, ring, tree, hypercube, fat hypercube (a hypercube with more than one processor at a node), or n-dimensional mesh.
Parallel computers based on interconnected networks need to have some kind of routing to enable the passing of messages between nodes that are not directly connected. The medium used for communication between the processors is likely to be hierarchical in large multiprocessor machines.
Classes of parallel computers
Parallel computers can be roughly classified according to the level at which the hardware supports parallelism. This classification is broadly analogous to the distance between basic computing nodes. These are not mutually exclusive; for example, clusters of symmetric multiprocessors are relatively common.
Multi-core computing
A multi-core processor is a processor that includes multiple
Symmetric multiprocessing
A symmetric multiprocessor (SMP) is a computer system with multiple identical processors that share memory and connect via a bus.[39] Bus contention prevents bus architectures from scaling. As a result, SMPs generally do not comprise more than 32 processors.[40] Because of the small size of the processors and the significant reduction in the requirements for bus bandwidth achieved by large caches, such symmetric multiprocessors are extremely cost-effective, provided that a sufficient amount of memory bandwidth exists.[39]
Distributed computing
A distributed computer (also known as a distributed memory multiprocessor) is a distributed memory computer system in which the processing elements are connected by a network. Distributed computers are highly scalable. The terms "concurrent computing", "parallel computing", and "distributed computing" have a lot of overlap, and no clear distinction exists between them.[41] The same system may be characterized both as "parallel" and "distributed"; the processors in a typical distributed system run concurrently in parallel.[42]
Cluster computing
A cluster is a group of loosely coupled computers that work together closely, so that in some respects they can be regarded as a single computer.
Because grid computing systems (described below) can easily handle embarrassingly parallel problems, modern clusters are typically designed to handle more difficult problems—problems that require nodes to share intermediate results with each other more often. This requires a high bandwidth and, more importantly, a low-latency interconnection network. Many historic and current supercomputers use customized high-performance network hardware specifically designed for cluster computing, such as the Cray Gemini network.[46] As of 2014, most current supercomputers use some off-the-shelf standard network hardware, often Myrinet, InfiniBand, or Gigabit Ethernet.
Massively parallel computing
A massively parallel processor (MPP) is a single computer with many networked processors. MPPs have many of the same characteristics as clusters, but MPPs have specialized interconnect networks (whereas clusters use commodity hardware for networking). MPPs also tend to be larger than clusters, typically having "far more" than 100 processors.[47] In an MPP, "each CPU contains its own memory and copy of the operating system and application. Each subsystem communicates with the others via a high-speed interconnect."[48]
Grid computing
Grid computing is the most distributed form of parallel computing. It makes use of computers communicating over the Internet to work on a given problem. Because of the low bandwidth and extremely high latency available on the Internet, distributed computing typically deals only with embarrassingly parallel problems.
Most grid computing applications use middleware (software that sits between the operating system and the application to manage network resources and standardize the software interface). The most common grid computing middleware is the Berkeley Open Infrastructure for Network Computing (BOINC). Often volunteer computing software makes use of "spare cycles", performing computations at times when a computer is idling.[49]
Cloud computing
The ubiquity of Internet brought the possibility of large-scale cloud computing.
Specialized parallel computers
Within parallel computing, there are specialized parallel devices that remain niche areas of interest. While not
Reconfigurable computing with field-programmable gate arrays
Reconfigurable computing is the use of a field-programmable gate array (FPGA) as a co-processor to a general-purpose computer. An FPGA is, in essence, a computer chip that can rewire itself for a given task.
FPGAs can be programmed with
AMD's decision to open its HyperTransport technology to third-party vendors has become the enabling technology for high-performance reconfigurable computing.[52] According to Michael R. D'Amour, Chief Operating Officer of DRC Computer Corporation, "when we first walked into AMD, they called us 'the socket stealers.' Now they call us their partners."[52]
General-purpose computing on graphics processing units (GPGPU)
General-purpose computing on graphics processing units (GPGPU) is a fairly recent trend in computer engineering research. GPUs are co-processors that have been heavily optimized for computer graphics processing.[53] Computer graphics processing is a field dominated by data parallel operations—particularly linear algebra matrix operations.
In the early days, GPGPU programs used the normal graphics APIs for executing programs. However, several new programming languages and platforms have been built to do general purpose computation on GPUs with both
Application-specific integrated circuits
Several application-specific integrated circuit (ASIC) approaches have been devised for dealing with parallel applications.[54][55][56]
Because an ASIC is (by definition) specific to a given application, it can be fully optimized for that application. As a result, for a given application, an ASIC tends to outperform a general-purpose computer. However, ASICs are created by UV photolithography. This process requires a mask set, which can be extremely expensive. A mask set can cost over a million US dollars.[57] (The smaller the transistors required for the chip, the more expensive the mask will be.) Meanwhile, performance increases in general-purpose computing over time (as described by Moore's law) tend to wipe out these gains in only one or two chip generations.[52] High initial cost, and the tendency to be overtaken by Moore's-law-driven general-purpose computing, has rendered ASICs unfeasible for most parallel computing applications. However, some have been built. One example is the PFLOPS RIKEN MDGRAPE-3 machine which uses custom ASICs for molecular dynamics simulation.
Vector processors
A vector processor is a CPU or computer system that can execute the same instruction on large sets of data. Vector processors have high-level operations that work on linear arrays of numbers or vectors. An example vector operation is A = B × C, where A, B, and C are each 64-element vectors of 64-bit
Software
Parallel programming languages
Efforts to standardize parallel programming include an open standard called OpenHMPP for hybrid multi-core parallel programming. The OpenHMPP directive-based programming model offers a syntax to efficiently offload computations on hardware accelerators and to optimize data movement to/from the hardware memory using remote procedure calls.
The rise of consumer GPUs has led to support for
Automatic parallelization
Automatic parallelization of a sequential program by a compiler is the "holy grail" of parallel computing, especially with the aforementioned limit of processor frequency. Despite decades of work by compiler researchers, automatic parallelization has had only limited success.[60]
Mainstream parallel programming languages remain either
.Application checkpointing
As a computer system grows in complexity, the
Algorithmic methods
As parallel computers become larger and faster, we are now able to solve problems that had previously taken too long to run. Fields as varied as bioinformatics (for protein folding and sequence analysis) and economics (for mathematical finance) have taken advantage of parallel computing. Common types of problems in parallel computing applications include:[62]
- Dense linear algebra
- Sparse linear algebra
- Spectral methods (such as Cooley–Tukey fast Fourier transform)
- N-body problems (such as Barnes–Hut simulation)
- Structured grid problems (such as Lattice Boltzmann methods)
- finite element analysis)
- Monte Carlo method
- brute-force cryptographic techniques)
- Graph traversal (such as sorting algorithms)
- Dynamic programming
- Branch and bound methods
- Graphical models (such as detecting hidden Markov models and constructing Bayesian networks)
- HBJ model, a concise message-passing model[63]
- Finite-state machine simulation
Fault tolerance
Parallel computing can also be applied to the design of
History
The origins of true (MIMD) parallelism go back to
In 1957,
In April 1958, Stanley Gill (Ferranti) discussed parallel programming and the need for branching and waiting.[72] Also in 1958, IBM researchers John Cocke and Daniel Slotnick discussed the use of parallelism in numerical calculations for the first time.[73] Burroughs Corporation introduced the D825 in 1962, a four-processor computer that accessed up to 16 memory modules through a crossbar switch.[74] In 1967, Amdahl and Slotnick published a debate about the feasibility of parallel processing at American Federation of Information Processing Societies Conference.[73] It was during this debate that Amdahl's law was coined to define the limit of speed-up due to parallelism.
In 1969, Honeywell introduced its first Multics system, a symmetric multiprocessor system capable of running up to eight processors in parallel.[73] C.mmp, a multi-processor project at Carnegie Mellon University in the 1970s, was among the first multiprocessors with more than a few processors. The first bus-connected multiprocessor with snooping caches was the Synapse N+1 in 1984.[67]
SIMD parallel computers can be traced back to the 1970s. The motivation behind early SIMD computers was to amortize the
Biological brain as massively parallel computer
In the early 1970s, at the MIT Computer Science and Artificial Intelligence Laboratory, Marvin Minsky and Seymour Papert started developing the Society of Mind theory, which views the biological brain as massively parallel computer. In 1986, Minsky published The Society of Mind, which claims that "mind is formed from many little agents, each mindless by itself".[76] The theory attempts to explain how what we call intelligence could be a product of the interaction of non-intelligent parts. Minsky says that the biggest source of ideas about the theory came from his work in trying to create a machine that uses a robotic arm, a video camera, and a computer to build with children's blocks.[77]
Similar models (which also view the biological brain as a massively parallel computer, i.e., the brain is made up of a constellation of independent or semi-independent agents) were also described by:
- Thomas R. Blakeslee,[78]
- Michael S. Gazzaniga,[79][80]
- Robert E. Ornstein,[81]
- Ernest Hilgard,[82][83]
- Michio Kaku,[84]
- George Ivanovich Gurdjieff,[85]
- Neurocluster Brain Model.[86]
See also
- Computer multitasking
- Concurrency (computer science)
- Content Addressable Parallel Processor
- List of distributed computing conferences
- Manchester dataflow machine
- Manycore
- Parallel programming model
- Parallelization contract
- Serializability
- Synchronous programming
- Transputer
- Vector processing
References
- ISBN 978-0-8053-0177-9.
- computer industryhas accepted that future performance increases must largely come from increasing the number of processors (or cores) on a die, rather than making a single core go faster."
- ^ Asanovic et al. Old [conventional wisdom]: Power is free, but transistors are expensive. New [conventional wisdom] is [that] power is expensive, but transistors are "free".
- Asanovic, Krste et al. (December 18, 2006). "The Landscape of Parallel Computing Research: A View from Berkeley"(PDF). University of California, Berkeley. Technical Report No. UCB/EECS-2006-183. "Old [conventional wisdom]: Increasing clock frequency is the primary method of improving processor performance. New [conventional wisdom]: Increasing parallelism is the primary method of improving processor performance… Even representatives from Intel, a company generally associated with the 'higher clock-speed is better' position, warned that traditional approaches to maximizing performance through maximizing clock speed have been pushed to their limits."
- ^ "Concurrency is not Parallelism", Waza conference Jan 11, 2012, Rob Pike (slides Archived 2015-07-30 at the Wayback Machine) (video)
- ^ "Parallelism vs. Concurrency". Haskell Wiki.
- ISBN 978-1-55860-428-5.
- ^ a b Barney, Blaise. "Introduction to Parallel Computing". Lawrence Livermore National Laboratory. Retrieved 2007-11-09.
- ISBN 9783642378010.
- ISBN 978-1-55860-724-8.
- ISBN 978-0-13-178609-7.
- ^ Flynn, Laurie J. (8 May 2004). "Intel Halts Development Of 2 New Microprocessors". New York Times. Retrieved 5 June 2012.
- ISBN 9783642378010.
- ISBN 9783642378010.
- )
- ISBN 978-0-201-83595-3.
- ^ Michael McCool; James Reinders; Arch Robison (2013). Structured Parallel Programming: Patterns for Efficient Computation. Elsevier. p. 61.
- S2CID 33937392. Archived from the originalon 2007-09-27.
- .
- ISBN 978-0-387-98716-3.
- ^ "Processes and Threads". Microsoft Developer Network. Microsoft Corp. 2018. Retrieved 2018-05-10.
- ^ Krauss, Kirk J (2018). "Thread Safety for Performance". Develop for Performance. Archived from the original on 2018-05-13. Retrieved 2018-05-10.
- ^ Tanenbaum, Andrew S. (2002-02-01). Introduction to Operating System Deadlocks. Pearson Education, Informit. Retrieved 2018-05-10.
{{cite book}}
:|website=
ignored (help) - ^ Cecil, David (2015-11-03). "Synchronization internals – the semaphore". Embedded. AspenCore. Retrieved 2018-05-10.
- ^ Preshing, Jeff (2012-06-08). "An Introduction to Lock-Free Programming". Preshing on Programming. Retrieved 2018-05-10.
- ^ "What's the opposite of "embarrassingly parallel"?". StackOverflow. Retrieved 2018-05-10.
- ^ Schwartz, David (2011-08-15). "What is thread contention?". StackOverflow. Retrieved 2018-05-10.
- ^ Kukanov, Alexey (2008-03-04). "Why a simple test can get parallel slowdown". Retrieved 2015-02-15.
- ^ Krauss, Kirk J (2018). "Threading for Performance". Develop for Performance. Archived from the original on 2018-05-13. Retrieved 2018-05-10.
- .
- ^ Patterson and Hennessy, p. 748.
- ISBN 978-1-55860-343-1.)
{{cite book}}
: CS1 maint: multiple names: authors list (link - ^ Culler et al. p. 15.
- ^ Patt, Yale (April 2004). "The Microprocessor Ten Years From Now: What Are The Challenges, How Do We Meet Them? Archived 2008-04-14 at the Wayback Machine (wmv). Distinguished Lecturer talk at Carnegie Mellon University. Retrieved on November 7, 2007.
- ^ Culler et al. p. 124.
- ^ Culler et al. p. 125.
- ^ Samuel Larsen; Saman Amarasinghe. "Exploiting Superword Level Parallelism with Multimedia Instruction Sets" (PDF).
- ^ a b Patterson and Hennessy, p. 713.
- ^ a b Hennessy and Patterson, p. 549.
- ^ Patterson and Hennessy, p. 714.
- ^ Ghosh (2007), p. 10. Keidar (2008).
- ^ Lynch (1996), p. xix, 1–2. Peleg (2000), p. 1.
- ^ What is clustering? Webopedia computer dictionary. Retrieved on November 7, 2007.
- ^ Beowulf definition. Archived 2012-10-10 at the Wayback Machine PC Magazine. Retrieved on November 7, 2007.
- ^ "List Statistics | TOP500 Supercomputer Sites". www.top500.org. Retrieved 2018-08-05.
- ^ "Interconnect" Archived 2015-01-28 at the Wayback Machine.
- ^ Hennessy and Patterson, p. 537.
- ^ MPP Definition. Archived 2013-05-11 at the Wayback Machine PC Magazine. Retrieved on November 7, 2007.
- S2CID 60622095.
- ISSN 2079-9292.
All simulated circuits were described in very high speed integrated circuit (VHSIC) hardware description language (VHDL). Hardware modeling was performed on Xilinx FPGA Artix 7 xc7a200tfbg484-2.
- S2CID 219990653.
- ^ a b c D'Amour, Michael R., Chief Operating Officer, DRC Computer Corporation. "Standard Reconfigurable Computing". Invited speaker at the University of Delaware, February 28, 2007.
- ^ Boggan, Sha'Kia and Daniel M. Pressel (August 2007). GPUs: An Emerging Platform for General-Purpose Computation Archived 2016-12-25 at the Wayback Machine (PDF). ARL-SR-154, U.S. Army Research Lab. Retrieved on November 7, 2007.
- ^ Maslennikov, Oleg (2002). "Systematic Generation of Executing Programs for Processor Elements in Parallel ASIC or FPGA-Based Systems and Their Transformation into VHDL-Descriptions of Processor Element Control Units". Lecture Notes in Computer Science, 2328/2002: p. 272.
- S2CID 61094111.
- S2CID 2976028.
- ^ Kahng, Andrew B. (June 21, 2004) "Scoping the Problem of DFM in the Semiconductor Industry Archived 2008-01-31 at the Wayback Machine." University of California, San Diego. "Future design for manufacturing (DFM) technology must reduce design [non-recoverable expenditure] cost and directly address manufacturing [non-recoverable expenditures]—the cost of a mask set and probe card—which is well over $1 million at the 90 nm technology node and creates a significant damper on semiconductor-based innovation."
- ^ a b Patterson and Hennessy, p. 751.
- ^ The Sidney Fernbach Award given to MPI inventor Bill Gropp Archived 2011-07-25 at the Wayback Machine refers to MPI as "the dominant HPC communications interface"
- ISBN 978-0-07-057064-1.
However, the holy grail of such research—automated parallelization of serial programs—has yet to materialize. While automated parallelization of certain classes of algorithms has been demonstrated, such success has largely been limited to scientific and numeric applications with predictable flow control (e.g., nested loop structures with statically determined iteration counts) and statically analyzable memory access patterns. (e.g., walks over large multidimensional arrays of float-point data).
- ISBN 0387097651page 265
- Asanovic, Krste, et al. (December 18, 2006). "The Landscape of Parallel Computing Research: A View from Berkeley"(PDF). University of California, Berkeley. Technical Report No. UCB/EECS-2006-183. See table on pages 17–19.
- ]
- ^ a b Patterson and Hennessy, pp. 749–50: "Although successful in pushing several technologies useful in later projects, the ILLIAC IV failed as a computer. Costs escalated from the $8 million estimated in 1966 to $31 million by 1972, despite the construction of only a quarter of the planned machine . It was perhaps the most infamous of supercomputers. The project started in 1965 and ran its first real application in 1976."
- ^ Menabrea, L. F. (1842). Sketch of the Analytic Engine Invented by Charles Babbage. Bibliothèque Universelle de Genève. Retrieved on November 7, 2007. quote: "when a long series of identical computations is to be performed, such as those required for the formation of numerical tables, the machine can be brought into play so as to give several results at the same time, which will greatly abridge the whole amount of the processes."
- ^ a b Patterson and Hennessy, p. 753.
- ^ R.W. Hockney, C.R. Jesshope. Parallel Computers 2: Architecture, Programming and Algorithms, Volume 2. 1988. p. 8 quote: "The earliest reference to parallelism in computer design is thought to be in General L. F. Menabrea's publication in… 1842, entitled Sketch of the Analytical Engine Invented by Charles Babbage".
- S2CID 34642285.
- ^ "Architecture Sketch of Bull Gamma 60 -- Mark Smotherman". www.feb-patrimoine.com. Retrieved 2023-08-14.
- ^ Tumlin, Smotherman (2023-08-14). "An Evaluation of the Design of the Gamma 60". ACONIT Computer History Museum. Department of Computer Science, Clemson University. Retrieved 2023-08-14.
- ^ "Parallel Programming", S. Gill, The Computer Journal Vol. 1 #1, pp2-10, British Computer Society, April 1958.
- ^ a b c d e Wilson, Gregory V. (1994). "The History of the Development of Parallel Computing". Virginia Tech/Norfolk State University, Interactive Learning with a Digital Library in Computer Science. Retrieved 2008-01-08.
- ^ Anthes, Gry (November 19, 2001). "The Power of Parallelism". Computerworld. Archived from the original on January 31, 2008. Retrieved 2008-01-08.
- ^ Patterson and Hennessy, p. 749.
- ISBN 978-0-671-60740-1.
- ISBN 978-0-671-60740-1.
- ISBN 9780306452628.
- Gazzaniga, Michael; LeDoux, Joseph(1978). The Integrated Mind. pp. 132–161.
- ISBN 9780465078509.
- Ornstein, Robert (1992). Evolution of Consciousness: The Origins of the Way We Think. pp. 2.
- ISBN 978-0-471-39602-4.
- ISBN 978-0-471-80572-4.
- ^ Kaku, Michio (2014). The Future of the Mind.
- Ouspenskii, Pyotr(1992). "Chapter 3". In Search of the Miraculous. Fragments of an Unknown Teaching. pp. 72–83.
- ^ "Official Neurocluster Brain Model site". Retrieved July 22, 2017.
Further reading
- Rodriguez, C.; Villagra, M.; Baran, B. (29 August 2008). "Asynchronous team algorithms for Boolean Satisfiability". Bio-Inspired Models of Network, Information and Computing Systems, 2007. Bionetics 2007. 2nd: 66–69. S2CID 15185219.
- Sechin, A.; Parallel Computing in Photogrammetry. GIM International. #1, 2016, pp. 21–23.