Grid computing
Grid computing is the use of widely distributed
Grids are a form of
Grid computing combines computers from multiple administrative domains to reach a common goal,[3] to solve a single task, and may then disappear just as quickly. The size of a grid may vary from small—confined to a network of computer workstations within a corporation, for example—to large, public collaborations across many companies and networks. "The notion of a confined grid may also be known as an intra-nodes cooperation whereas the notion of a larger, wider grid may thus refer to an inter-nodes cooperation".[4]
Coordinating applications on Grids can be a complex task, especially when coordinating the flow of information across distributed computing resources. Grid workflow systems have been developed as a specialized form of a workflow management system designed specifically to compose and execute a series of computational or data manipulation steps, or a workflow, in the grid context.
Comparison of grids and conventional supercomputers
“Distributed” or “grid” computing in general is a special type of parallel computing that relies on complete computers (with onboard CPUs, storage, power supplies, network interfaces, etc.) connected to a network (private, public or the Internet) by a conventional network interface producing commodity hardware, compared to the lower efficiency of designing and constructing a small number of custom supercomputers. The primary performance disadvantage is that the various processors and local storage areas do not have high-speed connections. This arrangement is thus well-suited to applications in which multiple parallel computations can take place independently, without the need to communicate intermediate results between processors.[5] The high-end scalability of geographically dispersed grids is generally favorable, due to the low need for connectivity between nodes relative to the capacity of the public Internet.[6]
There are also some differences between programming for a supercomputer and programming for a grid computing system. It can be costly and difficult to write programs that can run in the environment of a supercomputer, which may have a custom operating system, or require the program to address concurrency issues. If a problem can be adequately parallelized, a “thin” layer of “grid” infrastructure can allow conventional, standalone programs, given a different part of the same problem, to run on multiple machines. This makes it possible to write and debug on a single conventional machine and eliminates complications due to multiple instances of the same program running in the same shared memory and storage space at the same time.
Design considerations and variations
One feature of distributed grids is that they can be formed from computing resources belonging to one or multiple individuals or organizations (known as multiple administrative domains). This can facilitate commercial transactions, as in utility computing, or make it easier to assemble volunteer computing networks.
One disadvantage of this feature is that the computers which are actually performing the calculations might not be entirely trustworthy. The designers of the system must thus introduce measures to prevent malfunctions or malicious participants from producing false, misleading, or erroneous results, and from using the system as an attack vector. This often involves assigning work randomly to different nodes (presumably with different owners) and checking that at least two different nodes report the same answer for a given work unit. Discrepancies would identify malfunctioning and malicious nodes. However, due to the lack of central control over the hardware, there is no way to guarantee that
Another set of what could be termed social compatibility issues in the early days of grid computing related to the goals of grid developers to carry their innovation beyond the original field of high-performance computing and across disciplinary boundaries into new fields, like that of high-energy physics.[7]
The impacts of trust and availability on performance and development difficulty can influence the choice of whether to deploy onto a dedicated cluster, to idle machines internal to the developing organization, or to an open external network of volunteers or contractors. In many cases, the participating nodes must trust the central system not to abuse the access that is being granted, by interfering with the operation of other programs, mangling stored information, transmitting private data, or creating new security holes. Other systems employ measures to reduce the amount of trust “client” nodes must place in the central system such as placing applications in virtual machines.
Public systems or those crossing administrative domains (including different departments in the same organization) often result in the need to run on
In fact, the middleware can be seen as a layer between the hardware and the software. On top of the middleware, a number of technical areas have to be considered, and these may or may not be middleware independent. Example areas include
Market segmentation of the grid computing market
For the segmentation of the grid computing market, two perspectives need to be considered: the provider side and the user side:
The provider side
The overall grid market comprises several specific markets. These are the grid middleware market, the market for grid-enabled applications, the utility computing market, and the software-as-a-service (SaaS) market.
Grid middleware is a specific software product, which enables the sharing of heterogeneous resources, and Virtual Organizations. It is installed and integrated into the existing infrastructure of the involved company or companies and provides a special layer placed among the heterogeneous infrastructure and the specific user applications. Major grid middlewares are Globus Toolkit, gLite, and UNICORE.
Utility computing is referred to as the provision of grid computing and applications as service either as an open grid utility or as a hosting solution for one organization or a
.Grid-enabled applications are specific software applications that can utilize grid infrastructure. This is made possible by the use of grid middleware, as pointed out above.
Software as a service (SaaS) is “software that is owned, delivered and managed remotely by one or more providers.” (Gartner 2007) Additionally, SaaS applications are based on a single set of common code and data definitions. They are consumed in a one-to-many model, and SaaS uses a Pay As You Go (PAYG) model or a subscription model that is based on usage. Providers of SaaS do not necessarily own the computing resources themselves, which are required to run their SaaS. Therefore, SaaS providers may draw upon the utility computing market. The utility computing market provides computing resources for SaaS providers.
The user side
For companies on the demand or user side of the grid computing market, the different segments have significant implications for their IT deployment strategy. The IT deployment strategy as well as the type of IT investments made are relevant aspects for potential grid users and play an important role for grid adoption.
CPU scavenging
CPU-scavenging, cycle-scavenging, or shared computing creates a “grid” from the idle resources in a network of participants (whether worldwide or internal to an organization). Typically, this technique exploits the 'spare'
Many
Creating an Opportunistic Environment is another implementation of CPU-scavenging where special workload management system harvests the idle desktop computers for compute-intensive jobs, it also refers as Enterprise Desktop Grid (EDG). For instance, HTCondor[8] (the open-source high-throughput computing software framework for coarse-grained distributed rationalization of computationally intensive tasks) can be configured to only use desktop machines where the keyboard and mouse are idle to effectively harness wasted CPU power from otherwise idle desktop workstations. Like other full-featured batch systems, HTCondor provides a job queueing mechanism, scheduling policy, priority scheme, resource monitoring, and resource management. It can be used to manage workload on a dedicated cluster of computers as well or it can seamlessly integrate both dedicated resources (rack-mounted clusters) and non-dedicated desktop machines (cycle scavenging) into one computing environment.
History
The term grid computing originated in the early 1990s as a
CPU scavenging and volunteer computing were popularized beginning in 1997 by distributed.net and later in 1999 by SETI@home to harness the power of networked PCs worldwide, in order to solve CPU-intensive research problems.[11][12]
The ideas of the grid (including those from distributed computing, object-oriented programming, and Web services) were brought together by
In 2007 the term
Progress
In November 2006, Seidel received the Sidney Fernbach Award at the Supercomputing Conference in Tampa, Florida.[16] "For outstanding contributions to the development of software for HPC and Grid computing to enable the collaborative numerical investigation of complex problems in physics; in particular, modeling black hole collisions."[17] This award, which is one of the highest honors in computing, was awarded for his achievements in numerical relativity.
Fastest virtual supercomputers
- As of March 2020, Folding@home – 1.1 exaFLOPS.[18]
- As of April 7, 2020, BOINC – 29.8 PFLOPS.[19]
- As of November 2019, IceCube via OSG – 350 fp32 PFLOPS.[20]
- As of February 2018, Einstein@Home – 3.489 PFLOPS.[21]
- As of April 7, 2020, SETI@Home – 1.11 PFLOPS.[22]
- As of April 7, 2020, MilkyWay@Home – 1.465 PFLOPS.[23]
- As of March 2019, GIMPS – 0.558 PFLOPS.[24]
Also, as of March 2019, the
Projects and applications
Grid computing offers a way to solve
As of October 2016, over 4 million machines running the open-source Berkeley Open Infrastructure for Network Computing (BOINC) platform are members of the World Community Grid.[19] One of the projects using BOINC is SETI@home, which was using more than 400,000 computers to achieve 0.828 TFLOPS as of October 2016. As of October 2016 Folding@home, which is not part of BOINC, achieved more than 101 x86-equivalent petaflops on over 110,000 machines.[18]
The
The Enabling Grids for E-sciencE project, based in the European Union and included sites in Asia and the United States, was a follow-up project to the European DataGrid (EDG) and evolved into the European Grid Infrastructure. This, along with the Worldwide LHC Computing Grid[28] (WLCG), was developed to support experiments using the CERN Large Hadron Collider. A list of active sites participating within WLCG can be found online[29] as can real time monitoring of the EGEE infrastructure.[30] The relevant software and documentation is also publicly accessible.[31] There is speculation that dedicated fiber optic links, such as those installed by CERN to address the WLCG's data-intensive needs, may one day be available to home users thereby providing internet services at speeds up to 10,000 times faster than a traditional broadband connection.[32] The European Grid Infrastructure has been also used for other research activities and experiments such as the simulation of oncological clinical trials.[33]
The distributed.net project was started in 1997. The
In 2001,
Definitions
Today there are many definitions of grid computing:
- In his article “What is the Grid? A Three Point Checklist”,[3] Ian Foster lists these primary attributes:
- Computing resourcesare not administered centrally.
- Open standardsare used.
- Nontrivial quality of service is achieved.
- Plaszczak/Wellner[35] define grid technology as "the technology that enables resource virtualization, on-demand provisioning, and service (resource) sharing between organizations."
- IBM defines grid computing as “the ability, using a set of open standards and protocols, to gain access to applications and data, processing power, storage capacity and a vast array of other computing resources over the Internet. A grid is a type of parallel and distributed system that enables the sharing, selection, and aggregation of resources distributed across ‘multiple’ administrative domains based on their (resources) availability, capacity, performance, cost and users' quality-of-service requirements”.[36]
- An earlier example of the notion of computing as the utility was in 1965 by MIT's Fernando Corbató. Corbató and the other designers of the Multics operating system envisioned a computer facility operating “like a power company or water company”.[37]
- Buyya/Venugopal[38] define grid as "a type of parallel and distributed system that enables the sharing, selection, and aggregation of geographically distributed autonomous resources dynamically at runtime depending on their availability, capability, performance, cost, and users' quality-of-service requirements".
- CERN, one of the largest users of grid technology, talk of The Grid: “a service for sharing computer power and data storage capacity over the Internet.”[39]
See also
List of grid computing projects
Related concepts
- High-throughput computing
- Cloud computing
- Code mobility
- Jungle computing
- Sensor grid
- Utility computing
Alliances and organizations
- Global Grid Forum)
- Object Management Group
- SHIWA project
Production grids
- European Grid Infrastructure
- Enabling Grids for E-sciencE
- INFN Production Grid
- NorduGrid
- OurGrid
- Sun Grid
- Techila
- Xgrid
- Univa Grid Engine
International projects
Name | Region | Start | End |
---|---|---|---|
European Grid Infrastructure (EGI) | Europe | May 2010 | Dec 2014 |
Open Middleware Infrastructure Institute Europe (OMII-Europe)
|
Europe | May 2006 | May 2008 |
Enabling Grids for E-sciencE (EGEE, EGEE II and EGEE III) | Europe | March 2004 | April 2010 |
Grid enabled Remote Instrumentation with Distributed Control and Computation (GridCC)
|
Europe | September 2005 | September 2008 |
European Middleware Initiative (EMI) | Europe | May 2010 | active |
KnowARC
|
Europe | June 2006 | November 2009 |
Nordic Data Grid Facility | Scandinavia and Finland | June 2006 | December 2012 |
World Community Grid | Global | November 2004 | active |
XtreemOS
|
Europe | June 2006 | (May 2010) ext. to September 2010 |
OurGrid | Brazil | December 2004 | active |
National projects
- GridPP (UK)
- CNGrid (China)
- D-Grid (Germany)
- GARUDA (India)
- Calcutta, India)
- IsraGrid (Israel)
- INFN Grid(Italy)
- PL-Grid(Poland)
- National Grid Service (UK)
- Open Science Grid(USA)
- TeraGrid (USA)
Standards and APIs
- Distributed Resource Management Application API (DRMAA)
- A technology-agnostic information model for a uniform representation of Grid resources (GLUE)
- Grid Remote Procedure Call (GridRPC)
- Grid Security Infrastructure (GSI)
- Open Grid Services Architecture (OGSA)
- Common Object Request Broker Architecture (COBRA)
- Open Grid Services Infrastructure (OGSI)
- A Simple API for Grid Applications (SAGA)
- Web Services Resource Framework (WSRF)
Monitoring frameworks
References
- ^ What is grid computing? - Gridcafe Archived 2013-02-10 at the Wayback Machine. E-sciencecity.org. Retrieved 2013-09-18.
- ^ "Scale grid computing down to size". NetworkWorld.com. 2003-01-27. Archived from the original on 2017-02-17. Retrieved 2015-04-21.
- ^ a b "What is the Grid? A Three Point Checklist" (PDF). Archived from the original (PDF) on 2014-11-22. Retrieved 2010-10-21.
- ^ "Pervasive and Artificial Intelligence Group :: publications [Pervasive and Artificial Intelligence Research Group]". Diuf.unifr.ch. May 18, 2009. Archived from the original on July 7, 2011. Retrieved July 29, 2010.
- ^ Computational problems - Gridcafe Archived 2012-08-25 at the Wayback Machine. E-sciencecity.org. Retrieved 2013-09-18.
- ^ "What is grid computing?". IONOS Digitalguide. Archived from the original on 2022-01-28. Retrieved 2022-03-23.
- (PDF) from the original on 2022-03-28. Retrieved 2019-09-18.
- ^ "HTCondor - Home". research.cs.wisc.edu. Archived from the original on 2 March 2018. Retrieved 14 March 2018.
- ^ John McCarthy, speaking at the MIT Centennial in 1961
- ISBN 978-0-262-07196-3.
- S2CID 15439521.
- .
- ^ Johnson, Bridget (2019-11-06). "Grid Computing Pioneer Steve Tuecke Passes Away at 52". Archived from the original on 2022-11-04. Retrieved 2022-11-04.
- ^ "Father of the Grid". Archived from the original on 2012-03-01. Retrieved 2007-04-15.
- ^ Salem, M. (2007). Grid Computing: A New Paradigm for Healthcare Technologies/Applications. Retrieved 2022-08-30.
- ^ "Edward Seidel 2006 Sidney Fernbach Award Recipient". IEEE Computer Society Awards. IEEE Computer Society. Archived from the original on 15 August 2011. Retrieved 14 October 2011.
- ^ "Edward Seidel • IEEE Computer Society". www.computer.org. Archived from the original on 15 August 2011. Retrieved 14 March 2018.
- ^ a b Pande lab. "Client Statistics by OS". Folding@home. Stanford University. Archived from the original on April 12, 2020. Retrieved March 26, 2020.
- ^ a b "BOINCstats – BOINC combined credit overview". Archived from the original on January 22, 2013. Retrieved October 30, 2016.
- ^ "SDSC, Wisconsin U IceCube Center Conduct GPU Cloudburst Experiment". SDSC. Archived from the original on September 14, 2022. Retrieved April 22, 2022.
- ^ "Einstein@Home Credit overview". BOINC. Archived from the original on August 27, 2016. Retrieved October 30, 2016.
- ^ "SETI@Home Credit overview". BOINC. Archived from the original on July 3, 2013. Retrieved October 30, 2016.
- ^ "MilkyWay@Home Credit overview". BOINC. Archived from the original on May 20, 2012. Retrieved October 30, 2016.
- ^ "Internet PrimeNet Server Distributed Computing Technology for the Great Internet Mersenne Prime Search". GIMPS. Archived from the original on May 25, 2019. Retrieved March 12, 2019.
- ^ bitcoinwatch.com. "Bitcoin Network Statistics". Bitcoin. Archived from the original on January 20, 2023. Retrieved March 12, 2019.
- S2CID 219048576.
- ^ "beingrid.eu: Stromkosten Vergleiche -". beingrid.eu: Stromkosten Vergleiche. Archived from the original on 23 July 2011. Retrieved 14 March 2018.
- ^ "Welcome to the Worldwide LHC Computing Grid - WLCG". wlcg.web.cern.ch. Archived from the original on 25 July 2018. Retrieved 14 March 2018.
- ^ "GStat 2.0 – Summary View – GRID EGEE". Goc.grid.sinica.edu.tw. Archived from the original on March 20, 2008. Retrieved July 29, 2010.
- ^ "Real Time Monitor". Gridportal.hep.ph.ic.ac.uk. Archived from the original on December 16, 2009. Retrieved July 29, 2010.
- ^ "LCG – Deployment". Lcg.web.cern.ch. Archived from the original on November 17, 2010. Retrieved July 29, 2010.
- ^ "The Times & The Sunday Times". thetimes.co.uk. Archived from the original on 25 February 2021. Retrieved 14 March 2018.
- S2CID 206429690.
- ^ [1] Archived April 7, 2007, at the Wayback Machine
- ^ P Plaszczak, R Wellner, Grid computing, 2005, Elsevier/Morgan Kaufmann, San Francisco
- ^ IBM Solutions Grid for Business Partners: Helping IBM Business Partners to Grid-enable applications for the next phase of e-business on demand
- ^ Structure of the Multics Supervisor Archived 2014-01-16 at the Wayback Machine. Multicians.org. Retrieved 2013-09-18.
- ^ "A Gentle Introduction to Grid Computing and Technologies" (PDF). Archived (PDF) from the original on March 24, 2006. Retrieved May 6, 2005.
- ^ "The Grid Café – The place for everybody to learn about grid computing". CERN. Archived from the original on December 5, 2008. Retrieved December 3, 2008.
Bibliography
- ISBN 978-0-470-28768-2.
- Benedict, Shajulin; Vasudevan (2008). "A Niched Pareto GA approach for scheduling scientific workflows in wireless Grids". Journal of Computing and Information Technology. 16 (2): 101. .
- S2CID 7309750. Archived from the original(PDF) on 2008-02-28. Retrieved 2005-04-20.
- ISBN 978-1-55860-475-9.
- ISBN 978-0-12-742503-0.
- ISBN 978-0-470-85319-1.
- ISBN 978-0-470-09417-4. Archived from the originalon 2007-10-28. Retrieved 2005-04-26.
- .
- Smith, Roger (2005). "Grid Computing: A Brief Technology Analysis" (PDF). CTO Network Library. Archived from the original (PDF) on 2012-02-08.
- Buyya, Rajkumar (July 2005). "Grid Computing: Making the Global Cyberinfrastructure for eScience a Reality" (PDF). CSI Communications. 29 (1). Mumbai, India: Computer Society of India (CSI). Archived from the original (PDF) on 2006-02-28. Retrieved 2006-02-19.
- Berstis, Viktors. "Fundamentals of Grid Computing". IBM. Archived from the original on 2012-02-04.
- Elkhatib, Yehia (2011). Monitoring, Analysing and Predicting Network Performance in Grids (PDF) (Ph.D.). Lancaster University. Archived from the original (PDF) on 2015-06-23. Retrieved 2013-04-28.
- Ferreira, Luis; et al. (2016-09-30). "Grid Computing Products and Services". IBM.
- Ferreira, Luis; et al. (2016-09-30). "Introduction to Grid Computing with Globus". IBM.
- Jacob, Bart; et al. (2016-09-30). "Enabling Applications for Grid Computing". IBM.
- Ferreira, Luis; et al. "Grid Services Programming and Application Enablement". IBM. Archived from the original on 2012-02-04.
- Jacob, Bart; et al. (2016-09-30). Introduction to Grid Computing. IBM.
- Ferreira, Luis; et al. (2016-09-30). "Grid Computing in Research and Education". IBM.
- Ferreira, Luis; et al. "Globus Toolkit 3.0 Quick Start". IBM. Archived from the original on 2012-02-04. Retrieved 2006-04-27.
- Surridge, Mike; et al. "Experiences with GRIA – Industrial applications on a Web Services Grid" (PDF). IEEE. Archived from the original (PDF) on 2012-03-06.
- S2CID 16019948. Archived from the original(PDF) on 2007-01-07.
- Global Grids and Software Toolkits: A Study of Four Grid Middleware Technologies
- The Grid Technology Cookbook
- Francesco Lelli, Eric Frizziero, Michele Gulmini, Gaetano Maron, Salvatore Orlando, Andrea Petrucci and Silvano Squizzato. The many faces of the integration of instruments and the grid. International Journal of Web and Grid Services 2007 – Vol. 3, No.3 pp. 239 – 266 Electronic Edition
- Poess, Meikel; Nambiar, Raghunath (2005). Large Scale Data Warehouses on Grid (PDF). Archived from the original (PDF) on 2015-06-23. Retrieved 2010-11-05.
- .