Simple API for Grid Applications
GPL | |
Website | saga-project |
---|
The Simple API for Grid Applications (SAGA) is a family of related standards specified by the
Overview
The SAGA specification for distributed computing originally consisted of a single document, GFD.90, which was released in 2009.
The SAGA API does not strive to replace Globus or similar grid computing middleware systems, and does not target middleware developers, but application developers with no background on grid computing. Such developers typically wish to devote their time to their own goals and minimize the time spent coding infrastructure functionality. The API insulates application developers from middleware.
The specification of services, and the protocols to interact with them, is out of the scope of SAGA. Rather, the API seeks to hide the detail of any service infrastructures that may or may not be used to implement the functionality that the application developer needs. The API aligns, however, with all middleware standards within Open Grid Forum (OGF).[1]
The SAGA API defined a mechanism to specify additional API packages which expand its scope. The SAGA Core API itself defines a number of packages: job management, file management, replica management, remote procedure calls, and streams. SAGA covers the most important and frequently used distributed functionality and is supported and available on every major grid systems -
Standardization
The SAGA API is standardised in the SAGA Working Group the Open Grid Forum.[4] Based on a set of use cases [5] ,[6] the SAGA Core API specification[1] defines a set of general API principles (the 'SAGA Look and Feel', and a set of API packages which render commonly used Grid programming patterns (job management, file management and access, replica management etc.) The SAGA Core specification also defines how additional API packages are to be defined, and how they relate to the Core API, and to its 'Look and Feel'. Based on that, a number of API extensions have been defined, and are in various states of the standardisation process.[7][8][9][10]
All SAGA specifications are defined in (a flavor of) IDL, and thus object oriented, but language neutral. Different language bindings exist (Java, C++, Python), but are, at this point, not standardised. Nevertheless, different implementations of these language bindings have a relatively coherent API definition (in particular, the different Java implementations share the same abstract API classes).
The 'Look and Feel' part of the SAGA Core API specification covers the following areas:
- security and session management
- permission management
- asynchronous operations
- monitoring
- asynchronous notifications
- attribute management
- I/O buffer management
Architecture
SAGA is designed as an
- saga::advert - interface for Advert Serviceaccess
- saga::filesystem - interface for file and directory access
- saga::job - interface for job definition, management and control
- saga::namespace - abstract interface (used by advert, filesystem and replica interfaces)
- saga::replica - interface for replica management
- saga::rpc - interface for remote procedure calls client and servers
- saga::sd- interface for service discovery in distributed environments
- saga::stream - interface for data stream client and servers
The overall architecture of SAGA follows the
Supported middleware
The following table lists the distributed middleware systems that are currently supported by SAGA. The column labeled Adaptor Suite names the collection (release package) of the (set of) middleware adaptors that provides support for the middleware system.
Middleware System | SAGA Adaptor Suite | SAGA API Namespace |
---|---|---|
Amazon EC2 |
saga-adaptors-aws | saga::job |
Condor |
saga-adaptors-condor | saga::job |
Eucalyptus |
saga-adaptors-aws | saga::job |
Globus GRAM (2 and 5) | saga-adaptors-globus | saga::job |
Globus GridFTP | saga-adaptors-globus | saga::filesystem |
Globus RLS | saga-adaptors-globus | saga::replica |
HDFS | saga-adaptors-hdfs | saga::file |
Local file system | part of saga-core | saga::file |
Local fork |
part of saga-core | saga::job |
Nimbus | saga-adaptors-aws | saga::job |
PBS (Pro) | saga-adaptors-pbs | saga::job |
Platform LSF |
saga-adaptors-lsf | saga::job |
Advert Service |
part of saga-core | saga::advert |
SQL Replica Service | part of saga-core | saga::replica |
SSHFS | saga-adaptors-ssh | saga::file |
SSH | saga-adaptors-ssh | saga::job |
TORQUE |
saga-adaptors-torque | saga::job |
Implementations
Since the SAGA interface definitions are not bound to any specific programming language, several implementations of the SAGA standards exist in different programming languages. Apart from the implementation language, they differ from each other in their completeness in terms of standard coverage, as well as in their support for distributed middleware.
SAGA C++
SAGA C++[13] was the first complete implementation of the SAGA Core specification, written in C++. Currently the C++ implementation is not under active development.
RADICAL-SAGA(Python)
RADICAL-SAGA other clouds and local clusters.
JavaSAGA
JavaSAGA is a Java implementation of SAGA. This status of JavaSAGA remains uncertain.
import java.util.io.*
int main (int argc, char** argv)
{
namespace sa = saga::attributes;
namespace sja = saga::job::attributes;
try
{
saga::job::description jd;
jd.set_attribute (sja::description_executable, "/home/user/hello-mpi");
jd.set_attribute (sja::description_output, "/home/user/hello.out");
jd.set_attribute (sja::description_error, "/home/user/hello.err");
// Declare this as an MPI-style job
jd.set_attribute (sja::description_spmd_variation, "mpi");
// Name of the queue we want to use
jd.set_attribute (sja::description_queue, "checkpt");
jd.set_attribute (sja::description_spmd_variation, "mpi");
// Number of processors to request
jd.set_attribute (sja::description_number_of_processes, "32");
saga::job::service js("gram://my.globus.host/jobmanager-pbs");
saga::job::job j = js.create_job(jd);
j.run()
}
catch(saga::exception const & e)
{
std::cerr << "SAGA exception caught: " << e.what() << std::endl;
}
}
jSAGA
jSAGA [17] is another Java implementation of the SAGA Core specification. jSAGA is currently under active development.
DESHL
The DESHL
Examples
Job submission
A typical task in a distributed application is to submit a job to a local or remote
C++
#include <saga/saga.hpp>
int main (int argc, char** argv)
{
namespace sa = saga::attributes;
namespace sja = saga::job::attributes;
try
{
saga::job::description jd;
jd.set_attribute (sja::description_executable, "/home/user/hello-mpi");
jd.set_attribute (sja::description_output, "/home/user/hello.out");
jd.set_attribute (sja::description_error, "/home/user/hello.err");
// Declare this as an MPI-style job
jd.set_attribute (sja::description_spmd_variation, "mpi");
// Name of the queue we want to use
jd.set_attribute (sja::description_queue, "checkpt");
jd.set_attribute (sja::description_spmd_variation, "mpi");
// Number of processors to request
jd.set_attribute (sja::description_number_of_processes, "32");
saga::job::service js("gram://my.globus.host/jobmanager-pbs");
saga::job::job j = js.create_job(jd);
j.run()
}
catch(saga::exception const & e)
{
std::cerr << "SAGA exception caught: " << e.what() << std::endl;
}
}
Python
#!/usr/bin/env python3
import sys
import time
import bliss.saga as saga
def main(jobno: int, session, jobservice) -> None:
bfast_base_dir = saga.Url("sftp://india.futuregrid.org/N/u/oweidner/software/bfast/")
try:
workdir = "%s/tmp/run/%s" % (bfast_base_dir.path, str(int(time.time())))
basedir = saga.filesystem.Directory(bfast_base_dir, session=session)
basedir.make_dir(workdir)
jd = saga.job.Description()
jd.wall_time_limit = 5 # wall-time in minutes
jd.total_cpu_count = 1
jd.environment = {"BFAST_DIR": bfast_base_dir.path}
jd.working_directory = workdir
jd.executable = "$BFAST_DIR/bin/bfast"
jd.arguments = ["match", "-A 1",
"-r $BFAST_DIR/data/small/reads_5K/reads.10.fastq",
"-f $BFAST_DIR/data/small/reference/hg_2122.fa"]
myjob = js.create_job(jd)
myjob.run()
print("Job #%s started with ID '%s' and working directory: '%s'"
% (jobno, myjob.jobid, workdir))
myjob.wait()
print("Job #%s with ID '%s' finished (RC: %s). Output available in: '%s'"
% (jobno, myjob.jobid, myjob.exitcode, workdir))
basedir.close()
except saga.Exception, ex:
print(f"An error occurred during job execution: {ex}")
sys.exit(-1)
if __name__ == "__main__":
execution_host = saga.Url("pbs+ssh://india.futuregrid.org")
ctx = saga.Context()
ctx.type = saga.Context.SSH
ctx.userid = "oweidner" # like 'ssh username@host ...'
ctx.userkey = "/Users/oweidner/.ssh/rsa_work" # like ssh -i ...'
session = saga.Session()
session.contexts.append(ctx)
js = saga.job.Service(execution_host, session)
for i in range(0, 4):
main(i, session, js)
Grants
The work related to the SAGA Project is funded by the following grants: NSF-CHE 1125332 (CDI),[19] NSF-EPS 1003897 (LaSIGMA),[20] NSF-OCI 1007115 (ExTENCI).[21] Previous grants include: NSF-OCI 0710874 (HPCOPS), NIH grant number P20RR016456 and UK EPSRC grant number GR/D0766171/1 via OMII-UK[22]
External links
- SAGA-Bliss - A Python implementation of SAGA
- jSAGA - A Java implementation of SAGA
- SAGA C++ - A C++ implementation of SAGA
- SAGA-GLib - A Vala implementation of SAGA for GLib
- SAGA PROJECT
- POSIX
Notes
- ^ a b c T. Goodale; S. Jha; H. Kaiser; T. Kielmann; P. Kleijer; A. Merzky; J. Shalf; C. Smith. "A Simple API for Grid Applications (SAGA)" (PDF). OGF Document Series 90.
- ^ SAGA: A Simple API for Grid applications, High-Level Application Programming on the Grid Tom Goodale, Shantenu Jha, Harmut Kaiser, Thilo Kielmann, Pascal K leijer, Gregor von Laszewski, Craig Lee, Andre Merzky, Hrabri Rajic, John Shalf Computational Methods in Science and Technology, vol. 12 # 1, 2006
- ^ Grid Interoperability at the Application Level Using SAGA Shantenu Jha, Hartmut Kaiser, Andre Merzky, Ole Weidner E-SCIENCE ’07: Proceedings of the Third IEEE International Conference on e-Science and Grid Computing (e-Science 2007), 2007
- ^ "Overview - SAGA WG - Open Grid Forum".
- ^ Shantenu Jha, Andre Merzky: "A Collection of Use Cases for a Simple API for Grid Applications", OGF Informational Document, GFD.70 (pdf)
- ^ Shantenu Jha, Andre Merzky: "A Requirements Analysis for a Simple API for Grid Applications", OGF Informational Document, GFD.71 (pdf)
- ^ Steve Fisher, Anthony Wilson, Arumugam Paventhan: "SAGA API Extension: Service Discovery API", OGF Recommendation Document, GFD.144 (pdf)
- ^ Andre Merzky: "SAGA API Extension: Advert API", OGF Recommendation Document, GFD.177 (pdf)
- ^ Andre Merzky: "SAGA API Extension: Message API", OGF Recommendation Document, GFD.178 (pdf)
- ^ Steve Fisher, Anthony Wilson: "SAGA API Extension: Information System Navigator API", OGF Recommendation Document, GFD.195 (pdf)
- ^ "The SAGA C++ Reference API (Documentation)".
- ^ "SAGA: How it works". www.vimeo.com.
- ^ "SAGA C++".
- ^ "RADICAL-SAGA".
- ^ "Thank you for your interest in XSEDE".
- ^ "FutureGrid". Archived from the original on 2010-11-25.
- ^ "jSAGA".
- ^ "DESHL". Archived from the original on 2012-06-08.
- ^ NSF-CHE 1125332 (CDI)
- ^ NSF-EPS 1003897 (LaSIGMA)
- ^ NSF-OCI 1007115 (ExTENCI)
- ^ OMII-UK