The browser you are using is not supported by this website. All versions of Internet Explorer are no longer supported, either by us or Microsoft (read more here: https://www.microsoft.com/en-us/microsoft-365/windows/end-of-ie-support).

Please use a modern browser to fully experience our website, such as the newest versions of Edge, Chrome, Firefox or Safari etc.

profil image

Balazs Konya

Researcher, IT administrator

profil image

Integration of Nordugrid ARC with Galaxy and EGI IM

Author

  • Maiken Pedersen
  • Balazs Konya
  • Sebastian Luna-Valero
  • Björn Grüning

Editor

  • Abdulrahman Azab
  • Tomasz Malkiewicz

Summary, in English

In the scientific domain of high energy physics (HEP), the Worldwide LHC Computing Grid (WLCG) was created in order to handle the huge compute and storage needs of the experiments at Large Hadron Collider (LHC). Today WLCG combines about 1.4 million computer cores and 1.5 exabytes of storage from over 170 sites in 42 countries. What ties the sites together is the middleware installed at each site, one of these being the Nordugrid ARC (Advanced Resource Connector). ARC has been a great success and has served, and continues to serve the HEP community very well. Up until now though, we have had limited success in sharing our technology with other communities, despite the fact that many are faced with challenges that ARC solves: managing computation and storage across different infrastructure providers. With a network of ARC enabled compute sites - a user can submit a job from “anywhere” and automatically be routed to the best site depending on various matchmaking rules. One of the key strengths of ARC is its inbuilt data handling capabilities. ARC seamlessly downloads any remote input data to the computing site and makes sure all data is in place before the job is passed to the site’s local batch system. Once the job is done ARC can upload the data to a remote storage site, or it can be manually retrieved. In this paper we describe how we have integrated ARC with the Galaxy Project portal in the context of the EuroScienceGateway project. The Galaxy portal is a user-friendly job-submission and workflow platform that lets a user easily define and submit jobs to an underlying computing cluster, it allows reproducibility in addition to facilitating sharing of jobs and workflows. The Galaxy project has a large user-base from the bioinformatics communities, in addition to users from the climate, astrophysics and material science communities, to mention a few. With ARC integration in Galaxy, these new communities will seamlessly be able to enjoy the benefits of ARC by using Galaxy to submit jobs to their remote HPC system, instead of having to manually log into the HPC system and interact with the local batch system via scripting. We also present the ongoing work to make ARC available via the European Grid Infrastructure (EGI) Infrastructure Manager.

Department/s

  • Particle and nuclear physics
  • eSSENCE: The e-Science Collaboration

Publishing year

2025

Language

English

Pages

61-78

Publication/Series

Communications in Computer and Information Science

Volume

2398 CCIS

Document type

Conference paper

Publisher

Springer Science and Business Media B.V.

Topic

  • Subatomic Physics

Keywords

  • Cloud
  • Distributed computing
  • EGI
  • EuroScienceGateway Project
  • Galaxy Project
  • Grid
  • HPC
  • Middleware
  • Nordugrid ARC
  • Storage and compute

Conference name

6th Nordic e-Infrastructure Collaboration Conference, NeIC 2024

Conference date

2024-05-27 - 2024-05-29

Conference place

Tallinn, Estonia

Status

Published

ISBN/ISSN/Other

  • ISSN: 1865-0937
  • ISSN: 1865-0929
  • ISBN: 9783031862397