Standard

Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment. / Nilsson, Paul; Anisenkov, Alexey; Benjamin, Doug et al.

24TH INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS (CHEP 2019). ed. / C Doglioni; D Kim; GA Stewart; L Silvestris; P Jackson; W Kamleh. University of Lueneburg, Department of Economics and Social Sciences, Research Institute on Professions, 2020. (EPJ Web of Conferences; Vol. 245).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Research › peer-review

Harvard

Nilsson, P, Anisenkov, A, Benjamin, D, Guan, W, Javurek, T & Oleynik, D 2020, Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment. in C Doglioni, D Kim, GA Stewart, L Silvestris, P Jackson & W Kamleh (eds), 24TH INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS (CHEP 2019). EPJ Web of Conferences, vol. 245, University of Lueneburg, Department of Economics and Social Sciences, Research Institute on Professions, 24th International Conference on Computing in High Energy and Nuclear Physics (CHEP), Adelaide, Australia, 04.11.2019. https://doi.org/10.1051/epjconf/202024503025

APA

Nilsson, P., Anisenkov, A., Benjamin, D., Guan, W., Javurek, T., & Oleynik, D. (2020). Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment. In C. Doglioni, D. Kim, GA. Stewart, L. Silvestris, P. Jackson, & W. Kamleh (Eds.), 24TH INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS (CHEP 2019) (EPJ Web of Conferences; Vol. 245). University of Lueneburg, Department of Economics and Social Sciences, Research Institute on Professions. https://doi.org/10.1051/epjconf/202024503025

Vancouver

Nilsson P, Anisenkov A, Benjamin D, Guan W, Javurek T, Oleynik D. Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment. In Doglioni C, Kim D, Stewart GA, Silvestris L, Jackson P, Kamleh W, editors, 24TH INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS (CHEP 2019). University of Lueneburg, Department of Economics and Social Sciences, Research Institute on Professions. 2020. (EPJ Web of Conferences). doi: 10.1051/epjconf/202024503025

Author

Nilsson, Paul ; Anisenkov, Alexey ; Benjamin, Doug et al. / Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment. 24TH INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS (CHEP 2019). editor / C Doglioni ; D Kim ; GA Stewart ; L Silvestris ; P Jackson ; W Kamleh. University of Lueneburg, Department of Economics and Social Sciences, Research Institute on Professions, 2020. (EPJ Web of Conferences).

BibTeX

@inproceedings{1ab242f5f2ad42dfabccb94833a53035,
title = "Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment",
abstract = "The unprecedented computing resource needs of the ATLAS experiment at LHC have motivated the Collaboration to become a leader in exploiting High Performance Computers (HPCs). To meet the requirements of HPCs, the PanDA system has been equipped with two new components; Pilot 2 and Harvester, that were designed with HPCs in mind. While Harvester is a resource-facing service which provides resource provisioning and workload shaping, Pilot 2 is responsible for payload execution on the resource. The presentation focuses on Pilot 2, which is a complete rewrite of the original PanDA Pilot used by ATLAS and other experiments for well over a decade. Pilot 2 has a flexible and adaptive design that allows for plugins to be defined with streamlined workflows. In particular, it has plugins for specific hardware infrastructures (HPC/GPU clusters) as well as for dedicated workflows defined by the needs of an experiment. Examples of dedicated HPC workflows are discussed in which the Pilot either uses an MPI application for processing fine-grained event level service under the control of the Harvester service or acts like an MPI application itself and runs a set of job in an assemble. In addition to describing the technical details of these workflows, results are shown from its deployment on Titan (OLCF) and other HPCs in ATLAS.",
author = "Paul Nilsson and Alexey Anisenkov and Doug Benjamin and Wen Guan and Tomas Javurek and Danila Oleynik",
year = "2020",
doi = "10.1051/epjconf/202024503025",
language = "English",
series = "EPJ Web of Conferences",
publisher = "University of Lueneburg, Department of Economics and Social Sciences, Research Institute on Professions",
editor = "C Doglioni and D Kim and GA Stewart and L Silvestris and P Jackson and W Kamleh",
booktitle = "24TH INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS (CHEP 2019)",
address = "Germany",
note = "24th International Conference on Computing in High Energy and Nuclear Physics (CHEP) ; Conference date: 04-11-2019 Through 08-11-2019",

}

RIS

TY - GEN

T1 - Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment

AU - Nilsson, Paul

AU - Anisenkov, Alexey

AU - Benjamin, Doug

AU - Guan, Wen

AU - Javurek, Tomas

AU - Oleynik, Danila

PY - 2020

Y1 - 2020

N2 - The unprecedented computing resource needs of the ATLAS experiment at LHC have motivated the Collaboration to become a leader in exploiting High Performance Computers (HPCs). To meet the requirements of HPCs, the PanDA system has been equipped with two new components; Pilot 2 and Harvester, that were designed with HPCs in mind. While Harvester is a resource-facing service which provides resource provisioning and workload shaping, Pilot 2 is responsible for payload execution on the resource. The presentation focuses on Pilot 2, which is a complete rewrite of the original PanDA Pilot used by ATLAS and other experiments for well over a decade. Pilot 2 has a flexible and adaptive design that allows for plugins to be defined with streamlined workflows. In particular, it has plugins for specific hardware infrastructures (HPC/GPU clusters) as well as for dedicated workflows defined by the needs of an experiment. Examples of dedicated HPC workflows are discussed in which the Pilot either uses an MPI application for processing fine-grained event level service under the control of the Harvester service or acts like an MPI application itself and runs a set of job in an assemble. In addition to describing the technical details of these workflows, results are shown from its deployment on Titan (OLCF) and other HPCs in ATLAS.

AB - The unprecedented computing resource needs of the ATLAS experiment at LHC have motivated the Collaboration to become a leader in exploiting High Performance Computers (HPCs). To meet the requirements of HPCs, the PanDA system has been equipped with two new components; Pilot 2 and Harvester, that were designed with HPCs in mind. While Harvester is a resource-facing service which provides resource provisioning and workload shaping, Pilot 2 is responsible for payload execution on the resource. The presentation focuses on Pilot 2, which is a complete rewrite of the original PanDA Pilot used by ATLAS and other experiments for well over a decade. Pilot 2 has a flexible and adaptive design that allows for plugins to be defined with streamlined workflows. In particular, it has plugins for specific hardware infrastructures (HPC/GPU clusters) as well as for dedicated workflows defined by the needs of an experiment. Examples of dedicated HPC workflows are discussed in which the Pilot either uses an MPI application for processing fine-grained event level service under the control of the Harvester service or acts like an MPI application itself and runs a set of job in an assemble. In addition to describing the technical details of these workflows, results are shown from its deployment on Titan (OLCF) and other HPCs in ATLAS.

U2 - 10.1051/epjconf/202024503025

DO - 10.1051/epjconf/202024503025

M3 - Conference contribution

T3 - EPJ Web of Conferences

BT - 24TH INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS (CHEP 2019)

A2 - Doglioni, C

A2 - Kim, D

A2 - Stewart, GA

A2 - Silvestris, L

A2 - Jackson, P

A2 - Kamleh, W

PB - University of Lueneburg, Department of Economics and Social Sciences, Research Institute on Professions

T2 - 24th International Conference on Computing in High Energy and Nuclear Physics (CHEP)

Y2 - 4 November 2019 through 8 November 2019

ER -

ID: 34665337