Project Goal

This project is focused on the industry evolution of CERN’s EOS large-scale storage system. The goal is to simplify the usage, installation, and maintenance of the system. The project also aims to add native support for windows systems, expand documentation, and implement new features/ integration with other software packages.

Background

Within the CERN IT department, the Storage and Data Management group (IT-SD) is responsible for the operations and development of the main data storage infrastructure. This infrastructure crossed in 2023 the 1 exabyte of raw disk storage and it is used to store the physics data generated by the experiments at CERN, as well as the files of all members of personnel. EOS is a high-performance disk-based storage system developed at CERN. It is tailored to handle large data rates with low-latency metadata access from the experiments, while also running concurrent complex production workloads. EOS is also the key storage component behind CERNBox, CERN’s cloud-synchronisation service which allows to sync and share files on all major mobile and desktop platforms (Linux, Windows, macOS, Android, iOS), with the aim of providing offline availability to any data stored in the EOS infrastructure.

Progress

After successfully developing a native Windows client for EOS and a native drive interaction in the Windows environment, in 2023 Comtrade’s team invested in optimizing and benchmarking multiple clients and distributed setup to further improve their EOS Windows Native Client (EOSwnc) performance and demonstrate the full capability of this new client.

A comprehensive dedicated full-stack environment, hosted at Comtrade, was deployed. Rigorous testing of various storage solutions was conducted using identical hardware to ensure the utmost accuracy in results. EOS, Ceph, Hadoop Lustre and IBM Spectrum Scale were tested and compared, showing excellent overall results. Measurements were made both with CentOS7 and ALMA8.8.

Next Steps

The next primary objective is to identify a strategic industry partner to demonstrate the EOS software, along with the additional dedicated resources, including comprehensive documentation and software, developed and produced by Comtrade. This future collaboration will aim to pioneer the creation of an initial prototype for a full-storage appliance solution, with plans to present it to other prospective customers in the near future.

 

Project Coordinator: Luca Mascetti

Technical Team: Manuel Reis, Elvin Sindrilaru

Comtrade Collaboration Liaisons: Ivan Arizanović, Branko Blagojević, Svetlana Milenković, Gregor Molan, Anđelina Petrović

In partnership with: Comtrade