HPC Storage
The Confluence of Simulation and AI Changes Everything
Overview
HPC is transforming. Engineers and researchers now face exploding data growth—which far exceeds price/performance improvements in underlying storage media technologies. We also see a convergence of modeling and simulation, along with rapid growth of artificial intelligence (AI) workloads. These challenges have created new requirements for compute software, networking, and storage.
Called the Exascale Era by some, it represents far more than a performance milestone; it is a technology inflection point characterized by a new set of capabilities for a new set of workloads running in mission- or business-critical workflows on one machine. And it affects virtually every industry and field of inquiry.
Today, these new realities are putting intense demands on high-performance storage, which is impacting HPC users. For example, a recent study of the independent analyst firm Intersect360 found that the majority (61%) of HPC users are already running Machine Learning (ML) programs.
Storage technology that worked for Petascale Era workloads cannot power the Exascale era’s converged workflows because the I/O patterns and the characteristics of the currently deployed storage technologies could not be more different. As a result, sticking with your current storage infrastructure will leave you unable to keep up—both in terms of performance and budget.
The workload convergence of classic modeling and simulation with AI has fundamental implications for HPC storage as the I/O profiles of the different workloads could not be more different:
- Modeling and simulation: Mainly writing petabytes of large files in sequential order
- AI/machine learning: Mainly reading terabytes of files of all sizes in random order
HPE Parallel File System
HPE Parallel File System Storage offers cost-effective parallel storage for your HPC simulations, AI, and data analytics environments running on HPE ProLiant systems. HPE Parallel File System Storage provides multiples of performance and namespace scalability compared to standard scale-out NAS storage. This superior scalability increases utilization of your compute nodes by removing I/O bottlenecks. It also provides cost savings through storage island consolidation in a unified, high-performance namespace.
For customers requiring a parallel file system based on standard server building blocks, we also offer HPE Parallel File System Storage. It embeds IBM Spectrum Scale Erasure Code Edition, a General Parallel File System (GPFS) for the enterprise. A software-defined storage solution built on cost-effective HPE ProLiant DL Gen10 Plus servers, HPE Parallel File System Storage offers a robust set of enterprise-class storage features, including data availability (backup and disaster recovery), data accessibility (NFS, SMB, HDFS, Object storage), and data compliance (audit log, industry certifications). Key features include:
- Leading parallel file system based on HPE ProLiant DL325 Gen10 Plus servers
- Starts as low as 27 TB in just four rack units and scales to more than 25 PB in a single file system
- InfiniBand EDR/HDR or 100/200GbE connectivity
- Comprehensive support services (hardware and software) from HPE Pointnext Services
- Built with AMD EPYC processors with PCIe Gen4
Cray ClusterStor
Purpose-engineered to meet the demanding input/output requirements of supercomputers and HPC, this parallel storage solution offers optimal performance. Its efficient design also reduces the number of storage drives, meaning customers can cost-effectively accelerate time-to-insight by supporting more CPU/GPU compute nodes.
