Today’s proliferation of high-performance computing (HPC) initiatives — from AI and machine learning to complex research and real-time analytics — is increasing the demand for HPC storage systems. However, traditional data architectures and file systems are hitting their performance and scalability limits, causing leading enterprises to turn towards new disaggregated storage architectures.
This post will explore VAST’s Disaggregated, Shared-Everything (DASE) architecture, which offers high performance without the common complexity of parallel file systems. It will also discuss the advantages of disaggregated data over legacy data storage methods, and reveal some of the ways in which data-driven organizations and scientific institutions are leveraging DASE with strong results.
What Is Disaggregated, Shared-Everything (DASE)?
Disaggregated Shared-Everything architecture — or DASE, for short — is a modern data storage approach developed by VAST. DASE decouples the storage media from the CPUs that manage that media and provide storage services. This disaggregated storage, including all system metadata, is then shared by all servers in the cluster.
Why separate the storage and compute resources in this way? Simply put, it allows data storage capacity and computational power to be scaled independently as needed, which has significant cost advantages given the high demands of AI workloads. Additionally, DASE eliminates the need for complex storage tiering, and simplifies the overall data architecture for more efficient growth and maintenance.
With DASE, all VAST Data servers share metadata and capacity, giving clients unparalleled flexibility and cost advantages over other HPC storage systems.
Problems with Traditional Architectures
DASE was created because traditional data architectures couldn’t perform well in the age of heavy AI and HPC workloads. Common problems with legacy data systems include:
Wide Distribution: With a shared-nothing approach and an over-reliance on client-server models, legacy data systems quickly become heavily distributed, requiring frequent data requests and transfers which impact system speed and performance.
Parallel Complexity: Parallel file systems (PFS) such as Lustre, GPFS, and BeeGFS, while high-performing, are complex to set up and difficult to maintain. This puts a strain on system administrator workload as well as company finances.
Frequent Bottlenecks: Legacy data architectures were not designed to handle the immense data volumes of HPC and AI. Therefore, attempting to use them for this purpose requires significant tuning and restructuring — rarely producing a satisfactory outcome and often hindering the ability to scale.
Benefits of a Disaggregated Storage Architecture for HPC and AI
DASE is an AI-native disaggregated storage architecture that meets all the requirements of a modern HPC storage system. Developed to combine the advantages of traditional file systems without the inherent limitations, DASE offers a number of key benefits for today’s data-driven enterprises:
High Throughput: DASE and its flexible data storage clusters are equipped to handle exabyte-scale volumes of data, combining PFS performance with the simplicity of enterprise network-attached storage (NAS) systems.
Low Latency: Thanks to its lean, disaggregated design, DASE allows these high volumes of HPC data to flow through the system with minimal lag — further supporting scale and improving the energy efficiency of the overall system.
Reduced Overhead: The decoupled nature of the DASE architecture means that storage and computational resources are only bought as they’re needed. This eliminates the common phenomenon of having to pre-purchase large blocks of data or server capacity that may never get used, and preserves operational overhead for enterprises.
Greater Reliability: As a managed architecture powering the VAST AI Operating System (AI OS), DASE comes with built-in support for cloud, core, and edge workflows, as well as an unrivaled uptime — giving data teams back their time and peace of mind.
These clear advantages explain why AI and data teams are increasingly turning to DASE and the VAST AI OS for a wide variety of HPC storage use cases.
Real-World DASE Use Cases
How is a disaggregated storage architecture used for modern HPC storage systems? Let’s explore a few ways in which VAST Data clients are using DASE to get ahead.
AI Model Training
VAST’s DASE architecture simplifies the entire AI development workflow by providing a unified platform that can handle all data formats across each stage of the AI pipeline — from data capture and preparation to model training and serving. DASE removes the typical data bottlenecks and ensures a smooth, continuous flow of data throughout the pipeline with no lag or downtime, accelerating AI model training and development.
VAST client Hypertec Cloud is a leading provider of AI and high-performance Infrastructure-as-a-Service (IaaS) solutions. To deliver their secure, reliable, sustainable, and cost-effective AI and HPC IaaS solutions at scale across North America — with enough capacity to power hundreds of thousands of GPUs for the largest and most demanding AI training workloads — Hypertec Cloud relies on the VAST Data and its innovative DASE architecture.
Scientific Simulation and Research
DASE is uniquely capable of meeting and exceeding the rigorous data demands of leading scientific research. With all-flash performance that is orders of magnitude faster than the spinning disk operation of traditional data servers, computational jobs finish faster, accelerating time-to-discovery and making scientific HPC data centers more sustainable. Additionally, researchers are able to search for and find data more easily, and administrators can analyze application behavior and audit access with VAST’s intuitive UI and SQL interface for advanced queries.
The Texas Advanced Computing Center (TACC) has been pioneering computer systems for decades, with their research and development efforts producing some of the fastest computers in the world. To support their work in exascale AI and HPC system research, TACC has turned to the VAST Data and the power of DASE for their supercomputing support.
Cloud Bursting and Hybrid HPC Environments
DASE architecture also supports hybrid HPC environments that utilize techniques such as cloud bursting — the practice of extending data storage capacity beyond an internal file system and onto public cloud computing resources to handle temporary processing spikes and/or manage costs. In this way, the VAST Data AI OS enables data-driven organizations to seamlessly burst workloads to the cloud as needed — or as dictated by defined data access policies — while maintaining a unified data experience and simplified management process.
CINECA is Italy’s largest HPC/cloud infrastructure for public research, supporting scientists and engineers from 70 Italian universities and 50 research institutions. After years of working with complex parallel file systems, CINECA was looking for an easier-to-manage platform that could still deliver the performance they needed for their HPC, cloud, AI, and big data services. VAST Data offers CINECA a centralized, high-performance scale-out foundation for the highly complex computational workloads they are trusted to support.
DASE Is the Architectural Foundation, Not a Feature
At VAST, we’re so confident in disaggregated storage architectures that we’ve deployed DASE to underpin the VAST AI OS. As such, DASE is not a feature of our HPC storage system — it’s the entire foundation. Here’s a recap of the value DASE brings to modern high-performance computing that traditional file systems can’t match:
Higher data throughput and exabyte-scale
Ultra high data performance with low latency
Significantly less complexity and operational overhead
Maximum reliability, uptime, and support
The VAST AI OS accelerates HPC by delivering the performance and scale of parallel file systems with the simplicity of NAS, all at archive economics. With its multi-protocol support, modern analytics capabilities, and AI-native structure, DASE is the new, future-proof approach to HPC storage and compute convergence. Explore how the VAST AI OS supports HPC use cases, or reach out to our team today to talk to an architect.
