This is how you select the optimal container warehouse • The register
Commercially Enterprises are shortening their application development lifecycles to meet changing user demands, a trend that in turn is driving the adoption of microservices application architectures. And in the cloud-native era, they typically turn to Kubernetes containers (K8s) for this job.
Developed in 2014, Kubernetes is the portable, extensible, open-source platform designed to manage containers at scale. It also supports application extensions and failover, and allows the use of containers in production, which has helped promote container development.
A survey released in March 2022 by the Cloud Native Computing Foundation (CNCF) concluded that 96% of companies are using or evaluating K8s. These companies typically start by running stateless applications like web services on containers. But as the technology has evolved and IT departments have become more familiar with its benefits, usage has expanded to include stateful applications, including databases and middleware. The CNCF found that nearly 80% of customers plan to run stateful applications on containers, for example.
In order to store the data created by these stateful applications on a persistent disk, the K8s storage interface was separated from subsequent K8s release versions. It now functions as an independent storage interface standard – the Container Storage Interface (CSI) – for which many vendors have developed and released CSI plug-ins.
The result is that CSI-based storage can be integrated into containers to allow K8s to directly manage storage resources, including basic tasks like create, mount, delete and expand, as well as advanced operations like snapshot, clone, QoS and active-active Calls.
Adjusting storage for containerized applications
Based on the types of data stored, container storage can be categorized into SAN, NAS, and object storage, each of which is best suited for different environments.
In the early stages of container development, companies used local disks to store database applications like MySQL. At the time, datasets were small and containerized applications were not mission-critical, so investments were limited, even though these databases required high performance, availability, and low latency.
However, local disks are no longer preferred, largely because they do not support container failover in the event of a node exception, forcing manual intervention that can take several hours. In addition to poor availability and high maintenance overhead, local disk storage is also affected by resource isolation – different servers are stored in different locations, leading to difficulties in sharing the storage resource pool and inefficient capacity utilization.
SAN storage offers superior performance but does not meet the high availability requirements of the database. For example, because it is bound to the containers of a failed node, it cannot switch over automatically, requiring manual intervention again.
Some customers using MySQL databases also use enterprise-level NAS storage. In this case, automatic failover is possible because NAS supports multi-mounting. If a node fails and the containers fail, they can be redeployed to the target disk. Data stored on the NAS is also shared across multiple locations and does not need to be copied in failover scenarios. This reduces recovery times to minutes and can improve uptime up to 10x. Storage utilization is also optimized as shared NAS capacity offers a total cost of ownership that is up to 30% lower than equivalent local disk storage solutions.
New applications like AI training may require random reading/writing of billions of unstructured files ranging in size from several KB to several MB. They require simultaneous access to dozens and even thousands of server GPU resources to run, and the underlying memory needs low latency to accelerate GPU response times and increase GPU utilization.
SAN storage does not suffice here as it does not allow data to be shared across large clusters with thousands of computing servers. Likewise, object storage performs poorly on random reads/writes, and in archival applications only supports sequential reads/writes of cold data that is retained for long periods but accessed infrequently.
NAS storage bypasses limitations
NAS storage supports multi-node sharing and is the only applicable storage option in this scenario. For example, a common option for businesses is to implement a Ceph/GlusterFS-based distributed NAS solution with local server disks, with data distributed across multiple nodes. However, it is important to note that network latency issues can affect performance.
Huawei NAS Enterprise Storage can multiply the performance of this solution many times over with the same Ceph/GlusterFS configuration. For example, a large commercial bank uses Ceph distributed storage with local server disks, but the system only supported 20,000 IOPS in AI applications during a test. After replacing the existing storage system with Huawei’s all-flash solution OceanStor Dorado NAS, the bank found that the dual-controller performance easily reached 400,000 IOPS, a 20x increase in AI analysis efficiency.
The Huawei OceanStor Dorado offers leading NAS performance and reliability for workloads that handle large numbers of small files. It uses a globally shared distributed file system architecture and intelligent balancing technology to support file access across all controllers, eliminate cross-core and cross-controller transfer overheads, and reduce network latency.
Intelligent data layout technology can speed up file locating and transferring. OceanStor Dorado has demonstrated 30% better performance for applications that rely on input/output (I/O) of small files compared to the industry benchmark, enabling organizations to easily handle large volumes of small files stored in containers are.
It offers five levels of reliability covering disk, architecture, system, solution and cloud. Techniques include active-active NAS access, which operates two storage controllers simultaneously to enable fast failover and minimal disruption to NAS services in the event of a failure. It all adds up to a 7-9 reliability for always-on services and containerized data that’s always available. A more detailed description of Huawei’s OceanStor Dorado NAS all-flash storage system can be found here.
Because containers can run on any server within a cluster and can be switched from one server to another, container data must be shared between multiple nodes. Therefore, container storage solutions need to share data while handling random read/write operations generated by large amounts of small files, especially when it comes to supporting the development of new applications. Ultimately, a Kubernetes container can be summarized as an application-centric environment that stores its data in files, making Huawei’s OceanStor Dorado NAS all-flash storage system a good choice for the task.
Sponsored by Huawei.