Itzikr's Blog

Built on the 2 decades long history of Object Storage in Dell EMC, ECS is the ideal storage platform for powering the modern data lake.

As the leading object storage system, the ECS product line features capacity- and performance- optimized appliances, allowing your organization to both: establish deep archives with low TCO and also fuel next-gen workloads. The newest addition to our EX-Series appliance family, the all-flash EXF900, delivers up to 21x the performance over our capacity optimized EX300. This kind of performance means you can feed AI, analytics and cloud-native apps with more data, faster than ever before.

Couple ECS scale and speed with multiprotocol flexibility, and you get a versatile storage platform that can consolidate multiple workloads into a unified data lake.

Customers choose ECS time and again because it is an enterprise proven platform than can scale at the pace of your organization’s data growth.

We are now living in the data era, an age of untold potential and challenges.

The total amount of data generated around the planet is well documented. IDC predicts that the Global Datasphere will grow to 175 zettabytes by 2025, more than 90% of which is unstructured in nature.¹ This explosion of images, videos, social media data, internet traffic, and device telemetry has forced organizations to rethink their data storage strategies in recent years. By its very nature, unstructured data is more difficult to store, analyze and utilize than traditional structured data types since it isn’t organized neatly into databases that can be easily mined for insights. Unstructured data also presents challenges in making use of it in a timely manner. The data science behind the innovation and profitability locked away in unstructured data is a precious commodity. And if data is not easy to discover, access, and use, it can significantly hamper an organization’s ability to glean value from it at a speed that delivers competitive advantage.

And the pace of data growth is increasing as well. IDC predicts that by 2025, file and object capacity will grow by ~300%,² requiring organizations to invest in rapidly scalable and adaptable systems that can combat this unyielding growth.

Compounding the issue further, we are also in the midst of an application design revolution. Many companies are going through the process of re-platforming their workloads and developing new cloud-native applications that require modern storage solutions to function optimally. As organizations turn to new application architectures, microservices, containers, and the like, they are finding that traditional storage solutions and databases do not offer the simplicity and flexibility these new workloads require. By 2024, 70% of all new apps will be production-grade, cloud-native apps because of the adoption of technologies such as microservices, containers, dynamic orchestration, and DevOps.³

To that end, organizations require a flexible storage platform that aligns strategically with their application development processes. Those who don’t establish that foundation will lose to competitors who do.

Object storage is quickly becoming the de-facto primary storage for many organizations and the standard for cloud-native applications, which are becoming ubiquitous within the enterprise. This is due to some of the advantageous characteristics of object storage, namely its massive scalability, simplicity to manage and develop against, metadata, APIs and economics.

• Scalability: Designed with a horizontal, scale-out architecture and a single, global namespace, organizations can continuously add nodes to the storage pool without impacting its overall performance.
• Simplicity: Object storage is also much easier to manage and maintain. Once you hit a certain scale, block and file storage becomes untenable to utilize as you are forced to split up datasets into multiple LUNs to get an acceptable level of performance, introducing complexity and maintenance overhead.
• Metadata: Object storage is also unique in that organizations can tag objects with additional metadata, beyond what you get with a basic file system, so that data “objects” can be located quickly based on user-defined attributes and information. This is especially useful for developing business insights, in analytics scenarios and for training machine learning algorithms.
• APIs: Object storage also utilizes a higher order, born-in-the-cloud S3 (Simple Storage Service) API (popularized by Amazon) that makes it easier to integrate software both in the public cloud or on-premises. Most modern and cloud-native applications are being developed using the S3 API, allowing them to be moved between on-prem and cloud environments without a loss in functionality.
• Economics: Traditionally, object storage systems were seen as “cheap and deep” storage platforms for backups, content repositories and long-term data archives. Object storage has most often been used primarily as a second-tier repository to free up more performant and expensive tier-one storage. This is due to its proven record of being cost effective at scale. However, this view is beginning to shift as organizations begin to realize the advantages object storage can offer—flexibility, simplicity, cloud-native architectures, etc. This, combined with the continuous drop in the cost of all-flash media, has primed the object storage market for a revolution.

Object-based data continues to power cloud-native, AI, machine learning, IoT, and analytics applications using S3-based object as the underlying storage layer. These applications create enormous amounts of data, a challenge object storage is good at handling, but the performance was always lacking. Now, combined with fast, all-flash media becoming less cost prohibitive, object storage is being viewed as an optimized storage platform for these next-generation workloads as well.

Replication between regions now uses the standard cross-region replication feature in compliance with AWS’s S3 protocol

• Replication is now bucket-to-bucket
• Can be different for every bucket depending on redundancy / locality needs
• Can be unidirectional or bidirectional
• Can be part of a bucket (by prefix)
• Up to 1000 rules per bucket
• Network throttling and priority per policy / replication link / global
• Example New Use cases
  • Fan-in: 1000 bank branches back up surveillance video to core
  • Fan-out: core replicates billboard content to 1000 bank branches

Replicate data across any number of sites

Share data with siloed teams

Create a globally-connected data lake where data can be passed from edge, to core, to cloud

Ingest and do analytics at the edge, replicate the data to a core data center

Gives you more control for the developer and the application – pause, start, check, validate

Dell ObjectScale today’s key requirements for developers and IT. Built with a software-defined, containerized architecture ObjectScale delivers enterprise-class, high performance object storage in a Kubernetes-native package.

ObjectScale empowers organizations to move faster and respond effectively to rapidly changing business needs. It provides DevOps teams the freedom to deploy and scale S3-compatible object storage to support modern data applications that drive your business while ensuring exceptional simplicity, manageability and reliability.

In the following slides, we’ll explore three key benefits of ObjectScale:

• Simplify software-defined: Provide DevOps teams with Kubernetes-native object storage in a software-defined architecture.
• Power modern apps at scale: Support modern S3 workloads such as artificial intelligence, analytics and media content.
• Secure your object data: Protect at scale with technology you can trust from the #1 in object security.

Simpler out-of-the-box and overall experience

• Dell support for entire software stack, including Kubernetes (SW Bundle, available at RTS)
• Supports Red Hat OpenShift Container Platform 4.12
• Much simpler and faster to deploy and configure
• Simpler, more intuitive interface with streamlined bucket provisioning
• Now available in CloudIQ

Breakthrough performance and efficiency

• Large write performance up to 2GB/node
• Support for objects as large as 30TB
  • 6X more than the 5TB object support limit on AWS using Multipart Upload
  • Supporting use cases such as large HPC, analytics and AI projects as well as related backup requirements
  • More efficient read/write performance for small and medium-sized objects

Provides logical capacity usage for APEX

Enhanced Zero Trust features

• Broader, more flexible multi-tenancy
  and access
  • AD/LDAP support
  • Management IAM model
  • Remote Connect Assist via PAM
• More flexible erasure coding for better storage efficiency and reliability
  • Fewer nodes required
  • 4 nodes with 12+4 scheme
  • 2 nodes in Maintenance mode [10+2]

ObjectScale leverages the native orchestration capabilities of Kubernetes—scheduling, load-balancing, self-healing, resource optimization—to deliver enterprise-grade object storage in a simple, software-defined package.

By running ObjectScale in your data center, you can control shadow IT while freeing developers to support CI/CD processes and agile methodologies in a self-service manner via Kubernetes APIs. The end result is faster time-to-market, improved security, cost reduction and true data innovation.

ObjectScale provides a choice of deployment options: through a Software Bundle, where Dell managesthe software stack including Kubernetes, or As an Application where ObjectScale runs on Red Hat OpenShift you manage. These options allow developers to focus on code while IT professionals manage ObjectScale storage to policy.

• ObjectScale Community Edition is a free, community-supported option with no expiration. It provides up to 30TB of storage.

There are two options available by subscription or TLA:

• ObjectScale Software Bundle with Dell-supported Kubernetes runs on the customer’s HW and SLES O/S.
• ObjectScale
  As an Application with Red Hat OpenShift runs directly on disks/media and is deployable via Helm charts.

All these options provide the following benefits:

• Simpler pricing model that doesn’t change per region
• No data transfer, request and retrieval pricing tiers
• No data management or replication considerations
• No misaligned pricing choices

Ease of management

With the ObjectScale 1.2 update, you can deploy and configure the entire SW bundle software stack quickly via a single CLI.

IT operators can manage billons of objects, millions of buckets and petabytes of data with low overhead using a centralized, intuitive GUI and built-in reporting.

Storage clusters can be deployed in minutes and pushed to production or decommissioned at any time with a few simple clicks.

The GUI provides information on capacity utilization, logs, performance monitoring, and diagnostics across clusters. Native Grafana integration provides workload performance visualization.

ObjectScale makes use of an intelligent sizer tool to optimally configure the storage environment for your unique workload demands. Simply provide ObjectScale with key information such as capacity and the expected data growth rate and ObjectScale will automatically provision the right resources so you don’t have to guess.

RESTful management APIs also enable easy integration into existing management and directory solutions.

Available in CloudIQ

With ObjectScale 1.2, the software is now available in CloudIQ as part of your broader Dell infrastructure and data protection monitoring experience.

Cloud and List views in CloudIQ currently include:

• System Inventory
• System Capacity
• System Alerting
• Pool Inventory
• Pool Capacity
• Tenant Details

Built on a scale-out architecture, ObjectScale clusters expand from a few terabytes to petabytes and beyond without limits on the number of object stores, buckets or objects stored. Because ObjectScale is software-defined, it’s easy to keep pace with exponential data growth, wherever it occurs. Clusters start as small as three nodes and scale into the 1000s. With ObjectScale, support workloads at public cloud-scale with the reliability and control of a private cloud infrastructure.

S3-compatible – Implementing the latest S3 APIs

• Bucket-to-bucket replication
• Enables data redundancy and fault tolerance
• Enables data to be shared across regions
• Notifies users of bucket events
• Example: object creation / deletion
• Configurable at the bucket level
• Protects data from unauthorized access
• Sets retention policies for buckets & objects
• Enables compliance
• Retrieves only the data needed for the application
• Accelerates query performance
• Object formats: CSV, JSON, Parquet
• Adapter to replace HDFS
• Enables Hadoop
  to directly read & write S3 objects
• Native object advantages: cost, scale, durability, data protection

ObjectScale is optimal as enterprise-grade storage supporting large-scale event streams, log analytics and complex AI data pipelines.

Let’s say there is a large amount of telemetry data coming off edge devices and sensors such as manufacturing machines on a plant floor or geospatial data coming in from cars in an ADAS scenario.

• Organizations can ingest, analyze and label the telemetry coming from these sensors, accessing a single unified stream.
• Because of the cost-effectiveness, massive scalability, global accessibility and performance of ObjectScale, it is an excellent platform for long-term storage of the ingested data.
• ObjectScale can deliver datasets to the most demanding servers, efficiently exposing training algorithms to more data and supporting metadata search.
• The data can then be accessed by numerous applications, algorithms and additional data processing techniques—and then visualized through a range of available tools.
• The approach not only simplifies operations and reduces TCO—but also enables organizations to more easily mine data for unique insights and unlock new opportunities to further monetize their data.

• Supports an event-driven architecture
• Pushes notifications when objects are created, deleted, accessed, a policy changes, etc.
• Built-in notifications, push messaging and message queuing services are optimized
  for performance
• Invoke data processing services or conduct audits through events to enable serverless compute architectures
• Used for machine learning, log analytics, security and functions-as-a-service

• Rather than using compute resources to query and sort object data, offload query to ObjectScale
• Accelerates query performance
• Query pushdown supported applications such as Spark, Hive and Presto

Object formats: CSV, JSON, Parquet

With ObjectScale 1.2, the software introduces breakthrough performance and efficiency features to support the most demanding workloads, including large HPC, analytics and AI projects as well as related backup requirements. New features include:

• Large write performance up to 2GB/node — as well as more efficient read/write performance for small and medium-sized objects .
• Fast, stable support for objects as large as 30TB — 6X more than the Amazon S3 maximum object size using Multipart Upload.
• Logical capacity usage for APEX, enabling future capacity-based billing via the CloudIQ console and reports.

• ObjectScale Replication API compliant with AWS Cross-Region Replication
• Bucket-to-bucket copy of objects using asynchronous replication
• One-to-one, one-to-many and many-to-one bucket replication
• Enterprise grade Bidirectional or unidirectional with data validation built-in
• Can pause, restart, rate limit and check status of replication

Multiple site failure tolerance

• Stores objects in a write-once-read-many (WORM) model
• Prevents objects and/or buckets from being deleted or overwritten
• Objects can be locked for a fixed period of time or indefinitely at write time or by using policies
• Retention modes to meet business requirements: Governance, Compliance and Legal Hold

Identity & Access Management

• Empowers users to control “who” can perform “what” actions on “which” resources under “what conditions” (permissions and access)
• Comprehensive (expanded) IAM model with support for a wider set of granular permissions, conditions and policies
• Local or Management Users support
• External Auth Provider support
• Login workflows
• User Persona roles and responsibilities
• Password and account lockout configurations
• Identity federation via SAML 2.0 (SSO), multi-factor authentication, IAM roles and user policies
• Enables bucket, object and user tagging and tags to be referenced in IAM policies

Data integrity – Multiple protection mechanisms

Erasure coding

• Provides efficient data protection from a disk, node and rack failure
• More flexible erasure coding options available

Data validation

• Ensure data integrity throughout the IO lifecycle
• Write operation
• Read operation
• Data at rest

Data consistency

• Strong read-after-write consistency within object stores for GET, PUT and LIST operations
• Asynchronous replication across sites

ObjectScale is also a TCO-optimized S3 backup target and long-term archive. Featuring ObjectScale Lock for data immutability, data-at-rest-encryption (D@RE), global replication and erasure coding protection schemes, ObjectScale safeguards data from ransomware attacks and ensures resiliency from node or disk failures. With ObjectScale, everything from VMs to Microsoft Office 365 backups are secure and available. It’s business continuity at scale.

With ObjectScale 1.2, the software introduces even more in-built data protection and resiliency (Zero Trust features) to help you secure your object data.

• Broader, more flexible multi-tenancy and access capabilities
  • Active Directory (AD) and LDAP support for system users
  • Comprehensive Management IAM model including:
    • Local or Management Users support
    • External Auth Provider support
    • Login workflows
    • User Persona roles and responsibilities
    • Password and account lockout configurations
  • Remote Connect Assist via PAM (Pluggable Authentication Module), enabling Remote Secure Credentials token for remote support
• More flexible erasure coding for better storage efficiency and reliability
  • Fewer nodes required vs. release 1.0
  • Now supports 4 nodes with 12+4 scheme
  • Now supports 2 nodes in Maintenance mode

You can download the free, 1.2 community edition of Dell ObjectScale, using the link below Dell ObjectScale | Dell USA