Wow, we have just recently released CSM 1.1 (which you can read all about, here) and we are already releasing 1.2

Container Storage Modules (CSMs) build on top of the Container Storage Interface (CSI) foundation to deliver robust and powerful enterprise storage capabilities for Kubernetes clusters. CSMs are software modules for storage management that go beyond CSI functionality. The objective of these modules is to expose enterprise storage features to Kubernetes users. CSM aims at improving the monitoring, observability, usability, resiliency, and data mobility for stateful applications with Dell Technologies Storage portfolio. CSM, with the CSI plugins and the pioneering app-aware, app-consistent backup and recovery solutions, form the most comprehensive enterprise grade storage and data protection solutions for Kubernetes from Dell Technologies.

So first of all, we’ve updated our support matrix by adding more operating systems, container orchestrator platforms and recent versions:

	PowerMax	PowerFlex	Unity	PowerScale	PowerStore
Kubernetes	1.21, 1.22, 1.23	1.21, 1.22, 1.23	1.21, 1.22, 1.23	1.21, 1.22, 1.23	1.21, 1.22, 1.23
RHEL	7.x,8.x	7.x,8.x	7.x,8.x	7.x,8.x	7.x,8.x
Ubuntu	20.04	20.04	18.04, 20.04	18.04, 20.04	20.04
CentOS	7.8, 7.9	7.8, 7.9	7.8, 7.9	7.8, 7.9	7.8, 7.9
SLES	15SP3	15SP3	15SP3	15SP3	15SP3
Red Hat OpenShift	4.8, 4.8 EUS, 4.9	4.8, 4.8 EUS, 4.9	4.8, 4.8 EUS, 4.9	4.8, 4.8 EUS, 4.9	4.8, 4.8 EUS, 4.9
Mirantis Kubernetes Engine	3.4.x	3.4.x	3.5.x	3.4.x	3.4.x
Google Anthos	1.6	1.8	no	1.9	1.9
VMware Tanzu	no	no	NFS	NFS	NFS
Rancher Kubernetes Engine	yes	yes	yes	yes	yes
Amazon Elastic Kubernetes Service Anywhere	no	yes	no	no	no

In addition, we’ve updated the CSI drivers and added unique features for each storage product –

Features	PowerMax	PowerFlex	Unity	PowerScale	PowerStore
CSI Driver version	2.2.0	2.2.0	2.2.0	2.2.0	2.2.0
Static Provisioning	yes	yes	yes	yes	yes
Dynamic Provisioning	yes	yes	yes	yes	yes
Expand Persistent Volume	yes	yes	yes	yes	yes
Create VolumeSnapshot	yes	yes	yes	yes	yes
Create Volume from Snapshot	yes	yes	yes	yes	yes
Delete Snapshot	yes	yes	yes	yes	yes
Access Mode	RWO/ RWOP(FC/iSCSI) RWO/ RWX/ ROX/ RWOP(Raw block)	RWO/ROX/RWOP RWX (Raw block only)	RWO/ROX/RWOP RWX (Raw block & NFS only)	RWO/RWX/ROX/ RWOP	RWO/RWOP (FC/iSCSI) RWO/ RWX/ ROX/ RWOP (RawBlock, NFS)
CSI Volume Cloning	yes	yes	yes	yes	yes
CSI Raw Block Volume	yes	yes	yes	no	yes
CSI Ephemeral Volume	no	yes	yes	yes	yes
Topology	yes	yes	yes	yes	yes
Multi-array	yes	yes	yes	yes	yes
Volume Health Monitoring	yes	yes	yes	yes	yes

Resolved Issues:

The following issues were fixed in this release:

• Dependabot alerts on CSI drivers.
  

• Remove trivy scan from build to avoid podman build failures.
  
• Removing dirty bit flag from CSI PowerMax 2.2.0 image. Incorrect path for privatemount directory for unity NFS.)
  
• Documentation in values.yaml related to toleration for running on master node is incorrect – CSI driver for Unity-XT.
  
• Volume Health Monitoring section is missing under driver install using Operator.
  
• Error while creating RO volume from snapshot with different isiPaths.
  
• CSI Specification in documentation causes confusion for some users.
  
• Leader election timeout flags are not present in operator but present in helm for CSI-PowerScale.
  
• Tenant deletion should cancel tenant revocation.
  
• CSI request and response ID’s are not logged in the driver.
  
• CSM authorization / PowerMax / Misleading 401 error on quota violation.
  
• Couldn’t fence volumes.
  
• Scaled down pod and got files from a different volume – Intermittent.
  
• Improve deployment documentation for CSM Authorization.
  
• Fix the perf issue related to FQDN resolution during export update with clients on Isilon.
  
• CSI-PowerFlex logs do not contain CSI request IDs on the Request and Response lines.
  
• CSM Observability helm charts: Make app.kubernetes.io/name and name consistent.
  
• CSI-provisioner container panic issue.
  
• Go Mod tidy issues while building the image.
  
• Gosec for PowerMax is reporting failure.
  
• Verification of secrets repeated twice while installation of driver via helm.
  
• CSM Observability documentation is complicated and causing confusion.
  
• Documentation has references to using the CSM Installer as recommended method.
  
• Unity CSI node driver reports “invalid memory address or nil pointer dereference”.
  
• Force delete of pod kicks in late (pod in terminating state for a while).
  
• CSM Authorization sidecar install fails if k8s worker nodes are not in ~/.ssh/known_hosts.
  
• Container is terminated but Pod is stuck in terminating.
  
• Dell CSI Operator listed two times after upgrade (1.2.0 + 1.5.0).
  
• Failing to create replicated volumes with Integration tests.
  
• CSI-PowerScale installation fails when reverse DNS lookup is unavailable.
  
• Integration tests for replication is failing with Unsupported replication type.
  
• Issues while using PowerFlex secret for Observability.
  
• Replication Metro mode is not supported.
  
• Documentation improvement recommendations for PowerScale.
  
• Driver logs show dirty tag in version.
  
• Unit test for csm-metrics-powerstore fails periodically.
  

Now, let’s review the most important features per storage array

Container Storage Modules (CSM) 1.2 contains the following features/enhancements:

• CSM Operator support for PowerScale and Authorization
  

  Container Storage Modules (CSM) for Authorization is part of the open-source suite of Kubernetes storage enablers for Dell products.
  

  CSM for Authorization provides storage and Kubernetes administrators the ability to apply RBAC for Dell CSI Drivers. It does this by deploying a proxy between the CSI driver and the storage system to enforce role-based access and usage rules.
  

  Storage administrators of compatible storage platforms will be able to apply quota and RBAC rules that instantly and automatically restrict cluster tenants usage of storage resources. Users of storage through CSM for Authorization do not need to have storage admin root credentials to access the storage system
  

  Detailed information can be found here.
  

• CSM Replication support for PowerScale
  

  Container Storage Modules (CSM) for Replication is part of the open-source suite of Kubernetes storage enablers for Dell products.
  

  CSM for Replication project aims to bring Replication & Disaster Recovery capabilities of Dell Storage Arrays to Kubernetes clusters. It helps you replicate groups of volumes using the native replication technology available on the storage array and can provide you a way to restart applications in case of both planned and unplanned migration.
  

  Detailed information can be found here.
  

• CSM for Resiliency upgrade support – Ability to upgrade the Resiliency module via the Helm chart
  

  CSM for Resiliency can be upgraded as part of the Dell CSI driver upgrade process. The drivers can be upgraded either by a helm chart or by the Dell CSI Operator. Currently, only Helm chart upgrade is supported for CSM for Resiliency.
  

  For information on the PowerFlex CSI driver upgrade process, see PowerFlex CSI Driver.
  

  For information on the Unity CSI driver upgrade process, see Unity CSI Driver.
  

    To upgrade CSM for Resiliency with the driver using Helm Chart, the following steps are required.

Note: These steps refer to the values file and csi-install.sh script that were used during initial installation of the Dell CSI driver.

Steps

1. Update the podmon.image value in the values files to reference the new podmon image.
   
2. Run the csi-install script with the option –upgrade by running: cd ../dell-csi-helm-installer && ./csi-install.sh –namespace vxflexos –values ./myvalues.yaml –upgrade.
   

For additional information please visit the CSM documentation site: https://dell.github.io/csm-docs/docs/resiliency/upgrade/.

• CSM for Observability upgrade support – Ability to upgrade the Observability module via Helm chart and online installer
  

  CSM for Observability upgrade can be achieved either by Helm chart upgrade or using the online installer upgrade path.
  

  For additional information please visit the CSM documentation site: https://dell.github.io/csm-docs/docs/observability/upgrade/.

  Upgrade
  

  Helm Chart Upgrade
  

  Helm chart upgrade now supports upgrade for the Offline Installer if it was used as an initial deployment.
  

  Steps:
  

  helm repo update
  helm search repo dell
  helm upgrade –version $latest_chart_version karavi-observability dell/karavi-observability -n $namespace

Online Installer Upgrade

The Online Installer upgrade procedure can be used if the initial deployment was performed using the Online Installer or Helm.

Steps:

1. Change to the karavi-observability installer directory: cd karavi-observability/installer
   
2. Update values.yaml file as needed. Configuration options are outlined in the Helm chart deployment section.
   
3. Execute the./karavi-observability-install.sh script: ./karavi-observability-install.sh upgrade –namespace $namespace –values myvalues.yaml –version $latest_chart_version
   

• CSM for Resiliency supports Kubernetes Control Plane Failure
  

  CSM for Resiliency will taints node on K8S Control Plane Failure. This will happen when kubelet service is not running in node of a cluster.
  

  To enable this functionality either “skipArrayConnectionValidation” parameter is set to true in podmon controller args during CSI Driver install/upgrade.
  OR
  Configurable parameter “PODMON_SKIP_ARRAY_CONNECTION_VALIDATION” is set to true in configmap of CSI Driver.
  

• CSM Log Collector
  

CSM Log Collector collects all the necessary logs and push it to engineering team.

This application can be used to collect logs from any of the Kubernetes environment where Dell csi driver is running.

It basically collects following logs:

1. Controller logs including all side cars’ log
   
2. Node logs including all side cars’ log
   
3. Controller and Node describe logs
   
4. If normal pod is stuck or is not in running state, then will capture description of that particular POD
   
5. Status of all PVCs
   
6. Status of all PODs
   

And successful collection, all of the above logs will compress those to one tar file under “dell_csi_driver_logs” directory.

CSI Driver Capabilities

CSI PowerStore v2.2.0

• NVMe/TCP Support
  

  CSI Driver for Dell Powerstore 2.2.0 and above supports NVMe/TCP provisioning. To enable NVMe/TCP provisioning, blockProtocol on secret should be specified as NVMeTCP. In case block Protocol is specified as auto, the driver will be able to find the initiators on the host and choose the protocol accordingly. If the host has multiple protocols enabled, then FC gets the highest priority followed by iSCSI and then NVMeTCP.
  

  Note: NVMe/TCP is not supported on RHEL 7.x versions and CoreOS. NVMe/TCP is supported with Powerstore 2.1 and above.
  

• fsGroupPolicy Support
  

  CSI Drivers can indicate whether or not they support modifying a volume’s ownership or permissions when the volume is being mounted. This can be useful if the CSI Driver does not support the operation, or wishes to re-use volumes with constantly changing permissions.
  

  When creating the CSI Driver object, fsGroupPolicy is defined in the driver’s spec. The following shows the hostpath driver with None included, indicating that the volumes should not be modified when mounted:
  

  apiVersion: storage.k8s.io/v1

  kind: CSIDriver

  metadata:

  name: hostpath.csi.k8s.io

  spec:

  # Supports persistent and ephemeral inline volumes.

   volumeLifecycleModes:

  – Persistent

  – Ephemeral

  # To determine at runtime which mode a volume uses, pod info and its

  # “csi.storage.k8s.io/ephemeral” entry are needed.

  podInfoOnMount: true  fsGroupPolicy: None

  Supported Modes
  

  • The following modes are supported:
    
    • None: Indicates that volumes will be mounted with no modifications, as the CSI volume driver does not support these operations.
      
    • File: Indicates that the CSI volume driver supports volume ownership and permission change via fsGroup, and Kubernetes may use fsGroup to change permissions and ownership of the volume to match user requested fsGroup in the pod’s SecurityPolicy regardless of fstype or access mode.
      
    • ReadWriteOnceWithFSType: Indicates that volumes will be examined to determine if volume ownership and permissions should be modified to match the pod’s security policy. Changes will only occur if the fsType is defined and the persistent volume’s accessModes contains ReadWriteOnce.
      

• Support for configuring permissions using POSIX mode bits and NFSv4 ACLs on NFS mount directory.
  

  CSI PowerStore driver version 2.2.0 and later allows users to set user-defined permissions on NFS target mount directory using POSIX mode bits or NFSv4 ACLs.
  

  NFSv4 ACLs are supported for NFSv4 shares on NFSv4 enabled NAS servers only. Please ensure the order when providing the NFSv4 ACLs.
  

  To use this feature, provide permissions in nfsAcls parameter in values.yaml, secrets or NFS storage class.
  

  For example:
  

POSIX mode bits

nfsAcls: “0755”

NFSv4 ACLs

nfsAcls: “A::OWNER@:rwatTnNcCy,A::GROUP@:rxtncy,A::EVERYONE@:rxtncy,A::user@domain.com:rxtncy”

CSI Driver for PowerScale v2.2.0

• fsGroupPolicy Support
  

  CSI Drivers can indicate whether or not they support modifying a volume’s ownership or permissions when the volume is being mounted. This can be useful if the CSI Driver does not support the operation or wishes to re-use volumes with constantly changing permissions.
  

  When creating the CSI Driver object, fsGroupPolicy is defined in the driver’s spec. The following shows the hostpath driver with None included, indicating that the volumes should not be modified when mounted:
  

  apiVersion: storage.k8s.io/v1
  

  kind: CSIDriver
  

  metadata:
  

  name: hostpath.csi.k8s.io
  

  spec:
  

  # Supports persistent and ephemeral inline volumes.
  

  volumeLifecycleModes:
  

  – Persistent
  

  – Ephemeral
  

  # To determine at runtime which mode a volume uses, pod info and its
  

  # “csi.storage.k8s.io/ephemeral” entry are needed.
  

  podInfoOnMount: true
  

  fsGroupPolicy: None
  

  Supported Modes
  

  • The following modes are supported:
    
    • None: Indicates that volumes will be mounted with no modifications, as the CSI volume driver does not support these operations.
      
    • File: Indicates that the CSI volume driver supports volume ownership and permission change via fsGroup, and Kubernetes may use fsGroup to change permissions and ownership of the volume to match user requested fsGroup in the pod’s SecurityPolicy regardless of fstype or access mode.
      
    • ReadWriteOnceWithFSType: Indicates that volumes will be examined to determine if volume ownership and permissions should be modified to match the pod’s security policy. Changes will only occur if the fsType is defined and the persistent volume’s accessModes contains ReadWriteOnce.
      

• Support for session-based authentication along with basic authentication for PowerScale.
  

CSI PowerMax v2.2.0

• Support for new access modes in CSI Spec 1.5.
  

• Support for Volume Health Monitoring.
  

  CSI Volume Health Monitoring allows CSI Drivers to detect abnormal volume conditions from the underlying storage systems and report them as events on PVCs or Pods.
  

  The Kubernetes components that monitor the volumes and report events with volume health information include the following:
  

  • Kubelet, in addition to gathering the existing volume stats will watch the volume health of the PVCs on that node. If a PVC has an abnormal health condition, an event will be reported on the pod object using the PVC. If multiple pods are using the same PVC, events will be reported on all pods using that PVC.
    
  • An External Volume Health Monitor Controller watches volume health of the PVCs and reports events on the PVCs.
    

    Volume Health Monitoring feature is optional and by default this feature is disabled for drivers when installed via operator.
    

    To enable this feature, set X_CSI_HEALTH_MONITOR_ENABLED to true in the driver manifest under controller and node section. Also, install the external-health-monitor from sideCars section for controller plugin.
    

CSI PowerFlex v2.2.0

• Added support for Amazon Elastic Kubernetes Service Anywhere.
  

  Amazon EKS Anywhere is a new deployment option for Amazon EKS that allows customers to create and operate Kubernetes clusters on customer-managed infrastructure, supported by AWS. Customers can now run Amazon EKS Anywhere on their own on-premises infrastructure using VMware vSphere, with support for other deployment targets in the near future, including support for bare metal coming in this year.
  

  Dell Technologies and Amazon have partnered to validate Dell PowerFlex infrastructure with Amazon EKS Anywhere. EKS Anywhere is a deployment option enabling customers to create and operate Kubernetes clusters on-premises using VMware vSphere while making it possible to have connectivity and portability to AWS public cloud environments. EKS Anywhere provides operational consistency and tooling with AWS EKS.
  

  Detailed information can be found here.
  

A guest post by Tomer Eitan