This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Best Practices

Best Practices when running Capsule in production

This is general advice you should consider before making Kubernetes Distribution consideration. They are partly relevant for Multi-Tenancy with Capsule.

Authentication

User authentication for the platform should be handled via a central OIDC-compatible identity provider system (e.g., Keycloak, Azure AD, Okta, or any other OIDC-compliant provider). The rationale is that other central platform components, such as ArgoCD, Grafana, Headlamp, or Harbor, should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions.

Capsule relies on native Kubernetes RBAC, so it’s important to consider how the Kubernetes API handles user authentication.

OCI Pull-Cache

By default, Kubernetes clusters pull images directly from upstream registries like docker.io, quay.io, ghcr.io, or gcr.io. In production environments, this can lead to issues, especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled).

To ensure availability, performance, and control over container images, it’s essential to provide an on-premise OCI mirror. This mirror should be configured via the CRI (Container Runtime Interface) by defining it as a mirror endpoint in registries.conf for default registries (e.g., docker.io). This way, all nodes automatically benefit from caching without requiring developers to change image URLs.

Secrets Management

In more complex environments with multiple clusters and applications, managing secrets manually via YAML or Helm is no longer practical. Instead, a centralized secrets management system should be established, such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project OpenBao (formerly the Vault community fork).

To integrate these external secret stores with Kubernetes, the External Secrets Operator (ESO) is a recommended solution. It automatically syncs defined secrets from external sources as Kubernetes secrets, and supports dynamic rotation, access control, and auditing.

If no external secret store is available, there should at least be a secure way to store sensitive data in Git. In our ecosystem, we provide a solution based on SOPS (Secrets OPerationS) for this use case; called the sops-operator.

👉 Demonstration

1 - Admission Policies

Recommended Admission Policies to enforce best practices in multi-tenant environments.

As Capsule we try to provide a secure multi-tenant environment out of the box, there are however some additional Admission Policies you should consider to enforce best practices in your cluster. Since Capsule only covers the core multi-tenancy features, such as Namespaces, Resource Quotas, Network Policies, and Container Registries, Classes, you should consider using an additional Admission Controller to enforce best practices on workloads and other resources.

Custom

Create custom Policies and reuse data provided via Tenant Status to enforce your own rules.

Owner Validation

Class Validation

Let’s say we have the following namespaced ObjectBucketClaim resource:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: admission-class
  namespace: solar-production
  finalizers:
    - objectbucket.io/finalizer
  labels:
    bucket-provisioner: openshift-storage.ceph.rook.io-bucket
spec:
  additionalConfig:
    maxSize: 2G
  bucketName: test-some-uid
  generateBucketName: test
  objectBucketName: obc-test-test
  storageClassName: ocs-storagecluster-ceph-rgw

However since we are allowing Tenant Users to create these ObjectBucketClaims we might want to consider validating the storageClassName field to ensure that only allowed StorageClasses are used.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-tenant-class
spec:
  validationFailureAction: Enforce
  rules:
  - name: restrict-storage-class
    context:
      - name: classes
        apiCall:
          urlPath: "/apis/capsule.clastix.io/v1beta2/tenants"
          jmesPath: "items[?contains(status.namespaces, '{{ request.namespace }}')].status.classes | [0]"

      - name: storageClass
        variable:
          jmesPath: "request.object.spec.storageClassName || 'NONE'"
    match:
      resources:
        kinds:
        - ObjectBucketClaim
        namespaceSelector:
          matchExpressions:
          - key: capsule.clastix.io/tenant
            operator: Exists
    validate:
      message: "storageclass {{ storageClass }} is not allowed in tenant ({{classes.storage}})"
      deny:
        conditions:
          - key:   "{{classes.storage}}"
            operator: AnyNotIn
            value:  "{{ storageClass }}"

Workloads

Policies to harden workloads running in a multi-tenant environment.

Disallow Scheduling on Control Planes

If a Pods are not scoped to specific nodes, they could be scheduled on control plane nodes. You should disallow this by enforcing that Pods do not use tolerations for control plane nodes.

---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  owners:
  - name: alice
    kind: User
  nodeSelector:
    node-role.kubernetes.io/worker: ''
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: disallow-controlplane-scheduling
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        operations: ["CREATE","UPDATE"]
        scope: "Namespaced"
  validations:
    - expression: >
        // deny if any toleration targets control-plane taints
        !has(object.spec.tolerations) ||
        !object.spec.tolerations.exists(t,
          t.key in ['node-role.kubernetes.io/master','node-role.kubernetes.io/control-plane']
        )        
      message: "Pods may not use tolerations which schedule on control-plane nodes."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: disallow-controlplane-scheduling
spec:
  policyName: disallow-controlplane-scheduling
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: capsule.clastix.io/tenant
          operator: Exists
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-controlplane-scheduling
spec:
  validationFailureAction: Enforce
  rules:
  - name: restrict-controlplane-scheduling-master
    match:
      resources:
        kinds:
        - Pod
        namespaceSelector:
          matchExpressions:
          - key: capsule.clastix.io/tenant
            operator: Exists
    validate:
      message: Pods may not use tolerations which schedule on control plane nodes.
      pattern:
        spec:
          =(tolerations):
            - key: "!node-role.kubernetes.io/master"

  - name: restrict-controlplane-scheduling-control-plane
    match:
      resources:
        kinds:
        - Pod
        namespaceSelector:
          matchExpressions:
          - key: capsule.clastix.io/tenant
            operator: Exists
    validate:
      message: Pods may not use tolerations which schedule on control plane nodes.
      pattern:
        spec:
          =(tolerations):
            - key: "!node-role.kubernetes.io/control-plane"

Pod Disruption Budgets

Pod Disruption Budgets (PDBs) are a way to limit the number of concurrent disruptions to your Pods. In multi-tenant environments, it is recommended to enforce the usage of PDBs to ensure that tenants do not accidentally or maliciously block cluster operations.

MaxUnavailable

A PodDisruptionBudget which sets its maxUnavailable value to zero prevents all voluntary evictions including Node drains which may impact maintenance tasks. This policy enforces that if a PodDisruptionBudget specifies the maxUnavailable field it must be greater than zero.

---
# Source: https://kyverno.io/policies/other/pdb-maxunavailable/pdb-maxunavailable/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pdb-maxunavailable
  annotations:
    policies.kyverno.io/title: PodDisruptionBudget maxUnavailable Non-Zero
    policies.kyverno.io/category: Other
    kyverno.io/kyverno-version: 1.9.0
    kyverno.io/kubernetes-version: "1.24"
    policies.kyverno.io/subject: PodDisruptionBudget
    policies.kyverno.io/description: >-
      A PodDisruptionBudget which sets its maxUnavailable value to zero prevents
      all voluntary evictions including Node drains which may impact maintenance tasks.
      This policy enforces that if a PodDisruptionBudget specifies the maxUnavailable field
      it must be greater than zero.      
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: pdb-maxunavailable
      match:
        any:
          - resources:
              kinds:
                - PodDisruptionBudget
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      validate:
        message: "The value of maxUnavailable must be greater than zero."
        pattern:
          spec:
            =(maxUnavailable): ">0"
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: pdb-maxunavailable
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["policy"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["poddisruptionbudgets"]
    namespaceSelector:
      matchExpressions:
      - key: capsule.clastix.io/tenant
        operator: Exists
  validations:
  - expression: |
      !has(object.spec.maxUnavailable) ||
      string(object.spec.maxUnavailable).contains('%') ||
      object.spec.maxUnavailable > 0      
    message: "The value of maxUnavailable must be greater than zero or a percentage."
    reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: pdb-maxunavailable-binding
spec:
  policyName: pdb-maxunavailable
  validationActions: ["Deny"]

MinAvailable

When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget, if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget it may prevent voluntary disruptions including Node drains which may impact routine maintenance tasks and disrupt operations. This policy checks incoming Deployments and StatefulSets which have a matching PodDisruptionBudget to ensure these two values do not match.

---
# Source: https://kyverno.io/policies/other/pdb-minavailable/pdb-minavailable/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pdb-minavailable-check
  annotations:
    policies.kyverno.io/title: Check PodDisruptionBudget minAvailable
    policies.kyverno.io/category: Other
    kyverno.io/kyverno-version: 1.9.0
    kyverno.io/kubernetes-version: "1.24"
    policies.kyverno.io/subject: PodDisruptionBudget, Deployment, StatefulSet
    policies.kyverno.io/description: >-
      When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget,
      if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget
      it may prevent voluntary disruptions including Node drains which may impact routine maintenance
      tasks and disrupt operations. This policy checks incoming Deployments and StatefulSets which have
      a matching PodDisruptionBudget to ensure these two values do not match.      
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: pdb-minavailable
      match:
        any:
          - resources:
              kinds:
                - Deployment
                - StatefulSet
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      preconditions:
        all:
        - key: "{{`{{ request.operation | 'BACKGROUND' }}`}}"
          operator: AnyIn
          value:
          - CREATE
          - UPDATE
        - key: "{{`{{ request.object.spec.replicas | '1' }}`}}"
          operator: GreaterThan
          value: 0
      context:
        - name: minavailable
          apiCall:
            urlPath: "/apis/policy/v1/namespaces/{{`{{ request.namespace }}`}}/poddisruptionbudgets"
            jmesPath: "items[?label_match(spec.selector.matchLabels, `{{`{{ request.object.spec.template.metadata.labels }}`}}`)] | [0].spec.minAvailable | default(`0`)"
      validate:
        message: >-
          The matching PodDisruptionBudget for this resource has its minAvailable value equal to the replica count
          which is not permitted.          
        deny:
          conditions:
            any:
              - key: "{{`{{ request.object.spec.replicas }}`}}"
                operator: Equals
                value: "{{`{{ minavailable }}`}}"

Deployment Replicas higher than PDB

PodDisruptionBudget resources are useful to ensuring minimum availability is maintained at all times.Introducing a PDB where there are already matching Pod controllers may pose a problem if the author is unaware of the existing replica count. This policy ensures that the minAvailable value is not greater or equal to the replica count of any matching existing Deployment. If other Pod controllers should also be included in this check, additional rules may be added to the policy which match those controllers.

---
# Source: https://kyverno.io/policies/other/deployment-replicas-higher-than-pdb/deployment-replicas-higher-than-pdb/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: deployment-replicas-higher-than-pdb
  annotations:
    policies.kyverno.io/title: Ensure Deployment Replicas Higher Than PodDisruptionBudget
    policies.kyverno.io/category: Other
    policies.kyverno.io/subject: PodDisruptionBudget, Deployment
    kyverno.io/kyverno-version: 1.11.4
    kyverno.io/kubernetes-version: "1.27"
    policies.kyverno.io/description: >-
      PodDisruptionBudget resources are useful to ensuring minimum availability is maintained at all times.
      Introducing a PDB where there are already matching Pod controllers may pose a problem if the author
      is unaware of the existing replica count. This policy ensures that the minAvailable value is not
      greater or equal to the replica count of any matching existing Deployment. If other Pod controllers
      should also be included in this check, additional rules may be added to the policy which match those
      controllers.      
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: deployment-replicas-greater-minAvailable
    match:
      any:
      - resources:
          kinds:
          - PodDisruptionBudget
          operations:
          - CREATE
          - UPDATE
          namespaceSelector:
            matchExpressions:
            - key: capsule.clastix.io/tenant
              operator: Exists
    context:
    - name: deploymentreplicas
      apiCall:
        jmesPath: items[?label_match(`{{`{{ request.object.spec.selector.matchLabels }}`}}`, spec.template.metadata.labels)] || `[]`
        urlPath: /apis/apps/v1/namespaces/{{`{{request.namespace}}`}}/deployments
    preconditions:
      all:
      - key: '{{`{{ length(deploymentreplicas) }}`}}'
        operator: GreaterThan
        value: 0
      - key: '{{`{{ request.object.spec.minAvailable || "" }}`}}'
        operator: NotEquals
        value: ''
    validate:
      message: >-
        PodDisruption budget minAvailable ({{`{{ request.object.spec.minAvailable }}`}}) cannot be
        greater than or equal to the replica count of any matching existing Deployment.
        There are {{`{{ length(deploymentreplicas) }}`}} Deployments which match this labelSelector
        having {{`{{ deploymentreplicas[*].spec.replicas }}`}} replicas.        
      foreach:
        - list: deploymentreplicas
          deny:
            conditions:
              all:
              - key: "{{`{{ request.object.spec.minAvailable }}`}}"
                operator: GreaterThanOrEquals
                value: "{{`{{ element.spec.replicas }}`}}"

CNPG Cluster

When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget, if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget it may prevent voluntary disruptions including Node drains which may impact routine maintenance tasks and disrupt operations. This policy checks incoming CNPG Clusters and their .spec.enablePDB setting.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pdb-cnpg-cluster-validation
  annotations:
    policies.kyverno.io/title: Check PodDisruptionBudget minAvailable for cnpgCluster
    policies.kyverno.io/category: Other
    kyverno.io/kyverno-version: 1.9.0
    kyverno.io/kubernetes-version: "1.24"
    policies.kyverno.io/subject: PodDisruptionBudget, Cluster
    policies.kyverno.io/description: >-
      When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget,
      if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget
      it may prevent voluntary disruptions including Node drains which may impact routine maintenance
      tasks and disrupt operations. This policy checks incoming CNPG Clusters and their .spec.enablePDB setting.      
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: pdb-cnpg-cluster-validation
      match:
        any:
          - resources:
              kinds:
                - postgresql.cnpg.io/v1/Cluster
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      preconditions:
        any:
        - key: "{{request.operation || 'BACKGROUND'}}"
          operator: AnyIn
          value:
          - CREATE
          - UPDATE
      validate:
        message: >-
          Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2.          
        deny:
          conditions:
            all:
              - key: "{{request.object.spec.enablePDB }}"
                operator: Equals
                value: true
              - key: "{{request.object.spec.instances }}"
                operator: LessThan
                value: 2
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: pdb-cnpg-cluster-validation
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["postgresql.cnpg.io"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["clusters"]
    namespaceSelector:
      matchExpressions:
      - key: capsule.clastix.io/tenant
        operator: Exists
  validations:
  - expression: |
      !has(object.spec.enablePDB) ||
      object.spec.enablePDB == false ||
      (has(object.spec.instances) && object.spec.instances >= 2)      
    message: "Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2."
    messageExpression: |
      'Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2. Current instances: ' +
      string(has(object.spec.instances) ? object.spec.instances : 1)      
    reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: pdb-cnpg-cluster-validation-binding
spec:
  policyName: pdb-cnpg-cluster-validation
  validationActions: ["Deny"]

Mutate User Namespace

You should enforce the usage of User Namespaces. Most Helm-Charts currently don’t support this out of the box. With Kyverno you can enforce this on Pod level.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenants-user-namespace
spec:
  rules:
    - name: enforce-no-host-users
      match:
        any:
        - resources:
            kinds:
            - Pod
            namespaceSelector:
              matchExpressions:
              - key: capsule.clastix.io/tenant
                operator: Exists
            # selector:
            #   matchExpressions:
            #     - key: company.com/allow-host-users
            #       operator: NotIn
            #       values:
            #         - "true"
      preconditions:
        all:
        - key: "{{request.operation || 'BACKGROUND'}}"
          operator: AnyIn
          value:
            - CREATE
            - UPDATE
      skipBackgroundRequests: true
      mutate:
        patchStrategicMerge:
          spec:
            hostUsers: false

Note that users still can override this setting by adding the label company.com/allow-host-users=true to their namespace. You can change the label to your needs. This is because NFS does not support user namespaces and you might want to allow this for specific tenants.

Disallow Daemonsets

Tenant’s should not be allowed to create Daemonsets, unless they have dedicated nodes.

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: deny-daemonset-create
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        resources: ["daemonsets"]
        operations: ["CREATE"]
        scope: "Namespaced"
  validations:
    - expression: "false"
      message: "Creating DaemonSets is not allowed in this cluster."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: deny-daemonset-create-binding
spec:
  policyName: deny-daemonset-create
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: capsule.clastix.io/tenant
          operator: Exists
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenant-workload-restrictions
spec:
  validationFailureAction: Enforce
  rules:
  - name: block-daemonset-create
    match:
      any:
      - resources:
          kinds:
          - DaemonSet
          namespaceSelector:
            matchExpressions:
            - key: capsule.clastix.io/tenant
              operator: Exists
    preconditions:
      all:
      - key: "{{ request.operation || 'BACKGROUND' }}"
        operator: Equals
        value: CREATE
    validate:
      message: "Creating DaemonSets is not allowed in this cluster."
      deny:
        conditions:
          any:
          - key: "true"
            operator: Equals
            value: "true"

Enforce EmptDir Requests/Limits

By Defaults emptyDir Volumes do not have any limits. This could lead to a situation, where a tenant fills up the node disk. To avoid this, you can enforce limits on emptyDir volumes. You may also consider restricting the usage of emptyDir with the medium: Memory option, as this could lead to memory exhaustion on the node.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenant-workload-restrictions
spec:
  rules:
    - name: default-emptydir-sizelimit
      match:
        any:
        - resources:
            kinds:
            - Pod
            namespaceSelector:
              matchExpressions:
              - key: capsule.clastix.io/tenant
                operator: Exists
      mutate:
        foreach:
        - list: "request.object.spec.volumes[]"
          preconditions:
            all:
            - key: "{{element.keys(@)}}"
              operator: AnyIn
              value: emptyDir
            - key: "{{element.emptyDir.sizeLimit || ''}}"
              operator: Equals
              value: ''
          patchesJson6902: |-
            - path: "/spec/volumes/{{elementIndex}}/emptyDir/sizeLimit"
              op: add
              value: 250Mi

Block Ephemeral Containers

Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the kubectl debug functionality and attach a temporary container to an existing Pod. This may potentially be used to gain access to unauthorized information executing inside one or more containers in that Pod. This policy blocks the use of ephemeral containers.

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: block-ephemeral-containers
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      # 1) Regular Pods (ensure spec doesn't carry ephemeralContainers)
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        operations: ["CREATE","UPDATE"]
        scope: "Namespaced"
      # 2) Subresource used by `kubectl debug` to inject ephemeral containers
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods/ephemeralcontainers"]
        operations: ["UPDATE","CREATE"]  # UPDATE is typical, CREATE included for future-proofing
        scope: "Namespaced"
  validations:
    # Deny any request that targets the pods/ephemeralcontainers subresource
    - expression: request.subResource != "ephemeralcontainers"
      message: "Ephemeral (debug) containers are not permitted (subresource)."
    # For direct Pod create/update, allow only if the field is absent or empty
    - expression: >
        !has(object.spec.ephemeralContainers) ||
        size(object.spec.ephemeralContainers) == 0        
      message: "Ephemeral (debug) containers are not permitted in Pod specs."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: block-ephemeral-containers-binding
spec:
  policyName: block-ephemeral-containers
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: capsule.clastix.io/tenant
          operator: Exists
# Source: https://kyverno.io/policies/other/block-ephemeral-containers/block-ephemeral-containers/
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-ephemeral-containers
  annotations:
    policies.kyverno.io/title: Block Ephemeral Containers
    policies.kyverno.io/category: Other
    policies.kyverno.io/severity: medium
    kyverno.io/kyverno-version: 1.6.0
    policies.kyverno.io/minversion: 1.6.0
    kyverno.io/kubernetes-version: "1.23"
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the
      `kubectl debug` functionality and attach a temporary container to an existing Pod.
      This may potentially be used to gain access to unauthorized information executing inside
      one or more containers in that Pod. This policy blocks the use of ephemeral containers.      
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: block-ephemeral-containers
    match:
      any:
      - resources:
          kinds:
            - Pod
          namespaceSelector:
            matchExpressions:
            - key: capsule.clastix.io/tenant
              operator: Exists
    validate:
      message: "Ephemeral (debug) containers are not permitted."
      pattern:
        spec:
          X(ephemeralContainers): "null"

QOS Classes

You may consider the upstream policies, depending on your needs:

Certificates

Deny ClusterIssuer in Certificates

Often when working in multi-tenant environments, you want to ensure that tenants are not using ClusterIssuers to issue certificates, but rather use namespaced Issuers within their own namespace. This policy enforces that cert-manager.io/v1/Certificate resources do not reference ClusterIssuers and that the Issuer referenced is in the same namespace as the Certificate.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: certificates-only-local-issuer
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: deny-clusterissuer-in-certificates
      match:
        any:
          - resources:
              kinds:
                - cert-manager.io/v1/Certificate
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      validate:
        message: "Certificates must not reference ClusterIssuers; use a namespaced Issuer in the same namespace."
        deny:
          conditions:
            any:
              - key: "{{ request.object.spec.issuerRef.kind || 'Issuer' }}"
                operator: Equals
                value: "ClusterIssuer"

    - name: deny-cross-namespace-issuerref-in-certificates
      match:
        any:
          - resources:
              kinds:
                - cert-manager.io/v1/Certificate
      validate:
        message: "Certificates must reference an Issuer in the same namespace (spec.issuerRef.namespace must be empty or equal to the Certificate namespace)."
        deny:
          conditions:
            any:
              # If issuerRef.namespace is set and differs from the Certificate namespace -> deny
              - key: "{{request.object.spec.issuerRef.namespace || '' }}"
                operator: NotEquals
                value: ""
                # AND also not equal to request namespace
              - key: "{{ request.object.spec.issuerRef.namespace || request.namespace  }}"
                operator: NotEquals
                value: "{{ request.namespace }}"

Deny ClusterIssuer in Gateways

Deny to usage of ClusterIssuers in Gateways by checking for the cert-manager.io/cluster-issuer annotation. This ensures that tenants use namespaced issuer mechanisms instead.

This requires extra permissions to allow Kyverno to read Gateway resources:

admissionController:
  rbac:
    clusterRole:
      extraResources:
       - apiGroups: ["gateway.networking.k8s.io"]
         resources: ["*"]
         verbs: ["get", "list", "watch"]

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: gateways-deny-cluster-issuer-annotation
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: deny-cert-manager-cluster-issuer-annotation
      match:
        any:
          - resources:
              kinds:
                - gateway.networking.k8s.io/v1/Gateway
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      validate:
        message: "Gateways must not use cert-manager.io/cluster-issuer; use namespaced issuer mechanisms instead."
        deny:
          conditions:
            any:
              - key: "{{ request.object.metadata.annotations.\"cert-manager.io/cluster-issuer\" || '' }}"
                operator: NotEquals
                value: ""

Images

Allowed Registries

---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  containerRegistries:
    allowed:
    - "docker.io"
    - "public.ecr.aws"
    - "quay.io"
    - "mcr.microsoft.com"
# Or with a Kyverno Policy. Here the default registry is `docker.io`, when no registry prefix is specified:
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
  annotations:
    policies.kyverno.io/title: Restrict Image Registries
    policies.kyverno.io/category: Best Practices, EKS Best Practices
    policies.kyverno.io/severity: medium
spec:
  validationFailureAction: Audit
  background: true
  rules:
  - name: validate-registries
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Using unknown image registry."
      foreach:
      - list: "request.object.spec.initContainers"
        deny:
          conditions:
          - key: '{{images.initContainers."{{element.name}}".registry }}'
            operator: NotIn
            value:
            - "docker.io"
            - "public.ecr.aws"
            - "quay.io"
            - "mcr.microsoft.com"

      - list: "request.object.spec.ephemeralContainers"
        deny:
          conditions:
          - key: '{{images.ephemeralContainers."{{element.name}}".registry }}'
            operator: NotIn
            value:
            - "docker.io"
            - "public.ecr.aws"
            - "quay.io"
            - "mcr.microsoft.com"

      - list: "request.object.spec.containers"
        deny:
          conditions:
          - key: '{{images.containers."{{element.name}}".registry }}'
            operator: NotIn
            value:
            - "docker.io"
            - "public.ecr.aws"
            - "quay.io"
            - "mcr.microsoft.com"

Allowed PullPolicy

Read More.

---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  owners:
  - name: alice
    kind: User
  imagePullPolicies:
  - Always
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: always-pull-images
  annotations:
    policies.kyverno.io/title: Always Pull Images
    policies.kyverno.io/category: Sample
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/minversion: 1.6.0
    policies.kyverno.io/description: >-
      By default, images that have already been pulled can be accessed by other
      Pods without re-pulling them if the name and tag are known. In multi-tenant scenarios,
      this may be undesirable. This policy mutates all incoming Pods to set their
      imagePullPolicy to Always. An alternative to the Kubernetes admission controller
      AlwaysPullImages.      
spec:
  rules:
  - name: always-pull-images
    match:
      any:
      - resources:
          kinds:
          - Pod
          namespaceSelector:
            matchExpressions:
            - key: capsule.clastix.io/tenant
              operator: Exists
    mutate:
      patchStrategicMerge:
        spec:
          initContainers:
          - (name): "?*"
            imagePullPolicy: Always
          containers:
          - (name): "?*"
            imagePullPolicy: Always
          ephemeralContainers:
          - (name): "?*"
            imagePullPolicy: Always

Certificate Management

Selective ClusterIssuers

Allow certain ClusterIssuers within Tenants:

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: certificates-restrict-clusterissuer-by-tenant-label
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: allow-clusterissuer-only-when-managed-by-matches
      match:
        any:
          - resources:
              kinds:
                - cert-manager.io/v1/Certificate
              namespaceSelector:
                matchExpressions:
                  - key: capsule.clastix.io/tenant
                    operator: Exists
      context:
        - name: certManagedBy
          variable:
            jmesPath: request.object.metadata.labels."capsule.clastix.io/managed-by" || ''
        - name: clusterIssuerManagedBy
          apiCall:
            urlPath: /apis/cert-manager.io/v1/clusterissuers/{{ request.object.spec.issuerRef.name }}
            jmesPath: metadata.labels."company.com/tenant" || ''
            default: ""
      preconditions:
        all:
          - key: "{{ request.object.spec.issuerRef.kind || 'Issuer' }}"
            operator: Equals
            value: ClusterIssuer
          - key: "{{request.operation || 'BACKGROUND'}}"
            operator: AnyIn
            value:
            - CREATE
            - UPDATE
      validate:
        message: >-
          ClusterIssuer is only allowed when the Certificate label
          capsule.clastix.io/managed-by ({{certManagedBy}}) matches the referenced ClusterIssuer label
          company.com/tenant ({{clusterIssuerManagedBy}}).          
        deny:
          conditions:
            any:
              - key: "{{certManagedBy}}"
                operator: Equals
                value: ""
              - key: "{{clusterIssuerManagedBy}}"
                operator: Equals
                value: ""
              - key: "{{certManagedBy}}"
                operator: NotEquals
                value: "{{clusterIssuerManagedBy}}"

2 - Workloads

Control the security of the workloads running in the tenant namespaces

User Namespaces

A process running as root in a container can run as a different (non-root) user in the host; in other words, the process has full privileges for operations inside the user namespace, but is unprivileged for operations outside the namespace. Read More

Kubelet

On your Kubelet you must use the FeatureGates:

  • UserNamespacesSupport
  • UserNamespacesPodSecurityStandards (Optional)

Sysctls

user.max_user_namespaces: "11255"

Admission (Kyverno)

To make sure all the workloads are forced to use dedicated User Namespaces, we recommend to mutate pods at admission. See the following examples.

Kyverno

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-hostusers-spec
  annotations:
    policies.kyverno.io/title: Add HostUsers
    policies.kyverno.io/category: Security
    policies.kyverno.io/subject: Pod,User Namespace
    kyverno.io/kubernetes-version: "1.31"
    policies.kyverno.io/description: >-
      Do not use the host's user namespace. A new userns is created for the pod.
      Setting false is useful for mitigating container breakout vulnerabilities even allowing users to run their containers as root
      without actually having root privileges on the host. This field is
      alpha-level and is only honored by servers that enable the
      UserNamespacesSupport feature.      
spec:
  rules:
  - name: add-host-users
    match:
      any:
      - resources:
          kinds:
          - Pod
          namespaceSelector:
            matchExpressions:
            - key: capsule.clastix.io/tenant
              operator: Exists
    preconditions:
      all:
      - key: "{{request.operation || 'BACKGROUND'}}"
        operator: AnyIn
        value:
          - CREATE
          - UPDATE
    mutate:
      patchStrategicMerge:
        spec:
          hostUsers: false

Pod Security Standards

In Kubernetes, by default, workloads run with administrative access, which might be acceptable if there is only a single application running in the cluster or a single user accessing it. This is seldom required and you’ll consequently suffer a noisy neighbour effect along with large security blast radiuses.

Many of these concerns were addressed initially by PodSecurityPolicies which have been present in the Kubernetes APIs since the very early days.

The Pod Security Policies are deprecated in Kubernetes 1.21 and removed entirely in 1.25. As replacement, the Pod Security Standards and Pod Security Admission has been introduced. Capsule supports the new standard for tenants under its control as well as the oldest approach.

One of the issues with Pod Security Policies is that it is difficult to apply restrictive permissions on a granular level, increasing security risk. Also the Pod Security Policies get applied when the request is submitted and there is no way of applying them to pods that are already running. For these, and other reasons, the Kubernetes community decided to deprecate the Pod Security Policies.

As the Pod Security Policies get deprecated and removed, the Pod Security Standards is used in place. It defines three different policies to broadly cover the security spectrum. These policies are cumulative and range from highly-permissive to highly-restrictive:

  • Privileged: unrestricted policy, providing the widest possible level of permissions.
  • Baseline: minimally restrictive policy which prevents known privilege escalations.
  • Restricted: heavily restricted policy, following current Pod hardening best practices.

Kubernetes provides a built-in Admission Controller to enforce the Pod Security Standards at either:

  1. cluster level which applies a standard configuration to all namespaces in a cluster
  2. namespace level, one namespace at a time

For the first case, the cluster admin has to configure the Admission Controller and pass the configuration to the kube-apiserver by mean of the --admission-control-config-file extra argument, for example:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1beta1
    kind: PodSecurityConfiguration
    defaults:
      enforce: "baseline"
      enforce-version: "latest"
      warn: "restricted"
      warn-version: "latest"
      audit: "restricted"
      audit-version: "latest"
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces: [kube-system]

For the second case, he can just assign labels to the specific namespace he wants enforce the policy since the Pod Security Admission Controller is enabled by default starting from Kubernetes 1.23+:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted
  name: development

Capsule

According to the regular Kubernetes segregation model, the cluster admin has to operate either at cluster level or at namespace level. Since Capsule introduces a further segregation level (the Tenant abstraction), the cluster admin can implement Pod Security Standards at tenant level by simply forcing specific labels on all the namespaces created in the tenant.

You can distribute these profiles via namespace. Here’s how this could look like:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  namespaceOptions:
    additionalMetadataList:
    - namespaceSelector:
        matchExpressions:
          - key: projectcapsule.dev/low_security_profile
            operator: NotIn
            values: ["system"]
      labels:
        pod-security.kubernetes.io/enforce: restricted
        pod-security.kubernetes.io/warn: restricted
        pod-security.kubernetes.io/audit: restricted
    - namespaceSelector:
        matchExpressions:
          - key: company.com/env
            operator: In
            values: ["system"]
      labels:
        pod-security.kubernetes.io/enforce: privileged
        pod-security.kubernetes.io/warn: privileged
        pod-security.kubernetes.io/audit: privileged

All namespaces created by the tenant owner, will inherit the Pod Security labels:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    capsule.clastix.io/tenant: solar
    kubernetes.io/metadata.name: solar-development
    name: solar-development
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted
  name: solar-development
  ownerReferences:
  - apiVersion: capsule.clastix.io/v1beta2
    blockOwnerDeletion: true
    controller: true
    kind: Tenant
    name: solar

and the regular Pod Security Admission Controller does the magic:

kubectl --kubeconfig alice-wind.kubeconfig apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: solar-production
spec:
  containers:
  - image: nginx
    name: nginx
    ports:
    - containerPort: 80
    securityContext:
      privileged: true
EOF

The request gets denied:

Error from server (Forbidden): error when creating "STDIN":
pods "nginx" is forbidden: violates PodSecurity "baseline:latest": privileged
(container "nginx" must not set securityContext.privileged=true)

If the tenant owner tries to change or delete the above labels, Capsule will reconcile them to the original tenant manifest set by the cluster admin.

As additional security measure, the cluster admin can also prevent the tenant owner to make an improper usage of the above labels:

kubectl annotate tenant solar \
  capsule.clastix.io/forbidden-namespace-labels-regexp="pod-security.kubernetes.io\/(enforce|warn|audit)"

In that case, the tenant owner gets denied if she tries to use the labels:

kubectl --kubeconfig alice-solar.kubeconfig label ns solar-production \
    pod-security.kubernetes.io/enforce=restricted \
    --overwrite

Error from server (Label pod-security.kubernetes.io/audit is forbidden for namespaces in the current Tenant ...

Pod Security Policies

As stated in the documentation, “PodSecurityPolicies enable fine-grained authorization of pod creation and updates. A Pod Security Policy is a cluster-level resource that controls security sensitive aspects of the pod specification. The PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields.”

Using the Pod Security Policies, the cluster admin can impose limits on pod creation, for example the types of volume that can be consumed, the linux user that the process runs as in order to avoid running things as root, and more. From multi-tenancy point of view, the cluster admin has to control how users run pods in their tenants with a different level of permission on tenant basis.

Assume the Kubernetes cluster has been configured with Pod Security Policy Admission Controller enabled in the APIs server: --enable-admission-plugins=PodSecurityPolicy

The cluster admin creates a PodSecurityPolicy:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp:restricted
spec:
  privileged: false
  # Required to prevent escalations to root.
  allowPrivilegeEscalation: false

Then create a ClusterRole using or granting the said item

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp:restricted
rules:
- apiGroups: ['policy']
  resources: ['podsecuritypolicies']
  resourceNames: ['psp:restricted']
  verbs: ['use']

He can assign this role to all namespaces in a tenant by setting the tenant manifest:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  owners:
  - name: alice
    kind: User
  additionalRoleBindings:
  - clusterRoleName: psp:privileged
    subjects:
    - kind: "Group"
      apiGroup: "rbac.authorization.k8s.io"
      name: "system:authenticated"

With the given specification, Capsule will ensure that all tenant namespaces will contain a RoleBinding for the specified Cluster Role:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: 'capsule-solar-psp:privileged'
  namespace: solar-production
  labels:
    capsule.clastix.io/tenant: solar
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: 'system:authenticated'
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: 'psp:privileged'

Capsule admission controller forbids the tenant owner to run privileged pods in solar-production namespace and perform privilege escalation as declared by the above Cluster Role psp:privileged.

As tenant owner, creates a namespace:

kubectl --kubeconfig alice-solar.kubeconfig create ns solar-production

and create a pod with privileged permissions:

kubectl --kubeconfig alice-solar.kubeconfig apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: solar-production
spec:
  containers:
  - image: nginx
    name: nginx
    ports:
    - containerPort: 80
    securityContext:
      privileged: true
EOF

Since the assigned PodSecurityPolicy explicitly disallows privileged containers, the tenant owner will see her request to be rejected by the Pod Security Policy Admission Controller.

3 - Networking

Multi-Tenant Networking considerations

Network-Policies

It’s a best practice to not allow any traffic outside of a tenant (or a tenant’s namespace). For this we can use Tenant Replications to ensure we have for every namespace Networkpolicies in place.

The following NetworkPolicy is distributed to all namespaces which belong to a Capsule tenant:

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
  name: default-networkpolicies
  namespace: solar-system
spec:
  resyncPeriod: 60s
  resources:
    - rawItems:
        - apiVersion: networking.k8s.io/v1
          kind: NetworkPolicy
          metadata:
            name: default-policy
          spec:
            # Apply to all pods in this namespace
            podSelector: {}
            policyTypes:
              - Ingress
              - Egress
            ingress:
              # Allow traffic from the same namespace (intra-namespace communication)
              - from:
                  - podSelector: {}

              # Allow traffic from all namespaces within the tenant
              - from:
                  - namespaceSelector:
                      matchLabels:
                        capsule.clastix.io/tenant: "{{tenant.name}}"

              # Allow ingress from other namespaces labeled (System Namespaces, eg. Monitoring, Ingress)
              - from:
                  - namespaceSelector:
                      matchLabels:
                        company.com/system: "true"

            egress:
              # Allow DNS to kube-dns service IP (might be different in your setup)
              - to:
                  - ipBlock:
                      cidr: 10.96.0.10/32
                ports:
                  - protocol: UDP
                    port: 53
                  - protocol: TCP
                    port: 53

              # Allow traffic to all namespaces within the tenant
              - to:
                  - namespaceSelector:
                      matchLabels:
                        capsule.clastix.io/tenant: "{{tenant.name}}"

Deny Namespace Metadata

In the above example we allow traffic from namespaces with the label company.com/system: "true". This is meant for Kubernetes Operators to eg. scrape the workloads within a tenant. However without further enforcement any namespace can set this label and therefore gain access to any tenant namespace. To prevent this, we must restrict, who can declare this label on namespaces.

We can deny such labels on tenant basis. So in this scenario every tenant should disallow the use of these labels on namespaces:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  namespaceOptions:
    forbiddenLabels:
      denied:
          - company.com/system

Or you can implement a Kyverno-Policy, which solves this.

Non-Native Network-Policies

The same principle can be applied with alternative CNI solutions. In this example we are using Cilium:

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
  name: default-networkpolicies
  namespace: solar-system
spec:
  resyncPeriod: 60s
  resources:
    - rawItems:
        - apiVersion: cilium.io/v2
          kind: CiliumNetworkPolicy
          metadata:
            name: default-policy
          spec:
            endpointSelector: {}  # Apply to all pods in the namespace
            ingress:
              - fromEndpoints:
                  - matchLabels: {}  # Same namespace pods (intra-namespace)
              - fromEntities:
                  - cluster  # For completeness; can be used to allow internal cluster traffic if needed
              - fromEndpoints:
                  - matchLabels:
                      capsule.clastix.io/tenant: "{{tenant.name}}"  # Pods in other namespaces with same tenant
              - fromNamespaces:
                  - matchLabels:
                      company.com/system: "true"  # System namespaces (monitoring, ingress, etc.)

            egress:
              - toCIDR:
                  - 10.96.0.10/32  # kube-dns IP
                toPorts:
                  - ports:
                      - port: "53"
                        protocol: UDP
                      - port: "53"
                        protocol: TCP

              - toNamespaces:
                  - matchLabels:
                      capsule.clastix.io/tenant: "{{tenant.name}}"  # Egress to all tenant namespaces

4 - Container Images

Multi-Tenant Container Images considerations

Until this issue is resolved (might be in Kubernetes 1.34)

it’s recommended to use the ImagePullPolicy Always for private registries on shared nodes. This ensures that no images can be used which are already pulled to the node.