Initial Docs Push

docs/Virtualisation/pve-k8s-ceph-config.md

@@ -0,0 +1,127 @@
+# 📚 Complete Ceph CSI Deployment & Troubleshooting Guide
+
+This guide details the preparation and configuration necessary for successful dynamic provisioning of Ceph RBD (RWO) and CephFS (RWX) volumes in a Kubernetes cluster running on **MicroK8s**, backed by a Proxmox VE (PVE) Ceph cluster.
+
+---
+
+## 1. ⚙️ Ceph Cluster Preparation (Proxmox VE)
+
+These steps ensure the Ceph backend has the necessary pools and structure.
+
+* **Create Dedicated Pools:** Create OSD Pools for data, e.g., **`k8s_rbd`** (for RWO), and **`k8s_data`** and **`k8s_metadata`** (for CephFS).
+* **Create CephFS Metadata Servers (MDS):** Deploy **at least two** Metadata Server (MDS) instances.
+* **Create CephFS Filesystem:** Create the Ceph Filesystem (e.g., named **`k8s`**), linking the metadata and data pools.
+* **Create Subvolume Group (Mandatory Fix):** Create the dedicated Subvolume Group **`csi`** inside your CephFS. This is required by the CSI driver's default configuration and fixes the "No such file or directory" error during provisioning.
+    * **CLI Command:** `ceph fs subvolumegroup create k8s csi`
+
+---
+
+## 2. 🔑 Ceph User and Authorization (The Permission Fix)
+
+This addresses the persistent "Permission denied" errors during provisioning.
+
+* **Create and Configure Ceph User:** Create the user (`client.kubernetes`) and set permissions for all services. The **wildcard MGR cap** (`mgr "allow *"`) is critical for volume creation.
+    * **Final Correct Caps Command:**
+        ```bash
+        sudo ceph auth caps client.kubernetes \
+            mon 'allow r' \
+            mgr "allow *" \
+            mds 'allow rw' \
+            osd 'allow class-read object_prefix rbd_children, allow pool k8s_rbd rwx, allow pool k8s_metadata rwx, allow pool k8s_data rwx'
+        ```
+* **Export Key to Kubernetes Secrets:** Create and place two Secrets with the user key in the correct CSI provisioner namespaces:
+    * **RBD Secret:** `csi-rbd-secret` (in the RBD Provisioner namespace).
+    * **CephFS Secret:** `csi-cephfs-secret` (in the CephFS Provisioner namespace).
+    
+**The secrets should contain the keys: userID & userKey.  userID should omit the 'client.' from the ceph output.**
+
+---
+
+## 3. 🌐 Network Configuration and Bi-Directional Routing
+
+These steps ensure stable, bidirectional communication for volume staging and mounting.
+
+### A. PVE Host Firewall Configuration
+
+The PVE firewall must explicitly **allow inbound traffic** from the entire Kubernetes Pod Network to the Ceph service ports.
+
+| Protocol | Port(s) | Source | Purpose |
+| :--- | :--- | :--- | :--- |
+| **TCP** | **6789** | K8s Pod Network CIDR (e.g., `10.1.0.0/16`) | Monitor connection. |
+| **TCP** | **6800-7300** | K8s Pod Network CIDR | OSD/MDS/MGR data transfer. |
+
+Alternatively, you may find a 'ceph' macro you can use on the PVE Firewall, if so use the Macro instead of additional rules.
+
+### B. PVE Host Static Routing (Ceph $\rightarrow$ K8s)
+
+Add **persistent static routes** on **all PVE Ceph hosts** to allow Ceph to send responses back to the Pod Network.
+
+* **Action:** Edit `/etc/network/interfaces` on each PVE host:
+    ```ini
+    # Example:
+    post-up ip route add <POD_NETWORK_CIDR> via <K8S_NODE_IP> dev <PVE_INTERFACE>
+    # e.g., post-up ip route add 10.1.0.0/16 via 172.30.100.41 dev vmbr0
+    ```
+
+### C. K8s Node IP Forwarding (Gateway Function)
+
+Enable IP forwarding on **all Kubernetes nodes** so they can route incoming Ceph traffic to the correct Pods.
+
+* **Action:** Run on all K8s nodes:
+    ```bash
+    sudo sysctl net.ipv4.ip_forward=1
+    sudo sh -c 'echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/99-sysctl.conf'
+    ```
+    
+### D. K8s Static Routing (K8s $\rightarrow$ Ceph) - Conditional/Advanced ⚠️
+
+This routing is **only required** if the **Ceph Public Network** (the network Ceph Monitors/OSDs listen on) is **not reachable** by your Kubernetes Node's **default gateway**.
+
+* **Action:** This is implemented via a **Netplan configuration** on the Kubernetes nodes, using multiple routes with different metrics to provide load balancing and automatic failover.
+* **Example Netplan Configuration (`/etc/netplan/99-ceph-routes.yaml`):**
+    ```yaml
+    network:
+      version: 2
+      renderer: networkd
+      ethernets:
+        eth0: # Replace with your primary K8s network interface
+          routes:
+          # Route 1: Directs traffic destined for the first Ceph Monitor IP (10.15.16.1) 
+          # through three different PVE hosts (172.30.25.x) as gateways.
+          # The lowest metric (10) is preferred.
+          - to: 10.15.16.1/32
+            via: 172.30.25.10
+            metric: 10
+          - to: 10.15.16.1/32
+            via: 172.30.25.20
+            metric: 100
+          - to: 10.15.16.1/32
+            via: 172.30.25.30
+            metric: 100
+          
+          # Route 2: Directs traffic destined for the second Ceph Monitor IP (10.15.16.2)
+          # with a similar failover strategy.
+          - to: 10.15.16.2/32
+            via: 172.30.25.20
+            metric: 10
+          - to: 10.15.16.2/32
+            via: 172.30.25.10
+            metric: 100
+          - to: 10.15.16.2/32
+            via: 172.30.25.30
+            metric: 100
+    ```
+    
+Use route priorities (Lower is Higher) to prefer the most direct path, while still offering alternative gateways into the Ceph network where needed.
+
+---
+
+## 4. 🧩 MicroK8s CSI Driver Configuration (The Path Fix)
+
+This resolves the **`staging path does not exist on node`** error for the Node Plugin.
+
+* **Update `kubeletDir`:** When deploying the CSI driver (via Helm or YAML), the `kubeletDir` parameter must be set to the MicroK8s-specific path.
+    ```yaml
+    # Correct path for MicroK8s Kubelet root directory
+    kubeletDir: /var/snap/microk8s/common/var/lib/kubelet
+    ```