284 lines
9.1 KiB
Markdown
284 lines
9.1 KiB
Markdown
# Deploy Ceph RADOS Gateway in Kubernetes
|
|
## Pointing to a separate PVE Ceph Cluster
|
|
|
|
This guide outlines how to deploy a RADOS Gateway to enable an S3 API for a Ceph pool. I use this to provide S3 storage to my Kubernetes Cluster with the Ceph cluster hosted by Proxmox VE. Many conecpts are similar to the previous guide - Enable Ceph CSI for PVE Ceph, some steps will refer to that guide.
|
|
|
|
This guide makes the following assumptions:
|
|
* You are already runnung Ceph via PVE.
|
|
* You are using the PVE UI for Ceph actions where possible.
|
|
* You are deploying the RADOS Gateway to the `object-store` namespace in K8s.
|
|
* Flux is used to deploy to K8s using SOPS for secret encryption.
|
|
|
|
### 1. Ceph Pool & User Creation
|
|
|
|
These steps ensure that a Ceph Pool is created with appropriate Replication.
|
|
|
|
* Create the RGW Realm on a PVE Host from the Shell
|
|
* Create Realm: `radosgw-admin realm create --rgw-realm=default --default`
|
|
* Create Zonegroup: `radosgw-admin zonegroup create --rgw-zonegroup=default --master --default --endpoints=http://ceph-rgw.object-store.svc.cluster.local:8080`
|
|
* Create Zone: `radosgw-admin zone create --rgw-zone=default --master --default`
|
|
* Ensure Zone is included in Zonegroup: `radosgw-admin zonegroup add --rgw-zonegroup=default --rgw-zone=default`
|
|
* Update & Commit Period: `radosgw-admin period update --commit`
|
|
* Set the default realm: `radosgw-admin realm default --rgw-realm=default`
|
|
|
|
* The above commands will have created the following new pools
|
|
**You do not need to manually create these**
|
|
|
|
|Pool Name|Purpose|
|
|
| :----- | :----- |
|
|
|.rgw.root|
|
|
|default.rgw.log|
|
|
|default.rgw.control|
|
|
|default.rgw.meta|
|
|
|
|
* Create the two required Pools for index and data in the PVE UI:
|
|
|
|
|Pool Name | PG Autoscaler | Size | Min Size | Crush Rule |
|
|
| :------- | :------------ | :--- | :------- | :--------- |
|
|
| default.rgw.buckets.index | On | 3 | 2 | replicated_rule |
|
|
| default.rgw.buckets.data | On | 3 | 2 | replicated_rule |
|
|
|
|
* Enable RGW Application:
|
|
When the pool is created via PVE, it is registered by default as an RBD Pool,
|
|
run these commands to change it to an RGW pool.
|
|
* Disable RBD: `ceph osd pool application disable default.rgw.buckets.data rbd --yes-i-really-mean-it`
|
|
* Enable RGW: `ceph osd pool application enable default.rgw.buckets.data rgw`
|
|
* Check with: `ceph osd pool application get default.rgw.buckets.data`
|
|
* Repeat for index pool: `ceph osd pool application disable default.rgw.buckets.index rbd --yes-i-really-mean-it`
|
|
* Enable RGW: `ceph osd pool application enable default.rgw.buckets.index rgw`
|
|
* Check with: `ceph osd pool application get default.rgw.buckets.index`
|
|
|
|
* Create a user for the RADOS Gateway:
|
|
```
|
|
ceph auth get-or-create client.rgw.k8s.svc \
|
|
mon 'allow r' \
|
|
osd 'allow rwx pool=default.rgw.buckets.data, allow rwx pool=default.rgw.buckets.index, allow rwx pool=.rgw.root, allow rwx pool=default.rgw.meta, allow rwx pool=default.rgw.log, allow rwx pool=default.rgw.control' \
|
|
-o /etc/ceph/ceph.client.rgw.k8s.svc.keyring
|
|
```
|
|
|
|
### 2. Register Kubernetes Secrets
|
|
|
|
* Retreive the files, from the Ceph host, required for Kubernetes Secrets:
|
|
Retreive these files and store them **temporarily** on your workstation.
|
|
|
|
| File | Path | Purpose |
|
|
| :------- | :--- | :------ |
|
|
|ceph.conf|/etc/ceph/ceph.conf|Location of Ceph Monitors|
|
|
|Keyring | /etc/ceph/ceph.client.rgw.k8s.svc.keyring | Auth token |
|
|
|
|
* Create Secret manifests for deployment to K8s:
|
|
|
|
```
|
|
kubectl create secret generic ceph-config \
|
|
--namespace=object-store \
|
|
--from-file=ceph.conf=./conf \
|
|
--dry-run=client -o yaml > ceph-config-secret.yaml
|
|
```
|
|
```
|
|
kubectl create secret generic ceph-keyring \
|
|
--namespace=object-store \
|
|
--from-file=keyring=./keyring \
|
|
--dry-run=client -o yaml > ceph-keyring-secret.yaml
|
|
```
|
|
|
|
* Encrypt the secret manifests using sops:
|
|
* `sops encrypt --in-place ./ceph-config-secret.yaml`
|
|
* `sops encrypt --in-place ./ceph-keyring-secret.yaml`
|
|
|
|
### 3. Kubernetes Manifests
|
|
|
|
**These should be treated as examples, read through them and ensure they match your environment**
|
|
|
|
#### Namespace
|
|
|
|
```
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: object-store
|
|
```
|
|
|
|
#### Service
|
|
```
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: ceph-rgw-svc
|
|
namespace: object-store
|
|
labels:
|
|
app.kubernetes.io/name: ceph-rgw
|
|
app.kubernetes.io/component: gateway
|
|
spec:
|
|
# The ClusterIP DNS name used for the RGW initialization:
|
|
# http://ceph-rgw-svc.object-store.svc.cluster.local:8080
|
|
ports:
|
|
- port: 8080
|
|
targetPort: 8080
|
|
protocol: TCP
|
|
name: http-api
|
|
selector:
|
|
app: ceph-rgw
|
|
type: ClusterIP
|
|
```
|
|
|
|
#### Deployment
|
|
```
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: ceph-rgw
|
|
namespace: object-store
|
|
labels:
|
|
app.kubernetes.io/name: ceph-rgw
|
|
app.kubernetes.io/component: gateway
|
|
spec:
|
|
replicas: 2
|
|
selector:
|
|
matchLabels:
|
|
app: ceph-rgw
|
|
strategy:
|
|
type: RollingUpdate
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: ceph-rgw
|
|
spec:
|
|
# CRUCIAL: Enforce Pods to be on separate nodes for HA
|
|
affinity:
|
|
podAntiAffinity:
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
|
- labelSelector:
|
|
matchLabels:
|
|
app: ceph-rgw
|
|
topologyKey: "kubernetes.io/hostname"
|
|
containers:
|
|
- name: rgw
|
|
# Use the same Major:Minor as your PVE Hosts
|
|
image: quay.io/ceph/ceph:v18.2
|
|
# Arguments to start the RGW process on port 8080
|
|
args: [
|
|
"radosgw",
|
|
"-f", # Run in foreground
|
|
"--conf=/etc/ceph/ceph.conf", # Explicitly use the mounted config
|
|
"--name=client.rgw.k8s.svc", # The exact CephX user name we created
|
|
"--rgw-frontends=beast port=8080" # REQUIRED: Beast frontend for Ceph 18+
|
|
]
|
|
resources:
|
|
requests:
|
|
cpu: 500m
|
|
memory: 2Gi
|
|
limits:
|
|
cpu: 2000m
|
|
memory: 2Gi
|
|
ports:
|
|
- containerPort: 8080
|
|
name: rgw-http
|
|
# Ensure the Pod does not run as root unnecessarily
|
|
securityContext:
|
|
runAsUser: 167 # A common non-root user ID for Ceph containers
|
|
runAsGroup: 167
|
|
allowPrivilegeEscalation: false
|
|
volumeMounts:
|
|
- name: ceph-config-vol
|
|
mountPath: /etc/ceph/ceph.conf
|
|
subPath: ceph.conf
|
|
- name: ceph-keyring-vol
|
|
mountPath: /etc/ceph/ceph.client.rgw.k8s.svc.keyring
|
|
subPath: keyring
|
|
volumes:
|
|
- name: ceph-config-vol
|
|
secret:
|
|
secretName: ceph-config
|
|
items:
|
|
- key: ceph.conf
|
|
path: ceph.conf
|
|
- name: ceph-keyring-vol
|
|
secret:
|
|
secretName: ceph-keyring
|
|
items:
|
|
- key: keyring
|
|
path: ceph.client.rgw.k8s.svc.keyring
|
|
```
|
|
|
|
**Deploy these manifests to Flux**
|
|
|
|
### 4. RGW Admin Utility
|
|
|
|
**Do not commit this to Flux, run as and when required to manage RGW users and buckets**
|
|
|
|
#### Pod Manifest
|
|
```
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: rgw-admin-utility
|
|
namespace: object-store
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: rgw-admin-cli
|
|
# Use the same image as your RGW deployment for consistency
|
|
image: quay.io/ceph/ceph:v18.2
|
|
# Use the /bin/bash entrypoint to allow manual command execution
|
|
command: ["/bin/bash", "-c", "sleep 3600"]
|
|
# Environment variable to explicitly define the CephX user for CLI tools
|
|
env:
|
|
- name: CEPH_ARGS
|
|
value: "--name client.rgw.k8s.svc --keyring /etc/ceph/ceph.client.rgw.k8s.svc.keyring"
|
|
|
|
volumeMounts:
|
|
# Mount the ceph.conf Secret
|
|
- name: ceph-config-vol
|
|
mountPath: /etc/ceph/ceph.conf
|
|
subPath: ceph.conf
|
|
# Mount the keyring Secret to the file name radosgw-admin expects
|
|
- name: ceph-keyring-vol
|
|
mountPath: /etc/ceph/ceph.client.rgw.k8s.svc.keyring
|
|
subPath: keyring
|
|
|
|
volumes:
|
|
- name: ceph-config-vol
|
|
secret:
|
|
secretName: ceph-config
|
|
items:
|
|
- key: ceph.conf
|
|
path: ceph.conf
|
|
- name: ceph-keyring-vol
|
|
secret:
|
|
secretName: ceph-keyring
|
|
items:
|
|
- key: keyring
|
|
path: ceph.client.rgw.k8s.svc.keyring # Use the explicit filename
|
|
```
|
|
|
|
#### Managing RGW
|
|
|
|
* Deploy the Pod using `kubectl apply -f {filepath}`
|
|
* Exec into the pod `kubectl exec -it rgw-admin-utility -n object-store -- bash`
|
|
|
|
##### Create User
|
|
|
|
* `radosgw-admin user create --uid={uid} --display-name={display-name} --gen-key --gen-secret`
|
|
|
|
**CRITICAL:** *Copy the JSON output, save the access_key and secret_key*
|
|
|
|
##### Create Bucket
|
|
|
|
* `radosgw-admin bucket create --bucket={buket-name} --uid={owner-uid}`
|
|
|
|
##### Exit & Cleanup
|
|
|
|
* `exit`
|
|
* `kubectl delete pod rgw-admin-utility -n object-store`
|
|
|
|
### 5. Generate Secret for Client Access
|
|
|
|
Deploy this in the namespace of the appliation requiring the S3 API Access
|
|
|
|
```
|
|
kubectl create secret generic s3-credentials \
|
|
--namespace={application-namespace} \
|
|
--from-literal=S3_ACCESS_KEY={access-key-from-user-creation} \
|
|
--from-literal=S3_SECRET_KEY={secret-key-from-user-creation} \
|
|
--dry-run=client -o yaml > s3-secret.yaml
|
|
``` |