Hi Support Team,
User ID: CHZuknZ3yt
Workspace: ns-w9x6qbo4
Region: ap-southeast-1
K8s username: w9x6qbo4
Affected site: https://cpa.boy00720.qzz.io/home
Date observed: April 16, 2026 (Asia/Shanghai)
Summary:
The website is reachable, but recurring database backup jobs create unhealthy dp-backup pods because the backup PVC cannot be provisioned. This looks like a platform-side Object Storage and backup reconciliation problem.
What I observed:
- dp-backup pods became Unhealthy in the workspace.
- Main workload pods stayed healthy: sub2api, sub2api-pg-postgresql-0, sub2api-redis-redis-0, sub2api-redis-redis-sentinel-0.
- Backup PVC pvc-ap-southeast-lqwhxt stayed Pending.
- Earlier PVC event showed: The OSS Access Key Id you provided is disabled.
- I triggered Object Storage key rotation via the Object Storage app API, and the API returned 200.
- After that, ObjectStorageUser w9x6qbo4 still did not reconcile:
- spec.secretKeyVersion = 1776346940365
- status.secretKeyVersion = 0
- There were no ObjectStorageUser events in the namespace.
- The namespace object-storage-key secret already matches the Object Storage /api/user/init credentials, so this does not look like a secret sync mismatch.
- After the rotation request, the PVC retried provisioning but still remained Pending.
Steps to verify:
- Check workspace ns-w9x6qbo4 in region ap-southeast-1.
- Check backup PVC pvc-ap-southeast-lqwhxt and related dp-backup jobs and pods.
- Check ObjectStorageUser w9x6qbo4.
- Compare spec.secretKeyVersion and status.secretKeyVersion.
- Verify whether the Object Storage and backup controller has stalled reconciliation for this user.
Expected:
- ObjectStorageUser key rotation should reconcile successfully.
- Backup PVC should become Bound.
- Database backup jobs should run normally without creating recurring unhealthy pods.
Actual:
- Rotation request is accepted, but reconciliation stays stuck.
- Backup PVC remains Pending.
- Backup jobs keep producing unhealthy pods.
Please help check the Object Storage and backup controller side for user w9x6qbo4 in ap-southeast-1.