webgrip/homelab-cluster

Fork 0

[PR #67] [MERGED] fix: resolve cluster health audit findings #293

New issue

Closed

opened 2026-06-08 08:31:15 +00:00 by ryangr0 · 0 comments

ryangr0 commented

2026-06-08 08:31:15 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/webgrip/homelab-cluster/pull/67
Author: @Ryangr0
Created: 5/22/2026
Status: ✅ Merged
Merged: 5/22/2026
Merged by: @Ryangr0

Base: main ← Head: fix/cluster-health-audit-findings

📝 Commits (1)

136cce3 fix: resolve cluster health audit findings

📊 Changes

4 files changed (+26 additions, -10 deletions)

View changed files

📝 kubernetes/apps/cnpg-system/plugin-barman-cloud/app/helmrelease.yaml (+7 -0)
📝 kubernetes/apps/observability/kube-prometheus-stack/app/helmrelease.yaml (+1 -0)
📝 kubernetes/apps/renovate/renovate-operator/jobs/job-cleanup.cronjob.yaml (+2 -2)
📝 kubernetes/components/cnpg-monitoring/prometheus-rules.yaml (+16 -8)

📄 Description

Summary

Four concrete fixes surfaced by the cluster health audit agent run on 2026-05-22.

Changes

1. `renovate/job-cleanup` CronJob — daily failure fixed

Root cause (two bugs):

Flux postBuild.substituteFrom on ks-jobs.yaml silently erased ${ttl_days} → empty string (not a cluster-secrets key)
BusyBox date (alpine/k8s image) does not support GNU -d "-3 days" syntax

Fix: Replace with pure POSIX epoch arithmetic — no shell ${VAR} syntax, no GNU date dependency:

ttl_secs=$((3 * 86400))
cutoff_epoch=$(( $(date -u +%s) - ttl_secs ))

The cleanup job will now succeed and prune stale Renovate Jobs older than 3 days.

2. Alertmanager Discord notifications — unblocked

Root cause: RenovateProjectDependencyIssues fires for 5 repos simultaneously. Alertmanager groups them into one Discord message. The {{ range .Alerts }} template outputs full descriptions for each alert, exceeding Discord's 4096-char limit. Alertmanager had been silently dropping all warning/critical notifications for 9+ hours.

Fix: Add max_alerts: 5 to the Discord route. Notifications will be capped at 5 alerts per message; remaining alerts surface on the next interval.

3. `CNPGRestoreTestStale` / `CNPGRestoreTestFailed` — false-positive alerts silenced

Root cause: The restore-test CronJobs are intentionally suspend: true while CNPG disaster-recovery clusters are hibernated. The PrometheusRule fired unconditionally, treating suspended CronJobs as "stale".

Fix: Add unless on (namespace) kube_cronjob_spec_suspend{cronjob="cnpg-restore-test"} == 1 to both alert expressions. Alerts only fire when the CronJob is actually running and overdue.

4. `plugin-barman-cloud` — resource limits added

Context: The barman-cloud plugin deployment had 177 restarts over 37 days with no resource limits set. Periodic crashes cause the cnpg-disaster-recovery clusters to hit "context deadline exceeded" when CNPG tries to discover the plugin.

Fix: Add requests: 64Mi / limits: 256Mi to prevent unconstrained memory growth.

Testing

Shell script syntax validated with bash -n
All 4 YAML files validated (no tabs, valid structure)
Flux reconciliation will apply on next sync

Not fixed in this PR

Barman-cloud periodic API-server lease timeouts — monitoring for improvement after resource limits land
Tempo pod 15,555 restarts — separate investigation needed
pyroscope-alloy OOMKills — memory tuning needed (separate PR)
Orphaned test Longhorn volume — manual cleanup via Longhorn UI

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/webgrip/homelab-cluster/pull/67 **Author:** [@Ryangr0](https://github.com/Ryangr0) **Created:** 5/22/2026 **Status:** ✅ Merged **Merged:** 5/22/2026 **Merged by:** [@Ryangr0](https://github.com/Ryangr0) **Base:** `main` ← **Head:** `fix/cluster-health-audit-findings` --- ### 📝 Commits (1) - [`136cce3`](https://github.com/webgrip/homelab-cluster/commit/136cce30c5b5bf5791d0860171e63ae1bb6f4114) fix: resolve cluster health audit findings ### 📊 Changes **4 files changed** (+26 additions, -10 deletions) <details> <summary>View changed files</summary> 📝 `kubernetes/apps/cnpg-system/plugin-barman-cloud/app/helmrelease.yaml` (+7 -0) 📝 `kubernetes/apps/observability/kube-prometheus-stack/app/helmrelease.yaml` (+1 -0) 📝 `kubernetes/apps/renovate/renovate-operator/jobs/job-cleanup.cronjob.yaml` (+2 -2) 📝 `kubernetes/components/cnpg-monitoring/prometheus-rules.yaml` (+16 -8) </details> ### 📄 Description ## Summary Four concrete fixes surfaced by the cluster health audit agent run on 2026-05-22. --- ## Changes ### 1. `renovate/job-cleanup` CronJob — daily failure fixed **Root cause (two bugs):** - Flux `postBuild.substituteFrom` on `ks-jobs.yaml` silently erased `${ttl_days}` → empty string (not a `cluster-secrets` key) - BusyBox `date` (alpine/k8s image) does not support GNU `-d "-3 days"` syntax **Fix:** Replace with pure POSIX epoch arithmetic — no shell `${VAR}` syntax, no GNU date dependency: ```sh ttl_secs=$((3 * 86400)) cutoff_epoch=$(( $(date -u +%s) - ttl_secs )) ``` The cleanup job will now succeed and prune stale Renovate Jobs older than 3 days. --- ### 2. Alertmanager Discord notifications — unblocked **Root cause:** `RenovateProjectDependencyIssues` fires for 5 repos simultaneously. Alertmanager groups them into one Discord message. The `{{ range .Alerts }}` template outputs full descriptions for each alert, exceeding Discord's 4096-char limit. Alertmanager had been silently dropping **all** warning/critical notifications for 9+ hours. **Fix:** Add `max_alerts: 5` to the Discord route. Notifications will be capped at 5 alerts per message; remaining alerts surface on the next interval. --- ### 3. `CNPGRestoreTestStale` / `CNPGRestoreTestFailed` — false-positive alerts silenced **Root cause:** The restore-test CronJobs are intentionally `suspend: true` while CNPG disaster-recovery clusters are hibernated. The PrometheusRule fired unconditionally, treating suspended CronJobs as "stale". **Fix:** Add `unless on (namespace) kube_cronjob_spec_suspend{cronjob="cnpg-restore-test"} == 1` to both alert expressions. Alerts only fire when the CronJob is actually running and overdue. --- ### 4. `plugin-barman-cloud` — resource limits added **Context:** The barman-cloud plugin deployment had 177 restarts over 37 days with no resource limits set. Periodic crashes cause the `cnpg-disaster-recovery` clusters to hit "context deadline exceeded" when CNPG tries to discover the plugin. **Fix:** Add `requests: 64Mi` / `limits: 256Mi` to prevent unconstrained memory growth. --- ## Testing - Shell script syntax validated with `bash -n` - All 4 YAML files validated (no tabs, valid structure) - Flux reconciliation will apply on next sync ## Not fixed in this PR - Barman-cloud periodic API-server lease timeouts — monitoring for improvement after resource limits land - Tempo pod 15,555 restarts — separate investigation needed - pyroscope-alloy OOMKills — memory tuning needed (separate PR) - Orphaned `test` Longhorn volume — manual cleanup via Longhorn UI --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

ryangr0

2026-06-08 08:31:16 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

webgrip/homelab-cluster#293

No description provided.

Rows
Columns