diff --git a/.last-branch b/.last-branch deleted file mode 100644 index 26e0b09..0000000 --- a/.last-branch +++ /dev/null @@ -1 +0,0 @@ -v20260402-01 diff --git a/TODO-documentation.md b/TODO-documentation.md index dd1cd86..a1f4097 100644 --- a/TODO-documentation.md +++ b/TODO-documentation.md @@ -1288,7 +1288,7 @@ Static Files: - static/images/documentation/* # NEW: Screenshots folder Docs: -- changelog-claude.md # UPDATE: Document feature addition +- changelog-develop.md # UPDATE: Document feature addition ``` --- diff --git a/TODO-reports-improvements.md b/TODO-reports-improvements.md index 46602b9..3b5284b 100644 --- a/TODO-reports-improvements.md +++ b/TODO-reports-improvements.md @@ -927,7 +927,7 @@ containers/backupchecks/src/templates/main/ └── settings.html # Settings (add Reporting section) docs/ -└── changelog-claude.md # Changelog entry +└── changelog-develop.md # Changelog entry ``` --- diff --git a/TODO-smart-overrides.md b/TODO-smart-overrides.md new file mode 100644 index 0000000..96a6db8 --- /dev/null +++ b/TODO-smart-overrides.md @@ -0,0 +1,400 @@ +# Smart Overrides — Technisch Ontwerp + +> Backupchecks leert van operator-acties om herhaalwerk te verminderen. +> Gefaseerde aanpak: directe UI-verbetering → patroonherkenning → cross-job kennisbank. + +--- + +## Fase 1: "Wil je dit ook in de toekomst?" + +### Probleem + +De huidige "Mark as Success" knop in Run Checks maakt een override aan met een tijdvenster van ±1 minuut rond de specifieke run. Dezelfde fout morgen? Opnieuw handmatig markeren. + +### Oplossing + +Na "Mark as Success" toont de UI een vervolgdialoog waarin de operator de scope en duur kan bepalen. De backend accepteert deze extra parameters en maakt een bredere override aan. + +### UI-wijziging: vervolgdialoog + +Na een succesvolle `mark-success-override` API-call verschijnt een modal (of inline panel in de bestaande modal) met: + +**Scope-keuze (radio buttons):** + +- "Alleen deze run" ← huidige gedrag, default +- "Deze job, zelfde foutmelding" → object-level override op `job_id` + `match_error_contains` +- "Alle jobs met deze software/type en zelfde foutmelding" → global override op `backup_software` + `backup_type` + `match_error_contains` + +**Duur-keuze (radio buttons):** + +- "Eenmalig" ← huidige gedrag, ±1 minuut window +- "1 week" +- "1 maand" +- "Permanent (tot handmatig uitgeschakeld)" + +**Optioneel commentaar** (textarea, vooringevuld met de error-tekst als referentie) + +### API-wijziging + +`POST /api/run-checks/mark-success-override` krijgt optionele extra velden: + +```python +{ + "run_id": 123, + "scope": "job" | "global" | "run", # default: "run" (huidig gedrag) + "duration": "once" | "1w" | "1m" | "permanent", # default: "once" + "comment": "VSS snapshot timeout, known issue" +} +``` + +De backend-logica in `api_run_checks_mark_success_override()` verandert: + +- `scope="run"` → huidige gedrag (±1 min window) +- `scope="job"` → `Override(level="object", job_id=job.id, match_error_contains=..., start_at=now, end_at=now+duration)` +- `scope="global"` → `Override(level="global", backup_software=..., backup_type=..., match_error_contains=..., start_at=now, end_at=now+duration)` + +Bij `duration="permanent"` wordt `end_at=None` gezet. + +### Error-tekst extractie + +De vervolgdialoog moet de error-tekst tonen die als `match_error_contains` gebruikt wordt. Dit is al beschikbaar in de run-detail modal. De logica: + +1. Haal `run_object_links` op voor de run (bestaande query in `_apply_overrides_to_run`) +2. Filter op objecten met niet-success status +3. Neem de `error_message` van het eerste problematische object +4. Fallback naar `run.remark` + +Deze tekst wordt getoond in de dialoog zodat de operator kan zien *wat* er geleerd wordt, en eventueel kan aanpassen. + +### Database-wijzigingen + +Geen. Het bestaande `Override`-model ondersteunt alle benodigde velden al (`level`, `match_error_contains`, `match_error_mode`, `start_at`, `end_at`, `comment`). + +### Nieuwe audit logging + +Bij het aanmaken van een bredere override: log naar `AuditLog` met `event_type="override_from_review"` en details over scope, duur, en de oorspronkelijke run. + +--- + +## Fase 2: Patroonherkenning op reviews + +### Probleem + +Fase 1 vereist nog steeds een bewuste keuze van de operator. Als dezelfde fout 5× achter elkaar wordt gereviewed zonder ticket of opmerking, had het systeem dat al eerder moeten signaleren. + +### Oplossing + +Een achtergrondproces analyseert review-events en detecteert herhaalde patronen. Bij een drempel verschijnt een suggestie in de UI. + +### Nieuw model: `OverrideSuggestion` + +```python +class OverrideSuggestion(db.Model): + __tablename__ = "override_suggestions" + + id = db.Column(db.Integer, primary_key=True) + + # Fingerprint van het patroon (hash van genormaliseerde velden) + pattern_fingerprint = db.Column(db.String(64), nullable=False, index=True) + + # Leesbare omschrijving van het patroon + pattern_description = db.Column(db.Text, nullable=False) + + # Scope van de suggestie + suggested_level = db.Column(db.String(20), nullable=False) # global | object + suggested_backup_software = db.Column(db.String(255), nullable=True) + suggested_backup_type = db.Column(db.String(255), nullable=True) + suggested_job_id = db.Column(db.Integer, db.ForeignKey("jobs.id"), nullable=True) + suggested_match_status = db.Column(db.String(32), nullable=True) + suggested_match_error = db.Column(db.String(255), nullable=True) + + # Bewijs + match_count = db.Column(db.Integer, nullable=False, default=0) + sample_run_ids = db.Column(db.Text, nullable=True) # JSON list + first_seen_at = db.Column(db.DateTime, nullable=False) + last_seen_at = db.Column(db.DateTime, nullable=False) + + # Status + status = db.Column(db.String(20), nullable=False, default="pending") + # pending | accepted | dismissed | expired + resolved_by_user_id = db.Column(db.Integer, db.ForeignKey("users.id"), nullable=True) + resolved_at = db.Column(db.DateTime, nullable=True) + created_override_id = db.Column(db.Integer, db.ForeignKey("overrides.id"), nullable=True) + + created_at = db.Column(db.DateTime, nullable=False) + updated_at = db.Column(db.DateTime, nullable=False) +``` + +### Patroondetectie-logica + +**Trigger:** Na elke `mark-reviewed` actie (of als periodieke task, bijv. elk uur). + +**Algoritme:** + +``` +1. Query alle JobRuns die in de laatste 30 dagen reviewed zijn + ZONDER dat er een ticket aan gelinkt is + EN ZONDER dat er een override actief was + +2. Groepeer op: + - (backup_software, backup_type, status, genormaliseerde_error) → global kandidaat + - (job_id, status, genormaliseerde_error) → object kandidaat + +3. Voor elke groep met count >= DREMPEL (configureerbaar, default 3): + - Bereken fingerprint: sha256(level + scope-velden + genormaliseerde_error) + - Check of er al een OverrideSuggestion bestaat met deze fingerprint + → Ja: update match_count en last_seen_at + → Nee: maak nieuwe suggestie aan + +4. Suggesties ouder dan 90 dagen zonder nieuwe matches: status → expired +``` + +**Error-normalisatie** (voor groepering): + +```python +def normalize_error_for_grouping(error: str) -> str: + """Verwijder variabele delen uit error-berichten zodat + dezelfde fout met verschillende details als één patroon herkend wordt.""" + import re + s = (error or "").strip() + # Verwijder timestamps (diverse formaten) + s = re.sub(r'\d{4}[-/]\d{2}[-/]\d{2}[T ]\d{2}:\d{2}(:\d{2})?', '', s) + # Verwijder GUIDs + s = re.sub(r'[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}', '', s) + # Verwijder IP-adressen + s = re.sub(r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b', '', s) + # Verwijder paden (Windows en Unix) + s = re.sub(r'[A-Z]:\\[\w\\.-]+', '', s) + s = re.sub(r'/[\w/.-]+', '', s) + # Verwijder grote getallen (bijv. bytes, sizes) + s = re.sub(r'\b\d{4,}\b', '', s) + # Normaliseer whitespace + s = re.sub(r'\s+', ' ', s).strip() + return s +``` + +### UI: suggesties tonen + +**Optie A — Dashboard badge:** Een badge naast "Override Suggestions" in de sidebar of op het dashboard (zoals inbox_count). Operator klikt door naar een lijst. + +**Optie B — Inline in Run Checks:** Wanneer een operator een run opent die past bij een bestaand patroon, toon een banner: *"Dit patroon is al 5× handmatig gereviewed. Wil je een override aanmaken?"* met knoppen "Ja, maak override" en "Negeer". + +Aanbeveling: **beide**. De dashboard-badge geeft een overzicht, de inline-banner maakt het actionable op het moment dat het relevant is. + +**Suggesties-pagina** (`/override-suggestions`): + +- Lijst van pending suggesties, gesorteerd op match_count (hoogste eerst) +- Per suggestie: patroon-omschrijving, aantal matches, sample runs (klikbaar), voorgestelde override-config +- Acties: "Accepteer" (maakt Override aan met voorgestelde config), "Negeer" (dismissed), "Aanpassen" (opent override-aanmaakformulier met vooringevulde velden) + +### Configuratie + +Nieuwe velden op `SystemSettings`: + +```python +# Override Suggestions +suggestion_enabled = db.Column(db.Boolean, nullable=False, default=True) +suggestion_threshold = db.Column(db.Integer, nullable=False, default=3) +suggestion_lookback_days = db.Column(db.Integer, nullable=False, default=30) +suggestion_expiry_days = db.Column(db.Integer, nullable=False, default=90) +``` + +--- + +## Fase 3: Error-fingerprint kennisbank + +### Probleem + +Fase 2 werkt per installatie. Als klant A dezelfde Veeam VSS-fout heeft als klant B, moet de operator het bij beide apart afhandelen. + +### Oplossing + +Bouw een kennisbank van genormaliseerde error-fingerprints met hun classificatie. Nieuwe runs worden automatisch gematcht tegen de kennisbank. + +### Nieuw model: `ErrorPattern` + +```python +class ErrorPattern(db.Model): + __tablename__ = "error_patterns" + + id = db.Column(db.Integer, primary_key=True) + + # Genormaliseerde fingerprint (output van normalize_error_for_grouping + sha256) + fingerprint = db.Column(db.String(64), unique=True, nullable=False, index=True) + + # Leesbaar voorbeeld van de originele error + example_error = db.Column(db.Text, nullable=False) + + # Genormaliseerde versie (voor display) + normalized_error = db.Column(db.Text, nullable=False) + + # Classificatie + classification = db.Column(db.String(20), nullable=False, default="unknown") + # benign | actionable | critical | unknown + + # Automatische actie + auto_action = db.Column(db.String(20), nullable=False, default="none") + # none | treat_as_success | needs_review | escalate + + # Scope-beperking (optioneel: alleen voor specifieke software/type) + scope_backup_software = db.Column(db.String(255), nullable=True) + scope_backup_type = db.Column(db.String(255), nullable=True) + + # Herkomst + learned_from = db.Column(db.String(32), nullable=False, default="manual") + # manual | override | suggestion + source_override_id = db.Column(db.Integer, nullable=True) + + # Statistieken + times_matched = db.Column(db.Integer, nullable=False, default=0) + times_confirmed = db.Column(db.Integer, nullable=False, default=0) # keer dat operator OK gaf + times_rejected = db.Column(db.Integer, nullable=False, default=0) # keer dat operator NIET OK gaf + confidence = db.Column(db.Float, nullable=False, default=0.0) + + # Beheer + active = db.Column(db.Boolean, nullable=False, default=True) + created_by = db.Column(db.String(255), nullable=True) + notes = db.Column(db.Text, nullable=True) + + created_at = db.Column(db.DateTime, nullable=False) + updated_at = db.Column(db.DateTime, nullable=False) +``` + +### Automatisch leren van overrides + +Wanneer een Override wordt aangemaakt (via Fase 1-dialoog, Fase 2-suggestie, of handmatig): + +```python +def learn_from_override(override: Override): + """Extraheer een ErrorPattern uit een nieuwe override.""" + if not override.match_error_contains: + return # Geen error-tekst om van te leren + + normalized = normalize_error_for_grouping(override.match_error_contains) + fp = hashlib.sha256(normalized.encode()).hexdigest()[:16] + + existing = ErrorPattern.query.filter_by(fingerprint=fp).first() + if existing: + existing.times_confirmed += 1 + existing.confidence = min(1.0, existing.times_confirmed / (existing.times_confirmed + existing.times_rejected + 1)) + existing.updated_at = datetime.utcnow() + return + + pattern = ErrorPattern( + fingerprint=fp, + example_error=override.match_error_contains, + normalized_error=normalized, + classification="benign", + auto_action="treat_as_success" if override.treat_as_success else "none", + scope_backup_software=override.backup_software, + scope_backup_type=override.backup_type, + learned_from="override", + source_override_id=override.id, + times_matched=0, + times_confirmed=1, + confidence=0.5, # startwaarde bij eerste bevestiging + active=True, + created_by=override.created_by, + created_at=datetime.utcnow(), + updated_at=datetime.utcnow(), + ) + db.session.add(pattern) +``` + +### Integratie in de run-interpretatie + +De fingerprint-check wordt een extra laag in `_apply_overrides_to_run()`, ná de bestaande override-evaluatie: + +```python +# Na de bestaande override checks (regel ~502-516 in routes_shared.py): + +# Fase 3: Check error pattern kennisbank +if not override_applied: + error_text = _get_run_error_text(run, run_object_rows) + if error_text: + normalized = normalize_error_for_grouping(error_text) + fp = hashlib.sha256(normalized.encode()).hexdigest()[:16] + pattern = ErrorPattern.query.filter_by( + fingerprint=fp, active=True + ).first() + + if pattern and pattern.confidence >= CONFIDENCE_THRESHOLD: + # Check scope-beperking + if pattern.scope_backup_software and pattern.scope_backup_software.lower() != (job.backup_software or "").lower(): + pattern = None + if pattern and pattern.scope_backup_type and pattern.scope_backup_type.lower() != (job.backup_type or "").lower(): + pattern = None + + if pattern: + pattern.times_matched += 1 + if pattern.auto_action == "treat_as_success": + return "Success (auto)", True, "pattern", pattern.id, f"ErrorPattern id={pattern.id}" +``` + +**`CONFIDENCE_THRESHOLD`** (configureerbaar, default 0.7): voorkomt dat patronen met weinig bewijs automatisch worden toegepast. Een patroon begint op 0.5 en stijgt door bevestigingen. + +### Feedback-loop: confidence bijwerken + +Wanneer een operator een run reviewed die door een ErrorPattern als success was gemarkeerd: +- **Reviewed zonder correctie** → `pattern.times_confirmed += 1` +- **Operator maakt alsnog een ticket aan** → `pattern.times_rejected += 1` +- Herbereken: `confidence = confirmed / (confirmed + rejected + 1)` +- Bij confidence < 0.3: automatisch `active = False` zetten + +### UI: kennisbank-pagina + +**`/error-patterns`** (admin/operator): + +- Tabel met alle patronen, sorteerbaar op confidence, times_matched, classificatie +- Per patroon: genormaliseerde error, voorbeeld, classificatie, auto_action, confidence-balk, match-statistieken +- Inline edit voor classificatie en auto_action +- Bulk-acties: activeren/deactiveren +- Filter op software/type + +In de run-detail modal: als een ErrorPattern is toegepast, toon een klein label "Auto-classified: benign (confidence 85%)" met link naar het patroon. + +--- + +## Implementatievolgorde en afhankelijkheden + +``` +Fase 1 (directe waarde, ~2-3 dagen werk) +├── Backend: scope + duration parameters toevoegen aan mark-success-override +├── Frontend: vervolgdialoog in run_checks.html modal +├── Audit logging voor bredere overrides +└── Geen nieuwe modellen of migraties nodig + +Fase 2 (patroonherkenning, ~1-2 weken werk) +├── Migratie: override_suggestions tabel +├── Migratie: SystemSettings velden voor configuratie +├── Backend: patroondetectie-logica + normalize_error_for_grouping() +├── Backend: trigger na mark-reviewed OF periodieke task +├── Frontend: suggesties-pagina (/override-suggestions) +├── Frontend: dashboard badge + inline banner in Run Checks +└── Afhankelijk van: Fase 1 (optioneel, maar geeft betere data) + +Fase 3 (kennisbank, ~2-3 weken werk) +├── Migratie: error_patterns tabel +├── Backend: learn_from_override() hook +├── Backend: fingerprint-check in _apply_overrides_to_run() +├── Backend: confidence feedback-loop bij review-acties +├── Frontend: /error-patterns beheerpagina +├── Frontend: "Auto-classified" label in run-detail modal +└── Afhankelijk van: Fase 2 (voor normalize_error_for_grouping()) +``` + +--- + +## Risico's en mitigatie + +**False positives (patroon matcht verkeerd)** +Mitigatie: confidence-drempel, scope-beperkingen, en de mogelijkheid om patronen te deactiveren. Fase 3-patronen met `auto_action=treat_as_success` worden in de UI duidelijk gemarkeerd als "Auto" zodat operators ze kunnen herkennen en corrigeren. + +**Error-normalisatie te agressief** +Mitigatie: begin conservatief (alleen timestamps, GUIDs, IPs verwijderen). Breid de normalisatie geleidelijk uit op basis van ervaring. De kennisbank-pagina toont zowel de genormaliseerde als de originele tekst zodat operators kunnen beoordelen of de normalisatie klopt. + +**Performance bij grote datasets** +Mitigatie: `pattern_fingerprint` en `fingerprint` zijn geïndexeerd. De patroondetectie-query in Fase 2 draait periodiek (niet real-time). De fingerprint-lookup in Fase 3 is een simpele indexed lookup per run. + +**Operator-vertrouwen** +Mitigatie: het systeem suggereert en labelt, maar de operator heeft altijd het laatste woord. "Success (auto)" is visueel onderscheidbaar van "Success" en "Success (override)". Elke automatische actie is traceerbaar naar het bronpatroon. diff --git a/build-and-push.sh b/build-and-push.sh index 4de6431..6b4be0e 100755 --- a/build-and-push.sh +++ b/build-and-push.sh @@ -3,79 +3,54 @@ set -euo pipefail # ============================================================================ # build-and-push.sh -# Location: repo root (e.g. /docker/develop/backup-monitoring) # # Purpose: -# - Automatic version bump: -# 1 = patch, 2 = minor, 3 = major, t = test -# - Test builds: only update :dev (no commit/tag) -# - Release builds: update version.txt, commit, tag, push (to the current branch) -# - Build & push Docker images for each service under ./compose/* -# - Preflight checks: Docker daemon up, logged in to registry, valid names/tags -# - Summary: show all images + tags built and pushed -# - Branch visibility: -# - Shows currently checked out branch (authoritative) -# - Reads .last-branch for info (if present) when BRANCH is not set -# - Writes the current branch back to .last-branch at the end +# - Build & push Docker images for each service under ./containers/* +# - Two modes: +# t (test) = only push :dev +# r (release) = push :, :dev, :latest +# version is read from the top of changelog.md +# +# No git operations: committing and tagging is done manually. # # Usage: -# BRANCH= ./build-and-push.sh [bump] # BRANCH is optional; informative only -# ./build-and-push.sh [bump] -# If [bump] is omitted, you will be prompted (default = t). +# ./build-and-push.sh [mode] +# - mode = t -> test build, push :dev only +# - mode = r -> release build, version taken from changelog.md +# - omitted -> prompt (default: t) # -# Tagging rules: -# - Release build (1/2/3): push :, :dev, :latest -# - Test build (t): push only :dev (no :latest, no version tag) +# Requirements: +# - docs/changelog.md (relative to repo root), with the most recent release +# at the top as: +# ## vX.Y.Z — YYYY-MM-DD +# (the version is parsed from the first such line) +# - One Dockerfile per service under ./containers//Dockerfile # ============================================================================ DOCKER_REGISTRY="gitea.oskamp.info" DOCKER_NAMESPACE="ivooskamp" -VERSION_FILE="version.txt" -START_VERSION="v0.1.0" -COMPOSE_DIR="containers" -LAST_BRANCH_FILE=".last-branch" # stored in repo root +CHANGELOG_FILE="docs/changelog.md" +CONTAINERS_DIR="containers" # --- Input: prompt if missing ------------------------------------------------ -BUMP="${1:-}" -if [[ -z "${BUMP}" ]]; then - echo "Select bump type: [1] patch, [2] minor, [3] major, [t] test (default: t)" - read -r BUMP - BUMP="${BUMP:-t}" +MODE="${1:-}" +if [[ -z "${MODE}" ]]; then + echo "Select build type: [t] test build (push :dev only), [r] release build (default: t)" + read -r MODE + MODE="${MODE:-t}" fi -if [[ "$BUMP" != "1" && "$BUMP" != "2" && "$BUMP" != "3" && "$BUMP" != "t" ]]; then - echo "[ERROR] Unknown bump type '$BUMP' (use 1, 2, 3, or t)." - exit 1 -fi +case "$MODE" in + t|test) MODE="t" ;; + r|release) MODE="r" ;; + *) + echo "[ERROR] Unknown mode '$MODE' (use 't' for test or 'r' for release)." + exit 1 + ;; +esac # --- Helpers ----------------------------------------------------------------- -read_version() { - if [[ -f "$VERSION_FILE" ]]; then - tr -d ' \t\n\r' < "$VERSION_FILE" - else - echo "$START_VERSION" - fi -} - -write_version() { - echo "$1" > "$VERSION_FILE" -} - -bump_version() { - local cur="$1" - local kind="$2" - local core="${cur#v}" - IFS='.' read -r MA MI PA <<< "$core" - case "$kind" in - 1) PA=$((PA + 1));; - 2) MI=$((MI + 1)); PA=0;; - 3) MA=$((MA + 1)); MI=0; PA=0;; - *) echo "[ERROR] Unknown bump kind"; exit 1;; - esac - echo "v${MA}.${MI}.${PA}" -} - check_docker_ready() { if ! docker info >/dev/null 2>&1; then echo "[ERROR] Docker daemon not reachable. Is Docker running and do you have permission to use it?" @@ -117,14 +92,35 @@ validate_tag() { fi } -# --- Preflight --------------------------------------------------------------- -if [[ ! -d ".git" ]]; then - echo "[ERROR] Not a git repository (.git missing)." - exit 1 -fi +# Parse the first "## vX.Y.Z ..." heading from changelog.md. +# Accepts: ## v1.0.3 — 2026-04-24 +# ## v1.0.3 - 2026-04-24 +# ## v1.0.3 +read_version_from_changelog() { + if [[ ! -f "$CHANGELOG_FILE" ]]; then + echo "[ERROR] $CHANGELOG_FILE not found in $(pwd)." >&2 + exit 1 + fi + local line + # Match lines starting with "## v.." + line="$(grep -m1 -E '^##[[:space:]]+v[0-9]+\.[0-9]+\.[0-9]+' "$CHANGELOG_FILE" || true)" + if [[ -z "$line" ]]; then + echo "[ERROR] No release heading found in $CHANGELOG_FILE (expected e.g. '## v1.0.3 — 2026-04-24' near the top)." >&2 + exit 1 + fi + # Extract the vX.Y.Z token + local version + version="$(echo "$line" | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -n1)" + if [[ -z "$version" ]]; then + echo "[ERROR] Could not parse version from line: $line" >&2 + exit 1 + fi + echo "$version" +} -if [[ ! -d "$COMPOSE_DIR" ]]; then - echo "[ERROR] '$COMPOSE_DIR' directory missing. Expected ./compose// with a Dockerfile." +# --- Preflight --------------------------------------------------------------- +if [[ ! -d "$CONTAINERS_DIR" ]]; then + echo "[ERROR] '$CONTAINERS_DIR' directory missing. Expected ./${CONTAINERS_DIR}// with a Dockerfile." exit 1 fi @@ -132,91 +128,44 @@ check_docker_ready ensure_registry_login validate_repo_component "$DOCKER_NAMESPACE" -# Detect currently checked out branch (authoritative for this script) -DETECTED_BRANCH="$(git branch --show-current 2>/dev/null || true)" -if [[ -z "$DETECTED_BRANCH" ]]; then - DETECTED_BRANCH="$(git symbolic-ref --quiet --short HEAD 2>/dev/null || true)" -fi -if [[ -z "$DETECTED_BRANCH" ]]; then - # Try to derive from upstream - UPSTREAM_REF_DERIVED="$(git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || true)" - if [[ -n "$UPSTREAM_REF_DERIVED" ]]; then - DETECTED_BRANCH="${UPSTREAM_REF_DERIVED#origin/}" - fi -fi -if [[ -z "$DETECTED_BRANCH" ]]; then - DETECTED_BRANCH="main" -fi - -# Optional signals: BRANCH env and .last-branch (informational only) -ENV_BRANCH="${BRANCH:-}" -LAST_BRANCH_FILE_PATH="$(pwd)/$LAST_BRANCH_FILE" -LAST_BRANCH_VALUE="" -if [[ -z "$ENV_BRANCH" && -f "$LAST_BRANCH_FILE_PATH" ]]; then - LAST_BRANCH_VALUE="$(tr -d ' \t\n\r' < "$LAST_BRANCH_FILE_PATH")" -fi - -UPSTREAM_REF="$(git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "origin/$DETECTED_BRANCH")" -HEAD_SHA="$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")" - -echo "[INFO] Repo: $(pwd)" -echo "[INFO] Current branch: $DETECTED_BRANCH" -echo "[INFO] Upstream: $UPSTREAM_REF" -echo "[INFO] HEAD (sha): $HEAD_SHA" - -if [[ -n "$ENV_BRANCH" && "$ENV_BRANCH" != "$DETECTED_BRANCH" ]]; then - echo "[WARNING] BRANCH='$ENV_BRANCH' differs from checked out branch '$DETECTED_BRANCH'." - echo "[WARNING] This script does not switch branches; continuing on '$DETECTED_BRANCH'." -fi - -if [[ -n "$LAST_BRANCH_VALUE" && "$LAST_BRANCH_VALUE" != "$DETECTED_BRANCH" && -z "$ENV_BRANCH" ]]; then - echo "[INFO] .last-branch suggests '$LAST_BRANCH_VALUE', but current checkout is '$DETECTED_BRANCH'." - echo "[INFO] If you intended to build '$LAST_BRANCH_VALUE', switch branches first (use update-and-build.sh)." -fi - -# --- Versioning -------------------------------------------------------------- -CURRENT_VERSION="$(read_version)" -NEW_VERSION="$CURRENT_VERSION" -DO_TAG_AND_BUMP=true - -if [[ "$BUMP" == "t" ]]; then - echo "[INFO] Test build: keeping version $CURRENT_VERSION; will only update :dev." - DO_TAG_AND_BUMP=false +# Informational: show branch and HEAD if this happens to be a git repo. +BRANCH_INFO="" +HEAD_INFO="" +if [[ -d ".git" ]]; then + BRANCH_INFO="$(git branch --show-current 2>/dev/null || echo unknown)" + HEAD_INFO="$(git rev-parse --short HEAD 2>/dev/null || echo unknown)" + echo "[INFO] Repo: $(pwd)" + echo "[INFO] Current branch: $BRANCH_INFO" + echo "[INFO] HEAD (sha): $HEAD_INFO" else - NEW_VERSION="$(bump_version "$CURRENT_VERSION" "$BUMP")" - echo "[INFO] New version: $NEW_VERSION" + echo "[INFO] Repo: $(pwd) (not a git checkout)" fi -if $DO_TAG_AND_BUMP; then - validate_tag "$NEW_VERSION" +# --- Determine version (release only) ---------------------------------------- +VERSION="" +if [[ "$MODE" == "r" ]]; then + VERSION="$(read_version_from_changelog)" + echo "[INFO] Release version (from $CHANGELOG_FILE): $VERSION" + validate_tag "$VERSION" validate_tag "latest" + + # Ask for confirmation so you never accidentally re-push an old version or a wrong one. + read -r -p "Proceed building & pushing as ${VERSION}? [y/N] " CONFIRM + CONFIRM="${CONFIRM:-N}" + if [[ ! "$CONFIRM" =~ ^[Yy]$ ]]; then + echo "[INFO] Aborted by user." + exit 0 + fi +else + echo "[INFO] Test build: only :dev will be pushed." fi validate_tag "dev" -# --- Version update + VCS ops (release builds only) -------------------------- -if $DO_TAG_AND_BUMP; then - echo "[INFO] Writing $NEW_VERSION to $VERSION_FILE" - write_version "$NEW_VERSION" - - echo "[INFO] Git add + commit (branch: $DETECTED_BRANCH)" - git add "$VERSION_FILE" - git commit -m "Release $NEW_VERSION on branch $DETECTED_BRANCH (bump type $BUMP)" - - echo "[INFO] Git tag $NEW_VERSION" - git tag -a "$NEW_VERSION" -m "Release $NEW_VERSION" - - echo "[INFO] Git push + tags" - git push origin "$DETECTED_BRANCH" - git push --tags -else - echo "[INFO] Skipping commit/tagging (test build)." -fi - # --- Build & push per service ------------------------------------------------ shopt -s nullglob -services=( "$COMPOSE_DIR"/* ) +services=( "$CONTAINERS_DIR"/* ) if [[ ${#services[@]} -eq 0 ]]; then - echo "[ERROR] No services found under $COMPOSE_DIR" + echo "[ERROR] No services found under $CONTAINERS_DIR" exit 1 fi @@ -236,21 +185,21 @@ for svc_path in "${services[@]}"; do IMAGE_BASE="${DOCKER_REGISTRY}/${DOCKER_NAMESPACE}/${svc}" - if $DO_TAG_AND_BUMP; then + if [[ "$MODE" == "r" ]]; then echo "============================================================" - echo "[INFO] Building ${svc} -> tags: ${NEW_VERSION}, dev, latest" + echo "[INFO] Building ${svc} -> tags: ${VERSION}, dev, latest" echo "============================================================" docker build \ - -t "${IMAGE_BASE}:${NEW_VERSION}" \ + -t "${IMAGE_BASE}:${VERSION}" \ -t "${IMAGE_BASE}:dev" \ -t "${IMAGE_BASE}:latest" \ "$svc_path" - docker push "${IMAGE_BASE}:${NEW_VERSION}" + docker push "${IMAGE_BASE}:${VERSION}" docker push "${IMAGE_BASE}:dev" docker push "${IMAGE_BASE}:latest" - BUILT_IMAGES+=("${IMAGE_BASE}:${NEW_VERSION}" "${IMAGE_BASE}:dev" "${IMAGE_BASE}:latest") + BUILT_IMAGES+=("${IMAGE_BASE}:${VERSION}" "${IMAGE_BASE}:dev" "${IMAGE_BASE}:latest") else echo "============================================================" echo "[INFO] Test build ${svc} -> tag: dev" @@ -261,21 +210,27 @@ for svc_path in "${services[@]}"; do fi done -# --- Persist current branch to .last-branch ---------------------------------- -# (This helps script 1 to preselect next time, and is informative if you run script 2 standalone) -echo "$DETECTED_BRANCH" > "$LAST_BRANCH_FILE_PATH" - # --- Summary ----------------------------------------------------------------- echo "" echo "============================================================" -echo "[SUMMARY] Build & push complete (branch: $DETECTED_BRANCH)" -if $DO_TAG_AND_BUMP; then - echo "[INFO] Release version: $NEW_VERSION" +if [[ "$MODE" == "r" ]]; then + echo "[SUMMARY] Release build & push complete: $VERSION" else - echo "[INFO] Test build (no version bump)" + echo "[SUMMARY] Test build & push complete (:dev only)" +fi +if [[ -n "$BRANCH_INFO" ]]; then + echo "[INFO] Branch: $BRANCH_INFO HEAD: $HEAD_INFO" fi echo "[INFO] Images pushed:" for img in "${BUILT_IMAGES[@]}"; do echo " - $img" done echo "============================================================" +echo "" +echo "[REMINDER] No git operations were performed. If this was a release," +echo " commit and tag manually, e.g.:" +if [[ "$MODE" == "r" ]]; then + echo " git add -A && git commit -m \"Release ${VERSION}\"" + echo " git tag -a ${VERSION} -m \"Release ${VERSION}\"" + echo " git push && git push --tags" +fi diff --git a/containers/backupchecks/src/backend/app/cove_importer.py b/containers/backupchecks/src/backend/app/cove_importer.py index 2f21d2e..5c07167 100644 --- a/containers/backupchecks/src/backend/app/cove_importer.py +++ b/containers/backupchecks/src/backend/app/cove_importer.py @@ -541,6 +541,23 @@ def run_cove_import(settings, include_reasons: bool = False): break start += page_size + # Cove workstation offline detection (colorbar-based) — toggle in settings. + # Runs once per import cycle so the synthetic offline runs stay in sync with + # the freshly upserted colorbar data above. + try: + offline_changes = _apply_offline_detection_for_workstations(settings) + if offline_changes: + logger.info( + "Cove offline detection: updated %d workstation job(s)", + offline_changes, + ) + except Exception as exc: + logger.warning("Cove offline detection failed: %s", exc) + try: + db.session.rollback() + except Exception: + pass + # Update last import timestamp settings.cove_last_import_at = datetime.utcnow() try: @@ -553,6 +570,165 @@ def run_cove_import(settings, include_reasons: bool = False): return total, created, skipped, errors +def _parse_colorbar_codes(colorbar: str) -> list[int | None]: + """Parse a colorbar string into a list of status codes (oldest first). + + Non-numeric entries become None so the caller can ignore them when counting + streaks. + """ + stripped = (colorbar or "").strip() + if not stripped: + return [] + if " " in stripped or "," in stripped: + raw = re.split(r"[,\s]+", stripped) + else: + raw = list(stripped) + out: list[int | None] = [] + for c in raw: + try: + out.append(int(str(c).strip())) + except (ValueError, TypeError): + out.append(None) + return out + + +def _trailing_inactive_streak(codes: list[int | None]) -> int: + """Count consecutive trailing days where the colorbar code == 0 (no backup). + + Stops at the first non-zero, non-None code (a real status, success or not). + """ + streak = 0 + for code in reversed(codes): + if code == 0: + streak += 1 + else: + break + return streak + + +def _apply_offline_detection_for_workstations(settings) -> int: + """For every linked Cove workstation job, ensure a synthetic offline JobRun + exists when the colorbar shows extended inactivity. Returns the number of + job runs created/updated/removed. + + No-op when settings.cove_offline_detection_enabled is False. + """ + if not getattr(settings, "cove_offline_detection_enabled", False): + return 0 + + from .models import CoveAccount, Job, JobRun + + warn_days = max(1, int(getattr(settings, "cove_workstation_warning_days", 7) or 7)) + err_days = max(1, int(getattr(settings, "cove_workstation_error_days", 14) or 14)) + + # Linked Cove workstation jobs only. + rows = ( + db.session.query(CoveAccount, Job) + .join(Job, Job.id == CoveAccount.job_id) + .filter(CoveAccount.job_id.isnot(None)) + .filter(Job.archived.is_(False)) + .filter(Job.backup_software == "Cove Data Protection") + .all() + ) + + changes = 0 + for cove_acc, job in rows: + if (job.backup_type or "").strip().lower() != "workstation": + continue + try: + if _ensure_cove_offline_run(cove_acc, job, warn_days, err_days): + changes += 1 + except Exception as exc: + logger.warning( + "Cove offline detection: error on account %s: %s", + getattr(cove_acc, "account_id", "?"), + exc, + ) + try: + db.session.rollback() + except Exception: + pass + + if changes: + try: + db.session.commit() + except Exception: + db.session.rollback() + return changes + + +def _ensure_cove_offline_run(cove_acc, job, warn_days: int, err_days: int) -> bool: + """Create / update / remove the synthetic offline JobRun for one job. + + Uses a stable external_id `cove-offline-{account_id}` so the same row is + reused across import cycles. Reviewed runs are left untouched so previously + acknowledged alerts do not reappear or mutate. + + Returns True when a row was inserted, updated, or deleted. + """ + from .models import JobRun + + codes = _parse_colorbar_codes(getattr(cove_acc, "colorbar_28d", None) or "") + streak = _trailing_inactive_streak(codes) + + if streak >= err_days: + target_status = "Error" + elif streak >= warn_days: + target_status = "Warning" + else: + target_status = None + + external_id = f"cove-offline-{cove_acc.account_id}" + existing = ( + JobRun.query.filter_by(job_id=job.id, external_id=external_id).first() + ) + + if target_status is None: + # Condition cleared — drop unreviewed offline run; leave reviewed history alone. + if existing and existing.reviewed_at is None: + db.session.delete(existing) + return True + return False + + remark = ( + f"Cove workstation inactive for {streak} day(s) — " + f"no backup activity in colorbar. " + f"Account: {cove_acc.account_name or cove_acc.account_id} | " + f"Computer: {cove_acc.computer_name or '-'} | " + f"Customer: {cove_acc.customer_name or '-'}" + ) + now_utc = datetime.utcnow() + + if existing: + # Do not mutate a run the user has already reviewed. + if existing.reviewed_at is not None: + return False + changed = False + if existing.status != target_status: + existing.status = target_status + changed = True + if existing.remark != remark: + existing.remark = remark + changed = True + if changed: + existing.run_at = now_utc + return changed + + run = JobRun( + job_id=job.id, + mail_message_id=None, + run_at=now_utc, + status=target_status, + remark=remark, + missed=False, + override_applied=False, + source_type="cove_api", + external_id=external_id, + ) + db.session.add(run) + return True + + def _process_account(account: dict) -> str: """Upsert a Cove account into the staging table and create a JobRun if linked. diff --git a/containers/backupchecks/src/backend/app/main/routes_inbox.py b/containers/backupchecks/src/backend/app/main/routes_inbox.py index 9da9d33..00310f9 100644 --- a/containers/backupchecks/src/backend/app/main/routes_inbox.py +++ b/containers/backupchecks/src/backend/app/main/routes_inbox.py @@ -44,6 +44,10 @@ def inbox(): # Use location column if available; otherwise just return all if hasattr(MailMessage, "location"): query = query.filter(MailMessage.location == "inbox") + # Hide messages that belong to archived jobs; keep unlinked messages (job_id IS NULL). + query = query.outerjoin(Job, Job.id == MailMessage.job_id).filter( + (MailMessage.job_id.is_(None)) | (Job.archived.is_(False)) + ) if q: for pat in _patterns(q): query = query.filter( diff --git a/containers/backupchecks/src/backend/app/main/routes_run_checks.py b/containers/backupchecks/src/backend/app/main/routes_run_checks.py index df90961..2f28d3d 100644 --- a/containers/backupchecks/src/backend/app/main/routes_run_checks.py +++ b/containers/backupchecks/src/backend/app/main/routes_run_checks.py @@ -1094,6 +1094,16 @@ def _ensure_missed_runs_for_job(job: Job, start_from: date, end_inclusive: date) if getattr(job, "backup_type", "").lower() in ("cloud connect backup", "cloud connect agent"): return 0 + # Cove workstations are typically PCs that are powered off outside business hours. + # Schedule-based missed-run detection produces noise for "device was off" days, + # so skip entirely. Real Cove statuses (Failed/Warning/Not started) still surface + # via the regular import flow. Servers and Microsoft 365 remain unaffected. + if ( + (getattr(job, "backup_software", "") or "") == "Cove Data Protection" + and (getattr(job, "backup_type", "") or "").lower() == "workstation" + ): + return 0 + tz = _get_ui_timezone() resolved_schedule = _get_effective_schedule_for_job(job) schedule_map = resolved_schedule.get("weekly_map") or {i: [] for i in range(7)} @@ -1439,6 +1449,7 @@ def run_checks_page(): .select_from(Job) .outerjoin(Customer, Customer.id == Job.customer_id) .filter(Job.archived.is_(False)) + .filter(db.or_(Customer.id.is_(None), Customer.active.is_(True))) ) if q: for pat in _patterns(q): @@ -2969,49 +2980,31 @@ def api_run_checks_unmark_reviewed(): return jsonify({"status": "ok", "updated": updated, "skipped": skipped}) -@main_bp.post("/api/run-checks/mark-success-override") -@login_required -@roles_required("admin", "operator") -def api_run_checks_mark_success_override(): - """Create a time-bounded override so the selected run is treated as Success (override).""" - data = request.get_json(silent=True) or {} +def _get_run_error_text(run, obj_rows): + """Extract the most relevant error text from a run for override matching.""" + # First try: error messages from problem objects + for rr in obj_rows or []: + err = (rr.get("error_message") or "").strip() + if err: + return err[:255] + # Second try: run remark + remark = (getattr(run, "remark", None) or "").strip() + if remark: + return remark[:255] + # Last resort: legacy objects try: - run_id = int(data.get("run_id") or 0) + objs = list(run.objects) if hasattr(run, "objects") else [] except Exception: - run_id = 0 + objs = [] + for obj in objs or []: + em = (getattr(obj, "error_message", None) or "").strip() + if em: + return em[:255] + return "" - if run_id <= 0: - return jsonify({"status": "error", "message": "Invalid run_id."}), 400 - run = JobRun.query.get_or_404(run_id) - job = Job.query.get_or_404(run.job_id) - - # Do not allow overriding a missed placeholder run. - if bool(getattr(run, "missed", False)): - return jsonify({"status": "error", "message": "Missed runs cannot be marked as success."}), 400 - - # If it is already a success or already overridden, do nothing. - if bool(getattr(run, "override_applied", False)): - return jsonify({"status": "ok", "message": "Already overridden."}) - - if _status_is_success(getattr(run, "status", None)): - return jsonify({"status": "ok", "message": "Already successful."}) - - # Build a tight validity window around this run. - run_ts = getattr(run, "run_at", None) or getattr(run, "created_at", None) or datetime.utcnow() - start_at = run_ts - timedelta(minutes=1) - end_at = run_ts + timedelta(minutes=1) - - comment = (data.get("comment") or "").strip() - if not comment: - # Keep it short and consistent; Operators will typically include a ticket number separately. - comment = "Marked as success from Run Checks" - comment = comment[:2000] - - created_any = False - - # Prefer object-level overrides (scoped to this job) to avoid impacting other jobs. - obj_rows = [] +def _fetch_problem_objects(run_id): + """Fetch run object rows and filter to problem objects only.""" try: obj_rows = ( db.session.execute( @@ -3027,38 +3020,178 @@ def api_run_checks_mark_success_override(): ORDER BY co.object_name ASC """ ), - {"run_id": run.id}, + {"run_id": run_id}, ) .mappings() .all() ) except Exception: obj_rows = [] + return obj_rows - def _obj_is_problem(status: str | None) -> bool: - s = (status or "").strip().lower() - if not s: - return False - if "success" in s: - return False - if "override" in s: - return False - return True - for rr in obj_rows or []: - obj_name = (rr.get("object_name") or "").strip() - obj_status = (rr.get("status") or "").strip() - if (not obj_name) or (not _obj_is_problem(obj_status)): - continue +def _obj_is_problem(status): + s = (status or "").strip().lower() + if not s: + return False + if "success" in s: + return False + if "override" in s: + return False + return True - err = (rr.get("error_message") or "").strip() + +def _duration_to_end_at(duration, start_at): + """Convert a duration string to an end_at datetime. Returns None for permanent.""" + if duration == "1w": + return start_at + timedelta(weeks=1) + elif duration == "1m": + return start_at + timedelta(days=30) + elif duration == "permanent": + return None + # "once" or default: ±1 minute window (handled by caller) + return None + + +@main_bp.post("/api/run-checks/mark-success-override") +@login_required +@roles_required("admin", "operator") +def api_run_checks_mark_success_override(): + """Create a time-bounded override so the selected run is treated as Success (override). + + Supports two flows: + - scope="run" (default): creates a tight ±1 minute override for this specific run. + Returns error_text and run_info so the frontend can offer a broader override. + - scope="job": creates an object-level override on job_id + error text match. + - scope="global": creates a global override on backup_software + backup_type + error text match. + + Duration options: "once" (±1 min), "1w", "1m", "permanent". + """ + data = request.get_json(silent=True) or {} + try: + run_id = int(data.get("run_id") or 0) + except Exception: + run_id = 0 + + if run_id <= 0: + return jsonify({"status": "error", "message": "Invalid run_id."}), 400 + + run = JobRun.query.get_or_404(run_id) + job = Job.query.get_or_404(run.job_id) + + scope = (data.get("scope") or "run").strip().lower() + if scope not in ("run", "job", "global"): + scope = "run" + duration = (data.get("duration") or "once").strip().lower() + if duration not in ("once", "1w", "1m", "permanent"): + duration = "once" + + # Do not allow overriding a missed placeholder run. + if bool(getattr(run, "missed", False)): + return jsonify({"status": "error", "message": "Missed runs cannot be marked as success."}), 400 + + # For scope=run: if already overridden or successful, do nothing. + # For broader scopes (job/global): skip this check — the run-level override + # was already applied, and this call creates an additional broader override. + if scope == "run": + if bool(getattr(run, "override_applied", False)): + return jsonify({"status": "ok", "message": "Already overridden."}) + if _status_is_success(getattr(run, "status", None)): + return jsonify({"status": "ok", "message": "Already successful."}) + + comment = (data.get("comment") or "").strip() + if not comment: + comment = "Marked as success from Run Checks" + comment = comment[:2000] + + obj_rows = _fetch_problem_objects(run.id) + error_text = (data.get("error_text") or "").strip() + if not error_text: + error_text = _get_run_error_text(run, [rr for rr in obj_rows if _obj_is_problem(rr.get("status"))]) + + now = datetime.utcnow() + run_ts = getattr(run, "run_at", None) or getattr(run, "created_at", None) or now + + if scope == "run": + # Original behaviour: tight ±1 minute window per problem object. + start_at = run_ts - timedelta(minutes=1) + end_at = run_ts + timedelta(minutes=1) + created_any = False + + for rr in obj_rows or []: + obj_name = (rr.get("object_name") or "").strip() + obj_status = (rr.get("status") or "").strip() + if (not obj_name) or (not _obj_is_problem(obj_status)): + continue + err = (rr.get("error_message") or "").strip() + ov = Override( + level="object", + job_id=job.id, + object_name=obj_name, + match_status=(obj_status or None), + match_error_contains=(err[:255] if err else None), + match_error_mode=("contains" if err else None), + treat_as_success=True, + active=True, + comment=comment, + created_by=current_user.username, + start_at=start_at, + end_at=end_at, + ) + db.session.add(ov) + created_any = True + + if not created_any: + match_err = error_text[:255] if error_text else None + ov = Override( + level="global", + backup_software=job.backup_software or None, + backup_type=job.backup_type or None, + match_status=(getattr(run, "status", None) or None), + match_error_contains=match_err, + match_error_mode=("contains" if match_err else None), + treat_as_success=True, + active=True, + comment=comment, + created_by=current_user.username, + start_at=start_at, + end_at=end_at, + ) + db.session.add(ov) + + db.session.commit() + + try: + from .routes_shared import _recompute_override_flags_for_runs + _recompute_override_flags_for_runs(job_ids=[job.id], start_at=start_at, end_at=end_at, only_unreviewed=False) + except Exception: + pass + + # Return run info so the frontend can offer a broader override via the follow-up dialog. + return jsonify({ + "status": "ok", + "message": "Override created.", + "run_info": { + "run_id": run.id, + "job_id": job.id, + "backup_software": job.backup_software or "", + "backup_type": job.backup_type or "", + "job_name": job.job_name or "", + "customer_name": job.customer.name if job.customer else "", + "error_text": error_text, + }, + }) + + elif scope == "job": + # Object-level override scoped to this job, matching on error text. + start_at = now + end_at = _duration_to_end_at(duration, start_at) + match_err = error_text[:255] if error_text else None ov = Override( level="object", job_id=job.id, - object_name=obj_name, - match_status=(obj_status or None), - match_error_contains=(err[:255] if err else None), - match_error_mode=("contains" if err else None), + match_error_contains=match_err, + match_error_mode=("contains" if match_err else None), treat_as_success=True, active=True, comment=comment, @@ -3067,30 +3200,40 @@ def api_run_checks_mark_success_override(): end_at=end_at, ) db.session.add(ov) - created_any = True + db.session.commit() - # If we couldn't build a safe object-scoped override, fall back to a very tight global override. - if not created_any: - match_error_contains = (getattr(run, "remark", None) or "").strip() - if not match_error_contains: - # As a last resort, try to match any error message from legacy objects. - try: - objs = list(run.objects) if hasattr(run, "objects") else [] - except Exception: - objs = [] - for obj in objs or []: - em = (getattr(obj, "error_message", None) or "").strip() - if em: - match_error_contains = em - break + # Audit log for broader override + try: + from .routes_shared import _log_admin_event + duration_label = {"1w": "1 week", "1m": "1 month", "permanent": "permanent"}.get(duration, duration) + _log_admin_event( + "override_from_review", + f"Broader override created from Run Checks review (scope=job, duration={duration_label})", + details=f"run_id={run.id}, job_id={job.id}, job_name={job.job_name or ''}, error_match={match_err or 'none'}, override_id={ov.id}", + ) + except Exception: + pass + # Recompute for this job across the override window. + try: + from .routes_shared import _recompute_override_flags_for_runs + _recompute_override_flags_for_runs(job_ids=[job.id], start_at=start_at, end_at=end_at or (now + timedelta(days=365)), only_unreviewed=False) + except Exception: + pass + + return jsonify({"status": "ok", "message": "Job-level override created."}) + + elif scope == "global": + # Global override matching on backup_software + backup_type + error text. + start_at = now + end_at = _duration_to_end_at(duration, start_at) + match_err = error_text[:255] if error_text else None ov = Override( level="global", backup_software=job.backup_software or None, backup_type=job.backup_type or None, - match_status=(getattr(run, "status", None) or None), - match_error_contains=(match_error_contains[:255] if match_error_contains else None), - match_error_mode=("contains" if match_error_contains else None), + match_error_contains=match_err, + match_error_mode=("contains" if match_err else None), treat_as_success=True, active=True, comment=comment, @@ -3099,16 +3242,25 @@ def api_run_checks_mark_success_override(): end_at=end_at, ) db.session.add(ov) - created_any = True + db.session.commit() - db.session.commit() + # Audit log for broader override + try: + from .routes_shared import _log_admin_event + duration_label = {"1w": "1 week", "1m": "1 month", "permanent": "permanent"}.get(duration, duration) + _log_admin_event( + "override_from_review", + f"Broader override created from Run Checks review (scope=global, duration={duration_label})", + details=f"run_id={run.id}, software={job.backup_software or ''}, type={job.backup_type or ''}, error_match={match_err or 'none'}, override_id={ov.id}", + ) + except Exception: + pass - # Recompute flags so the overview and modal reflect the override immediately. - try: - from .routes_shared import _recompute_override_flags_for_runs + # Recompute broadly — all jobs could be affected by a global override. + try: + from .routes_shared import _recompute_override_flags_for_runs + _recompute_override_flags_for_runs(job_ids=None, start_at=start_at, end_at=end_at or (now + timedelta(days=365)), only_unreviewed=False) + except Exception: + pass - _recompute_override_flags_for_runs(job_ids=[job.id], start_at=start_at, end_at=end_at, only_unreviewed=False) - except Exception: - pass - - return jsonify({"status": "ok", "message": "Override created."}) + return jsonify({"status": "ok", "message": "Global override created."}) diff --git a/containers/backupchecks/src/backend/app/main/routes_search.py b/containers/backupchecks/src/backend/app/main/routes_search.py index be1c52d..39f5aef 100644 --- a/containers/backupchecks/src/backend/app/main/routes_search.py +++ b/containers/backupchecks/src/backend/app/main/routes_search.py @@ -476,6 +476,7 @@ def _build_run_checks_results(patterns: list[str], page: int) -> dict: .join(agg, agg.c.job_id == Job.id) .outerjoin(Customer, Customer.id == Job.customer_id) .filter(Job.archived.is_(False)) + .filter(db.or_(Customer.id.is_(None), Customer.active.is_(True))) ) match_expr = _contains_all_terms( diff --git a/containers/backupchecks/src/backend/app/main/routes_settings.py b/containers/backupchecks/src/backend/app/main/routes_settings.py index 3c4fa73..c219a9d 100644 --- a/containers/backupchecks/src/backend/app/main/routes_settings.py +++ b/containers/backupchecks/src/backend/app/main/routes_settings.py @@ -331,7 +331,7 @@ Backup failed. Please check the logs for details.""", if status_type not in email_sets: flash("Invalid status type.", "danger") - return redirect(url_for("main.settings", section="maintenance")) + return redirect(url_for("main.settings", section="testing")) emails = email_sets[status_type] created_count = 0 @@ -365,7 +365,195 @@ Backup failed. Please check the logs for details.""", print(f"[settings-test] Failed to generate test emails: {exc}") flash("Failed to generate test emails.", "danger") - return redirect(url_for("main.settings", section="maintenance")) + return redirect(url_for("main.settings", section="testing")) + + +# --------------------------------------------------------------------------- +# Test run generator — creates JobRuns with run_object_links for override testing +# --------------------------------------------------------------------------- + +_TEST_JOB_NAME = "Test Backup Job" +_TEST_CUSTOMER_NAME = "000 Test" +_TEST_BACKUP_SOFTWARE = "Veeam" +_TEST_BACKUP_TYPE = "Backup" + +_TEST_ERROR_SCENARIOS = { + "vss": "Cannot create snapshot: VSS error 0x800423f4", + "connection": "Cannot connect to host: connection timed out after 30s", + "diskspace": "Low free space on target datastore DS01 (2.1 GB remaining)", + "license": "License expired: Veeam Backup & Replication license has expired", + "network": "Network transfer timeout: failed to transfer data within 3600s", + "permission": "Access denied: insufficient permissions on target repository", +} + + +def _get_or_create_test_job(): + """Return the test job (and its customer), creating both if needed.""" + customer = Customer.query.filter_by(name=_TEST_CUSTOMER_NAME).first() + if not customer: + customer = Customer(name=_TEST_CUSTOMER_NAME, active=True) + db.session.add(customer) + db.session.flush() + + job = Job.query.filter_by(job_name=_TEST_JOB_NAME, customer_id=customer.id).first() + if not job: + job = Job( + customer_id=customer.id, + backup_software=_TEST_BACKUP_SOFTWARE, + backup_type=_TEST_BACKUP_TYPE, + job_name=_TEST_JOB_NAME, + auto_approve=True, + active=True, + ) + db.session.add(job) + db.session.flush() + + return job, customer + + +def _get_or_create_customer_object(customer_id, object_name): + """Return a customer_object row id, creating it if needed.""" + row = db.session.execute( + text("SELECT id FROM customer_objects WHERE customer_id = :cid AND object_name = :name"), + {"cid": customer_id, "name": object_name}, + ).fetchone() + if row: + return row[0] + db.session.execute( + text("INSERT INTO customer_objects (customer_id, object_name, first_seen_at, last_seen_at) VALUES (:cid, :name, NOW(), NOW())"), + {"cid": customer_id, "name": object_name}, + ) + db.session.flush() + row = db.session.execute( + text("SELECT id FROM customer_objects WHERE customer_id = :cid AND object_name = :name"), + {"cid": customer_id, "name": object_name}, + ).fetchone() + return row[0] + + +@main_bp.route("/settings/test-run/generate", methods=["POST"]) +@login_required +@roles_required("admin") +def settings_generate_test_run(): + """Generate a single test JobRun with 3 objects for override testing.""" + try: + status_type = (request.form.get("status") or "failed").strip().lower() + if status_type not in ("success", "warning", "failed"): + status_type = "failed" + + error_scenario = (request.form.get("error_scenario") or "vss").strip().lower() + custom_error = (request.form.get("custom_error") or "").strip() + + if error_scenario == "custom" and custom_error: + error_message = custom_error[:500] + else: + error_message = _TEST_ERROR_SCENARIOS.get(error_scenario, _TEST_ERROR_SCENARIOS["vss"]) + + job, customer = _get_or_create_test_job() + now = datetime.utcnow() + + # Determine run-level status + run_status_map = {"success": "Success", "warning": "Warning", "failed": "Failed"} + run_status = run_status_map[status_type] + + run = JobRun( + job_id=job.id, + status=run_status, + remark=error_message if status_type != "success" else "All backups completed successfully", + run_at=now, + source_type="test", + ) + db.session.add(run) + db.session.flush() + + # Create 3 objects: VM-APP01, VM-DB01, VM-WEB01 + object_defs = [ + ("VM-APP01", "Success", None), + ("VM-DB01", "Success", None), + ("VM-WEB01", "Success", None), + ] + + if status_type == "warning": + # First object gets the warning + object_defs[0] = ("VM-APP01", "Warning", error_message) + elif status_type == "failed": + # First object gets the failure + object_defs[0] = ("VM-APP01", "Failed", error_message) + + for obj_name, obj_status, obj_error in object_defs: + co_id = _get_or_create_customer_object(customer.id, obj_name) + db.session.execute( + text(""" + INSERT INTO run_object_links (run_id, customer_object_id, status, error_message, observed_at) + VALUES (:run_id, :co_id, :status, :error, :observed_at) + """), + {"run_id": run.id, "co_id": co_id, "status": obj_status, "error": obj_error, "observed_at": now}, + ) + + db.session.commit() + + flash(f"Generated {run_status} test run (id={run.id}) with error: {error_message[:80] if status_type != 'success' else 'none'}", "success") + + _log_admin_event( + event_type="maintenance_generate_test_run", + message=f"Generated {run_status} test run", + details=json.dumps({"run_id": run.id, "status": run_status, "error_scenario": error_scenario}), + ) + + except Exception as exc: + db.session.rollback() + print(f"[settings-test] Failed to generate test run: {exc}") + flash("Failed to generate test run.", "danger") + + return redirect(url_for("main.settings", section="testing")) + + +@main_bp.route("/settings/test-run/cleanup", methods=["POST"]) +@login_required +@roles_required("admin") +def settings_cleanup_test_runs(): + """Delete all test runs and associated objects for the test job.""" + try: + job = Job.query.filter_by(job_name=_TEST_JOB_NAME).first() + if not job: + flash("No test data found.", "info") + return redirect(url_for("main.settings", section="testing")) + + # Count before deleting + run_count = JobRun.query.filter_by(job_id=job.id).count() + + # Delete run_object_links via cascade (job_runs → run_object_links) + # Delete job runs + db.session.execute( + text("DELETE FROM run_object_links WHERE run_id IN (SELECT id FROM job_runs WHERE job_id = :jid)"), + {"jid": job.id}, + ) + db.session.execute( + text("DELETE FROM job_runs WHERE job_id = :jid"), + {"jid": job.id}, + ) + # Delete customer_objects for the test customer + customer = Customer.query.filter_by(name=_TEST_CUSTOMER_NAME).first() + if customer: + db.session.execute( + text("DELETE FROM customer_objects WHERE customer_id = :cid"), + {"cid": customer.id}, + ) + + db.session.commit() + flash(f"Deleted {run_count} test run(s) and associated objects.", "success") + + _log_admin_event( + event_type="maintenance_cleanup_test_runs", + message=f"Cleaned up {run_count} test runs", + ) + + except Exception as exc: + db.session.rollback() + print(f"[settings-test] Failed to cleanup test runs: {exc}") + flash("Failed to cleanup test runs.", "danger") + + return redirect(url_for("main.settings", section="testing")) @main_bp.route("/settings/objects/backfill", methods=["POST"]) @@ -936,6 +1124,30 @@ def settings(): except (ValueError, TypeError): pass + # Cove workstation offline detection (colorbar-based). + # The enable flag rides on the Cove form group, so update it whenever the + # form was submitted from this section. + if cove_form_touched: + settings.cove_offline_detection_enabled = bool( + request.form.get("cove_offline_detection_enabled") + ) + + if "cove_workstation_warning_days" in request.form: + try: + warn_days = int(request.form.get("cove_workstation_warning_days") or 7) + warn_days = max(1, min(warn_days, 28)) + settings.cove_workstation_warning_days = warn_days + except (ValueError, TypeError): + pass + + if "cove_workstation_error_days" in request.form: + try: + err_days = int(request.form.get("cove_workstation_error_days") or 14) + err_days = max(1, min(err_days, 28)) + settings.cove_workstation_error_days = err_days + except (ValueError, TypeError): + pass + # Microsoft Entra SSO if entra_form_touched: settings.entra_sso_enabled = bool(request.form.get("entra_sso_enabled")) diff --git a/containers/backupchecks/src/backend/app/migrations.py b/containers/backupchecks/src/backend/app/migrations.py index cb4ccdf..eca63b7 100644 --- a/containers/backupchecks/src/backend/app/migrations.py +++ b/containers/backupchecks/src/backend/app/migrations.py @@ -1280,6 +1280,37 @@ def migrate_cove_integration() -> None: print(f"[migrations] Failed to migrate Cove integration columns: {exc}") +def migrate_cove_offline_detection() -> None: + """Add Cove workstation offline-detection settings to system_settings. + + Adds: + - cove_offline_detection_enabled (BOOLEAN NOT NULL DEFAULT FALSE) + - cove_workstation_warning_days (INTEGER NOT NULL DEFAULT 7) + - cove_workstation_error_days (INTEGER NOT NULL DEFAULT 14) + """ + try: + engine = db.get_engine() + except Exception as exc: + print(f"[migrations] Could not get engine for Cove offline detection migration: {exc}") + return + + columns = [ + ("cove_offline_detection_enabled", "BOOLEAN NOT NULL DEFAULT FALSE"), + ("cove_workstation_warning_days", "INTEGER NOT NULL DEFAULT 7"), + ("cove_workstation_error_days", "INTEGER NOT NULL DEFAULT 14"), + ] + + try: + with engine.begin() as conn: + for column, ddl in columns: + if _column_exists_on_conn(conn, "system_settings", column): + continue + conn.execute(text(f'ALTER TABLE "system_settings" ADD COLUMN {column} {ddl}')) + print("[migrations] migrate_cove_offline_detection completed.") + except Exception as exc: + print(f"[migrations] Failed to migrate Cove offline detection columns: {exc}") + + def migrate_entra_sso_settings() -> None: """Add Microsoft Entra SSO columns to system_settings if missing.""" try: @@ -1484,6 +1515,7 @@ def run_migrations() -> None: migrate_rename_admin_logs_to_audit_logs() migrate_cove_integration() migrate_cove_accounts_table() + migrate_cove_offline_detection() migrate_cloud_connect_accounts_table() migrate_cc_accounts_repo_unique_key() migrate_cc_remove_synthetic_missed_runs() diff --git a/containers/backupchecks/src/backend/app/models.py b/containers/backupchecks/src/backend/app/models.py index 7ecc651..a425263 100644 --- a/containers/backupchecks/src/backend/app/models.py +++ b/containers/backupchecks/src/backend/app/models.py @@ -136,6 +136,13 @@ class SystemSettings(db.Model): cove_partner_id = db.Column(db.Integer, nullable=True) # stored after successful login cove_last_import_at = db.Column(db.DateTime, nullable=True) + # Cove workstation offline detection (colorbar-based). + # When enabled, Cove workstation jobs receive a synthetic warning/error + # JobRun if their 28-day colorbar shows extended inactivity. + cove_offline_detection_enabled = db.Column(db.Boolean, nullable=False, default=False) + cove_workstation_warning_days = db.Column(db.Integer, nullable=False, default=7) + cove_workstation_error_days = db.Column(db.Integer, nullable=False, default=14) + # Microsoft Entra SSO settings entra_sso_enabled = db.Column(db.Boolean, nullable=False, default=False) entra_tenant_id = db.Column(db.String(128), nullable=True) diff --git a/containers/backupchecks/src/templates/documentation/autotask/setup-configuration.html b/containers/backupchecks/src/templates/documentation/autotask/setup-configuration.html index c50675d..09375a3 100644 --- a/containers/backupchecks/src/templates/documentation/autotask/setup-configuration.html +++ b/containers/backupchecks/src/templates/documentation/autotask/setup-configuration.html @@ -27,7 +27,7 @@
  • In the Autotask section, enable integration.
  • Select environment (production or sandbox).
  • Fill in username, password, and tracking identifier.
    • -
  • Optional: set Autotask Base URL for links in notes/details.
    • +
  • Optional: set Backupchecks Base URL for deep-links from ticket notes/details back into Backupchecks.

    • Step 2: Configure Ticket Defaults

    diff --git a/containers/backupchecks/src/templates/documentation/backup-review/daily-jobs.html b/containers/backupchecks/src/templates/documentation/backup-review/daily-jobs.html index b0231fb..ceaaf29 100644 --- a/containers/backupchecks/src/templates/documentation/backup-review/daily-jobs.html +++ b/containers/backupchecks/src/templates/documentation/backup-review/daily-jobs.html @@ -301,6 +301,25 @@
  • Wait for more runs to be imported, then schedule inference will update automatically
    • +

    Cove Workstations Don't Show "Missed"

    + +
      +
    • + Cove Data Protection workstation jobs are intentionally excluded from + schedule-based missed-run detection. PCs are commonly powered off outside business hours, + which produced false-positive Missed alerts; they no longer appear here for those jobs. +
      • +
    • + Cove Server and Microsoft 365 jobs continue to use the regular + schedule-based missed-run logic. +
      • +
    • + For workstation inactivity, enable + colorbar-based offline detection + in Settings → Integrations → Cove instead. +
      • +
    +

    Job Shows "Missed" but Actually Ran

      diff --git a/containers/backupchecks/src/templates/documentation/backup-review/overrides.html b/containers/backupchecks/src/templates/documentation/backup-review/overrides.html index 01044f4..aeb28f6 100644 --- a/containers/backupchecks/src/templates/documentation/backup-review/overrides.html +++ b/containers/backupchecks/src/templates/documentation/backup-review/overrides.html @@ -159,7 +159,28 @@ When you create an override without a start date, it is applied retroactively to existing unreviewed runs. This means jobs that match the override will immediately show the "Treat as success" status in Daily Jobs, even if they ran before the override was created. -

      Creating an Override

      +

      Creating Overrides Directly From Run Checks

      + +

      The fastest way to create an override is from the run you are reviewing. After clicking + Mark as Success in the Run Checks modal, a follow-up dialog + "Apply override for future runs?" appears with:

      + +
        +
      • Scope: +
          +
        • Only this run — no override is created.
          • +
        • This job, same error message — creates a job-scoped override.
          • +
        • All jobs with same software/type and error — creates a global override.
          • +
        +
        • +
      • Duration: 1 week, 1 month, or permanent (until manually disabled).
        • +
      • The error text from the run's problem objects is pre-filled, ready to be trimmed.
        • +
      + +

      Broader overrides created this way are audit-logged with their scope, duration, and source + run. This path covers most day-to-day use cases without ever opening the Overrides page.

      + +

      Creating an Override (Overrides Page)

      To create a new override:

      diff --git a/containers/backupchecks/src/templates/documentation/backup-review/run-checks-modal.html b/containers/backupchecks/src/templates/documentation/backup-review/run-checks-modal.html index b3815e3..942eca0 100644 --- a/containers/backupchecks/src/templates/documentation/backup-review/run-checks-modal.html +++ b/containers/backupchecks/src/templates/documentation/backup-review/run-checks-modal.html @@ -132,9 +132,34 @@ Backup objects are only shown if the parser extracted them from the email. Not all backup software emails include object-level details. If no objects are shown, the parser didn't detect individual items in the email. +

      Cove / Cloud Connect Summary Panels

      + +

      For runs imported via the Cove API or Veeam Cloud Connect (no source email), the modal hides + the email section and shows a structured summary panel instead:

      + +
        +
      • Cove summary: account name, computer, customer, active datasources, last + session timestamp and status. Per-datasource objects appear in the Backup Objects table.
        • +
      • Cloud Connect summary: tenant, repository, used / quota / free storage and + a link to the source provider report.
        • +
      + +

      See Cove Data Protection + and Veeam Cloud Connect + for the underlying data flow.

      + +
      + 💡 Cove same-day suppression:
      + For Cove jobs, once the first complete success run on a given local day is recorded + (status Success with all object statuses Success), all newer Cove + runs on that same day are hidden from Run Checks — regardless of status. This prevents + duplicate review of the same day's backup activity. +
      +

      Email Content

      -

      The original email body from the backup software is displayed in an embedded iframe:

      +

      For email-imported runs, the original email body from the backup software is displayed in + an embedded iframe:

      • HTML emails are rendered with their original formatting
        • @@ -187,6 +212,36 @@
      • All runs for this job are immediately marked as reviewed and the job disappears from the Run Checks page
        • +

        Mark as Success (with optional override)

        + +

        For warning or failed runs that should be treated as successful, use Mark as Success:

        + +
          +
        1. Open the run details and click Mark as Success.
          2. +
        2. The run is recorded as success-by-override and removed from the unreviewed list.
          3. +
        3. A follow-up dialog asks "Apply override for future runs?" with two choices that + persist beyond this single run: +
            +
          • Only this run (default — no future override is created).
            • +
          • This job, same error message — creates a job-scoped override that + treats future occurrences with the same error text as success.
            • +
          • All jobs with same software/type and error — creates a global override + across the same backup software/type combination.
            • +
          +
          4. +
        4. Choose a duration: 1 week, 1 month, or permanent (until manually disabled).
          5. +
        5. The error text is pre-filled from the run's problem objects so you can review and trim it + before saving.
          6. +
        + +
        + 💡 When to use which scope:
        + Use "Only this run" for one-off events, "This job" for issues specific to a single environment + (e.g. a particular VM that always warns), and "All jobs" for known cosmetic warnings from a + backup product that recur across many customers. Broader overrides are audit-logged with + scope, duration, and source run details. +
        +

        Mark as Reviewed (Bulk)

        For efficiency, especially with successful backups:

        diff --git a/containers/backupchecks/src/templates/documentation/customers-jobs/job-schedules.html b/containers/backupchecks/src/templates/documentation/customers-jobs/job-schedules.html index 8e0ac1f..2b73745 100644 --- a/containers/backupchecks/src/templates/documentation/customers-jobs/job-schedules.html +++ b/containers/backupchecks/src/templates/documentation/customers-jobs/job-schedules.html @@ -140,6 +140,23 @@ New jobs without learned schedules will not appear on Daily Jobs until a pattern is established. +

        Cove Workstations: Missed-Run Detection Disabled

        + +

        + Schedule-based missed-run detection (synthetic Missed rows for past slots + without a real run) is not applied to Cove Data Protection workstation jobs. + Workstation devices are routinely powered off outside business hours, which produced + false-positive Missed alerts. Cove Server and Microsoft 365 jobs continue + to use the regular missed-run logic. +

        + +

        + For workstation inactivity, enable colorbar-based offline detection in + Settings → Integrations → Cove. See + Cove Data Protection + for details. +

        +

        Schedule Accuracy

        Schedule learning is based on pattern recognition and may not be 100% accurate in all cases:

        diff --git a/containers/backupchecks/src/templates/documentation/getting-started/what-is-backupchecks.html b/containers/backupchecks/src/templates/documentation/getting-started/what-is-backupchecks.html index 01de937..69f30c2 100644 --- a/containers/backupchecks/src/templates/documentation/getting-started/what-is-backupchecks.html +++ b/containers/backupchecks/src/templates/documentation/getting-started/what-is-backupchecks.html @@ -8,15 +8,11 @@ verify that backups are running successfully across their customer infrastructure.

        -
        - 📝 Coming Soon: - This page is under construction. Screenshots and additional content will be added in future updates. -
        -

        Key Features

        • Automated Mail Parsing: Import backup reports via email and automatically parse results
          • +
        • API Integrations: Direct API import for Cove Data Protection (N-able) and Veeam Cloud Connect — no email needed
        • Review Workflow: Review all backup jobs daily and mark them as reviewed (goal: clear the Run Checks queue)
        • Customer Management: Organize backups by customer and manage multiple backup jobs per customer
        • Autotask Integration: Manually create tickets in Autotask PSA for failed backups requiring follow-up
          • @@ -24,11 +20,6 @@
        • Role-Based Access: Admin, Operator, Reporter, and Viewer roles
        -
        - 💡 Note: - Screenshots will be added in a future update to illustrate the dashboard and key features. -
        -

        How It Works

        BackupChecks follows a simple workflow:

        @@ -60,40 +51,75 @@

        Supported Backup Software

        -

        BackupChecks supports parsing backup reports from:

        +

        BackupChecks ships with the following parsers and API integrations:

        - - + + - - + + - - - + + + - - + + + + + + + + + + + + - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -112,6 +138,8 @@
      • Backend: Flask (Python) application with PostgreSQL database
      • Frontend: Bootstrap 5 for responsive UI
      • Mail Import: Microsoft Graph API integration for automated email retrieval
        • +
      • Cove Data Protection: JSON-RPC API integration with background polling
        • +
      • Veeam Cloud Connect: Email-based per-tenant import with separate inbox flow
      • Autotask API: Optional integration for manual ticket creation
      • Reporting: Built-in report generation with scheduling
        • diff --git a/containers/backupchecks/src/templates/documentation/integrations/cove-data-protection.html b/containers/backupchecks/src/templates/documentation/integrations/cove-data-protection.html index 518c5f7..b6420e3 100644 --- a/containers/backupchecks/src/templates/documentation/integrations/cove-data-protection.html +++ b/containers/backupchecks/src/templates/documentation/integrations/cove-data-protection.html @@ -43,11 +43,63 @@
        • Cove runs are stored as API runs (source_type = cove_api).
          • -
        • Cove run details are visible in Job Detail and Run Checks with a Cove summary panel.
          • +
        • Cove run details are visible in Job Detail and Run Checks with a Cove summary panel (no mail section is shown for Cove runs).
        • No mail message is required for Cove runs.
        • Historical backfill can create runs from Cove 28-day colorbar data.
          • +
        • + Same-day duplicate suppression: once the first complete success + run for a Cove job/day is recorded (status Success with at least one persisted + object and all object statuses equal to Success), all newer Cove runs on that + same local day are hidden in Run Checks — regardless of whether they are + Success, Warning, or Failed/Error. Sort order in the modal + stays newest → oldest. +
        +

        Workstation Offline Handling

        + +

        + Cove workstation devices are commonly powered off outside business hours, which used to + produce false-positive missed-run alerts. Two mechanisms reduce this noise: +

        + +

        Schedule-based missed runs are skipped (always on)

        +
          +
        • + For jobs with backup_software = "Cove Data Protection" and + backup_type = "Workstation", the missed-run generator is disabled. + No synthetic Missed rows are created when a workstation is off during a + scheduled slot. +
          • +
        • + Cove Server and Microsoft 365 jobs keep the regular schedule-based + missed-run logic. +
          • +
        • + Real Cove statuses (Failed, Warning, Not started) + continue to surface via the normal import flow, so genuine problems still get reported. +
          • +
        + +

        Colorbar-based offline detection (optional toggle)

        +

        + Configure under Settings → Integrations → Cove → Workstation offline detection. +

        +
          +
        • Enable colorbar-based offline detection — off by default.
          • +
        • Warning after N inactive days — default 7 (range 1–28).
          • +
        • Error after N inactive days — default 14 (range 1–28).
          • +
        +

        + When enabled, every Cove import cycle checks each linked workstation's 28-day colorbar + (D09F08) and counts the consecutive trailing days with status 0 + (no backup activity). If the streak crosses the warning or error threshold, a single synthetic + JobRun is upserted for that account so the alert appears in Run Checks. The same + row is reused across cycles, so it escalates Warning → Error in place and is removed + automatically once activity resumes. Runs that have already been reviewed in Run Checks are + never mutated or removed, so acknowledged alerts do not reappear. +

        +

        Troubleshooting

          diff --git a/containers/backupchecks/src/templates/documentation/mail-import/inbox-management.html b/containers/backupchecks/src/templates/documentation/mail-import/inbox-management.html index de3138f..4a0b426 100644 --- a/containers/backupchecks/src/templates/documentation/mail-import/inbox-management.html +++ b/containers/backupchecks/src/templates/documentation/mail-import/inbox-management.html @@ -40,6 +40,25 @@ Once you approve an email, it immediately disappears from the inbox and the job appears in operational views (Jobs, Daily Jobs, Run Checks). The inbox only contains emails awaiting approval or deletion. +
          + 💡 Archived jobs are filtered out:
          + Mail messages whose linked job has been archived are hidden from the inbox. + Messages without a linked job remain visible. To process emails for an archived job, unarchive + the job first in Jobs. +
          + +
          + 💡 Parallel inboxes for API integrations:
          + This Inbox handles email-imported reports. API-based integrations have their own staging + pages with the same approve-and-link flow: +
            +
          • Cove Accounts — Cove Data Protection accounts awaiting link to a job + (see Cove Data Protection).
            • +
          • Cloud Connect Accounts — Veeam Cloud Connect tenants awaiting link + (see Veeam Cloud Connect).
            • +
          +
          +

          Table Columns

        SoftwareSupport LevelSourceType Notes
        Veeam Backup & ReplicationFullEmail notifications with detailed job statusEmail parserDetailed per-object job results
        Acronis Cyber ProtectFullBackup completion reportsVeeam Cloud ConnectEmail parserDaily provider report; per-tenant inbox flow
        Synology Active BackupFullCove Data Protection (N-able)API integrationJSON-RPC API; account-based inbox flow + 28-day colorbar history
        NAKIVO Backup & ReplicationEmail parserJob completion reports
        Synology Active Backup / Hyper BackupEmail parser Task result notifications
        QNAP Hybrid Backup SyncFullEmail parser Job completion emails
        Custom/OtherConfigurableSyncoveryEmail parserProfile result notifications
        BoxAfe (Microsoft 365 backup)Email parserTenant-level result reports
        R-Drive ImageEmail parserWorkstation/server image backup reports
        3CXEmail parser3CX phone system backup notifications
        NTFS AuditingEmail parserFile-system audit reports
        Custom / OtherManual Manual job configuration for non-standard formats
        diff --git a/containers/backupchecks/src/templates/documentation/settings/maintenance.html b/containers/backupchecks/src/templates/documentation/settings/maintenance.html index d7203eb..00301df 100644 --- a/containers/backupchecks/src/templates/documentation/settings/maintenance.html +++ b/containers/backupchecks/src/templates/documentation/settings/maintenance.html @@ -38,11 +38,24 @@
      • Delete orphaned: removes orphaned jobs and related run/email data.
        • +

        Generate Test Run

        + +

        Creates a single JobRun with three persisted run objects, attached to a fixed + test job (__test-override-job__) and customer that are auto-created on first use. + Designed for exercising the Smart Override flow in Run Checks without needing a live backup.

        + +
          +
        • Choose a status: Success, Warning, or Failed.
          • +
        • Choose an error scenario: VSS, connection timeout, disk space, license, + network, permissions, or a custom free-text message.
          • +
        • Delete all test runs removes all generated test data (runs and objects).
          • +
        +

        Generate Test Emails

          -
        • Generates one Veeam test email per selected status: Success, Warning, or Error.
          • -
        • Useful for parser testing and maintenance validation.
          • +
        • Generates one Veeam test email in the inbox per selected status: Success, Warning, or Error.
          • +
        • Useful for parser testing and inbox-flow validation.

        Jobs Maintenance

        diff --git a/containers/backupchecks/src/templates/documentation/users/login-authentication.html b/containers/backupchecks/src/templates/documentation/users/login-authentication.html index a421292..b54dbb8 100644 --- a/containers/backupchecks/src/templates/documentation/users/login-authentication.html +++ b/containers/backupchecks/src/templates/documentation/users/login-authentication.html @@ -7,14 +7,9 @@ Learn how to log in to BackupChecks and manage your authentication session.

        -
        - 📝 Coming Soon:
        - This page is under construction. Screenshots will be added in a future update. -
        -

        Logging In

        -

        BackupChecks uses a traditional username and password authentication system.

        +

        BackupChecks supports local username/password authentication and, optionally, Microsoft Entra (Azure AD) Single Sign-On.

        Login Steps

        @@ -23,15 +18,17 @@
      • You will be automatically redirected to the login page if not authenticated
      • Enter your username in the username field
      • Enter your password in the password field
        • -
      • Complete the captcha by solving the simple math problem (e.g., "3 + 5 = ?")
        • +
      • If the login captcha is enabled, solve the simple math problem (e.g., "3 + 5 = ?")
      • Click the Login button
        • -

        If your credentials are correct and the captcha is solved, you will be redirected to the dashboard.

        +

        If your credentials are correct (and the captcha, if shown, is solved), you will be redirected to the dashboard.

        - 📝 Future Change:
        - The captcha requirement is planned to become optional via a system setting. Since BackupChecks is typically deployed in restricted local environments, the captcha may be disabled to streamline the login process. + 💡 Captcha is configurable:
        + The login captcha is controlled by the login_captcha_enabled setting in + Settings → General. It is enabled by default; administrators can disable it + for restricted internal deployments where the extra step is unnecessary.

        First Login

        @@ -50,17 +47,24 @@ Bookmark the BackupChecks URL for quick access. The system will remember your last active role between sessions. -

        Authentication Method

        +

        Authentication Methods

        -

        BackupChecks uses local database authentication:

        +

        BackupChecks supports the following authentication methods:

          -
        • Username/Password: Credentials are stored securely in the database
          • +
        • Local Username/Password: Credentials are stored hashed in the database
          • +
        • Microsoft Entra SSO (OAuth/OIDC): Optional Single Sign-On via Microsoft Entra (Azure AD), configurable in Settings → Integrations → Entra SSO. See Entra SSO setup.
        • Session-Based: After login, a secure session is created and stored in a cookie
          • -
        • No SSO/OAuth: External authentication providers are not currently supported
        • No Two-Factor Authentication: 2FA is not currently implemented
        +
        + ⚠️ Entra SSO is implemented but not yet validated in production:
        + The integration is wired up end-to-end (login flow, token exchange, optional auto-provisioning, + group filtering) but has not been tested against a live tenant in our deployment. Treat it as a + preview feature until verified. Local username/password remains the recommended login method. +
        +
        💡 Note:
        User accounts are created and managed by administrators via Settings → User Management. diff --git a/containers/backupchecks/src/templates/main/run_checks.html b/containers/backupchecks/src/templates/main/run_checks.html index 8788667..b40177c 100644 --- a/containers/backupchecks/src/templates/main/run_checks.html +++ b/containers/backupchecks/src/templates/main/run_checks.html @@ -440,6 +440,7 @@
        Open job page +
        @@ -447,6 +448,77 @@
        +{# Smart Override follow-up dialog (Phase 1) #} + + +
        +
        Workstation offline detection
        +
        + Cove workstations are often powered off outside business hours, which causes + false-positive missed-run alerts. Schedule-based missed-run detection is + always disabled for Cove workstations. With this option enabled, a separate + check uses the 28-day Cove colorbar to flag a job only when the workstation + has truly been inactive for several days. +
        +
        +
        +
        + + +
        +
        +
        + + +
        Consecutive days without a successful backup before a Warning is raised.
        +
        +
        + + +
        Consecutive days without a successful backup before an Error is raised.
        +
        +
        +
        @@ -750,26 +783,10 @@
        -
        -
        -
        Generate test emails
        -
        -

        Generate Veeam test emails in the inbox for testing parsers and maintenance operations. Each button creates 1 Veeam Backup Job email with the specified status.

        -
        -
        - - -
        - - -
        - - -
        -
        -
        -
        + + +
        Jobs maintenance
        @@ -805,6 +822,74 @@
        +{% endif %} + +{% if section == 'testing' %} + +
        +
        Generate test run
        +
        +

        Generate a single test run with objects for testing the Smart Override flow in Run Checks. Each click creates one run for a fixed test job (__test-override-job__).

        +
        +
        +
        + +
        + + +
        +
        + + +
        +
        + + +
        +
        +
        + + + +
        +
        + +
        +
        + +
        +
        + + +
        +
        + +
        +
        Generate test emails
        +
        +

        Generate Veeam test emails in the inbox for testing parsers and inbox flow. Each button creates 1 Veeam Backup Job email with the specified status.

        +
        +
        + + +
        + + +
        + + +
        +
        +
        + {% endif %} {% if section == 'general' %} @@ -1083,4 +1168,31 @@ {% endif %} + + {% endblock %} diff --git a/docs/technical-notes-codex.md b/docs/TECHNICAL.md similarity index 89% rename from docs/technical-notes-codex.md rename to docs/TECHNICAL.md index d639d1d..e2be107 100644 --- a/docs/technical-notes-codex.md +++ b/docs/TECHNICAL.md @@ -1,6 +1,6 @@ # Technical Notes (Internal) -Last updated: 2026-03-23 +Last updated: 2026-04-16 ## Purpose Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis. @@ -337,9 +337,63 @@ Cove run rows in the job detail history table are clickable even without a mail - Once the first complete success exists on that day, all newer Cove runs for the same day are hidden in Run Checks (overview aggregation + details modal), regardless of status (`Success`, `Warning`, `Failed/Error`). - Sort order in the modal remains unchanged (`newest -> oldest`). +### Workstation Offline Handling (added 2026-05-01) +Two layered behaviours reduce false-positive alerts for PCs that are routinely powered off: + +1. **Schedule-based missed-run skip (always on)** — `_ensure_missed_runs_for_job` in `routes_run_checks.py` early-returns for Cove + Workstation jobs. Servers and Microsoft 365 keep the existing missed-run generation. Real Cove statuses (Failed/Warning/Not started) continue to surface through the normal import path. +2. **Colorbar-based offline detection (toggle in Settings → Integrations → Cove)** — when `cove_offline_detection_enabled = True`, `cove_importer._apply_offline_detection_for_workstations(settings)` runs once per import cycle. For every linked Cove workstation job it parses `cove_acc.colorbar_28d`, counts the trailing streak of `0` codes (no backup that day), and: + - `streak >= cove_workstation_error_days` → upsert synthetic `JobRun` with `status = "Error"`. + - `streak >= cove_workstation_warning_days` (and below error) → `status = "Warning"`. + - Otherwise → drop any existing unreviewed offline run for the job. + + The synthetic run uses a **stable external_id** `cove-offline-{account_id}` so it escalates in place (Warning → Error) and is removed when activity resumes. Reviewed runs (`reviewed_at IS NOT NULL`) are never mutated or deleted, so previously acknowledged alerts do not reappear. + ### Migrations - `migrate_cove_integration()` — adds 8 columns to `system_settings`, `cove_account_id` to `jobs`, `source_type` + `external_id` to `job_runs`, dedup index on `job_runs.external_id` - `migrate_cove_accounts_table()` — creates `cove_accounts` table with indexes +- `migrate_cove_offline_detection()` — adds `cove_offline_detection_enabled`, `cove_workstation_warning_days`, `cove_workstation_error_days` to `system_settings` + +--- + +## Override System + +### Overview +Overrides allow operators to mark specific backup runs (or patterns of runs) as success, suppressing repeated alerts for known non-critical issues. The system operates at two levels: **object-level** (scoped to a specific job) and **global** (scoped to backup software/type). + +### Override Model (`overrides` table) +- `level`: `"global"` or `"object"` +- `backup_software`, `backup_type`: scope for global overrides (nullable) +- `job_id`, `object_name`: scope for object overrides (nullable) +- `match_status`, `match_error_contains`, `match_error_mode`: matching criteria (`contains`/`exact`/`starts_with`/`ends_with`) +- `treat_as_success`, `active`: behavior flags +- `start_at`, `end_at`: validity window (`end_at=NULL` for permanent overrides) +- `comment`, `created_by`, `updated_by`: audit metadata + +### Override Evaluation (`_apply_overrides_to_run` in `routes_shared.py`) +- Called per run to determine display status +- Object-level overrides take precedence over global overrides +- Uses wildcard matching (fnmatch) for `object_name`, configurable text matching for errors +- Returns 5-tuple: `(display_status, override_applied, override_level, override_id, override_reason)` +- `_recompute_override_flags_for_runs()` batch-updates `JobRun.override_applied` flag after override changes + +### Smart Overrides — Phase 1 (2026-04-16) +"Mark as Success" in Run Checks now offers a follow-up dialog to create broader overrides for recurring issues. + +**API** (`POST /api/run-checks/mark-success-override`): +- `scope` parameter: `"run"` (default, ±1 min window), `"job"` (object-level on job_id + error match), `"global"` (software + type + error match) +- `duration` parameter: `"once"` (±1 min), `"1w"`, `"1m"`, `"permanent"` (end_at=NULL) +- `error_text` parameter: pre-filled from problem objects, can be adjusted by operator +- `comment` parameter: optional operator note +- For `scope=run`: returns `run_info` object with error_text and job metadata so frontend can populate the follow-up dialog +- For `scope=job/global`: creates the broader override and audit-logs it with `event_type=override_from_review` + +**Frontend flow** (`run_checks.html`): +1. Operator clicks "Mark as Success" → scope=run override created → follow-up modal appears +2. Modal shows scope choice (run/job/global), duration choice (1w/1m/permanent), error text preview +3. If operator chooses a broader scope → second API call creates the additional override +4. If operator dismisses → page reloads to reflect the initial run-level override + +**No database migrations required** — all fields already exist on the Override model. --- @@ -735,6 +789,20 @@ File: `build-and-push.sh` ## Recent Changes +### 2026-04-16 +- **Smart Overrides Phase 1** (`main/routes_run_checks.py`, `run_checks.html`): + - Extended `mark-success-override` API with `scope` (run/job/global) and `duration` (once/1w/1m/permanent) parameters. + - Added follow-up dialog modal in Run Checks for creating broader overrides after "Mark as Success". + - Broader overrides (job/global) are audit-logged with `event_type=override_from_review`. + - Restored "Mark as Success" button in Run Checks modal footer. + - Extracted helper functions: `_get_run_error_text()`, `_fetch_problem_objects()`, `_obj_is_problem()`, `_duration_to_end_at()`. +- **Test run generator** (`main/routes_settings.py`, `settings.html`): + - `POST /settings/test-run/generate`: creates a single `JobRun` with 3 objects in `run_object_links` for a fixed test job (`__test-override-job__` under `__Test Customer__`). Operator chooses status (success/warning/failed) and error scenario (VSS, connection timeout, disk space, license, network, permissions, or custom text). + - `POST /settings/test-run/cleanup`: deletes all runs and `customer_objects` for the test job. + - New card on Settings → Maintenance with status radio buttons, error scenario dropdown, and cleanup button. +- **Bug fix: inactive customer filter** (`main/routes_run_checks.py`, `main/routes_search.py`): + - Run Checks overview query was missing `Customer.active` filter — jobs for inactive customers were still shown. Added to both the overview aggregation query and the Search page Run Checks section. + ### 2026-04-13 - **Run Checks Cove daily suppression** (`main/routes_run_checks.py`): - Added Cove-specific filtering to suppress repeated same-day runs after the first complete success run. diff --git a/docs/changelog-claude.md b/docs/changelog-develop.md similarity index 92% rename from docs/changelog-claude.md rename to docs/changelog-develop.md index ed41e22..367e3da 100644 --- a/docs/changelog-claude.md +++ b/docs/changelog-develop.md @@ -1,6 +1,61 @@ -# Changelog - Claude Code +# Changelog - Develop -This file documents all changes made to this project via Claude Code. +This file is the long-form append-only development log. It captures every change in detail and is summarised into `docs/changelog.md` at release time. Release markers (`## YYYY-MM-DD — Released as vX.Y.Z`) indicate which entries are already published. + +## 2026-05-01 — Released as v0.3.0 + +## [2026-05-01] + +### Documentation +- In-app documentation refresh after several months without updates: + - **`integrations/cove-data-protection.html`**: added "Workstation Offline Handling" section covering the schedule-missed exclusion (always-on) and the new colorbar-based offline detection toggle with warning/error day thresholds. Same-day duplicate-suppression behaviour now documented. + - **`backup-review/daily-jobs.html`** and **`customers-jobs/job-schedules.html`**: noted that Cove workstation jobs are excluded from schedule-based missed-run detection. + - **`backup-review/run-checks-modal.html`**: documented the Cove and Cloud Connect summary panels shown for API-imported runs (mail section is hidden), Cove same-day suppression, and the Smart Overrides Phase 1 "Apply override for future runs?" follow-up dialog after Mark as Success (scope + duration choices). + - **`backup-review/overrides.html`**: added "Creating Overrides Directly From Run Checks" section. + - **`mail-import/inbox-management.html`**: noted that messages linked to archived jobs are now hidden from the inbox; cross-linked Cove Accounts and Cloud Connect Accounts staging pages. + - **`getting-started/what-is-backupchecks.html`**: removed stale "Coming Soon" callouts; expanded supported-software table to reflect actual parsers (Veeam, Veeam Cloud Connect, Cove API, NAKIVO, Synology, QNAP, Syncovery, BoxAfe, R-Drive, 3CX, NTFS Auditing) and architecture overview now lists Cove + Cloud Connect components. + - **`users/login-authentication.html`**: removed "No SSO/OAuth" claim; added Microsoft Entra SSO as an authentication method with an explicit "implemented but not yet validated in production" caveat. Captcha section corrected to reflect the existing `login_captcha_enabled` setting. + - **`settings/maintenance.html`**: added "Generate Test Run" subsection (test job auto-creation, status + scenario choices, "Delete all test runs" action). + - **`autotask/setup-configuration.html`**: aligned terminology with the Settings page (`Backupchecks Base URL`). + +### Changed +- Cove workstation jobs no longer receive schedule-based "Missed" runs. Workstations are commonly powered off outside business hours; the synthetic missed-run generator (`_ensure_missed_runs_for_job` in `routes_run_checks.py`) now early-returns for jobs where `backup_software == "Cove Data Protection"` and `backup_type == "Workstation"`. Servers and Microsoft 365 Cove jobs remain unaffected. Real Cove statuses (Failed / Warning / Not started) still surface via the normal import flow. + +### Added +- Cove workstation **offline detection** based on the 28-day colorbar (`D09F08`), configurable in Settings → Integrations → Cove: + - **Enable colorbar-based offline detection** (off by default). + - **Warning after N inactive days** (default 7, range 1–28). + - **Error after N inactive days** (default 14, range 1–28). + - When enabled, `cove_importer.run_cove_import` runs `_apply_offline_detection_for_workstations` once per cycle. For each linked Cove workstation job it counts the consecutive trailing colorbar days with status `0` (no backup). When the streak crosses the warning/error threshold, a synthetic `JobRun` is upserted with stable `external_id = "cove-offline-{account_id}"` (so the same row escalates Warning → Error and is removed once activity resumes). Reviewed runs are never mutated, so acknowledged offline alerts do not reappear. +- New `system_settings` columns: `cove_offline_detection_enabled` (bool), `cove_workstation_warning_days` (int), `cove_workstation_error_days` (int) — added by `migrate_cove_offline_detection()`. + +## [2026-04-24] + +### Fixed +- Inbox no longer lists mail messages linked to archived jobs. The inbox query now excludes messages whose `job_id` points to a job with `archived=True` (messages without a linked job remain visible). + +## [2026-04-16] + +### Added +- Smart Overrides Phase 1: "Apply override for future runs?" follow-up dialog after Mark as Success in Run Checks: + - After marking a run as success, a follow-up dialog offers to create a broader override for future occurrences of the same error. + - **Scope options**: "Only this run" (default, existing behavior), "This job, same error message" (object-level override), "All jobs with same software/type and error" (global override). + - **Duration options**: 1 week, 1 month, or permanent (until manually disabled). + - Error text is pre-filled from the run's problem objects and shown in the dialog for operator review. + - Broader overrides (job/global scope) are audit-logged with `event_type=override_from_review` including scope, duration, and source run details. + - No database migrations required — uses existing Override model fields (`level`, `match_error_contains`, `start_at`, `end_at`). +- Restored "Mark as Success" button in Run Checks modal footer (was previously removed from HTML but still referenced in JavaScript). + +### Added +- Settings → Maintenance: new "Generate test run (override testing)" card: + - Generates a single JobRun with 3 objects in `run_object_links` for a fixed test job (`__test-override-job__`). + - Operator chooses status (Success/Warning/Failed) and error scenario (VSS, connection timeout, disk space, license, network, permissions, or custom). + - Test job and customer are auto-created on first use. + - "Delete all test runs" button cleans up all generated test data. + +### Fixed +- Run Checks overview was not filtering out jobs belonging to inactive customers; only the missed-run sweep query had the `Customer.active` filter. Added `Customer.active` filter to the overview aggregation query. +- Search page Run Checks section had the same missing `Customer.active` filter; now consistent with other views. ## [2026-04-13] diff --git a/docs/changelog.md b/docs/changelog.md index 3b10155..c390f8e 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -1,3 +1,33 @@ +## v0.3.0 + +This release bundles all changes made since `v0.2.5`. Highlights: Smart Overrides Phase 1 (create overrides directly from Run Checks), Cove workstation offline handling (no more false-positive Missed alerts for powered-off PCs, plus an optional colorbar-based offline-detection toggle), a test-run generator in Settings → Maintenance, and a full in-app documentation refresh. + +### Added +- **Smart Overrides Phase 1** — after marking a Run Checks run as Success, a follow-up dialog offers to create a broader override for future occurrences: + - Scope: only this run, this job + same error, or all jobs with same software/type + error. + - Duration: 1 week, 1 month, or permanent. + - Error text is pre-filled from the run's problem objects; broader overrides are audit-logged with scope, duration, and source run. + - No database migration required (uses existing `Override` model). +- **Cove workstation offline detection** (Settings → Integrations → Cove) — optional colorbar-based check that flags Cove workstations as Warning / Error after a configurable number of consecutive inactive days. Disabled by default; thresholds default to 7 / 14 days. Synthetic offline runs use a stable `external_id` per account so they escalate in place and clear automatically once activity resumes; reviewed runs are never mutated. + - New `system_settings` columns: `cove_offline_detection_enabled`, `cove_workstation_warning_days`, `cove_workstation_error_days` (`migrate_cove_offline_detection`). +- **Settings → Maintenance: Generate test run** card — creates a single `JobRun` with three persisted objects on a fixed test job (`__test-override-job__`) so operators can exercise the Smart Override flow without a real backup. "Delete all test runs" button cleans up. +- Run Checks "Mark as Success" button restored in the modal footer (the JS hook existed but the button was missing). + +### Changed +- **Cove workstation jobs are excluded from schedule-based missed-run detection.** Workstations are routinely powered off outside business hours, which produced false-positive Missed alerts. Cove Server and Microsoft 365 jobs are unaffected; real Cove statuses (Failed / Warning / Not started) still surface via the normal import flow. +- In-app documentation refreshed across `getting-started`, `users`, `mail-import`, `integrations` (Cove), `settings`, `backup-review`, `customers-jobs`, and `autotask` to match current behaviour. Includes Smart Overrides documentation, Cove/Cloud Connect summary panels in the Run Checks modal, archived-jobs filter on the Inbox, captcha & Entra SSO authentication options (SSO marked as implemented but not yet validated in production), expanded supported-software list, and Cove offline-handling sections. + +### Fixed +- **Run Checks Cove same-day suppression** — once the first complete success run for a Cove job/day is recorded (status Success with all object statuses Success), all newer Cove runs on that same local day are hidden from Run Checks (overview aggregation + modal details), regardless of status. Sort order in the modal stays newest → oldest. +- **Inbox excludes mail messages from archived jobs** — messages whose `job_id` points to an archived job are filtered out (messages without a linked job remain visible). +- **Run Checks / Search overview filters out inactive customers** — both queries now apply the `Customer.active` filter that was previously only on the missed-run sweep. + +### Build / Tooling +- Adopted the shared `docker-build-and-push` script from `/docker/develop/shared-integrations/tooling/`. The new contract: + - Modes are `t` (test → push `:dev`) and `r` (release → push `:`, `:dev`, `:latest`); bump types `1`/`2`/`3` are no longer used. + - Release version is read from the first `## vX.Y.Z` heading in this file; the script no longer maintains `version.txt`. + - The script performs no git operations — commit, tag, and push are run manually after the registry has accepted the images. + ## v0.2.5 This release bundles all changes made since `v0.2.4`, including schedule management improvements, Autotask/remark synchronization, Run Checks stability updates, and a full documentation refresh. diff --git a/version.txt b/version.txt deleted file mode 100644 index b88fb90..0000000 --- a/version.txt +++ /dev/null @@ -1 +0,0 @@ -v0.2.5