Update technical notes to v0.2.1

- Add schedule inference / missed run detection section (90d/180d windows, cadence guard, MIN_OCCURRENCES)
- Add jobs export/import schema v2 section (Cove + Cloud Connect account links)
- Add inbox batch re-parse section (POST /inbox/reparse-batch, progress modal)
- Add CloudConnectAccount to data model, Job model CC link note
- Add inject_inbox_count context processor to architecture section
- Add Layout v2 UI notes
- Extend Quick References with Cloud Connect, Inbox, Settings routes
- Add Recent Changes entries for v0.2.0 and v0.2.1
- Update last-updated date to 2026-03-20

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Ivo Oskamp 2026-03-20 16:32:07 +01:00
parent 15347e0715
commit 46ef515d4f

View File

@ -1,6 +1,6 @@
# Technical Notes (Internal) # Technical Notes (Internal)
Last updated: 2026-02-23 (late) Last updated: 2026-03-20
## Purpose ## Purpose
Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis. Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis.
@ -33,6 +33,8 @@ Internal technical snapshot of the `backupchecks` repository for faster onboardi
- Background tasks: - Background tasks:
- `start_auto_importer(app)` starts the automatic mail importer thread. - `start_auto_importer(app)` starts the automatic mail importer thread.
- `start_cove_importer(app)` starts the Cove Data Protection polling thread (started only when `cove_import_enabled` is set). - `start_cove_importer(app)` starts the Cove Data Protection polling thread (started only when `cove_import_enabled` is set).
- Global template context:
- `inject_inbox_count()` context processor injects `inbox_count` into every template for authenticated users (sidebar badge).
- Health endpoint: - Health endpoint:
- `GET /health` returns `{ "status": "ok" }`. - `GET /health` returns `{ "status": "ok" }`.
@ -80,6 +82,7 @@ File: `containers/backupchecks/src/backend/app/models.py`
- `Customer`, `Job`, `JobRun`, `Override` - `Customer`, `Job`, `JobRun`, `Override`
- `MailMessage`, `MailObject` - `MailMessage`, `MailObject`
- `CoveAccount` (Cove staging table — see Cove integration section) - `CoveAccount` (Cove staging table — see Cove integration section)
- `CloudConnectAccount` (Cloud Connect staging table — see Cloud Connect integration section)
- `Ticket`, `TicketScope`, `TicketJobRun` - `Ticket`, `TicketScope`, `TicketJobRun`
- `Remark`, `RemarkScope`, `RemarkJobRun` - `Remark`, `RemarkScope`, `RemarkJobRun`
- `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment` - `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`
@ -111,6 +114,7 @@ Note: always use direct SQL (`DELETE FROM`) for bulk deletions — ORM-level del
- `backup_software` - e.g., "Veeam", "Synology", "Cove Data Protection" - `backup_software` - e.g., "Veeam", "Synology", "Cove Data Protection"
- `backup_type` - e.g., "Backup Job", "Active Backup" - `backup_type` - e.g., "Backup Job", "Active Backup"
- `cove_account_id` - (nullable int) links this job to a Cove AccountId - `cove_account_id` - (nullable int) links this job to a Cove AccountId
- Cloud Connect accounts link back via `CloudConnectAccount.job_id` (no FK column on `jobs` — the link is on the staging table side)
**JobRun model:** **JobRun model:**
- `source_type` - NULL = email (backwards compat), `"cove_api"` for Cove-imported runs - `source_type` - NULL = email (backwards compat), `"cove_api"` for Cove-imported runs
@ -152,6 +156,31 @@ Note: always use direct SQL (`DELETE FROM`) for bulk deletions — ORM-level del
- Hostname extraction from multiple patterns - Hostname extraction from multiple patterns
- Returns: backup_type "Updates", job_name "Synology Automatic Update" - Returns: backup_type "Updates", job_name "Synology Automatic Update"
## Schedule Inference and Missed Run Detection
### Overview
File: `containers/backupchecks/src/backend/app/main/routes_shared.py`
Missed runs are detected via `_ensure_missed_runs_for_job()` which is called from Run Checks on page load (throttled: max once per 10 minutes per job via in-memory dict). It infers the expected schedule from recent run history and creates `JobRun` records with `missed=True` for any slots that are overdue.
### Weekly Schedule Inference (`_infer_schedule_map_from_runs`)
- **Window**: last **90 days** only (older runs are excluded to handle schedule changes)
- **MIN_OCCURRENCES**: **5** hits on a weekday+time slot to count as expected (raised from 3 to reduce false positives during transitional periods)
- **Cadence guard**: if median gap between runs ≥ 20 days, weekly inference is **skipped** entirely → monthly inference handles the job instead. Prevents monthly jobs from accumulating enough weekly hits after long operation.
- **Key rule**: time-of-day changes or frequency changes stop generating missed runs on old slots within 90 days (no more stale slot false positives)
### Monthly Schedule Inference (`_infer_monthly_schedule_from_runs`)
- **Window**: last **180 days** (enough for ≥ 3 monthly occurrences, but forgotten within 6 months after a schedule change)
- Infers day-of-month + time-of-day from historical runs
- Used when weekly cadence guard fires (median gap ≥ 20 days)
### Important Rules
- **Never** extend the window without considering stale slot false positives
- Schedule changes (time, frequency) take effect in missed run detection within the window period (90d weekly, 180d monthly)
- Informational parsers (`3CX / Update`, `3CX / SSL Certificate`) are excluded from all schedule inference
---
## Cove Data Protection Integration ## Cove Data Protection Integration
### Overview ### Overview
@ -502,7 +531,14 @@ Visible on Logging page under `event_type = "ticket_link_debug"`. Remove after d
## UI and UX Notes ## UI and UX Notes
### Navbar ### Layout v2 (2026-03-20)
- Complete sidebar-first redesign replacing the top navbar layout:
- `layout.css` rewritten with IBM Plex Sans/Mono fonts and CSS custom properties (design tokens)
- Fixed dark sidebar (220 px wide)
- `base.html` updated with Google Fonts preload and sidebar-aware structure
- Sandbox banner: semi-transparent (`rgba(220,53,69,0.45)`) instead of solid red
### Navbar (pre-v0.2.0 reference — replaced by sidebar in v0.2.0)
- Fixed-top positioning - Fixed-top positioning
- Collapses on mobile (hamburger menu) - Collapses on mobile (hamburger menu)
- Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top) - Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top)
@ -572,6 +608,58 @@ Visible on Logging page under `event_type = "ticket_link_debug"`. Remove after d
- Per-section limit (`SEARCH_LIMIT_PER_SECTION = 10`), with total count per section. - Per-section limit (`SEARCH_LIMIT_PER_SECTION = 10`), with total count per section.
- No schema migration required for V1. - No schema migration required for V1.
## Jobs Export / Import
### Schema Versions
- **v1** (`approved_jobs_export_v1`): job fields only, no account links
- **v2** (`approved_jobs_export_v2`): same as v1 plus per-job `cove_account` and `cloud_connect_account` objects
### Export (v2)
Each job entry contains:
```json
{
"cove_account": {"account_id": 1234, "account_name": "...", "computer_name": "..."},
"cloud_connect_account": {"user": "...", "section": "Backup", "repo_name": "..."}
}
```
`null` when not linked. File: `routes_settings.py``export_jobs()`.
### Import (v1 + v2)
- Accepts both schema versions (detected via `export_type` field)
- For v2 files: after creating/updating the job, the importer:
1. Looks up `CoveAccount` by `account_id` (fallback: `account_name` + `computer_name`)
2. Looks up `CloudConnectAccount` by `user` + `section` + `repo_name`
3. Links the account to the job — only if the account is not yet linked to a different job
- File: `routes_settings.py``import_jobs()`
## Inbox Batch Re-parse
### Endpoint
`POST /inbox/reparse-batch` — JSON in/out, login required, admin/operator only.
**Request body:**
```json
{"last_id": <int|null>, "total": <int|null>}
```
- `last_id`: keyset cursor from previous batch (process messages with `id < last_id`)
- `total`: total count from first call (avoids re-counting on every batch)
**Response:**
```json
{"processed": 50, "total": 847, "parsed_ok": 42, "auto_approved": 12, "no_match": 8, "errors": 0, "last_id": 1234, "done": false}
```
### Batch Parameters
- `batch_size`: 50 messages per call
- `time_budget_s`: 8 seconds per call (stops processing mid-batch if exceeded)
- Auto-approve logic is identical to `inbox_reparse_all` (including VSPC multi-company handling)
### Frontend (inbox.html)
- "Re-parse all" button opens a Bootstrap modal (`data-bs-backdrop="static"` — cannot close while running)
- JS loop: `fetch → process response → setTimeout(100ms) → repeat` until `done: true`
- Progress bar + live counters update after each batch
- Close button appears only when `done: true` or on error
## Feedback Module with Screenshots ## Feedback Module with Screenshots
- Models: `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`. - Models: `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`.
- Attachments: - Attachments:
@ -633,11 +721,46 @@ File: `build-and-push.sh`
- Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py` - Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py`
- Cove importer: `containers/backupchecks/src/backend/app/cove_importer.py` - Cove importer: `containers/backupchecks/src/backend/app/cove_importer.py`
- Cove routes: `containers/backupchecks/src/backend/app/main/routes_cove.py` - Cove routes: `containers/backupchecks/src/backend/app/main/routes_cove.py`
- Cloud Connect importer: `containers/backupchecks/src/backend/app/cloud_connect_importer.py`
- Cloud Connect routes: `containers/backupchecks/src/backend/app/main/routes_cloud_connect.py`
- Inbox routes: `containers/backupchecks/src/backend/app/main/routes_inbox.py`
- Settings routes: `containers/backupchecks/src/backend/app/main/routes_settings.py`
- Compose stack: `deploy/backupchecks-stack.yml` - Compose stack: `deploy/backupchecks-stack.yml`
- Build script: `build-and-push.sh` - Build script: `build-and-push.sh`
## Recent Changes ## Recent Changes
### 2026-03-20 (v0.2.1)
- **Missed run false positive fix** (`routes_shared.py`):
- Weekly inference window: last 90 days only (was unbounded). Eliminates stale slot false positives after time-of-day or frequency changes.
- Cadence guard: if median gap between runs ≥ 20 days, skip weekly inference and let monthly inference handle the job. Fixes monthly jobs accumulating enough weekly hits after ~21 months.
- Monthly inference window: last 180 days (was unbounded).
- `MIN_OCCURRENCES` raised from 3 → 5 for weekly inference.
- **Objects sort fix** (`run_checks.html`, `job_detail.html`):
- `objectSeverityRank`: `|| err` on rank-0 check caused Warning items with `error_message` to rank as Critical. Fixed: only `error`/`failed`/`failure` status → rank 0; `|| err` moved to rank 1.
- **Mail iframe height fix** (`run_checks.html`):
- Flex rules were on `#rcm_body_iframe` but the iframe is not a direct flex child of `.rcm-mail-panel`. Fixed by moving `flex: 1 1 auto; min-height: 0` to the wrapper `#rcm_mail_iframe_body` and setting `height: 100%` on the iframe itself.
- **Inbox sidebar badge on all pages** (`__init__.py`):
- Added `inject_inbox_count()` Flask context processor — injects `inbox_count` into every template for authenticated users. Previously only injected in the dashboard route.
- **Jobs export/import schema v2** (`routes_settings.py`):
- Export: includes `cove_account` and `cloud_connect_account` per job.
- Import: accepts v1 and v2; links Cove/CC accounts on import if not yet linked to a different job.
- **Inbox re-parse progress modal** (`routes_inbox.py`, `inbox.html`):
- New `POST /inbox/reparse-batch` endpoint: 50 messages per call, 8 s time budget, keyset pagination, full auto-approve logic (including VSPC multi-company). Returns JSON progress.
- "Re-parse all" button replaced with modal trigger; JS loop calls batch endpoint until `done: true` and updates live progress bar + stats.
### 2026-03-20 (v0.2.0)
- **Layout v2**: complete sidebar-first redesign (`layout.css`, `base.html`). IBM Plex Sans/Mono fonts, CSS design tokens, fixed 220 px dark sidebar.
- **Veeam Cloud Connect importer**: HTML parser for daily report emails → `cloud_connect_accounts` staging table → `JobRun` records for linked accounts. `CloudConnectAccount` model + migrations. `/cloud-connect/accounts` review page. Sidebar link for admin/operator.
- **Cove historical run backfill**: `_backfill_colorbar_runs()` reconstructs up to 27 days of history from the `D09F08` colorbar on first run creation. Idempotent via `external_id = "cove-colorbar-{account_id}-{date}"`.
- **Cove run details popup**: Cove runs in job detail are clickable; popup fetches `/cove/run/<run_id>/detail` (structured Cove summary, per-datasource objects, mail section hidden).
- **Run Checks user preferences**: per-user sort mode + filter defaults stored in DB; `POST /run-checks/preferences`; User Settings page section.
- **Login captcha toggle**: Settings → General → Security card; `login_captcha_enabled` column with `DEFAULT TRUE`.
- **Cloud Connect unique key**: changed from `(user, section)` to `(user, section, repo_name)` — supports multiple repos per user.
- **Cloud Connect run detail popup**: shows structured CC summary instead of raw email; raw email accessible via toggle.
- **Entra SSO**: implemented (marked untested). Login/callback/logout flow, optional auto-provisioning, tenant/domain + security-group restrictions.
- **Fixes**: login page layout with flash messages; "Delete all jobs" timeout (replaced ORM with direct SQL); archived job auto-matching; Cove link sync between `cove_accounts.job_id``jobs.cove_account_id`; Cove run creation transaction scope.
### 2026-02-23 ### 2026-02-23
- **Cove Data Protection full integration**: - **Cove Data Protection full integration**:
- `cove_importer.py` Cove API client (login, paginated enumeration, status mapping, deduplication, per-datasource object persistence) - `cove_importer.py` Cove API client (login, paginated enumeration, status mapping, deduplication, per-datasource object persistence)