diff --git a/docs/technical-notes-codex.md b/docs/technical-notes-codex.md index e01a6b8..3842e10 100644 --- a/docs/technical-notes-codex.md +++ b/docs/technical-notes-codex.md @@ -1,6 +1,6 @@ # Technical Notes (Internal) -Last updated: 2026-02-23 (late) +Last updated: 2026-03-20 ## Purpose Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis. @@ -33,6 +33,8 @@ Internal technical snapshot of the `backupchecks` repository for faster onboardi - Background tasks: - `start_auto_importer(app)` starts the automatic mail importer thread. - `start_cove_importer(app)` starts the Cove Data Protection polling thread (started only when `cove_import_enabled` is set). +- Global template context: + - `inject_inbox_count()` context processor injects `inbox_count` into every template for authenticated users (sidebar badge). - Health endpoint: - `GET /health` returns `{ "status": "ok" }`. @@ -80,6 +82,7 @@ File: `containers/backupchecks/src/backend/app/models.py` - `Customer`, `Job`, `JobRun`, `Override` - `MailMessage`, `MailObject` - `CoveAccount` (Cove staging table — see Cove integration section) + - `CloudConnectAccount` (Cloud Connect staging table — see Cloud Connect integration section) - `Ticket`, `TicketScope`, `TicketJobRun` - `Remark`, `RemarkScope`, `RemarkJobRun` - `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment` @@ -111,6 +114,7 @@ Note: always use direct SQL (`DELETE FROM`) for bulk deletions — ORM-level del - `backup_software` - e.g., "Veeam", "Synology", "Cove Data Protection" - `backup_type` - e.g., "Backup Job", "Active Backup" - `cove_account_id` - (nullable int) links this job to a Cove AccountId +- Cloud Connect accounts link back via `CloudConnectAccount.job_id` (no FK column on `jobs` — the link is on the staging table side) **JobRun model:** - `source_type` - NULL = email (backwards compat), `"cove_api"` for Cove-imported runs @@ -152,6 +156,31 @@ Note: always use direct SQL (`DELETE FROM`) for bulk deletions — ORM-level del - Hostname extraction from multiple patterns - Returns: backup_type "Updates", job_name "Synology Automatic Update" +## Schedule Inference and Missed Run Detection + +### Overview +File: `containers/backupchecks/src/backend/app/main/routes_shared.py` + +Missed runs are detected via `_ensure_missed_runs_for_job()` which is called from Run Checks on page load (throttled: max once per 10 minutes per job via in-memory dict). It infers the expected schedule from recent run history and creates `JobRun` records with `missed=True` for any slots that are overdue. + +### Weekly Schedule Inference (`_infer_schedule_map_from_runs`) +- **Window**: last **90 days** only (older runs are excluded to handle schedule changes) +- **MIN_OCCURRENCES**: **5** hits on a weekday+time slot to count as expected (raised from 3 to reduce false positives during transitional periods) +- **Cadence guard**: if median gap between runs ≥ 20 days, weekly inference is **skipped** entirely → monthly inference handles the job instead. Prevents monthly jobs from accumulating enough weekly hits after long operation. +- **Key rule**: time-of-day changes or frequency changes stop generating missed runs on old slots within 90 days (no more stale slot false positives) + +### Monthly Schedule Inference (`_infer_monthly_schedule_from_runs`) +- **Window**: last **180 days** (enough for ≥ 3 monthly occurrences, but forgotten within 6 months after a schedule change) +- Infers day-of-month + time-of-day from historical runs +- Used when weekly cadence guard fires (median gap ≥ 20 days) + +### Important Rules +- **Never** extend the window without considering stale slot false positives +- Schedule changes (time, frequency) take effect in missed run detection within the window period (90d weekly, 180d monthly) +- Informational parsers (`3CX / Update`, `3CX / SSL Certificate`) are excluded from all schedule inference + +--- + ## Cove Data Protection Integration ### Overview @@ -502,7 +531,14 @@ Visible on Logging page under `event_type = "ticket_link_debug"`. Remove after d ## UI and UX Notes -### Navbar +### Layout v2 (2026-03-20) +- Complete sidebar-first redesign replacing the top navbar layout: + - `layout.css` rewritten with IBM Plex Sans/Mono fonts and CSS custom properties (design tokens) + - Fixed dark sidebar (220 px wide) + - `base.html` updated with Google Fonts preload and sidebar-aware structure +- Sandbox banner: semi-transparent (`rgba(220,53,69,0.45)`) instead of solid red + +### Navbar (pre-v0.2.0 reference — replaced by sidebar in v0.2.0) - Fixed-top positioning - Collapses on mobile (hamburger menu) - Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top) @@ -572,6 +608,58 @@ Visible on Logging page under `event_type = "ticket_link_debug"`. Remove after d - Per-section limit (`SEARCH_LIMIT_PER_SECTION = 10`), with total count per section. - No schema migration required for V1. +## Jobs Export / Import + +### Schema Versions +- **v1** (`approved_jobs_export_v1`): job fields only, no account links +- **v2** (`approved_jobs_export_v2`): same as v1 plus per-job `cove_account` and `cloud_connect_account` objects + +### Export (v2) +Each job entry contains: +```json +{ + "cove_account": {"account_id": 1234, "account_name": "...", "computer_name": "..."}, + "cloud_connect_account": {"user": "...", "section": "Backup", "repo_name": "..."} +} +``` +`null` when not linked. File: `routes_settings.py` → `export_jobs()`. + +### Import (v1 + v2) +- Accepts both schema versions (detected via `export_type` field) +- For v2 files: after creating/updating the job, the importer: + 1. Looks up `CoveAccount` by `account_id` (fallback: `account_name` + `computer_name`) + 2. Looks up `CloudConnectAccount` by `user` + `section` + `repo_name` + 3. Links the account to the job — only if the account is not yet linked to a different job +- File: `routes_settings.py` → `import_jobs()` + +## Inbox Batch Re-parse + +### Endpoint +`POST /inbox/reparse-batch` — JSON in/out, login required, admin/operator only. + +**Request body:** +```json +{"last_id": , "total": } +``` +- `last_id`: keyset cursor from previous batch (process messages with `id < last_id`) +- `total`: total count from first call (avoids re-counting on every batch) + +**Response:** +```json +{"processed": 50, "total": 847, "parsed_ok": 42, "auto_approved": 12, "no_match": 8, "errors": 0, "last_id": 1234, "done": false} +``` + +### Batch Parameters +- `batch_size`: 50 messages per call +- `time_budget_s`: 8 seconds per call (stops processing mid-batch if exceeded) +- Auto-approve logic is identical to `inbox_reparse_all` (including VSPC multi-company handling) + +### Frontend (inbox.html) +- "Re-parse all" button opens a Bootstrap modal (`data-bs-backdrop="static"` — cannot close while running) +- JS loop: `fetch → process response → setTimeout(100ms) → repeat` until `done: true` +- Progress bar + live counters update after each batch +- Close button appears only when `done: true` or on error + ## Feedback Module with Screenshots - Models: `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`. - Attachments: @@ -633,11 +721,46 @@ File: `build-and-push.sh` - Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py` - Cove importer: `containers/backupchecks/src/backend/app/cove_importer.py` - Cove routes: `containers/backupchecks/src/backend/app/main/routes_cove.py` +- Cloud Connect importer: `containers/backupchecks/src/backend/app/cloud_connect_importer.py` +- Cloud Connect routes: `containers/backupchecks/src/backend/app/main/routes_cloud_connect.py` +- Inbox routes: `containers/backupchecks/src/backend/app/main/routes_inbox.py` +- Settings routes: `containers/backupchecks/src/backend/app/main/routes_settings.py` - Compose stack: `deploy/backupchecks-stack.yml` - Build script: `build-and-push.sh` ## Recent Changes +### 2026-03-20 (v0.2.1) +- **Missed run false positive fix** (`routes_shared.py`): + - Weekly inference window: last 90 days only (was unbounded). Eliminates stale slot false positives after time-of-day or frequency changes. + - Cadence guard: if median gap between runs ≥ 20 days, skip weekly inference and let monthly inference handle the job. Fixes monthly jobs accumulating enough weekly hits after ~21 months. + - Monthly inference window: last 180 days (was unbounded). + - `MIN_OCCURRENCES` raised from 3 → 5 for weekly inference. +- **Objects sort fix** (`run_checks.html`, `job_detail.html`): + - `objectSeverityRank`: `|| err` on rank-0 check caused Warning items with `error_message` to rank as Critical. Fixed: only `error`/`failed`/`failure` status → rank 0; `|| err` moved to rank 1. +- **Mail iframe height fix** (`run_checks.html`): + - Flex rules were on `#rcm_body_iframe` but the iframe is not a direct flex child of `.rcm-mail-panel`. Fixed by moving `flex: 1 1 auto; min-height: 0` to the wrapper `#rcm_mail_iframe_body` and setting `height: 100%` on the iframe itself. +- **Inbox sidebar badge on all pages** (`__init__.py`): + - Added `inject_inbox_count()` Flask context processor — injects `inbox_count` into every template for authenticated users. Previously only injected in the dashboard route. +- **Jobs export/import schema v2** (`routes_settings.py`): + - Export: includes `cove_account` and `cloud_connect_account` per job. + - Import: accepts v1 and v2; links Cove/CC accounts on import if not yet linked to a different job. +- **Inbox re-parse progress modal** (`routes_inbox.py`, `inbox.html`): + - New `POST /inbox/reparse-batch` endpoint: 50 messages per call, 8 s time budget, keyset pagination, full auto-approve logic (including VSPC multi-company). Returns JSON progress. + - "Re-parse all" button replaced with modal trigger; JS loop calls batch endpoint until `done: true` and updates live progress bar + stats. + +### 2026-03-20 (v0.2.0) +- **Layout v2**: complete sidebar-first redesign (`layout.css`, `base.html`). IBM Plex Sans/Mono fonts, CSS design tokens, fixed 220 px dark sidebar. +- **Veeam Cloud Connect importer**: HTML parser for daily report emails → `cloud_connect_accounts` staging table → `JobRun` records for linked accounts. `CloudConnectAccount` model + migrations. `/cloud-connect/accounts` review page. Sidebar link for admin/operator. +- **Cove historical run backfill**: `_backfill_colorbar_runs()` reconstructs up to 27 days of history from the `D09F08` colorbar on first run creation. Idempotent via `external_id = "cove-colorbar-{account_id}-{date}"`. +- **Cove run details popup**: Cove runs in job detail are clickable; popup fetches `/cove/run//detail` (structured Cove summary, per-datasource objects, mail section hidden). +- **Run Checks user preferences**: per-user sort mode + filter defaults stored in DB; `POST /run-checks/preferences`; User Settings page section. +- **Login captcha toggle**: Settings → General → Security card; `login_captcha_enabled` column with `DEFAULT TRUE`. +- **Cloud Connect unique key**: changed from `(user, section)` to `(user, section, repo_name)` — supports multiple repos per user. +- **Cloud Connect run detail popup**: shows structured CC summary instead of raw email; raw email accessible via toggle. +- **Entra SSO**: implemented (marked untested). Login/callback/logout flow, optional auto-provisioning, tenant/domain + security-group restrictions. +- **Fixes**: login page layout with flash messages; "Delete all jobs" timeout (replaced ORM with direct SQL); archived job auto-matching; Cove link sync between `cove_accounts.job_id` ↔ `jobs.cove_account_id`; Cove run creation transaction scope. + ### 2026-02-23 - **Cove Data Protection full integration**: - `cove_importer.py` – Cove API client (login, paginated enumeration, status mapping, deduplication, per-datasource object persistence)