Update technical notes to v0.2.1
- Add schedule inference / missed run detection section (90d/180d windows, cadence guard, MIN_OCCURRENCES) - Add jobs export/import schema v2 section (Cove + Cloud Connect account links) - Add inbox batch re-parse section (POST /inbox/reparse-batch, progress modal) - Add CloudConnectAccount to data model, Job model CC link note - Add inject_inbox_count context processor to architecture section - Add Layout v2 UI notes - Extend Quick References with Cloud Connect, Inbox, Settings routes - Add Recent Changes entries for v0.2.0 and v0.2.1 - Update last-updated date to 2026-03-20 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
15347e0715
commit
46ef515d4f
@ -1,6 +1,6 @@
|
|||||||
# Technical Notes (Internal)
|
# Technical Notes (Internal)
|
||||||
|
|
||||||
Last updated: 2026-02-23 (late)
|
Last updated: 2026-03-20
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis.
|
Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis.
|
||||||
@ -33,6 +33,8 @@ Internal technical snapshot of the `backupchecks` repository for faster onboardi
|
|||||||
- Background tasks:
|
- Background tasks:
|
||||||
- `start_auto_importer(app)` starts the automatic mail importer thread.
|
- `start_auto_importer(app)` starts the automatic mail importer thread.
|
||||||
- `start_cove_importer(app)` starts the Cove Data Protection polling thread (started only when `cove_import_enabled` is set).
|
- `start_cove_importer(app)` starts the Cove Data Protection polling thread (started only when `cove_import_enabled` is set).
|
||||||
|
- Global template context:
|
||||||
|
- `inject_inbox_count()` context processor injects `inbox_count` into every template for authenticated users (sidebar badge).
|
||||||
- Health endpoint:
|
- Health endpoint:
|
||||||
- `GET /health` returns `{ "status": "ok" }`.
|
- `GET /health` returns `{ "status": "ok" }`.
|
||||||
|
|
||||||
@ -80,6 +82,7 @@ File: `containers/backupchecks/src/backend/app/models.py`
|
|||||||
- `Customer`, `Job`, `JobRun`, `Override`
|
- `Customer`, `Job`, `JobRun`, `Override`
|
||||||
- `MailMessage`, `MailObject`
|
- `MailMessage`, `MailObject`
|
||||||
- `CoveAccount` (Cove staging table — see Cove integration section)
|
- `CoveAccount` (Cove staging table — see Cove integration section)
|
||||||
|
- `CloudConnectAccount` (Cloud Connect staging table — see Cloud Connect integration section)
|
||||||
- `Ticket`, `TicketScope`, `TicketJobRun`
|
- `Ticket`, `TicketScope`, `TicketJobRun`
|
||||||
- `Remark`, `RemarkScope`, `RemarkJobRun`
|
- `Remark`, `RemarkScope`, `RemarkJobRun`
|
||||||
- `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`
|
- `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`
|
||||||
@ -111,6 +114,7 @@ Note: always use direct SQL (`DELETE FROM`) for bulk deletions — ORM-level del
|
|||||||
- `backup_software` - e.g., "Veeam", "Synology", "Cove Data Protection"
|
- `backup_software` - e.g., "Veeam", "Synology", "Cove Data Protection"
|
||||||
- `backup_type` - e.g., "Backup Job", "Active Backup"
|
- `backup_type` - e.g., "Backup Job", "Active Backup"
|
||||||
- `cove_account_id` - (nullable int) links this job to a Cove AccountId
|
- `cove_account_id` - (nullable int) links this job to a Cove AccountId
|
||||||
|
- Cloud Connect accounts link back via `CloudConnectAccount.job_id` (no FK column on `jobs` — the link is on the staging table side)
|
||||||
|
|
||||||
**JobRun model:**
|
**JobRun model:**
|
||||||
- `source_type` - NULL = email (backwards compat), `"cove_api"` for Cove-imported runs
|
- `source_type` - NULL = email (backwards compat), `"cove_api"` for Cove-imported runs
|
||||||
@ -152,6 +156,31 @@ Note: always use direct SQL (`DELETE FROM`) for bulk deletions — ORM-level del
|
|||||||
- Hostname extraction from multiple patterns
|
- Hostname extraction from multiple patterns
|
||||||
- Returns: backup_type "Updates", job_name "Synology Automatic Update"
|
- Returns: backup_type "Updates", job_name "Synology Automatic Update"
|
||||||
|
|
||||||
|
## Schedule Inference and Missed Run Detection
|
||||||
|
|
||||||
|
### Overview
|
||||||
|
File: `containers/backupchecks/src/backend/app/main/routes_shared.py`
|
||||||
|
|
||||||
|
Missed runs are detected via `_ensure_missed_runs_for_job()` which is called from Run Checks on page load (throttled: max once per 10 minutes per job via in-memory dict). It infers the expected schedule from recent run history and creates `JobRun` records with `missed=True` for any slots that are overdue.
|
||||||
|
|
||||||
|
### Weekly Schedule Inference (`_infer_schedule_map_from_runs`)
|
||||||
|
- **Window**: last **90 days** only (older runs are excluded to handle schedule changes)
|
||||||
|
- **MIN_OCCURRENCES**: **5** hits on a weekday+time slot to count as expected (raised from 3 to reduce false positives during transitional periods)
|
||||||
|
- **Cadence guard**: if median gap between runs ≥ 20 days, weekly inference is **skipped** entirely → monthly inference handles the job instead. Prevents monthly jobs from accumulating enough weekly hits after long operation.
|
||||||
|
- **Key rule**: time-of-day changes or frequency changes stop generating missed runs on old slots within 90 days (no more stale slot false positives)
|
||||||
|
|
||||||
|
### Monthly Schedule Inference (`_infer_monthly_schedule_from_runs`)
|
||||||
|
- **Window**: last **180 days** (enough for ≥ 3 monthly occurrences, but forgotten within 6 months after a schedule change)
|
||||||
|
- Infers day-of-month + time-of-day from historical runs
|
||||||
|
- Used when weekly cadence guard fires (median gap ≥ 20 days)
|
||||||
|
|
||||||
|
### Important Rules
|
||||||
|
- **Never** extend the window without considering stale slot false positives
|
||||||
|
- Schedule changes (time, frequency) take effect in missed run detection within the window period (90d weekly, 180d monthly)
|
||||||
|
- Informational parsers (`3CX / Update`, `3CX / SSL Certificate`) are excluded from all schedule inference
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Cove Data Protection Integration
|
## Cove Data Protection Integration
|
||||||
|
|
||||||
### Overview
|
### Overview
|
||||||
@ -502,7 +531,14 @@ Visible on Logging page under `event_type = "ticket_link_debug"`. Remove after d
|
|||||||
|
|
||||||
## UI and UX Notes
|
## UI and UX Notes
|
||||||
|
|
||||||
### Navbar
|
### Layout v2 (2026-03-20)
|
||||||
|
- Complete sidebar-first redesign replacing the top navbar layout:
|
||||||
|
- `layout.css` rewritten with IBM Plex Sans/Mono fonts and CSS custom properties (design tokens)
|
||||||
|
- Fixed dark sidebar (220 px wide)
|
||||||
|
- `base.html` updated with Google Fonts preload and sidebar-aware structure
|
||||||
|
- Sandbox banner: semi-transparent (`rgba(220,53,69,0.45)`) instead of solid red
|
||||||
|
|
||||||
|
### Navbar (pre-v0.2.0 reference — replaced by sidebar in v0.2.0)
|
||||||
- Fixed-top positioning
|
- Fixed-top positioning
|
||||||
- Collapses on mobile (hamburger menu)
|
- Collapses on mobile (hamburger menu)
|
||||||
- Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top)
|
- Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top)
|
||||||
@ -572,6 +608,58 @@ Visible on Logging page under `event_type = "ticket_link_debug"`. Remove after d
|
|||||||
- Per-section limit (`SEARCH_LIMIT_PER_SECTION = 10`), with total count per section.
|
- Per-section limit (`SEARCH_LIMIT_PER_SECTION = 10`), with total count per section.
|
||||||
- No schema migration required for V1.
|
- No schema migration required for V1.
|
||||||
|
|
||||||
|
## Jobs Export / Import
|
||||||
|
|
||||||
|
### Schema Versions
|
||||||
|
- **v1** (`approved_jobs_export_v1`): job fields only, no account links
|
||||||
|
- **v2** (`approved_jobs_export_v2`): same as v1 plus per-job `cove_account` and `cloud_connect_account` objects
|
||||||
|
|
||||||
|
### Export (v2)
|
||||||
|
Each job entry contains:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"cove_account": {"account_id": 1234, "account_name": "...", "computer_name": "..."},
|
||||||
|
"cloud_connect_account": {"user": "...", "section": "Backup", "repo_name": "..."}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
`null` when not linked. File: `routes_settings.py` → `export_jobs()`.
|
||||||
|
|
||||||
|
### Import (v1 + v2)
|
||||||
|
- Accepts both schema versions (detected via `export_type` field)
|
||||||
|
- For v2 files: after creating/updating the job, the importer:
|
||||||
|
1. Looks up `CoveAccount` by `account_id` (fallback: `account_name` + `computer_name`)
|
||||||
|
2. Looks up `CloudConnectAccount` by `user` + `section` + `repo_name`
|
||||||
|
3. Links the account to the job — only if the account is not yet linked to a different job
|
||||||
|
- File: `routes_settings.py` → `import_jobs()`
|
||||||
|
|
||||||
|
## Inbox Batch Re-parse
|
||||||
|
|
||||||
|
### Endpoint
|
||||||
|
`POST /inbox/reparse-batch` — JSON in/out, login required, admin/operator only.
|
||||||
|
|
||||||
|
**Request body:**
|
||||||
|
```json
|
||||||
|
{"last_id": <int|null>, "total": <int|null>}
|
||||||
|
```
|
||||||
|
- `last_id`: keyset cursor from previous batch (process messages with `id < last_id`)
|
||||||
|
- `total`: total count from first call (avoids re-counting on every batch)
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{"processed": 50, "total": 847, "parsed_ok": 42, "auto_approved": 12, "no_match": 8, "errors": 0, "last_id": 1234, "done": false}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Batch Parameters
|
||||||
|
- `batch_size`: 50 messages per call
|
||||||
|
- `time_budget_s`: 8 seconds per call (stops processing mid-batch if exceeded)
|
||||||
|
- Auto-approve logic is identical to `inbox_reparse_all` (including VSPC multi-company handling)
|
||||||
|
|
||||||
|
### Frontend (inbox.html)
|
||||||
|
- "Re-parse all" button opens a Bootstrap modal (`data-bs-backdrop="static"` — cannot close while running)
|
||||||
|
- JS loop: `fetch → process response → setTimeout(100ms) → repeat` until `done: true`
|
||||||
|
- Progress bar + live counters update after each batch
|
||||||
|
- Close button appears only when `done: true` or on error
|
||||||
|
|
||||||
## Feedback Module with Screenshots
|
## Feedback Module with Screenshots
|
||||||
- Models: `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`.
|
- Models: `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`.
|
||||||
- Attachments:
|
- Attachments:
|
||||||
@ -633,11 +721,46 @@ File: `build-and-push.sh`
|
|||||||
- Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py`
|
- Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py`
|
||||||
- Cove importer: `containers/backupchecks/src/backend/app/cove_importer.py`
|
- Cove importer: `containers/backupchecks/src/backend/app/cove_importer.py`
|
||||||
- Cove routes: `containers/backupchecks/src/backend/app/main/routes_cove.py`
|
- Cove routes: `containers/backupchecks/src/backend/app/main/routes_cove.py`
|
||||||
|
- Cloud Connect importer: `containers/backupchecks/src/backend/app/cloud_connect_importer.py`
|
||||||
|
- Cloud Connect routes: `containers/backupchecks/src/backend/app/main/routes_cloud_connect.py`
|
||||||
|
- Inbox routes: `containers/backupchecks/src/backend/app/main/routes_inbox.py`
|
||||||
|
- Settings routes: `containers/backupchecks/src/backend/app/main/routes_settings.py`
|
||||||
- Compose stack: `deploy/backupchecks-stack.yml`
|
- Compose stack: `deploy/backupchecks-stack.yml`
|
||||||
- Build script: `build-and-push.sh`
|
- Build script: `build-and-push.sh`
|
||||||
|
|
||||||
## Recent Changes
|
## Recent Changes
|
||||||
|
|
||||||
|
### 2026-03-20 (v0.2.1)
|
||||||
|
- **Missed run false positive fix** (`routes_shared.py`):
|
||||||
|
- Weekly inference window: last 90 days only (was unbounded). Eliminates stale slot false positives after time-of-day or frequency changes.
|
||||||
|
- Cadence guard: if median gap between runs ≥ 20 days, skip weekly inference and let monthly inference handle the job. Fixes monthly jobs accumulating enough weekly hits after ~21 months.
|
||||||
|
- Monthly inference window: last 180 days (was unbounded).
|
||||||
|
- `MIN_OCCURRENCES` raised from 3 → 5 for weekly inference.
|
||||||
|
- **Objects sort fix** (`run_checks.html`, `job_detail.html`):
|
||||||
|
- `objectSeverityRank`: `|| err` on rank-0 check caused Warning items with `error_message` to rank as Critical. Fixed: only `error`/`failed`/`failure` status → rank 0; `|| err` moved to rank 1.
|
||||||
|
- **Mail iframe height fix** (`run_checks.html`):
|
||||||
|
- Flex rules were on `#rcm_body_iframe` but the iframe is not a direct flex child of `.rcm-mail-panel`. Fixed by moving `flex: 1 1 auto; min-height: 0` to the wrapper `#rcm_mail_iframe_body` and setting `height: 100%` on the iframe itself.
|
||||||
|
- **Inbox sidebar badge on all pages** (`__init__.py`):
|
||||||
|
- Added `inject_inbox_count()` Flask context processor — injects `inbox_count` into every template for authenticated users. Previously only injected in the dashboard route.
|
||||||
|
- **Jobs export/import schema v2** (`routes_settings.py`):
|
||||||
|
- Export: includes `cove_account` and `cloud_connect_account` per job.
|
||||||
|
- Import: accepts v1 and v2; links Cove/CC accounts on import if not yet linked to a different job.
|
||||||
|
- **Inbox re-parse progress modal** (`routes_inbox.py`, `inbox.html`):
|
||||||
|
- New `POST /inbox/reparse-batch` endpoint: 50 messages per call, 8 s time budget, keyset pagination, full auto-approve logic (including VSPC multi-company). Returns JSON progress.
|
||||||
|
- "Re-parse all" button replaced with modal trigger; JS loop calls batch endpoint until `done: true` and updates live progress bar + stats.
|
||||||
|
|
||||||
|
### 2026-03-20 (v0.2.0)
|
||||||
|
- **Layout v2**: complete sidebar-first redesign (`layout.css`, `base.html`). IBM Plex Sans/Mono fonts, CSS design tokens, fixed 220 px dark sidebar.
|
||||||
|
- **Veeam Cloud Connect importer**: HTML parser for daily report emails → `cloud_connect_accounts` staging table → `JobRun` records for linked accounts. `CloudConnectAccount` model + migrations. `/cloud-connect/accounts` review page. Sidebar link for admin/operator.
|
||||||
|
- **Cove historical run backfill**: `_backfill_colorbar_runs()` reconstructs up to 27 days of history from the `D09F08` colorbar on first run creation. Idempotent via `external_id = "cove-colorbar-{account_id}-{date}"`.
|
||||||
|
- **Cove run details popup**: Cove runs in job detail are clickable; popup fetches `/cove/run/<run_id>/detail` (structured Cove summary, per-datasource objects, mail section hidden).
|
||||||
|
- **Run Checks user preferences**: per-user sort mode + filter defaults stored in DB; `POST /run-checks/preferences`; User Settings page section.
|
||||||
|
- **Login captcha toggle**: Settings → General → Security card; `login_captcha_enabled` column with `DEFAULT TRUE`.
|
||||||
|
- **Cloud Connect unique key**: changed from `(user, section)` to `(user, section, repo_name)` — supports multiple repos per user.
|
||||||
|
- **Cloud Connect run detail popup**: shows structured CC summary instead of raw email; raw email accessible via toggle.
|
||||||
|
- **Entra SSO**: implemented (marked untested). Login/callback/logout flow, optional auto-provisioning, tenant/domain + security-group restrictions.
|
||||||
|
- **Fixes**: login page layout with flash messages; "Delete all jobs" timeout (replaced ORM with direct SQL); archived job auto-matching; Cove link sync between `cove_accounts.job_id` ↔ `jobs.cove_account_id`; Cove run creation transaction scope.
|
||||||
|
|
||||||
### 2026-02-23
|
### 2026-02-23
|
||||||
- **Cove Data Protection full integration**:
|
- **Cove Data Protection full integration**:
|
||||||
- `cove_importer.py` – Cove API client (login, paginated enumeration, status mapping, deduplication, per-datasource object persistence)
|
- `cove_importer.py` – Cove API client (login, paginated enumeration, status mapping, deduplication, per-datasource object persistence)
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user