From 8deecd4c110e35470322b3f089fe7f286b4e9efa Mon Sep 17 00:00:00 2001 From: Ivo Oskamp Date: Mon, 23 Feb 2026 10:49:44 +0100 Subject: [PATCH] Update technical-notes-codex.md with Cove integration and recent changes - Add complete Cove Data Protection integration section: API details, column codes, status mapping, inbox flow, CoveAccount model, routes, migrations, background thread, settings UI - Update Data Model section: CoveAccount model, Job.cove_account_id, JobRun.source_type + external_id, SystemSettings Cove fields - Update Application Architecture: cove_importer_service background thread - Add debug logging snippet for ticket linking issues - Add Recent Changes entry for 2026-02-23 - Add Cove files to Quick References Co-Authored-By: Claude Sonnet 4.6 --- docs/technical-notes-codex.md | 172 +++++++++++++++++++++++++++++++++- 1 file changed, 169 insertions(+), 3 deletions(-) diff --git a/docs/technical-notes-codex.md b/docs/technical-notes-codex.md index dbacd80..3a6d0c1 100644 --- a/docs/technical-notes-codex.md +++ b/docs/technical-notes-codex.md @@ -1,6 +1,6 @@ # Technical Notes (Internal) -Last updated: 2026-02-19 +Last updated: 2026-02-23 ## Purpose Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis. @@ -30,8 +30,9 @@ Internal technical snapshot of the `backupchecks` repository for faster onboardi - Database initialization at startup: - `db.create_all()` - `run_migrations()` -- Background task: +- Background tasks: - `start_auto_importer(app)` starts the automatic mail importer thread. + - `start_cove_importer(app)` starts the Cove Data Protection polling thread (started only when `cove_import_enabled` is set). - Health endpoint: - `GET /health` returns `{ "status": "ok" }`. @@ -71,11 +72,13 @@ File: `containers/backupchecks/src/backend/app/models.py` - System settings: - `SystemSettings` with Graph/mail settings, import settings, UI timezone, dashboard policy, sandbox flag. - Autotask configuration and cache fields are present. + - Cove Data Protection fields: `cove_enabled`, `cove_api_url`, `cove_api_username`, `cove_api_password`, `cove_import_enabled`, `cove_import_interval_minutes`, `cove_partner_id`, `cove_last_import_at`. - Logging: - `AuditLog` (legacy alias `AdminLog`). - Domain: - `Customer`, `Job`, `JobRun`, `Override` - `MailMessage`, `MailObject` + - `CoveAccount` (Cove staging table — see Cove integration section) - `Ticket`, `TicketScope`, `TicketJobRun` - `Remark`, `RemarkScope`, `RemarkJobRun` - `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment` @@ -101,8 +104,13 @@ Critical deletion order to avoid constraint violations: **Job model:** - `customer_id` - FK to Customer - `job_name` - parsed from email -- `backup_software` - e.g., "Veeam", "Synology" +- `backup_software` - e.g., "Veeam", "Synology", "Cove Data Protection" - `backup_type` - e.g., "Backup Job", "Active Backup" +- `cove_account_id` - (nullable int) links this job to a Cove AccountId + +**JobRun model:** +- `source_type` - NULL = email (backwards compat), `"cove_api"` for Cove-imported runs +- `external_id` - deduplication key for Cove runs: `"cove-{account_id}-{run_ts}"` ## Parser Architecture - Folder: `containers/backupchecks/src/backend/app/parsers/` @@ -140,6 +148,121 @@ Critical deletion order to avoid constraint violations: - Hostname extraction from multiple patterns - Returns: backup_type "Updates", job_name "Synology Automatic Update" +## Cove Data Protection Integration + +### Overview +Cove (N-able) Data Protection is a cloud backup platform. Backupchecks integrates with it via the Cove JSON-RPC API, following the same inbox-style staging flow as email imports. + +### Files +- `containers/backupchecks/src/backend/app/cove_importer.py` – API client, account processing, JobRun creation +- `containers/backupchecks/src/backend/app/cove_importer_service.py` – background polling thread +- `containers/backupchecks/src/backend/app/main/routes_cove.py` – `/cove/accounts` routes +- `containers/backupchecks/src/templates/main/cove_accounts.html` – inbox-style accounts page + +### API Details +- Endpoint: `https://api.backup.management/jsonapi` (JSON-RPC 2.0) +- **Login**: `POST` with `{"jsonrpc":"2.0","id":"jsonrpc","method":"Login","params":{"username":"...","password":"..."}}` + - Returns `visa` at top level (`data["visa"]`), **not** inside `result` + - Returns `PartnerId` inside `result` +- **EnumerateAccountStatistics**: `POST` with visa in payload, `query` (lowercase) with `PartnerId`, `StartRecordNumber`, `RecordsCount`, `Columns` +- Settings format per account: `[{"D09F00": "5"}, {"I1": "device name"}, ...]` — list of single-key dicts, flatten with `dict.update(item)` + +### Column Codes +| Code | Meaning | +|------|---------| +| `I1` | Account/device name | +| `I18` | Computer name | +| `I8` | Customer/partner name | +| `I78` | Active datasource label | +| `D09F00` | Overall last session status code | +| `D09F09` | Last successful session timestamp (Unix) | +| `D09F15` | Last session end timestamp (Unix) | +| `D09F08` | 28-day colorbar string | +| `D1F00/F15` | Files & Folders status/timestamp | +| `D10F00/F15` | VssMsSql | +| `D11F00/F15` | VssSharePoint | +| `D19F00/F15` | M365 Exchange | +| `D20F00/F15` | M365 OneDrive | +| `D5F00/F15` | M365 SharePoint | +| `D23F00/F15` | M365 Teams | + +### Status Code Mapping +| Cove code | Meaning | Backupchecks status | +|-----------|---------|---------------------| +| 1 | In process | Warning | +| 2 | Failed | Error | +| 3 | Aborted | Error | +| 5 | Completed | Success | +| 6 | Interrupted | Error | +| 7 | Not started | Warning | +| 8 | Completed with errors | Warning | +| 9 | In progress with faults | Warning | +| 10 | Over quota | Error | +| 11 | No selection | Warning | +| 12 | Restarted | Warning | + +### Inbox-Style Flow (mirrors email import) +1. Cove importer fetches all accounts via paginated `EnumerateAccountStatistics` (250/page). +2. Every account is upserted into the `cove_accounts` staging table (always, regardless of job link). +3. Accounts without a `job_id` appear on `/cove/accounts` ("Cove Accounts" page) for admin action. +4. Admin can: + - **Create new job** – creates a `Job` with `backup_software="Cove Data Protection"` and links it. + - **Link to existing job** – sets `job.cove_account_id` and `cove_acc.job_id`. +5. On the next import, linked accounts generate `JobRun` records (deduplicated via `external_id`). +6. Per-datasource objects are persisted to `customer_objects`, `job_object_links`, `run_object_links`. + +### CoveAccount Model +```python +class CoveAccount(db.Model): + __tablename__ = "cove_accounts" + id # PK + account_id # Cove AccountId (unique) + account_name # I1 + computer_name # I18 + customer_name # I8 + datasource_types # I78 + last_status_code # D09F00 (int) + last_run_at # D09F15 (datetime) + colorbar_28d # D09F08 + job_id # FK → jobs.id (nullable — None = unmatched) + first_seen_at + last_seen_at + job # relationship → Job +``` + +### Deduplication +`external_id = f"cove-{account_id}-{run_ts}"` where `run_ts` is the Unix timestamp from `D09F15`. +Before creating a `JobRun`, check `JobRun.query.filter_by(external_id=external_id).first()`. + +### Background Thread +`cove_importer_service.py` — same pattern as `auto_importer_service.py`: +- Thread name: `"cove_importer"` +- Checks `settings.cove_import_enabled` +- Interval: `settings.cove_import_interval_minutes` (default 30) +- Calls `run_cove_import(settings)` which returns `(total, created, skipped, errors)` + +### Settings UI +Settings → Integrations → Cove section: +- Enable toggle, API URL, username, password (masked, only overwritten if non-empty) +- Import enabled + interval +- "Test Connection" button (AJAX → `POST /settings/cove/test-connection`) returns `{ok, partner_id, message}` +- "Run import now" button (→ `POST /settings/cove/run-now`) triggers manual import + +### Routes +| Route | Method | Description | +|-------|--------|-------------| +| `/cove/accounts` | GET | Inbox-style page: unmatched + matched accounts | +| `/cove/accounts//link` | POST | `action=create` or `action=link` | +| `/cove/accounts//unlink` | POST | Removes job link, puts account back in unmatched | +| `/settings/cove/test-connection` | POST | AJAX: verify credentials, save partner_id | +| `/settings/cove/run-now` | POST | Manual import trigger | + +### Migrations +- `migrate_cove_integration()` — adds 8 columns to `system_settings`, `cove_account_id` to `jobs`, `source_type` + `external_id` to `job_runs`, dedup index on `job_runs.external_id` +- `migrate_cove_accounts_table()` — creates `cove_accounts` table with indexes + +--- + ## Ticketing and Autotask (Critical Rules) ### Two Ticket Types @@ -208,6 +331,30 @@ All pages use **explicit link-based queries** (no date-based logic): 2. Job-level scope: `remarks JOIN remark_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date` (with timezone-safe fallback from `start_date`) - Prevents duplicates by tracking seen remark IDs +### Debug Logging for Ticket Linking (Reference) +If you need to debug ticket linking issues, add this to `link_open_internal_tickets_to_run` in `ticketing_utils.py` after the rows query: + +```python +try: + from .models import AuditLog + details = [] + if rows: + for tid, code, t_resolved, ts_resolved in rows: + details.append(f"ticket_id={tid}, code={code}, t.resolved_at={t_resolved}, ts.resolved_at={ts_resolved}") + else: + details.append("No open tickets found for this job") + audit = AuditLog( + user="system", event_type="ticket_link_debug", + message=f"link_open_internal_tickets_to_run called: run_id={run.id}, job_id={job.id}, found={len(rows)} ticket(s)", + details="\n".join(details) + ) + db.session.add(audit) + db.session.commit() +except Exception: + pass +``` +Visible on Logging page under `event_type = "ticket_link_debug"`. Remove after debugging. + ### Resolved vs Deleted - **Resolved**: Ticket completed in Autotask (tracked in internal `tickets.resolved_at`) - Stops propagating to new runs @@ -358,11 +505,30 @@ File: `build-and-push.sh` - Parsers: `containers/backupchecks/src/backend/app/parsers/registry.py` - Ticketing utilities: `containers/backupchecks/src/backend/app/ticketing_utils.py` - Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py` +- Cove importer: `containers/backupchecks/src/backend/app/cove_importer.py` +- Cove routes: `containers/backupchecks/src/backend/app/main/routes_cove.py` - Compose stack: `deploy/backupchecks-stack.yml` - Build script: `build-and-push.sh` ## Recent Changes +### 2026-02-23 +- **Cove Data Protection full integration**: + - `cove_importer.py` – Cove API client (login, paginated enumeration, status mapping, deduplication, per-datasource object persistence) + - `cove_importer_service.py` – background polling thread (same pattern as `auto_importer_service.py`) + - `CoveAccount` staging model + `migrate_cove_accounts_table()` migration + - `SystemSettings` – 8 new Cove fields, `Job` – `cove_account_id`, `JobRun` – `source_type` + `external_id` + - `routes_cove.py` – inbox-style `/cove/accounts` with link/unlink routes + - `cove_accounts.html` – unmatched accounts shown first with Bootstrap modals (create job / link to existing), matched accounts with Unlink + - Settings > Integrations: Cove section with test connection (AJAX) and manual import trigger + - Navbar: "Cove Accounts" link for admin/operator when `cove_enabled` +- **Cove API key findings** (from test script + N-able support): + - Visa is returned at top level of login response, not inside `result` + - Settings per account are a list of single-key dicts `[{"D09F00":"5"}, ...]` — flatten with `flat.update(item)` + - EnumerateAccountStatistics params must use lowercase `query` key and `RecordsCount` (not `RecordCount`) + - Login params must use lowercase `username`/`password` + - D02/D03 are legacy; use D10/D11 or D09 (Total) instead + ### 2026-02-19 - **Added 3CX Update parser support**: `threecx.py` now recognizes subject `3CX Notification: Update Successful - ` and stores it as informational with: - `backup_software = 3CX`