Update technical-notes-codex.md with Cove integration and recent changes

- Add complete Cove Data Protection integration section: API details, column codes, status mapping, inbox flow, CoveAccount model, routes, migrations, background thread, settings UI - Update Data Model section: CoveAccount model, Job.cove_account_id, JobRun.source_type + external_id, SystemSettings Cove fields - Update Application Architecture: cove_importer_service background thread - Add debug logging snippet for ticket linking issues - Add Recent Changes entry for 2026-02-23 - Add Cove files to Quick References Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-23 10:49:44 +01:00 · 2026-02-23 10:49:44 +01:00 · 8deecd4c11
commit 8deecd4c11
parent 9b19283c97
1 changed files with 169 additions and 3 deletions
--- a/docs/technical-notes-codex.md
+++ b/docs/technical-notes-codex.md
@ -1,6 +1,6 @@
 # Technical Notes (Internal)

-Last updated: 2026-02-19
+Last updated: 2026-02-23

 ## Purpose
 Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis.
@ -30,8 +30,9 @@ Internal technical snapshot of the `backupchecks` repository for faster onboardi
 - Database initialization at startup:
  - `db.create_all()`
  - `run_migrations()`
- Background task:
+- Background tasks:
  - `start_auto_importer(app)` starts the automatic mail importer thread.
+  - `start_cove_importer(app)` starts the Cove Data Protection polling thread (started only when `cove_import_enabled` is set).
 - Health endpoint:
  - `GET /health` returns `{ "status": "ok" }`.

@ -71,11 +72,13 @@ File: `containers/backupchecks/src/backend/app/models.py`
 - System settings:
  - `SystemSettings` with Graph/mail settings, import settings, UI timezone, dashboard policy, sandbox flag.
  - Autotask configuration and cache fields are present.
+  - Cove Data Protection fields: `cove_enabled`, `cove_api_url`, `cove_api_username`, `cove_api_password`, `cove_import_enabled`, `cove_import_interval_minutes`, `cove_partner_id`, `cove_last_import_at`.
 - Logging:
  - `AuditLog` (legacy alias `AdminLog`).
 - Domain:
  - `Customer`, `Job`, `JobRun`, `Override`
  - `MailMessage`, `MailObject`
+  - `CoveAccount` (Cove staging table — see Cove integration section)
  - `Ticket`, `TicketScope`, `TicketJobRun`
  - `Remark`, `RemarkScope`, `RemarkJobRun`
  - `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`
@ -101,8 +104,13 @@ Critical deletion order to avoid constraint violations:
 **Job model:**
 - `customer_id` - FK to Customer
 - `job_name` - parsed from email
- `backup_software` - e.g., "Veeam", "Synology"
+- `backup_software` - e.g., "Veeam", "Synology", "Cove Data Protection"
 - `backup_type` - e.g., "Backup Job", "Active Backup"
+- `cove_account_id` - (nullable int) links this job to a Cove AccountId
+
+**JobRun model:**
+- `source_type` - NULL = email (backwards compat), `"cove_api"` for Cove-imported runs
+- `external_id` - deduplication key for Cove runs: `"cove-{account_id}-{run_ts}"`

 ## Parser Architecture
 - Folder: `containers/backupchecks/src/backend/app/parsers/`
@ -140,6 +148,121 @@ Critical deletion order to avoid constraint violations:
 - Hostname extraction from multiple patterns
 - Returns: backup_type "Updates", job_name "Synology Automatic Update"

+## Cove Data Protection Integration
+
+### Overview
+Cove (N-able) Data Protection is a cloud backup platform. Backupchecks integrates with it via the Cove JSON-RPC API, following the same inbox-style staging flow as email imports.
+
+### Files
+- `containers/backupchecks/src/backend/app/cove_importer.py` – API client, account processing, JobRun creation
+- `containers/backupchecks/src/backend/app/cove_importer_service.py` – background polling thread
+- `containers/backupchecks/src/backend/app/main/routes_cove.py` – `/cove/accounts` routes
+- `containers/backupchecks/src/templates/main/cove_accounts.html` – inbox-style accounts page
+
+### API Details
+- Endpoint: `https://api.backup.management/jsonapi` (JSON-RPC 2.0)
+- **Login**: `POST` with `{"jsonrpc":"2.0","id":"jsonrpc","method":"Login","params":{"username":"...","password":"..."}}`
+  - Returns `visa` at top level (`data["visa"]`), **not** inside `result`
+  - Returns `PartnerId` inside `result`
+- **EnumerateAccountStatistics**: `POST` with visa in payload, `query` (lowercase) with `PartnerId`, `StartRecordNumber`, `RecordsCount`, `Columns`
+- Settings format per account: `[{"D09F00": "5"}, {"I1": "device name"}, ...]` — list of single-key dicts, flatten with `dict.update(item)`
+
+### Column Codes
+| Code | Meaning |
+|------|---------|
+| `I1` | Account/device name |
+| `I18` | Computer name |
+| `I8` | Customer/partner name |
+| `I78` | Active datasource label |
+| `D09F00` | Overall last session status code |
+| `D09F09` | Last successful session timestamp (Unix) |
+| `D09F15` | Last session end timestamp (Unix) |
+| `D09F08` | 28-day colorbar string |
+| `D1F00/F15` | Files & Folders status/timestamp |
+| `D10F00/F15` | VssMsSql |
+| `D11F00/F15` | VssSharePoint |
+| `D19F00/F15` | M365 Exchange |
+| `D20F00/F15` | M365 OneDrive |
+| `D5F00/F15` | M365 SharePoint |
+| `D23F00/F15` | M365 Teams |
+
+### Status Code Mapping
+| Cove code | Meaning | Backupchecks status |
+|-----------|---------|---------------------|
+| 1 | In process | Warning |
+| 2 | Failed | Error |
+| 3 | Aborted | Error |
+| 5 | Completed | Success |
+| 6 | Interrupted | Error |
+| 7 | Not started | Warning |
+| 8 | Completed with errors | Warning |
+| 9 | In progress with faults | Warning |
+| 10 | Over quota | Error |
+| 11 | No selection | Warning |
+| 12 | Restarted | Warning |
+
+### Inbox-Style Flow (mirrors email import)
+1. Cove importer fetches all accounts via paginated `EnumerateAccountStatistics` (250/page).
+2. Every account is upserted into the `cove_accounts` staging table (always, regardless of job link).
+3. Accounts without a `job_id` appear on `/cove/accounts` ("Cove Accounts" page) for admin action.
+4. Admin can:
+   - **Create new job** – creates a `Job` with `backup_software="Cove Data Protection"` and links it.
+   - **Link to existing job** – sets `job.cove_account_id` and `cove_acc.job_id`.
+5. On the next import, linked accounts generate `JobRun` records (deduplicated via `external_id`).
+6. Per-datasource objects are persisted to `customer_objects`, `job_object_links`, `run_object_links`.
+
+### CoveAccount Model
+```python
+class CoveAccount(db.Model):
+    __tablename__ = "cove_accounts"
+    id            # PK
+    account_id    # Cove AccountId (unique)
+    account_name  # I1
+    computer_name # I18
+    customer_name # I8
+    datasource_types # I78
+    last_status_code # D09F00 (int)
+    last_run_at   # D09F15 (datetime)
+    colorbar_28d  # D09F08
+    job_id        # FK → jobs.id (nullable — None = unmatched)
+    first_seen_at
+    last_seen_at
+    job           # relationship → Job
+```
+
+### Deduplication
+`external_id = f"cove-{account_id}-{run_ts}"` where `run_ts` is the Unix timestamp from `D09F15`.
+Before creating a `JobRun`, check `JobRun.query.filter_by(external_id=external_id).first()`.
+
+### Background Thread
+`cove_importer_service.py` — same pattern as `auto_importer_service.py`:
+- Thread name: `"cove_importer"`
+- Checks `settings.cove_import_enabled`
+- Interval: `settings.cove_import_interval_minutes` (default 30)
+- Calls `run_cove_import(settings)` which returns `(total, created, skipped, errors)`
+
+### Settings UI
+Settings → Integrations → Cove section:
+- Enable toggle, API URL, username, password (masked, only overwritten if non-empty)
+- Import enabled + interval
+- "Test Connection" button (AJAX → `POST /settings/cove/test-connection`) returns `{ok, partner_id, message}`
+- "Run import now" button (→ `POST /settings/cove/run-now`) triggers manual import
+
+### Routes
+| Route | Method | Description |
+|-------|--------|-------------|
+| `/cove/accounts` | GET | Inbox-style page: unmatched + matched accounts |
+| `/cove/accounts/<id>/link` | POST | `action=create` or `action=link` |
+| `/cove/accounts/<id>/unlink` | POST | Removes job link, puts account back in unmatched |
+| `/settings/cove/test-connection` | POST | AJAX: verify credentials, save partner_id |
+| `/settings/cove/run-now` | POST | Manual import trigger |
+
+### Migrations
+- `migrate_cove_integration()` — adds 8 columns to `system_settings`, `cove_account_id` to `jobs`, `source_type` + `external_id` to `job_runs`, dedup index on `job_runs.external_id`
+- `migrate_cove_accounts_table()` — creates `cove_accounts` table with indexes
+
+---
+
 ## Ticketing and Autotask (Critical Rules)

 ### Two Ticket Types
@ -208,6 +331,30 @@ All pages use **explicit link-based queries** (no date-based logic):
  2. Job-level scope: `remarks JOIN remark_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date` (with timezone-safe fallback from `start_date`)
 - Prevents duplicates by tracking seen remark IDs

+### Debug Logging for Ticket Linking (Reference)
+If you need to debug ticket linking issues, add this to `link_open_internal_tickets_to_run` in `ticketing_utils.py` after the rows query:
+
+```python
+try:
+    from .models import AuditLog
+    details = []
+    if rows:
+        for tid, code, t_resolved, ts_resolved in rows:
+            details.append(f"ticket_id={tid}, code={code}, t.resolved_at={t_resolved}, ts.resolved_at={ts_resolved}")
+    else:
+        details.append("No open tickets found for this job")
+    audit = AuditLog(
+        user="system", event_type="ticket_link_debug",
+        message=f"link_open_internal_tickets_to_run called: run_id={run.id}, job_id={job.id}, found={len(rows)} ticket(s)",
+        details="\n".join(details)
+    )
+    db.session.add(audit)
+    db.session.commit()
+except Exception:
+    pass
+```
+Visible on Logging page under `event_type = "ticket_link_debug"`. Remove after debugging.
+
 ### Resolved vs Deleted
 - **Resolved**: Ticket completed in Autotask (tracked in internal `tickets.resolved_at`)
  - Stops propagating to new runs
@ -358,11 +505,30 @@ File: `build-and-push.sh`
 - Parsers: `containers/backupchecks/src/backend/app/parsers/registry.py`
 - Ticketing utilities: `containers/backupchecks/src/backend/app/ticketing_utils.py`
 - Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py`
+- Cove importer: `containers/backupchecks/src/backend/app/cove_importer.py`
+- Cove routes: `containers/backupchecks/src/backend/app/main/routes_cove.py`
 - Compose stack: `deploy/backupchecks-stack.yml`
 - Build script: `build-and-push.sh`

 ## Recent Changes

+### 2026-02-23
+- **Cove Data Protection full integration**:
+  - `cove_importer.py` – Cove API client (login, paginated enumeration, status mapping, deduplication, per-datasource object persistence)
+  - `cove_importer_service.py` – background polling thread (same pattern as `auto_importer_service.py`)
+  - `CoveAccount` staging model + `migrate_cove_accounts_table()` migration
+  - `SystemSettings` – 8 new Cove fields, `Job` – `cove_account_id`, `JobRun` – `source_type` + `external_id`
+  - `routes_cove.py` – inbox-style `/cove/accounts` with link/unlink routes
+  - `cove_accounts.html` – unmatched accounts shown first with Bootstrap modals (create job / link to existing), matched accounts with Unlink
+  - Settings > Integrations: Cove section with test connection (AJAX) and manual import trigger
+  - Navbar: "Cove Accounts" link for admin/operator when `cove_enabled`
+- **Cove API key findings** (from test script + N-able support):
+  - Visa is returned at top level of login response, not inside `result`
+  - Settings per account are a list of single-key dicts `[{"D09F00":"5"}, ...]` — flatten with `flat.update(item)`
+  - EnumerateAccountStatistics params must use lowercase `query` key and `RecordsCount` (not `RecordCount`)
+  - Login params must use lowercase `username`/`password`
+  - D02/D03 are legacy; use D10/D11 or D09 (Total) instead
+
 ### 2026-02-19
 - **Added 3CX Update parser support**: `threecx.py` now recognizes subject `3CX Notification: Update Successful - <host>` and stores it as informational with:
  - `backup_software = 3CX`