backupchecks/docs/technical-notes-codex.md

# Technical Notes (Internal)

Last updated: 2026-02-23 (late)

## Purpose
Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis.

## Repository Overview
- Application: Flask web app with SQLAlchemy and Flask-Migrate.
- Runtime: Containerized (Docker), deployed via Docker Compose stack.
- Primary source code location: `containers/backupchecks/src`.
- The project also contains extensive functional documentation in `docs/` and multiple roadmap TODO files at repository root.

## Main Structure
- `containers/backupchecks/Dockerfile`: Python 3.12-slim image, starts `gunicorn` with `backend.app:create_app()`.
- `containers/backupchecks/requirements.txt`: Flask stack + PostgreSQL driver + reporting libraries (`reportlab`, `Markdown`).
- `containers/backupchecks/src/backend/app`: backend domain logic, routes, parsers, models, migrations.
- `containers/backupchecks/src/templates`: Jinja templates for auth/main/documentation pages.
- `containers/backupchecks/src/static`: CSS, images, favicon.
- `deploy/backupchecks-stack.yml`: compose stack with `backupchecks`, `postgres`, `adminer`.
- `build-and-push.sh`: release/test build script with version bumping, tags, and image push.
- `docs/`: functional design, changelogs, migration notes, API notes.

## Application Architecture (Current Observation)
- Factory pattern: `create_app()` in `containers/backupchecks/src/backend/app/__init__.py`.
- Blueprints:
  - `auth_bp` for authentication.
  - `main_bp` for core functionality.
  - `doc_bp` for internal documentation pages.
- Database initialization at startup:
  - `db.create_all()`
  - `run_migrations()`
- Background tasks:
  - `start_auto_importer(app)` starts the automatic mail importer thread.
  - `start_cove_importer(app)` starts the Cove Data Protection polling thread (started only when `cove_import_enabled` is set).
- Health endpoint:
  - `GET /health` returns `{ "status": "ok" }`.

## Functional Processing Flow
- Import:
  - Email is fetched via Microsoft Graph API.
- Parse:
  - Parser selection through registry + software-specific parser implementations.
- Approve:
  - New jobs first appear in Inbox for initial customer assignment.
- Auto-process:
  - Subsequent emails for known jobs automatically create `JobRun` records.
- Monitor:
  - Runs appear in Daily Jobs and Run Checks.
- Review:
  - Manual review removes items from the unreviewed operational queue.

## Configuration and Runtime
- Config is built from environment variables in `containers/backupchecks/src/backend/app/config.py`.
- Important variables:
  - `APP_SECRET_KEY`
  - `APP_ENV`
  - `APP_PORT`
  - `POSTGRES_DB`
  - `POSTGRES_USER`
  - `POSTGRES_PASSWORD`
  - `DB_HOST`
  - `DB_PORT`
- Database URI pattern:
  - `postgresql+psycopg2://<user>:<pass>@<host>:<port>/<db>`
- Default timezone in config: `Europe/Amsterdam`.

## Data Model (High-level)
File: `containers/backupchecks/src/backend/app/models.py`
- Auth/users:
  - `User` with role(s), active role in session.
- System settings:
  - `SystemSettings` with Graph/mail settings, import settings, UI timezone, dashboard policy, sandbox flag.
  - Autotask configuration and cache fields are present.
  - Cove Data Protection fields: `cove_enabled`, `cove_api_url`, `cove_api_username`, `cove_api_password`, `cove_import_enabled`, `cove_import_interval_minutes`, `cove_partner_id`, `cove_last_import_at`.
  - Microsoft Entra SSO fields: `entra_sso_enabled`, `entra_tenant_id`, `entra_client_id`, `entra_client_secret`, `entra_redirect_uri`, `entra_allowed_domain`, `entra_allowed_group_ids`, `entra_auto_provision_users`.
- Logging:
  - `AuditLog` (legacy alias `AdminLog`).
- Domain:
  - `Customer`, `Job`, `JobRun`, `Override`
  - `MailMessage`, `MailObject`
  - `CoveAccount` (Cove staging table — see Cove integration section)
  - `Ticket`, `TicketScope`, `TicketJobRun`
  - `Remark`, `RemarkScope`, `RemarkJobRun`
  - `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`

### Foreign Key Relationships & Deletion Order
Critical deletion order to avoid constraint violations:
1. Clean auxiliary tables (ticket_job_runs, remark_job_runs, scopes, overrides)
2. Unlink mails from jobs (UPDATE mail_messages SET job_id = NULL)
3. Delete mail_objects
4. Delete jobs (cascades to job_runs)
5. Delete mails

### Key Model Fields
**MailMessage model:**
- `from_address` (NOT `sender`!) - sender email
- `subject` - email subject
- `text_body` - plain text content
- `html_body` - HTML content
- `received_at` - timestamp
- `location` - inbox/processed/deleted
- `job_id` - link to Job (nullable)

**Job model:**
- `customer_id` - FK to Customer
- `job_name` - parsed from email
- `backup_software` - e.g., "Veeam", "Synology", "Cove Data Protection"
- `backup_type` - e.g., "Backup Job", "Active Backup"
- `cove_account_id` - (nullable int) links this job to a Cove AccountId

**JobRun model:**
- `source_type` - NULL = email (backwards compat), `"cove_api"` for Cove-imported runs
- `external_id` - deduplication key for Cove runs: `"cove-{account_id}-{run_ts}"`

## Parser Architecture
- Folder: `containers/backupchecks/src/backend/app/parsers/`
- Two layers:
  - `registry.py`:
    - matching/documentation/visibility on `/parsers`.
    - examples must stay generic (no customer names).
  - parser files (`veeam.py`, `synology.py`, etc.):
    - actual detection and parsing logic.
    - return structured output: software, type, job name, status, objects.
- Practical rule:
  - extend patterns by adding, not replacing (backward compatibility).

### Parser Types
**Informational Parsers:**
- DSM Updates, Account Protection, Firmware Updates
- Set appropriate backup_type (e.g., "Updates", "Firmware Update")
- Do NOT participate in schedule learning
- Usually still visible in Run Checks for awareness
- Exception: non-backup 3CX informational types (`Update`, `SSL Certificate`) are hidden from Run Checks

**Regular Parsers:**
- Backup jobs (Veeam, Synology Active Backup, NAKIVO, etc.)
- Participate in schedule learning (daily/weekly/monthly detection)
- Generate missed runs when expected runs don't occur

**Example: Synology Updates Parser (synology.py)**
- Handles multiple update notification types under same job:
  - DSM automatic update cancelled
  - Packages out-of-date
  - Combined notifications (DSM + packages)
- Detection patterns:
  - DSM: "Automatische DSM-update", "DSM-update op", "automatic DSM update"
  - Packages: "Packages on", "out-of-date", "Package Center"
- Hostname extraction from multiple patterns
- Returns: backup_type "Updates", job_name "Synology Automatic Update"

## Cove Data Protection Integration

### Overview
Cove (N-able) Data Protection is a cloud backup platform. Backupchecks integrates with it via the Cove JSON-RPC API, following the same inbox-style staging flow as email imports.

### Files
- `containers/backupchecks/src/backend/app/cove_importer.py` – API client, account processing, JobRun creation
- `containers/backupchecks/src/backend/app/cove_importer_service.py` – background polling thread
- `containers/backupchecks/src/backend/app/main/routes_cove.py` – `/cove/accounts` routes
- `containers/backupchecks/src/templates/main/cove_accounts.html` – inbox-style accounts page

### API Details
- Endpoint: `https://api.backup.management/jsonapi` (JSON-RPC 2.0)
- **Login**: `POST` with `{"jsonrpc":"2.0","id":"jsonrpc","method":"Login","params":{"username":"...","password":"..."}}`
  - Returns `visa` at top level (`data["visa"]`), **not** inside `result`
  - Returns `PartnerId` inside `result`
- **EnumerateAccountStatistics**: `POST` with visa in payload, `query` (lowercase) with `PartnerId`, `StartRecordNumber`, `RecordsCount`, `Columns`
- Settings format per account: `[{"D09F00": "5"}, {"I1": "device name"}, ...]` — list of single-key dicts, flatten with `dict.update(item)`

### Column Codes
| Code | Meaning |
|------|---------|
| `I1` | Account/device name |
| `I18` | Computer name |
| `I8` | Customer/partner name |
| `I78` | Active datasource label |
| `D09F00` | Overall last session status code |
| `D09F09` | Last successful session timestamp (Unix) |
| `D09F15` | Last session end timestamp (Unix) |
| `D09F08` | 28-day colorbar string |
| `D1F00/F15` | Files & Folders status/timestamp |
| `D10F00/F15` | VssMsSql |
| `D11F00/F15` | VssSharePoint |
| `D19F00/F15` | M365 Exchange |
| `D20F00/F15` | M365 OneDrive |
| `D5F00/F15` | M365 SharePoint |
| `D23F00/F15` | M365 Teams |

### Status Code Mapping
| Cove code | Meaning | Backupchecks status |
|-----------|---------|---------------------|
| 1 | In process | Warning |
| 2 | Failed | Error |
| 3 | Aborted | Error |
| 5 | Completed | Success |
| 6 | Interrupted | Error |
| 7 | Not started | Warning |
| 8 | Completed with errors | Warning |
| 9 | In progress with faults | Warning |
| 10 | Over quota | Error |
| 11 | No selection | Warning |
| 12 | Restarted | Warning |

### Inbox-Style Flow (mirrors email import)
1. Cove importer fetches all accounts via paginated `EnumerateAccountStatistics` (250/page).
2. Every account is upserted into the `cove_accounts` staging table (always, regardless of job link).
3. Accounts without a `job_id` appear on `/cove/accounts` ("Cove Accounts" page) for admin action.
4. Admin can:
   - **Create new job** – creates a `Job` with `backup_software="Cove Data Protection"` and links it.
   - **Link to existing job** – sets `job.cove_account_id` and `cove_acc.job_id`.
5. Linking an account triggers an immediate import attempt; linked accounts then generate `JobRun` records (deduplicated per job via `job_id + external_id`).
6. Per-datasource objects are persisted to `customer_objects`, `job_object_links`, `run_object_links`.

### CoveAccount Model
```python
class CoveAccount(db.Model):
    __tablename__ = "cove_accounts"
    id            # PK
    account_id    # Cove AccountId (unique)
    account_name  # I1
    computer_name # I18
    customer_name # I8
    datasource_types # I78
    last_status_code # D09F00 (int)
    last_run_at   # D09F15 (datetime)
    colorbar_28d  # D09F08
    job_id        # FK → jobs.id (nullable — None = unmatched)
    first_seen_at
    last_seen_at
    job           # relationship → Job
```

### Deduplication
`external_id = f"cove-{account_id}-{run_ts}"` where `run_ts` is Unix timestamp from `D09F15` (fallback to `D09F09` when needed).

Deduplication is enforced per linked job:
- check `JobRun.query.filter_by(job_id=job.id, external_id=external_id).first()`
- this prevents cross-job collisions when accounts are relinked.

### Run Enrichment
- Cove-created `JobRun.remark` contains account/computer/customer and last status/timestamp summary.
- Per-datasource run object records include:
  - mapped Backupchecks status
  - readable status details in `error_message`
  - datasource-level session timestamp in `observed_at`

### Cove Accounts UI Notes
- `/cove/accounts` derives display fields to align with existing job logic:
  - `backup_software`: `Cove Data Protection`
  - `backup_type`: `Server`, `Workstation`, or `Microsoft 365`
  - `job_name`: based on Cove account/computer fallback
  - readable datasource labels instead of raw `I78` code stream
- `computer_name` is shown in both unmatched and matched account tables.

### Background Thread
`cove_importer_service.py` — same pattern as `auto_importer_service.py`:
- Thread name: `"cove_importer"`
- Checks `settings.cove_import_enabled`
- Interval: `settings.cove_import_interval_minutes` (default 30)
- Calls `run_cove_import(settings)` which returns `(total, created, skipped, errors)`

### Settings UI
Settings → Integrations → Cove section:
- Enable toggle, API URL, username, password (masked, only overwritten if non-empty)
- Import enabled + interval
- "Test Connection" button (AJAX → `POST /settings/cove/test-connection`) returns `{ok, partner_id, message}`
- "Run import now" button (→ `POST /settings/cove/run-now`) triggers manual import

### Routes
| Route | Method | Description |
|-------|--------|-------------|
| `/cove/accounts` | GET | Inbox-style page: unmatched + matched accounts |
| `/cove/accounts/<id>/link` | POST | `action=create` or `action=link` |
| `/cove/accounts/<id>/unlink` | POST | Removes job link, puts account back in unmatched |
| `/settings/cove/test-connection` | POST | AJAX: verify credentials, save partner_id |
| `/settings/cove/run-now` | POST | Manual import trigger |

### Migrations
- `migrate_cove_integration()` — adds 8 columns to `system_settings`, `cove_account_id` to `jobs`, `source_type` + `external_id` to `job_runs`, dedup index on `job_runs.external_id`
- `migrate_cove_accounts_table()` — creates `cove_accounts` table with indexes

---

## Ticketing and Autotask (Critical Rules)

### Two Ticket Types
1. **Internal Tickets** (tickets table)
   - Created manually or via Autotask integration
   - Stored in `tickets` table with `ticket_code` (e.g., "T20250123.0001")
   - Linked to runs via `ticket_job_runs` many-to-many table
   - Scoped to jobs via `ticket_scopes` table
   - Have `resolved_at` field for resolution tracking
   - **Auto-propagation**: Automatically linked to new runs via `link_open_internal_tickets_to_run`

2. **Autotask Tickets** (job_runs columns)
   - Created via Run Checks modal → "Create Autotask Ticket"
   - Stored directly in JobRun columns: `autotask_ticket_id`, `autotask_ticket_number`, etc.
- When created, also creates matching internal ticket for legacy UI compatibility

## Microsoft Entra SSO (Current State)

### Status
- Implemented but marked **Untested in Backupchecks**.

### Routes
- `GET /auth/entra/login` – starts Entra auth code flow.
- `GET /auth/entra/callback` – exchanges code, maps/provisions local user, logs in session.
- `/auth/logout` – Entra-aware logout redirect when user authenticated via Entra.

### Access Controls
- Optional tenant/domain restriction (`entra_allowed_domain`).
- Optional Entra security-group allowlist (`entra_allowed_group_ids`) based on group object IDs.
- Group overage / missing groups claim blocks login intentionally when group gate is enabled.

### Local User Mapping
- Primary mapping by `preferred_username`/UPN/email.
- Optional auto-provision (`entra_auto_provision_users`) creates local Viewer users for unknown identities.

### Documentation
- Built-in docs page: `/documentation/settings/entra-sso`
- Includes configuration steps and explicit untested warning.

## Navbar Notes (Latest)

- To reduce split-screen overflow, nav is compacted by grouping:
  - admin-only links under `Admin` dropdown
  - secondary non-admin links under `More` dropdown
- Primary operational links remain visible (notably `Run Checks`).
- Viewer role now exposes `Customers` and `Jobs` directly in navbar.
   - Have `autotask_ticket_deleted_at` field for deletion tracking
   - Resolution tracked via matching internal ticket's `resolved_at` field
   - **Auto-propagation**: Linked to new runs via two-strategy approach

### Ticket Propagation to New Runs
When a new JobRun is created (via email import OR missed run generation), `link_open_internal_tickets_to_run` ensures:

**Strategy 1: Internal ticket linking**
- Query finds tickets where: `COALESCE(ts.resolved_at, t.resolved_at) IS NULL`
- Creates `ticket_job_runs` links automatically
- Tickets remain visible until explicitly resolved
- **NO date-based logic** - resolved = immediately hidden from new runs

**Strategy 2: Autotask ticket propagation (independent)**
1. Check if internal ticket code exists → find matching Autotask run → copy ticket info
2. If no match, directly search for most recent Autotask ticket on job where:
   - `autotask_ticket_deleted_at IS NULL` (not deleted in PSA)
   - Internal ticket `resolved_at IS NULL` (not resolved in PSA)
3. Copy `autotask_ticket_id`, `autotask_ticket_number`, `created_at`, `created_by_user_id` to new run

### Where Ticket Linking is Called
`link_open_internal_tickets_to_run` is invoked in three locations:
1. **Email-based runs**: `routes_inbox.py` and `mail_importer.py` - after creating JobRun from parsed email
2. **Missed runs**: `routes_run_checks.py` in `_ensure_missed_runs_for_job` - after creating missed JobRun records
   - Weekly schedule: After creating weekly missed run (with flush to get run.id)
   - Monthly schedule: After creating monthly missed run (with flush to get run.id)
   - **Critical**: Without this call, missed runs don't get ticket propagation!

### Display Logic - Link-Based System
All pages use **explicit link-based queries** (no date-based logic):

**Job Details Page:**
- **Two sources** for ticket display:
  1. Direct links (`ticket_job_runs WHERE job_run_id = X`) → always show (audit trail)
  2. Active window (`ticket_scopes WHERE job_id = Y AND resolved_at IS NULL`) → only unresolved
- Result: Old runs keep their ticket references, new runs don't get resolved tickets

**Run Checks Main Page (Indicators 🎫):**
- Query: `ticket_scopes JOIN tickets WHERE job_id = X AND resolved_at IS NULL`
- Only shows indicator if unresolved tickets exist for the job

**Run Checks Popup Modal:**
- API: `/api/job-runs/<run_id>/alerts`
- **Two-source ticket display**:
  1. Direct links: `tickets JOIN ticket_job_runs WHERE job_run_id = X`
  2. Job-level scope: `tickets JOIN ticket_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date`
- Prevents duplicates by tracking seen ticket IDs
- Shows newly created tickets immediately (via scope) without waiting for resolve action
- **Two-source remark display**:
  1. Direct links: `remarks JOIN remark_job_runs WHERE job_run_id = X`
  2. Job-level scope: `remarks JOIN remark_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date` (with timezone-safe fallback from `start_date`)
- Prevents duplicates by tracking seen remark IDs

### Debug Logging for Ticket Linking (Reference)
If you need to debug ticket linking issues, add this to `link_open_internal_tickets_to_run` in `ticketing_utils.py` after the rows query:

```python
try:
    from .models import AuditLog
    details = []
    if rows:
        for tid, code, t_resolved, ts_resolved in rows:
            details.append(f"ticket_id={tid}, code={code}, t.resolved_at={t_resolved}, ts.resolved_at={ts_resolved}")
    else:
        details.append("No open tickets found for this job")
    audit = AuditLog(
        user="system", event_type="ticket_link_debug",
        message=f"link_open_internal_tickets_to_run called: run_id={run.id}, job_id={job.id}, found={len(rows)} ticket(s)",
        details="\n".join(details)
    )
    db.session.add(audit)
    db.session.commit()
except Exception:
    pass
```
Visible on Logging page under `event_type = "ticket_link_debug"`. Remove after debugging.

### Resolved vs Deleted
- **Resolved**: Ticket completed in Autotask (tracked in internal `tickets.resolved_at`)
  - Stops propagating to new runs
  - Ticket still exists in PSA
  - Synced via PSA polling
- **Deleted**: Ticket removed from Autotask (tracked in `job_runs.autotask_ticket_deleted_at`)
  - Also stops propagating
  - Ticket no longer exists in PSA
  - Rare operation

### Critical Rules
- ❌ **NEVER** use date-based resolved logic: `resolved_at >= run_date` OR `active_from_date <= run_date`
- ✅ Only show tickets that are ACTUALLY LINKED via `ticket_job_runs` table
- ✅ Resolved tickets stop linking immediately when resolved
- ✅ Old links preserved for audit trail (visible on old runs)
- ✅ All queries must use explicit JOIN to link tables
- ✅ Consistency: All pages use same "resolved = NULL" logic
- ✅ **CRITICAL**: Preserve description field during Autotask updates - must include "description" in optional_fields list

## UI and UX Notes

### Navbar
- Fixed-top positioning
- Collapses on mobile (hamburger menu)
- Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top)
- Role-based menu items (Admin sees more than Operator/Viewer)

### Status Badges
- Success: Green
- Warning: Yellow/Orange
- Failed/Error: Red
- Override applied: Blue badge
- Reviewed: Checkmark indicator

### Ticket Copy Functionality
- Copy button (⧉) available on both Run Checks and Job Details pages
- Allows quick copying of ticket numbers to clipboard
- Cross-browser compatible with three-tier fallback mechanism:
  1. **Modern Clipboard API**: `navigator.clipboard.writeText()` - works in modern browsers with HTTPS
  2. **Legacy execCommand**: `document.execCommand('copy')` - fallback for older browsers and Edge
  3. **Prompt fallback**: `window.prompt()` - last resort if clipboard access fails
- Visual feedback: button changes to ✓ checkmark for 800ms after successful copy
- Implementation uses hidden textarea for execCommand method to ensure compatibility
- No user interaction required in modern browsers (direct copy)

### Checkbox Behavior
- All checkboxes on Inbox and Run Checks pages use `autocomplete="off"`
- Prevents browser from auto-selecting checkboxes after page reload
- Fixes issue where deleting items would cause same number of new items to be selected

### Customers to Jobs Navigation (2026-02-16)
- Customers page links each customer name to filtered Jobs view:
  - `GET /jobs?customer_id=<customer_id>`
- Jobs route behavior:
  - Accepts optional `customer_id` query parameter in `routes_jobs.py`.
  - If set: returns jobs for that customer only.
  - If not set: keeps default filter that hides jobs linked to inactive customers.
- Jobs UI behavior:
  - Shows active filter banner with selected customer name.
  - Provides "Clear filter" action back to unfiltered `/jobs`.
- Templates touched:
  - `templates/main/customers.html`
  - `templates/main/jobs.html`

### Global Grouped Search (2026-02-16)
- New route:
  - `GET /search` in `main/routes_search.py`
- New UI:
  - Navbar search form in `templates/layout/base.html`
  - Grouped result page in `templates/main/search.html`
- Search behavior:
  - Case-insensitive matching (`ILIKE`).
  - `*` wildcard is supported and translated to SQL `%`.
  - Automatic contains behavior is applied per term (`*term*`) when wildcard not explicitly set.
  - Multi-term queries use AND across terms and OR across configured columns within each section.
  - Per-section pagination is supported via query params: `p_inbox`, `p_customers`, `p_jobs`, `p_daily_jobs`, `p_run_checks`, `p_tickets`, `p_remarks`, `p_overrides`, `p_reports`.
  - Pagination keeps search state for all sections while browsing one section.
  - "Open <section>" links pass `q` to destination overview pages so page-level filtering matches the search term.
- Grouped sections:
  - Inbox, Customers, Jobs, Daily Jobs, Run Checks, Tickets, Remarks, Existing overrides, Reports.
- Daily Jobs search result details:
  - Meta now includes expected run time, success indicator, and run count for the selected day.
  - Link now opens Daily Jobs with modal auto-open using `open_job_id` query parameter (same modal flow as clicking a row in Daily Jobs).
- Access control:
  - Search results are role-aware and only show sections/data the active role can access.
  - `run_checks` results are restricted to `admin`/`operator`.
  - `reports` supports `admin`/`operator`/`viewer`/`reporter`.
- Current performance strategy:
  - Per-section limit (`SEARCH_LIMIT_PER_SECTION = 10`), with total count per section.
  - No schema migration required for V1.

## Feedback Module with Screenshots
- Models: `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`.
- Attachments:
  - multiple uploads, type validation, per-file size limits, storage in database (BYTEA).

## Validation Snapshot
- 2026-02-16: Test build + push succeeded via `update-and-build.sh t`.
- Pushed image: `gitea.oskamp.info/ivooskamp/backupchecks:dev`.
- 2026-02-16: Test build + push succeeded on branch `v20260216-02-global-search`.
- Pushed image digest: `sha256:6996675b9529426fe2ad58b5f353479623f3ebe24b34552c17ad0421d8a7ee0f`.
- 2026-02-16: Additional test build + push cycles succeeded on `v20260216-02-global-search`.
- Latest pushed image digest: `sha256:8ec8bfcbb928e282182fa223ce8bf7f92112d20e79f4a8602d015991700df5d7`.
- 2026-02-16: Additional test build + push cycles succeeded after search enhancements.
- Latest pushed image digest: `sha256:b36b5cdd4bc7c4dadedca0534f1904a6e12b5b97abc4f12bc51e42921976f061`.
- Delete strategy:
  - soft delete by default,
  - permanent delete only for admins and only after soft delete.

## Deployment and Operations
- Stack exposes:
  - app on `8080`
  - adminer on `8081`
- PostgreSQL persistent volume:
  - `/docker/appdata/backupchecks/backupchecks-postgres:/var/lib/postgresql/data`
- `deploy/backupchecks-stack.yml` also contains example `.env` variables at the bottom.

## Build/Release Flow
File: `build-and-push.sh`
- Bump options:
  - `1` patch, `2` minor, `3` major, `t` test.
- Release build:
  - update `version.txt`
  - commit + tag + push
  - docker push of `:<version>`, `:dev`, `:latest`
- Test build:
  - only `:dev`
  - no commit/tag.
- Services are discovered under `containers/*` with Dockerfile-per-service.

## Technical Observations / Attention Points
- `README.md` is currently empty; quick-start entry context is missing.
- `LICENSE` is currently empty.
- `docs/architecture.md` is currently empty.
- `deploy/backupchecks-stack.yml` contains hardcoded example values (`Changeme`), with risk if used without proper secrets management.
- The app performs DB initialization + migrations at startup; for larger schema changes this can impact startup time/robustness.
- There is significant parser and ticketing complexity; route changes carry regression risk without targeted testing.
- For Autotask update calls, the `description` field must be explicitly preserved to prevent unintended NULL overwrite.
- Security hygiene remains important:
  - no customer names in parser examples/source,
  - no hardcoded credentials.

## Quick References
- App entrypoint: `containers/backupchecks/src/backend/app/main.py`
- App factory: `containers/backupchecks/src/backend/app/__init__.py`
- Config: `containers/backupchecks/src/backend/app/config.py`
- Models: `containers/backupchecks/src/backend/app/models.py`
- Parsers: `containers/backupchecks/src/backend/app/parsers/registry.py`
- Ticketing utilities: `containers/backupchecks/src/backend/app/ticketing_utils.py`
- Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py`
- Cove importer: `containers/backupchecks/src/backend/app/cove_importer.py`
- Cove routes: `containers/backupchecks/src/backend/app/main/routes_cove.py`
- Compose stack: `deploy/backupchecks-stack.yml`
- Build script: `build-and-push.sh`

## Recent Changes

### 2026-02-23
- **Cove Data Protection full integration**:
  - `cove_importer.py` – Cove API client (login, paginated enumeration, status mapping, deduplication, per-datasource object persistence)
  - `cove_importer_service.py` – background polling thread (same pattern as `auto_importer_service.py`)
  - `CoveAccount` staging model + `migrate_cove_accounts_table()` migration
  - `SystemSettings` – 8 new Cove fields, `Job` – `cove_account_id`, `JobRun` – `source_type` + `external_id`
  - `routes_cove.py` – inbox-style `/cove/accounts` with link/unlink routes
  - `cove_accounts.html` – unmatched accounts shown first with Bootstrap modals (create job / link to existing), matched accounts with Unlink
  - Settings > Integrations: Cove section with test connection (AJAX) and manual import trigger
  - Navbar: "Cove Accounts" link for admin/operator when `cove_enabled`
- **Cove API key findings** (from test script + N-able support):
  - Visa is returned at top level of login response, not inside `result`
  - Settings per account are a list of single-key dicts `[{"D09F00":"5"}, ...]` — flatten with `flat.update(item)`
  - EnumerateAccountStatistics params must use lowercase `query` key and `RecordsCount` (not `RecordCount`)
  - Login params must use lowercase `username`/`password`
  - D02/D03 are legacy; use D10/D11 or D09 (Total) instead

### 2026-02-19
- **Added 3CX Update parser support**: `threecx.py` now recognizes subject `3CX Notification: Update Successful - <host>` and stores it as informational with:
  - `backup_software = 3CX`
  - `backup_type = Update`
  - `overall_status = Success`
- **3CX informational schedule behavior**:
  - `3CX / Update` and `3CX / SSL Certificate` are excluded from schedule inference in `routes_shared.py` (no Expected/Missed generation).
- **Run Checks visibility scope (3CX-only)**:
  - Run Checks now hides only non-backup 3CX informational jobs (`Update`, `SSL Certificate`).
  - Other backup software/types remain visible and unchanged.
- **Fixed remark visibility mismatch**:
  - `/api/job-runs/<run_id>/alerts` now loads remarks from both:
    1. `remark_job_runs` (explicit run links),
    2. `remark_scopes` (active job-scoped remarks),
  - with duplicate prevention by remark ID.
  - This resolves cases where the remark indicator appeared but remarks were not shown in Run Checks modal or Job Details modal.

### 2026-02-13
- **Fixed missed runs ticket propagation**: Added `link_open_internal_tickets_to_run` calls in `_ensure_missed_runs_for_job` (routes_run_checks.py) after creating both weekly and monthly missed JobRun records. Previously only email-based runs got ticket linking, causing missed runs to not show internal tickets or Autotask tickets. Required `db.session.flush()` before linking to ensure run.id is available.
- **Fixed checkbox auto-selection**: Added `autocomplete="off"` to all checkboxes on Inbox and Run Checks pages. Prevents browser from automatically re-selecting checkboxes after page reload following delete actions.

### 2026-02-12
- **Fixed Run Checks modal ticket display**: Implemented two-source display logic (ticket_job_runs + ticket_scopes). Previously only showed tickets after they were resolved (when ticket_job_runs entry was created). Now shows tickets immediately upon creation via scope query.
- **Fixed copy button in Edge**: Moved clipboard functions inside IIFE scope for proper closure access (Edge is stricter than Firefox about scope resolution).

### 2026-02-10
- **Added screenshot support to Feedback system**: Multiple file upload, inline display, two-stage delete (soft delete for audit trail, permanent delete for cleanup).
- **Completed transition to link-based ticket system**: All pages now use JOIN queries, no date-based logic. Added cross-browser copy ticket functionality with three-tier fallback mechanism to both Run Checks and Job Details pages.