backupchecks/docs/technical-notes-codex.md

# Technical Notes (Internal)

Last updated: 2026-02-19

## Purpose
Internal technical snapshot of the `backupchecks` repository for faster onboarding, troubleshooting, and change impact analysis.

## Repository Overview
- Application: Flask web app with SQLAlchemy and Flask-Migrate.
- Runtime: Containerized (Docker), deployed via Docker Compose stack.
- Primary source code location: `containers/backupchecks/src`.
- The project also contains extensive functional documentation in `docs/` and multiple roadmap TODO files at repository root.

## Main Structure
- `containers/backupchecks/Dockerfile`: Python 3.12-slim image, starts `gunicorn` with `backend.app:create_app()`.
- `containers/backupchecks/requirements.txt`: Flask stack + PostgreSQL driver + reporting libraries (`reportlab`, `Markdown`).
- `containers/backupchecks/src/backend/app`: backend domain logic, routes, parsers, models, migrations.
- `containers/backupchecks/src/templates`: Jinja templates for auth/main/documentation pages.
- `containers/backupchecks/src/static`: CSS, images, favicon.
- `deploy/backupchecks-stack.yml`: compose stack with `backupchecks`, `postgres`, `adminer`.
- `build-and-push.sh`: release/test build script with version bumping, tags, and image push.
- `docs/`: functional design, changelogs, migration notes, API notes.

## Application Architecture (Current Observation)
- Factory pattern: `create_app()` in `containers/backupchecks/src/backend/app/__init__.py`.
- Blueprints:
  - `auth_bp` for authentication.
  - `main_bp` for core functionality.
  - `doc_bp` for internal documentation pages.
- Database initialization at startup:
  - `db.create_all()`
  - `run_migrations()`
- Background task:
  - `start_auto_importer(app)` starts the automatic mail importer thread.
- Health endpoint:
  - `GET /health` returns `{ "status": "ok" }`.

## Functional Processing Flow
- Import:
  - Email is fetched via Microsoft Graph API.
- Parse:
  - Parser selection through registry + software-specific parser implementations.
- Approve:
  - New jobs first appear in Inbox for initial customer assignment.
- Auto-process:
  - Subsequent emails for known jobs automatically create `JobRun` records.
- Monitor:
  - Runs appear in Daily Jobs and Run Checks.
- Review:
  - Manual review removes items from the unreviewed operational queue.

## Configuration and Runtime
- Config is built from environment variables in `containers/backupchecks/src/backend/app/config.py`.
- Important variables:
  - `APP_SECRET_KEY`
  - `APP_ENV`
  - `APP_PORT`
  - `POSTGRES_DB`
  - `POSTGRES_USER`
  - `POSTGRES_PASSWORD`
  - `DB_HOST`
  - `DB_PORT`
- Database URI pattern:
  - `postgresql+psycopg2://<user>:<pass>@<host>:<port>/<db>`
- Default timezone in config: `Europe/Amsterdam`.

## Data Model (High-level)
File: `containers/backupchecks/src/backend/app/models.py`
- Auth/users:
  - `User` with role(s), active role in session.
- System settings:
  - `SystemSettings` with Graph/mail settings, import settings, UI timezone, dashboard policy, sandbox flag.
  - Autotask configuration and cache fields are present.
- Logging:
  - `AuditLog` (legacy alias `AdminLog`).
- Domain:
  - `Customer`, `Job`, `JobRun`, `Override`
  - `MailMessage`, `MailObject`
  - `Ticket`, `TicketScope`, `TicketJobRun`
  - `Remark`, `RemarkScope`, `RemarkJobRun`
  - `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`

### Foreign Key Relationships & Deletion Order
Critical deletion order to avoid constraint violations:
1. Clean auxiliary tables (ticket_job_runs, remark_job_runs, scopes, overrides)
2. Unlink mails from jobs (UPDATE mail_messages SET job_id = NULL)
3. Delete mail_objects
4. Delete jobs (cascades to job_runs)
5. Delete mails

### Key Model Fields
**MailMessage model:**
- `from_address` (NOT `sender`!) - sender email
- `subject` - email subject
- `text_body` - plain text content
- `html_body` - HTML content
- `received_at` - timestamp
- `location` - inbox/processed/deleted
- `job_id` - link to Job (nullable)

**Job model:**
- `customer_id` - FK to Customer
- `job_name` - parsed from email
- `backup_software` - e.g., "Veeam", "Synology"
- `backup_type` - e.g., "Backup Job", "Active Backup"

## Parser Architecture
- Folder: `containers/backupchecks/src/backend/app/parsers/`
- Two layers:
  - `registry.py`:
    - matching/documentation/visibility on `/parsers`.
    - examples must stay generic (no customer names).
  - parser files (`veeam.py`, `synology.py`, etc.):
    - actual detection and parsing logic.
    - return structured output: software, type, job name, status, objects.
- Practical rule:
  - extend patterns by adding, not replacing (backward compatibility).

### Parser Types
**Informational Parsers:**
- DSM Updates, Account Protection, Firmware Updates
- Set appropriate backup_type (e.g., "Updates", "Firmware Update")
- Do NOT participate in schedule learning
- Usually still visible in Run Checks for awareness
- Exception: non-backup 3CX informational types (`Update`, `SSL Certificate`) are hidden from Run Checks

**Regular Parsers:**
- Backup jobs (Veeam, Synology Active Backup, NAKIVO, etc.)
- Participate in schedule learning (daily/weekly/monthly detection)
- Generate missed runs when expected runs don't occur

**Example: Synology Updates Parser (synology.py)**
- Handles multiple update notification types under same job:
  - DSM automatic update cancelled
  - Packages out-of-date
  - Combined notifications (DSM + packages)
- Detection patterns:
  - DSM: "Automatische DSM-update", "DSM-update op", "automatic DSM update"
  - Packages: "Packages on", "out-of-date", "Package Center"
- Hostname extraction from multiple patterns
- Returns: backup_type "Updates", job_name "Synology Automatic Update"

## Ticketing and Autotask (Critical Rules)

### Two Ticket Types
1. **Internal Tickets** (tickets table)
   - Created manually or via Autotask integration
   - Stored in `tickets` table with `ticket_code` (e.g., "T20250123.0001")
   - Linked to runs via `ticket_job_runs` many-to-many table
   - Scoped to jobs via `ticket_scopes` table
   - Have `resolved_at` field for resolution tracking
   - **Auto-propagation**: Automatically linked to new runs via `link_open_internal_tickets_to_run`

2. **Autotask Tickets** (job_runs columns)
   - Created via Run Checks modal → "Create Autotask Ticket"
   - Stored directly in JobRun columns: `autotask_ticket_id`, `autotask_ticket_number`, etc.
   - When created, also creates matching internal ticket for legacy UI compatibility
   - Have `autotask_ticket_deleted_at` field for deletion tracking
   - Resolution tracked via matching internal ticket's `resolved_at` field
   - **Auto-propagation**: Linked to new runs via two-strategy approach

### Ticket Propagation to New Runs
When a new JobRun is created (via email import OR missed run generation), `link_open_internal_tickets_to_run` ensures:

**Strategy 1: Internal ticket linking**
- Query finds tickets where: `COALESCE(ts.resolved_at, t.resolved_at) IS NULL`
- Creates `ticket_job_runs` links automatically
- Tickets remain visible until explicitly resolved
- **NO date-based logic** - resolved = immediately hidden from new runs

**Strategy 2: Autotask ticket propagation (independent)**
1. Check if internal ticket code exists → find matching Autotask run → copy ticket info
2. If no match, directly search for most recent Autotask ticket on job where:
   - `autotask_ticket_deleted_at IS NULL` (not deleted in PSA)
   - Internal ticket `resolved_at IS NULL` (not resolved in PSA)
3. Copy `autotask_ticket_id`, `autotask_ticket_number`, `created_at`, `created_by_user_id` to new run

### Where Ticket Linking is Called
`link_open_internal_tickets_to_run` is invoked in three locations:
1. **Email-based runs**: `routes_inbox.py` and `mail_importer.py` - after creating JobRun from parsed email
2. **Missed runs**: `routes_run_checks.py` in `_ensure_missed_runs_for_job` - after creating missed JobRun records
   - Weekly schedule: After creating weekly missed run (with flush to get run.id)
   - Monthly schedule: After creating monthly missed run (with flush to get run.id)
   - **Critical**: Without this call, missed runs don't get ticket propagation!

### Display Logic - Link-Based System
All pages use **explicit link-based queries** (no date-based logic):

**Job Details Page:**
- **Two sources** for ticket display:
  1. Direct links (`ticket_job_runs WHERE job_run_id = X`) → always show (audit trail)
  2. Active window (`ticket_scopes WHERE job_id = Y AND resolved_at IS NULL`) → only unresolved
- Result: Old runs keep their ticket references, new runs don't get resolved tickets

**Run Checks Main Page (Indicators 🎫):**
- Query: `ticket_scopes JOIN tickets WHERE job_id = X AND resolved_at IS NULL`
- Only shows indicator if unresolved tickets exist for the job

**Run Checks Popup Modal:**
- API: `/api/job-runs/<run_id>/alerts`
- **Two-source ticket display**:
  1. Direct links: `tickets JOIN ticket_job_runs WHERE job_run_id = X`
  2. Job-level scope: `tickets JOIN ticket_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date`
- Prevents duplicates by tracking seen ticket IDs
- Shows newly created tickets immediately (via scope) without waiting for resolve action
- **Two-source remark display**:
  1. Direct links: `remarks JOIN remark_job_runs WHERE job_run_id = X`
  2. Job-level scope: `remarks JOIN remark_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date` (with timezone-safe fallback from `start_date`)
- Prevents duplicates by tracking seen remark IDs

### Resolved vs Deleted
- **Resolved**: Ticket completed in Autotask (tracked in internal `tickets.resolved_at`)
  - Stops propagating to new runs
  - Ticket still exists in PSA
  - Synced via PSA polling
- **Deleted**: Ticket removed from Autotask (tracked in `job_runs.autotask_ticket_deleted_at`)
  - Also stops propagating
  - Ticket no longer exists in PSA
  - Rare operation

### Critical Rules
- ❌ **NEVER** use date-based resolved logic: `resolved_at >= run_date` OR `active_from_date <= run_date`
- ✅ Only show tickets that are ACTUALLY LINKED via `ticket_job_runs` table
- ✅ Resolved tickets stop linking immediately when resolved
- ✅ Old links preserved for audit trail (visible on old runs)
- ✅ All queries must use explicit JOIN to link tables
- ✅ Consistency: All pages use same "resolved = NULL" logic
- ✅ **CRITICAL**: Preserve description field during Autotask updates - must include "description" in optional_fields list

## UI and UX Notes

### Navbar
- Fixed-top positioning
- Collapses on mobile (hamburger menu)
- Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top)
- Role-based menu items (Admin sees more than Operator/Viewer)

### Status Badges
- Success: Green
- Warning: Yellow/Orange
- Failed/Error: Red
- Override applied: Blue badge
- Reviewed: Checkmark indicator

### Ticket Copy Functionality
- Copy button (⧉) available on both Run Checks and Job Details pages
- Allows quick copying of ticket numbers to clipboard
- Cross-browser compatible with three-tier fallback mechanism:
  1. **Modern Clipboard API**: `navigator.clipboard.writeText()` - works in modern browsers with HTTPS
  2. **Legacy execCommand**: `document.execCommand('copy')` - fallback for older browsers and Edge
  3. **Prompt fallback**: `window.prompt()` - last resort if clipboard access fails
- Visual feedback: button changes to ✓ checkmark for 800ms after successful copy
- Implementation uses hidden textarea for execCommand method to ensure compatibility
- No user interaction required in modern browsers (direct copy)

### Checkbox Behavior
- All checkboxes on Inbox and Run Checks pages use `autocomplete="off"`
- Prevents browser from auto-selecting checkboxes after page reload
- Fixes issue where deleting items would cause same number of new items to be selected

### Customers to Jobs Navigation (2026-02-16)
- Customers page links each customer name to filtered Jobs view:
  - `GET /jobs?customer_id=<customer_id>`
- Jobs route behavior:
  - Accepts optional `customer_id` query parameter in `routes_jobs.py`.
  - If set: returns jobs for that customer only.
  - If not set: keeps default filter that hides jobs linked to inactive customers.
- Jobs UI behavior:
  - Shows active filter banner with selected customer name.
  - Provides "Clear filter" action back to unfiltered `/jobs`.
- Templates touched:
  - `templates/main/customers.html`
  - `templates/main/jobs.html`

### Global Grouped Search (2026-02-16)
- New route:
  - `GET /search` in `main/routes_search.py`
- New UI:
  - Navbar search form in `templates/layout/base.html`
  - Grouped result page in `templates/main/search.html`
- Search behavior:
  - Case-insensitive matching (`ILIKE`).
  - `*` wildcard is supported and translated to SQL `%`.
  - Automatic contains behavior is applied per term (`*term*`) when wildcard not explicitly set.
  - Multi-term queries use AND across terms and OR across configured columns within each section.
  - Per-section pagination is supported via query params: `p_inbox`, `p_customers`, `p_jobs`, `p_daily_jobs`, `p_run_checks`, `p_tickets`, `p_remarks`, `p_overrides`, `p_reports`.
  - Pagination keeps search state for all sections while browsing one section.
  - "Open <section>" links pass `q` to destination overview pages so page-level filtering matches the search term.
- Grouped sections:
  - Inbox, Customers, Jobs, Daily Jobs, Run Checks, Tickets, Remarks, Existing overrides, Reports.
- Daily Jobs search result details:
  - Meta now includes expected run time, success indicator, and run count for the selected day.
  - Link now opens Daily Jobs with modal auto-open using `open_job_id` query parameter (same modal flow as clicking a row in Daily Jobs).
- Access control:
  - Search results are role-aware and only show sections/data the active role can access.
  - `run_checks` results are restricted to `admin`/`operator`.
  - `reports` supports `admin`/`operator`/`viewer`/`reporter`.
- Current performance strategy:
  - Per-section limit (`SEARCH_LIMIT_PER_SECTION = 10`), with total count per section.
  - No schema migration required for V1.

## Feedback Module with Screenshots
- Models: `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`.
- Attachments:
  - multiple uploads, type validation, per-file size limits, storage in database (BYTEA).

## Validation Snapshot
- 2026-02-16: Test build + push succeeded via `update-and-build.sh t`.
- Pushed image: `gitea.oskamp.info/ivooskamp/backupchecks:dev`.
- 2026-02-16: Test build + push succeeded on branch `v20260216-02-global-search`.
- Pushed image digest: `sha256:6996675b9529426fe2ad58b5f353479623f3ebe24b34552c17ad0421d8a7ee0f`.
- 2026-02-16: Additional test build + push cycles succeeded on `v20260216-02-global-search`.
- Latest pushed image digest: `sha256:8ec8bfcbb928e282182fa223ce8bf7f92112d20e79f4a8602d015991700df5d7`.
- 2026-02-16: Additional test build + push cycles succeeded after search enhancements.
- Latest pushed image digest: `sha256:b36b5cdd4bc7c4dadedca0534f1904a6e12b5b97abc4f12bc51e42921976f061`.
- Delete strategy:
  - soft delete by default,
  - permanent delete only for admins and only after soft delete.

## Deployment and Operations
- Stack exposes:
  - app on `8080`
  - adminer on `8081`
- PostgreSQL persistent volume:
  - `/docker/appdata/backupchecks/backupchecks-postgres:/var/lib/postgresql/data`
- `deploy/backupchecks-stack.yml` also contains example `.env` variables at the bottom.

## Build/Release Flow
File: `build-and-push.sh`
- Bump options:
  - `1` patch, `2` minor, `3` major, `t` test.
- Release build:
  - update `version.txt`
  - commit + tag + push
  - docker push of `:<version>`, `:dev`, `:latest`
- Test build:
  - only `:dev`
  - no commit/tag.
- Services are discovered under `containers/*` with Dockerfile-per-service.

## Technical Observations / Attention Points
- `README.md` is currently empty; quick-start entry context is missing.
- `LICENSE` is currently empty.
- `docs/architecture.md` is currently empty.
- `deploy/backupchecks-stack.yml` contains hardcoded example values (`Changeme`), with risk if used without proper secrets management.
- The app performs DB initialization + migrations at startup; for larger schema changes this can impact startup time/robustness.
- There is significant parser and ticketing complexity; route changes carry regression risk without targeted testing.
- For Autotask update calls, the `description` field must be explicitly preserved to prevent unintended NULL overwrite.
- Security hygiene remains important:
  - no customer names in parser examples/source,
  - no hardcoded credentials.

## Quick References
- App entrypoint: `containers/backupchecks/src/backend/app/main.py`
- App factory: `containers/backupchecks/src/backend/app/__init__.py`
- Config: `containers/backupchecks/src/backend/app/config.py`
- Models: `containers/backupchecks/src/backend/app/models.py`
- Parsers: `containers/backupchecks/src/backend/app/parsers/registry.py`
- Ticketing utilities: `containers/backupchecks/src/backend/app/ticketing_utils.py`
- Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py`
- Compose stack: `deploy/backupchecks-stack.yml`
- Build script: `build-and-push.sh`

## Recent Changes

### 2026-02-19
- **Added 3CX Update parser support**: `threecx.py` now recognizes subject `3CX Notification: Update Successful - <host>` and stores it as informational with:
  - `backup_software = 3CX`
  - `backup_type = Update`
  - `overall_status = Success`
- **3CX informational schedule behavior**:
  - `3CX / Update` and `3CX / SSL Certificate` are excluded from schedule inference in `routes_shared.py` (no Expected/Missed generation).
- **Run Checks visibility scope (3CX-only)**:
  - Run Checks now hides only non-backup 3CX informational jobs (`Update`, `SSL Certificate`).
  - Other backup software/types remain visible and unchanged.
- **Fixed remark visibility mismatch**:
  - `/api/job-runs/<run_id>/alerts` now loads remarks from both:
    1. `remark_job_runs` (explicit run links),
    2. `remark_scopes` (active job-scoped remarks),
  - with duplicate prevention by remark ID.
  - This resolves cases where the remark indicator appeared but remarks were not shown in Run Checks modal or Job Details modal.

### 2026-02-13
- **Fixed missed runs ticket propagation**: Added `link_open_internal_tickets_to_run` calls in `_ensure_missed_runs_for_job` (routes_run_checks.py) after creating both weekly and monthly missed JobRun records. Previously only email-based runs got ticket linking, causing missed runs to not show internal tickets or Autotask tickets. Required `db.session.flush()` before linking to ensure run.id is available.
- **Fixed checkbox auto-selection**: Added `autocomplete="off"` to all checkboxes on Inbox and Run Checks pages. Prevents browser from automatically re-selecting checkboxes after page reload following delete actions.

### 2026-02-12
- **Fixed Run Checks modal ticket display**: Implemented two-source display logic (ticket_job_runs + ticket_scopes). Previously only showed tickets after they were resolved (when ticket_job_runs entry was created). Now shows tickets immediately upon creation via scope query.
- **Fixed copy button in Edge**: Moved clipboard functions inside IIFE scope for proper closure access (Edge is stricter than Firefox about scope resolution).

### 2026-02-10
- **Added screenshot support to Feedback system**: Multiple file upload, inline display, two-stage delete (soft delete for audit trail, permanent delete for cleanup).
- **Completed transition to link-based ticket system**: All pages now use JOIN queries, no date-based logic. Added cross-browser copy ticket functionality with three-tier fallback mechanism to both Run Checks and Job Details pages.