backupchecks/docs/technical-notes-codex.md

Technical Notes (Internal)

Last updated: 2026-02-16

Purpose

Internal technical snapshot of the backupchecks repository for faster onboarding, troubleshooting, and change impact analysis.

Repository Overview

• Application: Flask web app with SQLAlchemy and Flask-Migrate.
• Runtime: Containerized (Docker), deployed via Docker Compose stack.
• Primary source code location: containers/backupchecks/src.
• The project also contains extensive functional documentation in docs/ and multiple roadmap TODO files at repository root.

Main Structure

• containers/backupchecks/Dockerfile: Python 3.12-slim image, starts gunicorn with backend.app:create_app().
• containers/backupchecks/requirements.txt: Flask stack + PostgreSQL driver + reporting libraries (reportlab, Markdown).
• containers/backupchecks/src/backend/app: backend domain logic, routes, parsers, models, migrations.
• containers/backupchecks/src/templates: Jinja templates for auth/main/documentation pages.
• containers/backupchecks/src/static: CSS, images, favicon.
• deploy/backupchecks-stack.yml: compose stack with backupchecks, postgres, adminer.
• build-and-push.sh: release/test build script with version bumping, tags, and image push.
• docs/: functional design, changelogs, migration notes, API notes.

Application Architecture (Current Observation)

• Factory pattern: create_app() in containers/backupchecks/src/backend/app/__init__.py.
• Blueprints:
  • auth_bp for authentication.
  • main_bp for core functionality.
  • doc_bp for internal documentation pages.
• Database initialization at startup:
  • db.create_all()
  • run_migrations()
• Background task:
  • start_auto_importer(app) starts the automatic mail importer thread.
• Health endpoint:
  • GET /health returns { "status": "ok" }.

Functional Processing Flow

• Import:
  • Email is fetched via Microsoft Graph API.
• Parse:
  • Parser selection through registry + software-specific parser implementations.
• Approve:
  • New jobs first appear in Inbox for initial customer assignment.
• Auto-process:
  • Subsequent emails for known jobs automatically create JobRun records.
• Monitor:
  • Runs appear in Daily Jobs and Run Checks.
• Review:
  • Manual review removes items from the unreviewed operational queue.

Configuration and Runtime

• Config is built from environment variables in containers/backupchecks/src/backend/app/config.py.
• Important variables:
  • APP_SECRET_KEY
  • APP_ENV
  • APP_PORT
  • POSTGRES_DB
  • POSTGRES_USER
  • POSTGRES_PASSWORD
  • DB_HOST
  • DB_PORT
• Database URI pattern:
  • postgresql+psycopg2://<user>:<pass>@<host>:<port>/<db>
• Default timezone in config: Europe/Amsterdam.

Data Model (High-level)

File: containers/backupchecks/src/backend/app/models.py

• Auth/users:
  • User with role(s), active role in session.
• System settings:
  • SystemSettings with Graph/mail settings, import settings, UI timezone, dashboard policy, sandbox flag.
  • Autotask configuration and cache fields are present.
• Logging:
  • AuditLog (legacy alias AdminLog).
• Domain:
  • Customer, Job, JobRun, Override
  • MailMessage, MailObject
  • Ticket, TicketScope, TicketJobRun
  • Remark, RemarkScope, RemarkJobRun
  • FeedbackItem, FeedbackVote, FeedbackReply, FeedbackAttachment

Foreign Key Relationships & Deletion Order

Critical deletion order to avoid constraint violations:

1. Clean auxiliary tables (ticket_job_runs, remark_job_runs, scopes, overrides)
2. Unlink mails from jobs (UPDATE mail_messages SET job_id = NULL)
3. Delete mail_objects
4. Delete jobs (cascades to job_runs)
5. Delete mails

Key Model Fields

MailMessage model:

• from_address (NOT sender!) - sender email
• subject - email subject
• text_body - plain text content
• html_body - HTML content
• received_at - timestamp
• location - inbox/processed/deleted
• job_id - link to Job (nullable)

Job model:

• customer_id - FK to Customer
• job_name - parsed from email
• backup_software - e.g., "Veeam", "Synology"
• backup_type - e.g., "Backup Job", "Active Backup"

Parser Architecture

• Folder: containers/backupchecks/src/backend/app/parsers/
• Two layers:
  • registry.py:
    • matching/documentation/visibility on /parsers.
    • examples must stay generic (no customer names).
  • parser files (veeam.py, synology.py, etc.):
    • actual detection and parsing logic.
    • return structured output: software, type, job name, status, objects.
• Practical rule:
  • extend patterns by adding, not replacing (backward compatibility).

Parser Types

Informational Parsers:

• DSM Updates, Account Protection, Firmware Updates
• Set appropriate backup_type (e.g., "Updates", "Firmware Update")
• Do NOT participate in schedule learning
• Still visible in Run Checks for awareness

Regular Parsers:

• Backup jobs (Veeam, Synology Active Backup, NAKIVO, etc.)
• Participate in schedule learning (daily/weekly/monthly detection)
• Generate missed runs when expected runs don't occur

Example: Synology Updates Parser (synology.py)

• Handles multiple update notification types under same job:
  • DSM automatic update cancelled
  • Packages out-of-date
  • Combined notifications (DSM + packages)
• Detection patterns:
  • DSM: "Automatische DSM-update", "DSM-update op", "automatic DSM update"
  • Packages: "Packages on", "out-of-date", "Package Center"
• Hostname extraction from multiple patterns
• Returns: backup_type "Updates", job_name "Synology Automatic Update"

Ticketing and Autotask (Critical Rules)

Two Ticket Types

1. Internal Tickets (tickets table)

   • Created manually or via Autotask integration
   • Stored in tickets table with ticket_code (e.g., "T20250123.0001")
   • Linked to runs via ticket_job_runs many-to-many table
   • Scoped to jobs via ticket_scopes table
   • Have resolved_at field for resolution tracking
   • Auto-propagation: Automatically linked to new runs via link_open_internal_tickets_to_run

2. Autotask Tickets (job_runs columns)

   • Created via Run Checks modal → "Create Autotask Ticket"
   • Stored directly in JobRun columns: autotask_ticket_id, autotask_ticket_number, etc.
   • When created, also creates matching internal ticket for legacy UI compatibility
   • Have autotask_ticket_deleted_at field for deletion tracking
   • Resolution tracked via matching internal ticket's resolved_at field
   • Auto-propagation: Linked to new runs via two-strategy approach

Ticket Propagation to New Runs

When a new JobRun is created (via email import OR missed run generation), link_open_internal_tickets_to_run ensures:

Strategy 1: Internal ticket linking

• Query finds tickets where: COALESCE(ts.resolved_at, t.resolved_at) IS NULL
• Creates ticket_job_runs links automatically
• Tickets remain visible until explicitly resolved
• NO date-based logic - resolved = immediately hidden from new runs

Strategy 2: Autotask ticket propagation (independent)

1. Check if internal ticket code exists → find matching Autotask run → copy ticket info
2. If no match, directly search for most recent Autotask ticket on job where:
   • autotask_ticket_deleted_at IS NULL (not deleted in PSA)
   • Internal ticket resolved_at IS NULL (not resolved in PSA)
3. Copy autotask_ticket_id, autotask_ticket_number, created_at, created_by_user_id to new run

Where Ticket Linking is Called

link_open_internal_tickets_to_run is invoked in three locations:

1. Email-based runs: routes_inbox.py and mail_importer.py - after creating JobRun from parsed email
2. Missed runs: routes_run_checks.py in _ensure_missed_runs_for_job - after creating missed JobRun records
   • Weekly schedule: After creating weekly missed run (with flush to get run.id)
   • Monthly schedule: After creating monthly missed run (with flush to get run.id)
   • Critical: Without this call, missed runs don't get ticket propagation!

Display Logic - Link-Based System

All pages use explicit link-based queries (no date-based logic):

Job Details Page:

• Two sources for ticket display:
  1. Direct links (ticket_job_runs WHERE job_run_id = X) → always show (audit trail)
  2. Active window (ticket_scopes WHERE job_id = Y AND resolved_at IS NULL) → only unresolved
• Result: Old runs keep their ticket references, new runs don't get resolved tickets

Run Checks Main Page (Indicators 🎫):

• Query: ticket_scopes JOIN tickets WHERE job_id = X AND resolved_at IS NULL
• Only shows indicator if unresolved tickets exist for the job

Run Checks Popup Modal:

• API: /api/job-runs/<run_id>/alerts
• Two-source ticket display:
  1. Direct links: tickets JOIN ticket_job_runs WHERE job_run_id = X
  2. Job-level scope: tickets JOIN ticket_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date
• Prevents duplicates by tracking seen ticket IDs
• Shows newly created tickets immediately (via scope) without waiting for resolve action
• Same for remarks: remarks JOIN remark_job_runs WHERE job_run_id = X

Resolved vs Deleted

• Resolved: Ticket completed in Autotask (tracked in internal tickets.resolved_at)
  • Stops propagating to new runs
  • Ticket still exists in PSA
  • Synced via PSA polling
• Deleted: Ticket removed from Autotask (tracked in job_runs.autotask_ticket_deleted_at)
  • Also stops propagating
  • Ticket no longer exists in PSA
  • Rare operation

Critical Rules

• ❌ NEVER use date-based resolved logic: resolved_at >= run_date OR active_from_date <= run_date
• ✅ Only show tickets that are ACTUALLY LINKED via ticket_job_runs table
• ✅ Resolved tickets stop linking immediately when resolved
• ✅ Old links preserved for audit trail (visible on old runs)
• ✅ All queries must use explicit JOIN to link tables
• ✅ Consistency: All pages use same "resolved = NULL" logic
• ✅ CRITICAL: Preserve description field during Autotask updates - must include "description" in optional_fields list

UI and UX Notes

Navbar

• Fixed-top positioning
• Collapses on mobile (hamburger menu)
• Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top)
• Role-based menu items (Admin sees more than Operator/Viewer)

Status Badges

• Success: Green
• Warning: Yellow/Orange
• Failed/Error: Red
• Override applied: Blue badge
• Reviewed: Checkmark indicator

Ticket Copy Functionality

• Copy button (⧉) available on both Run Checks and Job Details pages
• Allows quick copying of ticket numbers to clipboard
• Cross-browser compatible with three-tier fallback mechanism:
  1. Modern Clipboard API: navigator.clipboard.writeText() - works in modern browsers with HTTPS
  2. Legacy execCommand: document.execCommand('copy') - fallback for older browsers and Edge
  3. Prompt fallback: window.prompt() - last resort if clipboard access fails
• Visual feedback: button changes to ✓ checkmark for 800ms after successful copy
• Implementation uses hidden textarea for execCommand method to ensure compatibility
• No user interaction required in modern browsers (direct copy)

Checkbox Behavior

• All checkboxes on Inbox and Run Checks pages use autocomplete="off"
• Prevents browser from auto-selecting checkboxes after page reload
• Fixes issue where deleting items would cause same number of new items to be selected

Customers to Jobs Navigation (2026-02-16)

• Customers page links each customer name to filtered Jobs view:
  • GET /jobs?customer_id=<customer_id>
• Jobs route behavior:
  • Accepts optional customer_id query parameter in routes_jobs.py.
  • If set: returns jobs for that customer only.
  • If not set: keeps default filter that hides jobs linked to inactive customers.
• Jobs UI behavior:
  • Shows active filter banner with selected customer name.
  • Provides "Clear filter" action back to unfiltered /jobs.
• Templates touched:
  • templates/main/customers.html
  • templates/main/jobs.html

Feedback Module with Screenshots

• Models: FeedbackItem, FeedbackVote, FeedbackReply, FeedbackAttachment.
• Attachments:
  • multiple uploads, type validation, per-file size limits, storage in database (BYTEA).

Validation Snapshot

• 2026-02-16: Test build + push succeeded via update-and-build.sh t.
• Pushed image: gitea.oskamp.info/ivooskamp/backupchecks:dev.
• Delete strategy:
  • soft delete by default,
  • permanent delete only for admins and only after soft delete.

Deployment and Operations

• Stack exposes:
  • app on 8080
  • adminer on 8081
• PostgreSQL persistent volume:
  • /docker/appdata/backupchecks/backupchecks-postgres:/var/lib/postgresql/data
• deploy/backupchecks-stack.yml also contains example .env variables at the bottom.

Build/Release Flow

File: build-and-push.sh

• Bump options:
  • 1 patch, 2 minor, 3 major, t test.
• Release build:
  • update version.txt
  • commit + tag + push
  • docker push of :<version>, :dev, :latest
• Test build:
  • only :dev
  • no commit/tag.
• Services are discovered under containers/* with Dockerfile-per-service.

Technical Observations / Attention Points

• README.md is currently empty; quick-start entry context is missing.
• LICENSE is currently empty.
• docs/architecture.md is currently empty.
• deploy/backupchecks-stack.yml contains hardcoded example values (Changeme), with risk if used without proper secrets management.
• The app performs DB initialization + migrations at startup; for larger schema changes this can impact startup time/robustness.
• There is significant parser and ticketing complexity; route changes carry regression risk without targeted testing.
• For Autotask update calls, the description field must be explicitly preserved to prevent unintended NULL overwrite.
• Security hygiene remains important:
  • no customer names in parser examples/source,
  • no hardcoded credentials.

Quick References

• App entrypoint: containers/backupchecks/src/backend/app/main.py
• App factory: containers/backupchecks/src/backend/app/__init__.py
• Config: containers/backupchecks/src/backend/app/config.py
• Models: containers/backupchecks/src/backend/app/models.py
• Parsers: containers/backupchecks/src/backend/app/parsers/registry.py
• Ticketing utilities: containers/backupchecks/src/backend/app/ticketing_utils.py
• Run Checks routes: containers/backupchecks/src/backend/app/main/routes_run_checks.py
• Compose stack: deploy/backupchecks-stack.yml
• Build script: build-and-push.sh

Recent Changes

2026-02-13

• Fixed missed runs ticket propagation: Added link_open_internal_tickets_to_run calls in _ensure_missed_runs_for_job (routes_run_checks.py) after creating both weekly and monthly missed JobRun records. Previously only email-based runs got ticket linking, causing missed runs to not show internal tickets or Autotask tickets. Required db.session.flush() before linking to ensure run.id is available.
• Fixed checkbox auto-selection: Added autocomplete="off" to all checkboxes on Inbox and Run Checks pages. Prevents browser from automatically re-selecting checkboxes after page reload following delete actions.

2026-02-12

• Fixed Run Checks modal ticket display: Implemented two-source display logic (ticket_job_runs + ticket_scopes). Previously only showed tickets after they were resolved (when ticket_job_runs entry was created). Now shows tickets immediately upon creation via scope query.
• Fixed copy button in Edge: Moved clipboard functions inside IIFE scope for proper closure access (Edge is stricter than Firefox about scope resolution).

2026-02-10

• Added screenshot support to Feedback system: Multiple file upload, inline display, two-stage delete (soft delete for audit trail, permanent delete for cleanup).
• Completed transition to link-based ticket system: All pages now use JOIN queries, no date-based logic. Added cross-browser copy ticket functionality with three-tier fallback mechanism to both Run Checks and Job Details pages.