ivooskamp/backupchecks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3dae203cf6
backupchecks/docs/technical-notes-codex.md

47 KiB
Raw Blame History

Technical Notes (Internal)

Last updated: 2026-03-23

Purpose

Internal technical snapshot of the backupchecks repository for faster onboarding, troubleshooting, and change impact analysis.

Repository Overview

  • Application: Flask web app with SQLAlchemy and Flask-Migrate.
  • Runtime: Containerized (Docker), deployed via Docker Compose stack.
  • Primary source code location: containers/backupchecks/src.
  • The project also contains extensive functional documentation in docs/ and multiple roadmap TODO files at repository root.

Main Structure

  • containers/backupchecks/Dockerfile: Python 3.12-slim image, starts gunicorn with backend.app:create_app().
  • containers/backupchecks/requirements.txt: Flask stack + PostgreSQL driver + reporting libraries (reportlab, Markdown).
  • containers/backupchecks/src/backend/app: backend domain logic, routes, parsers, models, migrations.
  • containers/backupchecks/src/templates: Jinja templates for auth/main/documentation pages.
  • containers/backupchecks/src/static: CSS, images, favicon.
  • deploy/backupchecks-stack.yml: compose stack with backupchecks, postgres, adminer.
  • build-and-push.sh: release/test build script with version bumping, tags, and image push.
  • docs/: functional design, changelogs, migration notes, API notes.

Application Architecture (Current Observation)

  • Factory pattern: create_app() in containers/backupchecks/src/backend/app/__init__.py.
  • Blueprints:
    • auth_bp for authentication.
    • main_bp for core functionality.
    • doc_bp for internal documentation pages.
  • Database initialization at startup:
    • db.create_all()
    • run_migrations()
  • Background tasks:
    • start_auto_importer(app) starts the automatic mail importer thread.
    • start_cove_importer(app) starts the Cove Data Protection polling thread (started only when cove_import_enabled is set).
  • Global template context:
    • inject_inbox_count() context processor injects inbox_count into every template for authenticated users (sidebar badge).
  • Health endpoint:
    • GET /health returns { "status": "ok" }.

Functional Processing Flow

  • Import:
    • Email is fetched via Microsoft Graph API.
  • Parse:
    • Parser selection through registry + software-specific parser implementations.
  • Approve:
    • New jobs first appear in Inbox for initial customer assignment.
  • Auto-process:
    • Subsequent emails for known jobs automatically create JobRun records.
  • Monitor:
    • Runs appear in Daily Jobs and Run Checks.
  • Review:
    • Manual review removes items from the unreviewed operational queue.

Configuration and Runtime

  • Config is built from environment variables in containers/backupchecks/src/backend/app/config.py.
  • Important variables:
    • APP_SECRET_KEY
    • APP_ENV
    • APP_PORT
    • POSTGRES_DB
    • POSTGRES_USER
    • POSTGRES_PASSWORD
    • DB_HOST
    • DB_PORT
  • Database URI pattern:
    • postgresql+psycopg2://<user>:<pass>@<host>:<port>/<db>
  • Default timezone in config: Europe/Amsterdam.

Data Model (High-level)

File: containers/backupchecks/src/backend/app/models.py

  • Auth/users:
    • User with role(s), active role in session.
  • System settings:
    • SystemSettings with Graph/mail settings, import settings, UI timezone, dashboard policy, sandbox flag.
    • Autotask configuration and cache fields are present.
    • Cove Data Protection fields: cove_enabled, cove_api_url, cove_api_username, cove_api_password, cove_import_enabled, cove_import_interval_minutes, cove_partner_id, cove_last_import_at.
    • Microsoft Entra SSO fields: entra_sso_enabled, entra_tenant_id, entra_client_id, entra_client_secret, entra_redirect_uri, entra_allowed_domain, entra_allowed_group_ids, entra_auto_provision_users.
  • Logging:
    • AuditLog (legacy alias AdminLog).
  • Domain:
    • Customer, Job, JobRun, Override
    • MailMessage, MailObject
    • CoveAccount (Cove staging table — see Cove integration section)
    • CloudConnectAccount (Cloud Connect staging table — see Cloud Connect integration section)
    • Ticket, TicketScope, TicketJobRun
    • Remark, RemarkScope, RemarkJobRun
    • FeedbackItem, FeedbackVote, FeedbackReply, FeedbackAttachment

Foreign Key Relationships & Deletion Order

Critical deletion order to avoid constraint violations (used in "Delete all jobs" maintenance route):

  1. Unlink staging accounts: UPDATE cove_accounts SET job_id = NULL, UPDATE cloud_connect_accounts SET job_id = NULL
  2. Unlink mails: UPDATE mail_messages SET job_id = NULL, location = 'inbox'
  3. Delete FK tables referencing job_runs: remark_job_runs, ticket_job_runs, run_object_links, job_run_review_events
  4. Delete FK tables referencing jobs: job_object_links, ticket_scopes, remark_scopes, overrides
  5. DELETE FROM job_runs
  6. DELETE FROM jobs

Note: always use direct SQL (DELETE FROM) for bulk deletions — ORM-level deletes load all objects into Python memory and time out on large datasets.

Key Model Fields

MailMessage model:

  • from_address (NOT sender!) - sender email
  • subject - email subject
  • text_body - plain text content
  • html_body - HTML content
  • received_at - timestamp
  • location - inbox/processed/deleted
  • job_id - link to Job (nullable)

Job model:

  • customer_id - FK to Customer
  • job_name - parsed from email
  • backup_software - e.g., "Veeam", "Synology", "Cove Data Protection"
  • backup_type - e.g., "Backup Job", "Active Backup"
  • cove_account_id - (nullable int) links this job to a Cove AccountId
  • Cloud Connect accounts link back via CloudConnectAccount.job_id (no FK column on jobs — the link is on the staging table side)

JobRun model:

  • source_type - NULL = email (backwards compat), "cove_api" for Cove-imported runs
  • external_id - deduplication key for Cove runs: "cove-{account_id}-{run_ts}"

Parser Architecture

  • Folder: containers/backupchecks/src/backend/app/parsers/
  • Two layers:
    • registry.py:
      • matching/documentation/visibility on /parsers.
      • examples must stay generic (no customer names).
    • parser files (veeam.py, synology.py, etc.):
      • actual detection and parsing logic.
      • return structured output: software, type, job name, status, objects.
  • Practical rule:
    • extend patterns by adding, not replacing (backward compatibility).

Parser Types

Informational Parsers:

  • DSM Updates, Account Protection, Firmware Updates
  • Set appropriate backup_type (e.g., "Updates", "Firmware Update")
  • Do NOT participate in schedule learning
  • Usually still visible in Run Checks for awareness
  • Exception: non-backup 3CX informational types (Update, SSL Certificate) are hidden from Run Checks

Regular Parsers:

  • Backup jobs (Veeam, Synology Active Backup, NAKIVO, etc.)
  • Participate in schedule learning (daily/weekly/monthly detection)
  • Generate missed runs when expected runs don't occur

Example: Synology Updates Parser (synology.py)

  • Handles multiple update notification types under same job:
    • DSM automatic update cancelled
    • Packages out-of-date
    • Combined notifications (DSM + packages)
  • Detection patterns:
    • DSM: "Automatische DSM-update", "DSM-update op", "automatic DSM update"
    • Packages: "Packages on", "out-of-date", "Package Center"
  • Hostname extraction from multiple patterns
  • Returns: backup_type "Updates", job_name "Synology Automatic Update"

Schedule Inference and Missed Run Detection

Overview

File: containers/backupchecks/src/backend/app/main/routes_shared.py

Missed runs are detected via _ensure_missed_runs_for_job() which is called from Run Checks on page load (throttled: max once per 10 minutes per job via in-memory dict). It infers the expected schedule from recent run history and creates JobRun records with missed=True for any slots that are overdue.

Weekly Schedule Inference (_infer_schedule_map_from_runs)

  • Window: last 90 days only (older runs are excluded to handle schedule changes)
  • MIN_OCCURRENCES: 5 hits on a weekday+time slot to count as expected (raised from 3 to reduce false positives during transitional periods)
  • Cadence guard: if median gap between runs ≥ 20 days, weekly inference is skipped entirely → monthly inference handles the job instead. Prevents monthly jobs from accumulating enough weekly hits after long operation.
  • Key rule: time-of-day changes or frequency changes stop generating missed runs on old slots within 90 days (no more stale slot false positives)

Monthly Schedule Inference (_infer_monthly_schedule_from_runs)

  • Window: last 180 days (enough for ≥ 3 monthly occurrences, but forgotten within 6 months after a schedule change)
  • Infers day-of-month + time-of-day from historical runs
  • Used when weekly cadence guard fires (median gap ≥ 20 days)

Important Rules

  • Never extend the window without considering stale slot false positives
  • Schedule changes (time, frequency) take effect in missed run detection within the window period (90d weekly, 180d monthly)
  • Informational parsers (3CX / Update, 3CX / SSL Certificate) are excluded from all schedule inference

Cove Data Protection Integration

Overview

Cove (N-able) Data Protection is a cloud backup platform. Backupchecks integrates with it via the Cove JSON-RPC API, following the same inbox-style staging flow as email imports.

Files

  • containers/backupchecks/src/backend/app/cove_importer.py – API client, account processing, JobRun creation
  • containers/backupchecks/src/backend/app/cove_importer_service.py – background polling thread
  • containers/backupchecks/src/backend/app/main/routes_cove.py – /cove/accounts routes
  • containers/backupchecks/src/templates/main/cove_accounts.html – inbox-style accounts page

API Details

  • Endpoint: https://api.backup.management/jsonapi (JSON-RPC 2.0)
  • Login: POST with {"jsonrpc":"2.0","id":"jsonrpc","method":"Login","params":{"username":"...","password":"..."}}
    • Returns visa at top level (data["visa"]), not inside result
    • Returns PartnerId inside result
  • EnumerateAccountStatistics: POST with visa in payload, query (lowercase) with PartnerId, StartRecordNumber, RecordsCount, Columns
  • Settings format per account: [{"D09F00": "5"}, {"I1": "device name"}, ...] — list of single-key dicts, flatten with dict.update(item)

Column Codes

Code Meaning
I1 Account/device name
I18 Computer name
I8 Customer/partner name
I78 Active datasource label
D09F00 Overall last session status code
D09F09 Last successful session timestamp (Unix)
D09F15 Last session end timestamp (Unix)
D09F08 28-day colorbar string
D1F00/F15 Files & Folders status/timestamp
D10F00/F15 VssMsSql
D11F00/F15 VssSharePoint
D19F00/F15 M365 Exchange
D20F00/F15 M365 OneDrive
D5F00/F15 M365 SharePoint
D23F00/F15 M365 Teams

Status Code Mapping

Cove code Meaning Backupchecks status
1 In process Warning
2 Failed Error
3 Aborted Error
5 Completed Success
6 Interrupted Error
7 Not started Warning
8 Completed with errors Warning
9 In progress with faults Warning
10 Over quota Error
11 No selection Warning
12 Restarted Warning

Inbox-Style Flow (mirrors email import)

  1. Cove importer fetches all accounts via paginated EnumerateAccountStatistics (250/page).
  2. Every account is upserted into the cove_accounts staging table (always, regardless of job link).
  3. Accounts without a job_id appear on /cove/accounts ("Cove Accounts" page) for admin action.
  4. Admin can:
    • Create new job – creates a Job with backup_software="Cove Data Protection" and links it.
    • Link to existing job – sets job.cove_account_id and cove_acc.job_id.
  5. Linking an account triggers an immediate import attempt; linked accounts then generate JobRun records (deduplicated per job via job_id + external_id).
  6. Per-datasource objects are persisted to customer_objects, job_object_links, run_object_links.

CoveAccount Model

class CoveAccount(db.Model):
    __tablename__ = "cove_accounts"
    id            # PK
    account_id    # Cove AccountId (unique)
    account_name  # I1
    computer_name # I18
    customer_name # I8
    datasource_types # I78
    last_status_code # D09F00 (int)
    last_run_at   # D09F15 (datetime)
    colorbar_28d  # D09F08
    job_id        # FK → jobs.id (nullable — None = unmatched)
    first_seen_at
    last_seen_at
    job           # relationship → Job

Deduplication

external_id = f"cove-{account_id}-{run_ts}" where run_ts is Unix timestamp from D09F15 (fallback to D09F09 when needed).

Deduplication is enforced per linked job:

  • check JobRun.query.filter_by(job_id=job.id, external_id=external_id).first()
  • this prevents cross-job collisions when accounts are relinked.

Historical (colorbar) runs use external_id = f"cove-colorbar-{account_id}-{date_str}" (e.g. cove-colorbar-4378343-2026-03-15).

Historical Backfill (28-day colorbar)

When a new run is created for a job, _backfill_colorbar_runs() is called to reconstruct up to 27 additional days of history from the D09F08 colorbar field.

  • Each character in the colorbar = one day's status (oldest first, position 0 = 27 days ago, last position = today)
  • Status 0 = no backup that day → skipped
  • run_at = same time-of-day as the real run, but on the historical date
  • Idempotent: external_id deduplication prevents duplicates on subsequent imports
  • Only creates runs for days where a backup actually ran (non-zero status code)

Run Enrichment

  • Cove-created JobRun.remark contains account/computer/customer and last status/timestamp summary.
  • Per-datasource run object records include:
    • mapped Backupchecks status
    • readable status details in error_message
    • datasource-level session timestamp in observed_at

Cove Accounts UI Notes

  • /cove/accounts derives display fields to align with existing job logic:
    • backup_software: Cove Data Protection
    • backup_type: Server, Workstation, or Microsoft 365
    • job_name: based on Cove account/computer fallback
    • readable datasource labels instead of raw I78 code stream
  • computer_name is shown in both unmatched and matched account tables.

Background Thread

cove_importer_service.py — same pattern as auto_importer_service.py:

  • Thread name: "cove_importer"
  • Checks settings.cove_import_enabled
  • Interval: settings.cove_import_interval_minutes (default 30)
  • Calls run_cove_import(settings) which returns (total, created, skipped, errors)

Settings UI

Settings → Integrations → Cove section:

  • Enable toggle, API URL, username, password (masked, only overwritten if non-empty)
  • Import enabled + interval
  • "Test Connection" button (AJAX → POST /settings/cove/test-connection) returns {ok, partner_id, message}
  • "Run import now" button (→ POST /settings/cove/run-now) triggers manual import

Routes

Route Method Description
/cove/accounts GET Inbox-style page: unmatched + matched accounts
/cove/accounts/<id>/link POST action=create or action=link
/cove/accounts/<id>/unlink POST Removes job link, puts account back in unmatched
/cove/run/<run_id>/detail GET JSON: structured Cove run details for job detail popup
/settings/cove/test-connection POST AJAX: verify credentials, save partner_id
/settings/cove/run-now POST Manual import trigger

Job Detail Popup (Cove runs)

Cove run rows in the job detail history table are clickable even without a mail message:

  • Row has data-source-type="cove_api" and data-run-id
  • JS detects source_type === "cove_api" and fetches /cove/run/<run_id>/detail instead of inbox_message_detail
  • Response includes: meta (account name as subject, backup_software, run_at as received_at), cove_summary (account, computer, customer, datasources, last run, status), objects (per-datasource run_object_links)
  • Mail section hidden entirely; Cove summary panel shown instead

Run Checks Popup (Cove runs)

  • routes_run_checks.py returns cove_summary in the run payload for source_type="cove_api" runs
  • Includes: account_name, computer_name, customer_name, readable datasource labels, last_run_at, status
  • run_checks.html shows the Cove summary panel and hides the mail section

Migrations

  • migrate_cove_integration() — adds 8 columns to system_settings, cove_account_id to jobs, source_type + external_id to job_runs, dedup index on job_runs.external_id
  • migrate_cove_accounts_table() — creates cove_accounts table with indexes

Veeam Cloud Connect Integration

Overview

Veeam Cloud Connect sends a daily HTML report email (one email per provider, covering all tenants). The importer parses the HTML table and upserts each tenant row into the cloud_connect_accounts staging table. Linked accounts create JobRun records; unlinked accounts appear on the Cloud Connect Accounts review page.

Files

  • app/cloud_connect_importer.py — HTML parser + upsert logic
  • app/main/routes_cloud_connect.py — /cloud-connect/accounts page, link/unlink/scan-inbox routes
  • templates/main/cloud_connect_accounts.html — accounts review page

Email Structure

  • One email covers all tenants (unlike Cove, which sends one email per account)
  • Sections: Backup, Replication, Agent (detected from <p> tags with font-size: 18px)
  • Each section has a table with columns: User, #VM / #WS+#Server (Agent), Repo Name, Repository, Total quota, Used space, Free space, Last active, Expiry

Status Mapping (row background colour)

Colour Status
#fb9895 / #ff9999 / #f4cccc / #ffb3b3 Failed
#ffd96c / #fff2cc / #ffe599 / #f9cb9c Warning
white / no background Success

⚠ TODO — Last active detection logic: The importer currently trusts Veeam's row colour as the sole status indicator (white = Success). An earlier version downgraded white rows to Warning when "Last active" exceeded 3 days, but this was removed because Veeam itself determines row colour. It is still an open question whether backupchecks should apply its own independent "last active" threshold on top of Veeam's colour — e.g. to catch cases where Veeam shows a white row for a backup that hasn't run in a long time. Needs review before production use.

Staging Table: cloud_connect_accounts

Unique key: (user, section, repo_name) — one row per tenant × section × repository.

A single user can have multiple repositories (e.g. a standard repo + an immutable repo), each stored as a separate account row and each linkable to a separate Backupchecks job.

Deduplication

external_id = f"vcc-{user}-{section}-{repo_slug}-{report_date}" — one JobRun per job per repository per report date. Re-importing the same email updates the run status and refreshes run_object_links.

Run Enrichment

  • source_type = "cloud_connect" on every JobRun created by the importer
  • _persist_cc_objects() upserts the repository as a customer_object (type cloud_connect_repo) and links it via run_object_links — mirrors the Cove datasource pattern and enables per-run reporting
  • The run_at is set to the mail's received_at (not today) so historical re-imports land on the correct date

Job Detail Popup (job_detail.html)

For CC runs the popup shows a structured Cloud Connect summary panel (User/Section/Repository/Used/Quota/Free/Last active/Status) instead of the raw report email. The report email is still accessible via a collapsible "Source report email" toggle. Only the single per-run repository object is shown (from run_object_links), not all tenants from the shared mail.

Scan Inbox

POST /cloud-connect/accounts/scan-inbox — re-processes all stored CC report emails (location ≠ deleted). Safe to run multiple times; deduplication prevents duplicate runs.

Migrations

  • migrate_cloud_connect_accounts_table() — creates cloud_connect_accounts table with (user, section) unique key
  • migrate_cc_accounts_repo_unique_key() — extends unique key to (user, section, repo_name), makes repo_name NOT NULL DEFAULT ''

Ticketing and Autotask (Critical Rules)

Two Ticket Types

  1. Internal Tickets (tickets table)

    • Created manually or via Autotask integration
    • Stored in tickets table with ticket_code (e.g., "T20250123.0001")
    • Linked to runs via ticket_job_runs many-to-many table
    • Scoped to jobs via ticket_scopes table
    • Have resolved_at field for resolution tracking
    • Auto-propagation: Automatically linked to new runs via link_open_internal_tickets_to_run

  2. Autotask Tickets (job_runs columns)

    • Created via Run Checks modal → "Create Autotask Ticket"
    • Stored directly in JobRun columns: autotask_ticket_id, autotask_ticket_number, etc.
  • When created, also creates matching internal ticket for legacy UI compatibility

Microsoft Entra SSO (Current State)

Status

  • Implemented but marked Untested in Backupchecks.

Routes

  • GET /auth/entra/login – starts Entra auth code flow.
  • GET /auth/entra/callback – exchanges code, maps/provisions local user, logs in session.
  • /auth/logout – Entra-aware logout redirect when user authenticated via Entra.

Access Controls

  • Optional tenant/domain restriction (entra_allowed_domain).
  • Optional Entra security-group allowlist (entra_allowed_group_ids) based on group object IDs.
  • Group overage / missing groups claim blocks login intentionally when group gate is enabled.

Local User Mapping

  • Primary mapping by preferred_username/UPN/email.
  • Optional auto-provision (entra_auto_provision_users) creates local Viewer users for unknown identities.

Documentation

  • Built-in docs page: /documentation/settings/entra-sso
  • Includes configuration steps and explicit untested warning.

Navbar Notes (Latest)

  • To reduce split-screen overflow, nav is compacted by grouping:
    • admin-only links under Admin dropdown
    • secondary non-admin links under More dropdown
  • Primary operational links remain visible (notably Run Checks).
  • Viewer role now exposes Customers and Jobs directly in navbar.
    • Have autotask_ticket_deleted_at field for deletion tracking
    • Resolution tracked via matching internal ticket's resolved_at field
    • Auto-propagation: Linked to new runs via two-strategy approach

Ticket Propagation to New Runs

When a new JobRun is created (via email import OR missed run generation), link_open_internal_tickets_to_run ensures:

Strategy 1: Internal ticket linking

  • Query finds tickets where: COALESCE(ts.resolved_at, t.resolved_at) IS NULL
  • Creates ticket_job_runs links automatically
  • Tickets remain visible until explicitly resolved
  • NO date-based logic - resolved = immediately hidden from new runs

Strategy 2: Autotask ticket propagation (independent)

  1. Check if internal ticket code exists → find matching Autotask run → copy ticket info
  2. If no match, directly search for most recent Autotask ticket on job where:
    • autotask_ticket_deleted_at IS NULL (not deleted in PSA)
    • Internal ticket resolved_at IS NULL (not resolved in PSA)
  3. Copy autotask_ticket_id, autotask_ticket_number, created_at, created_by_user_id to new run

Where Ticket Linking is Called

link_open_internal_tickets_to_run is invoked in three locations:

  1. Email-based runs: routes_inbox.py and mail_importer.py - after creating JobRun from parsed email
  2. Missed runs: routes_run_checks.py in _ensure_missed_runs_for_job - after creating missed JobRun records
    • Weekly schedule: After creating weekly missed run (with flush to get run.id)
    • Monthly schedule: After creating monthly missed run (with flush to get run.id)
    • Critical: Without this call, missed runs don't get ticket propagation!

Display Logic - Link-Based System

All pages use explicit link-based queries (no date-based logic):

Job Details Page:

  • Two sources for ticket display:
    1. Direct links (ticket_job_runs WHERE job_run_id = X) → always show (audit trail)
    2. Active window (ticket_scopes WHERE job_id = Y AND resolved_at IS NULL) → only unresolved
  • Result: Old runs keep their ticket references, new runs don't get resolved tickets

Run Checks Main Page (Indicators 🎫):

  • Query: ticket_scopes JOIN tickets WHERE job_id = X AND resolved_at IS NULL
  • Only shows indicator if unresolved tickets exist for the job

Run Checks Popup Modal:

  • API: /api/job-runs/<run_id>/alerts
  • Two-source ticket display:
    1. Direct links: tickets JOIN ticket_job_runs WHERE job_run_id = X
    2. Job-level scope: tickets JOIN ticket_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date
  • Prevents duplicates by tracking seen ticket IDs
  • Shows newly created tickets immediately (via scope) without waiting for resolve action
  • Two-source remark display:
    1. Direct links: remarks JOIN remark_job_runs WHERE job_run_id = X
    2. Job-level scope: remarks JOIN remark_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date (with timezone-safe fallback from start_date)
  • Prevents duplicates by tracking seen remark IDs

Debug Logging for Ticket Linking (Reference)

If you need to debug ticket linking issues, add this to link_open_internal_tickets_to_run in ticketing_utils.py after the rows query:

try:
    from .models import AuditLog
    details = []
    if rows:
        for tid, code, t_resolved, ts_resolved in rows:
            details.append(f"ticket_id={tid}, code={code}, t.resolved_at={t_resolved}, ts.resolved_at={ts_resolved}")
    else:
        details.append("No open tickets found for this job")
    audit = AuditLog(
        user="system", event_type="ticket_link_debug",
        message=f"link_open_internal_tickets_to_run called: run_id={run.id}, job_id={job.id}, found={len(rows)} ticket(s)",
        details="\n".join(details)
    )
    db.session.add(audit)
    db.session.commit()
except Exception:
    pass

Visible on Logging page under event_type = "ticket_link_debug". Remove after debugging.

Resolved vs Deleted

  • Resolved: Ticket completed in Autotask (tracked in internal tickets.resolved_at)
    • Stops propagating to new runs
    • Ticket still exists in PSA
    • Synced via PSA polling
  • Deleted: Ticket removed from Autotask (tracked in job_runs.autotask_ticket_deleted_at)
    • Also stops propagating
    • Ticket no longer exists in PSA
    • Rare operation

Critical Rules

  • ❌ NEVER use date-based resolved logic: resolved_at >= run_date OR active_from_date <= run_date
  • ✅ Only show tickets that are ACTUALLY LINKED via ticket_job_runs table
  • ✅ Resolved tickets stop linking immediately when resolved
  • ✅ Old links preserved for audit trail (visible on old runs)
  • ✅ All queries must use explicit JOIN to link tables
  • ✅ Consistency: All pages use same "resolved = NULL" logic
  • ✅ CRITICAL: Preserve description field during Autotask updates - must include "description" in optional_fields list

UI and UX Notes

Layout v2 (2026-03-20)

  • Complete sidebar-first redesign replacing the top navbar layout:
    • layout.css rewritten with IBM Plex Sans/Mono fonts and CSS custom properties (design tokens)
    • Fixed dark sidebar (220 px wide)
    • base.html updated with Google Fonts preload and sidebar-aware structure
  • Sandbox banner: semi-transparent (rgba(220,53,69,0.45)) instead of solid red

Navbar (pre-v0.2.0 reference — replaced by sidebar in v0.2.0)

  • Fixed-top positioning
  • Collapses on mobile (hamburger menu)
  • Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top)
  • Role-based menu items (Admin sees more than Operator/Viewer)

Status Badges

  • Success: Green
  • Warning: Yellow/Orange
  • Failed/Error: Red
  • Override applied: Blue badge
  • Reviewed: Checkmark indicator

Ticket Copy Functionality

  • Copy button (⧉) available on both Run Checks and Job Details pages
  • Allows quick copying of ticket numbers to clipboard
  • Cross-browser compatible with three-tier fallback mechanism:
    1. Modern Clipboard API: navigator.clipboard.writeText() - works in modern browsers with HTTPS
    2. Legacy execCommand: document.execCommand('copy') - fallback for older browsers and Edge
    3. Prompt fallback: window.prompt() - last resort if clipboard access fails
  • Visual feedback: button changes to ✓ checkmark for 800ms after successful copy
  • Implementation uses hidden textarea for execCommand method to ensure compatibility
  • No user interaction required in modern browsers (direct copy)

Checkbox Behavior

  • All checkboxes on Inbox and Run Checks pages use autocomplete="off"
  • Prevents browser from auto-selecting checkboxes after page reload
  • Fixes issue where deleting items would cause same number of new items to be selected

Customers to Jobs Navigation (2026-02-16)

  • Customers page links each customer name to filtered Jobs view:
    • GET /jobs?customer_id=<customer_id>
  • Jobs route behavior:
    • Accepts optional customer_id query parameter in routes_jobs.py.
    • If set: returns jobs for that customer only.
    • If not set: keeps default filter that hides jobs linked to inactive customers.
  • Jobs UI behavior:
    • Shows active filter banner with selected customer name.
    • Provides "Clear filter" action back to unfiltered /jobs.
  • Templates touched:
    • templates/main/customers.html
    • templates/main/jobs.html

Global Grouped Search (2026-02-16)

  • New route:
    • GET /search in main/routes_search.py
  • New UI:
    • Navbar search form in templates/layout/base.html
    • Grouped result page in templates/main/search.html
  • Search behavior:
    • Case-insensitive matching (ILIKE).
    • * wildcard is supported and translated to SQL %.
    • Automatic contains behavior is applied per term (*term*) when wildcard not explicitly set.
    • Multi-term queries use AND across terms and OR across configured columns within each section.
    • Per-section pagination is supported via query params: p_inbox, p_customers, p_jobs, p_daily_jobs, p_run_checks, p_tickets, p_remarks, p_overrides, p_reports.
    • Pagination keeps search state for all sections while browsing one section.
    • "Open
      " links pass q to destination overview pages so page-level filtering matches the search term.
  • Grouped sections:
    • Inbox, Customers, Jobs, Daily Jobs, Run Checks, Tickets, Remarks, Existing overrides, Reports.
  • Daily Jobs search result details:
    • Meta now includes expected run time, success indicator, and run count for the selected day.
    • Link now opens Daily Jobs with modal auto-open using open_job_id query parameter (same modal flow as clicking a row in Daily Jobs).
  • Access control:
    • Search results are role-aware and only show sections/data the active role can access.
    • run_checks results are restricted to admin/operator.
    • reports supports admin/operator/viewer/reporter.
  • Current performance strategy:
    • Per-section limit (SEARCH_LIMIT_PER_SECTION = 10), with total count per section.
    • No schema migration required for V1.

Jobs Export / Import

Schema Versions

  • v1 (approved_jobs_export_v1): job fields only, no account links
  • v2 (approved_jobs_export_v2): same as v1 plus per-job cove_account and cloud_connect_account objects

Export (v2)

Each job entry contains:

{
  "cove_account": {"account_id": 1234, "account_name": "...", "computer_name": "..."},
  "cloud_connect_account": {"user": "...", "section": "Backup", "repo_name": "..."}
}

null when not linked. File: routes_settings.py → export_jobs().

Import (v1 + v2)

  • Accepts both schema versions (detected via export_type field)
  • For v2 files: after creating/updating the job, the importer:
    1. Looks up CoveAccount by account_id (fallback: account_name + computer_name)
    2. Looks up CloudConnectAccount by user + section + repo_name
    3. Links the account to the job — only if the account is not yet linked to a different job
  • File: routes_settings.py → import_jobs()

Inbox Batch Re-parse

Endpoint

POST /inbox/reparse-batch — JSON in/out, login required, admin/operator only.

Request body:

{"last_id": <int|null>, "total": <int|null>}
  • last_id: keyset cursor from previous batch (process messages with id < last_id)
  • total: total count from first call (avoids re-counting on every batch)

Response:

{"processed": 50, "total": 847, "parsed_ok": 42, "auto_approved": 12, "no_match": 8, "errors": 0, "last_id": 1234, "done": false}

Batch Parameters

  • batch_size: 50 messages per call
  • time_budget_s: 8 seconds per call (stops processing mid-batch if exceeded)
  • Auto-approve logic is identical to inbox_reparse_all (including VSPC multi-company handling)

Frontend (inbox.html)

  • "Re-parse all" button opens a Bootstrap modal (data-bs-backdrop="static" — cannot close while running)
  • JS loop: fetch → process response → setTimeout(100ms) → repeat until done: true
  • Progress bar + live counters update after each batch
  • Close button appears only when done: true or on error

Feedback Module with Screenshots

  • Models: FeedbackItem, FeedbackVote, FeedbackReply, FeedbackAttachment.
  • Attachments:
    • multiple uploads, type validation, per-file size limits, storage in database (BYTEA).

Validation Snapshot

  • 2026-02-16: Test build + push succeeded via update-and-build.sh t.
  • Pushed image: gitea.oskamp.info/ivooskamp/backupchecks:dev.
  • 2026-02-16: Test build + push succeeded on branch v20260216-02-global-search.
  • Pushed image digest: sha256:6996675b9529426fe2ad58b5f353479623f3ebe24b34552c17ad0421d8a7ee0f.
  • 2026-02-16: Additional test build + push cycles succeeded on v20260216-02-global-search.
  • Latest pushed image digest: sha256:8ec8bfcbb928e282182fa223ce8bf7f92112d20e79f4a8602d015991700df5d7.
  • 2026-02-16: Additional test build + push cycles succeeded after search enhancements.
  • Latest pushed image digest: sha256:b36b5cdd4bc7c4dadedca0534f1904a6e12b5b97abc4f12bc51e42921976f061.
  • Delete strategy:
    • soft delete by default,
    • permanent delete only for admins and only after soft delete.

Deployment and Operations

  • Stack exposes:
    • app on 8080
    • adminer on 8081
  • PostgreSQL persistent volume:
    • /docker/appdata/backupchecks/backupchecks-postgres:/var/lib/postgresql/data
  • deploy/backupchecks-stack.yml also contains example .env variables at the bottom.

Build/Release Flow

File: build-and-push.sh

  • Bump options:
    • 1 patch, 2 minor, 3 major, t test.
  • Release build:
    • update version.txt
    • commit + tag + push
    • docker push of :<version>, :dev, :latest
  • Test build:
    • only :dev
    • no commit/tag.
  • Services are discovered under containers/* with Dockerfile-per-service.

Technical Observations / Attention Points

  • README.md is currently empty; quick-start entry context is missing.
  • LICENSE is currently empty.
  • docs/architecture.md is currently empty.
  • deploy/backupchecks-stack.yml contains hardcoded example values (Changeme), with risk if used without proper secrets management.
  • The app performs DB initialization + migrations at startup; for larger schema changes this can impact startup time/robustness.
  • There is significant parser and ticketing complexity; route changes carry regression risk without targeted testing.
  • For Autotask update calls, the description field must be explicitly preserved to prevent unintended NULL overwrite.
  • Security hygiene remains important:
    • no customer names in parser examples/source,
    • no hardcoded credentials.

Quick References

  • App entrypoint: containers/backupchecks/src/backend/app/main.py
  • App factory: containers/backupchecks/src/backend/app/__init__.py
  • Config: containers/backupchecks/src/backend/app/config.py
  • Models: containers/backupchecks/src/backend/app/models.py
  • Parsers: containers/backupchecks/src/backend/app/parsers/registry.py
  • Ticketing utilities: containers/backupchecks/src/backend/app/ticketing_utils.py
  • Run Checks routes: containers/backupchecks/src/backend/app/main/routes_run_checks.py
  • Cove importer: containers/backupchecks/src/backend/app/cove_importer.py
  • Cove routes: containers/backupchecks/src/backend/app/main/routes_cove.py
  • Cloud Connect importer: containers/backupchecks/src/backend/app/cloud_connect_importer.py
  • Cloud Connect routes: containers/backupchecks/src/backend/app/main/routes_cloud_connect.py
  • Inbox routes: containers/backupchecks/src/backend/app/main/routes_inbox.py
  • Settings routes: containers/backupchecks/src/backend/app/main/routes_settings.py
  • Compose stack: deploy/backupchecks-stack.yml
  • Build script: build-and-push.sh

Recent Changes

2026-03-23

  • Synology ABB parser fix (parsers/synology.py): ABB completion regex now also matches has been completed phrasing.
  • Job name parsing corrected for ABB mails: messages like backup task dc001 on DS220p has been completed no longer fall back to generic Synology Active Backup parsing; job_name stays dc001 instead of bracketed subject prefix values.
  • Validation: test build ran successfully via ./build-and-push.sh t; pushed gitea.oskamp.info/ivooskamp/backupchecks:dev with digest sha256:19014477f2ae14eac0a62f07e11c923c83f9cd5e478873290bdcca37e6ab257c.
  • Run Checks Autotask (Cove object source fix) (main/routes_run_checks.py): ticket creation now reads object details from run_object_links + customer_objects first (same data source as Run Checks modal), then falls back to legacy job_objects/mail_objects.
  • Autotask ticket object filtering tightened (main/routes_run_checks.py): ticket description now lists only problem objects (failed/error/warning/missed) and excludes completed/success objects (including Completed (...) text variants).
  • Validation: latest test build ran successfully via ./build-and-push.sh t; pushed gitea.oskamp.info/ivooskamp/backupchecks:dev with digest sha256:b9bb6d50f131118ebccaed5834513ca83ec7592bd622ecea42c1ce2dd7bf0cfc.
  • Autotask link-existing ticket note update (main/routes_run_checks.py): linking a run to an existing Autotask ticket now posts an informational ticket note that another alert/run was linked (with customer/job/run context and Backupchecks deep-link).
  • Autotask link-existing API response enriched (main/routes_run_checks.py): response now includes note_posted and note_warning so UI/operators can see if the extra note call succeeded.
  • Customers job-filter link visual alignment (templates/main/customers.html, static/css/layout.css): customer links to filtered Jobs view now use sidebar text colors and hover behavior (bc-sidebar-link-inline) instead of Bootstrap link-primary blue.
  • Validation: test build ran successfully via ./build-and-push.sh t; pushed gitea.oskamp.info/ivooskamp/backupchecks:dev with digest sha256:2ff1675996b27bf409687bf5c52e2a3cb3314728ce1c67bc3ffc14fbd0562427.
  • Validation: test build ran successfully via ./build-and-push.sh t; pushed gitea.oskamp.info/ivooskamp/backupchecks:dev with digest sha256:f87d871caa3501251c31a66f61eb94b249faf0d3ab85da3f0c02c6036855849b.

2026-03-20 (v0.2.1)

  • Missed run false positive fix (routes_shared.py):
    • Weekly inference window: last 90 days only (was unbounded). Eliminates stale slot false positives after time-of-day or frequency changes.
    • Cadence guard: if median gap between runs ≥ 20 days, skip weekly inference and let monthly inference handle the job. Fixes monthly jobs accumulating enough weekly hits after ~21 months.
    • Monthly inference window: last 180 days (was unbounded).
    • MIN_OCCURRENCES raised from 3 → 5 for weekly inference.
  • Objects sort fix (run_checks.html, job_detail.html):
    • objectSeverityRank: || err on rank-0 check caused Warning items with error_message to rank as Critical. Fixed: only error/failed/failure status → rank 0; || err moved to rank 1.
  • Mail iframe height fix (run_checks.html):
    • Flex rules were on #rcm_body_iframe but the iframe is not a direct flex child of .rcm-mail-panel. Fixed by moving flex: 1 1 auto; min-height: 0 to the wrapper #rcm_mail_iframe_body and setting height: 100% on the iframe itself.
  • Inbox sidebar badge on all pages (__init__.py):
    • Added inject_inbox_count() Flask context processor — injects inbox_count into every template for authenticated users. Previously only injected in the dashboard route.
  • Jobs export/import schema v2 (routes_settings.py):
    • Export: includes cove_account and cloud_connect_account per job.
    • Import: accepts v1 and v2; links Cove/CC accounts on import if not yet linked to a different job.
  • Inbox re-parse progress modal (routes_inbox.py, inbox.html):
    • New POST /inbox/reparse-batch endpoint: 50 messages per call, 8 s time budget, keyset pagination, full auto-approve logic (including VSPC multi-company). Returns JSON progress.
    • "Re-parse all" button replaced with modal trigger; JS loop calls batch endpoint until done: true and updates live progress bar + stats.

2026-03-20 (v0.2.0)

  • Layout v2: complete sidebar-first redesign (layout.css, base.html). IBM Plex Sans/Mono fonts, CSS design tokens, fixed 220 px dark sidebar.
  • Veeam Cloud Connect importer: HTML parser for daily report emails → cloud_connect_accounts staging table → JobRun records for linked accounts. CloudConnectAccount model + migrations. /cloud-connect/accounts review page. Sidebar link for admin/operator.
  • Cove historical run backfill: _backfill_colorbar_runs() reconstructs up to 27 days of history from the D09F08 colorbar on first run creation. Idempotent via external_id = "cove-colorbar-{account_id}-{date}".
  • Cove run details popup: Cove runs in job detail are clickable; popup fetches /cove/run/<run_id>/detail (structured Cove summary, per-datasource objects, mail section hidden).
  • Run Checks user preferences: per-user sort mode + filter defaults stored in DB; POST /run-checks/preferences; User Settings page section.
  • Login captcha toggle: Settings → General → Security card; login_captcha_enabled column with DEFAULT TRUE.
  • Cloud Connect unique key: changed from (user, section) to (user, section, repo_name) — supports multiple repos per user.
  • Cloud Connect run detail popup: shows structured CC summary instead of raw email; raw email accessible via toggle.
  • Entra SSO: implemented (marked untested). Login/callback/logout flow, optional auto-provisioning, tenant/domain + security-group restrictions.
  • Fixes: login page layout with flash messages; "Delete all jobs" timeout (replaced ORM with direct SQL); archived job auto-matching; Cove link sync between cove_accounts.job_id ↔ jobs.cove_account_id; Cove run creation transaction scope.

2026-02-23

  • Cove Data Protection full integration:
    • cove_importer.py – Cove API client (login, paginated enumeration, status mapping, deduplication, per-datasource object persistence)
    • cove_importer_service.py – background polling thread (same pattern as auto_importer_service.py)
    • CoveAccount staging model + migrate_cove_accounts_table() migration
    • SystemSettings – 8 new Cove fields, Job – cove_account_id, JobRun – source_type + external_id
    • routes_cove.py – inbox-style /cove/accounts with link/unlink routes
    • cove_accounts.html – unmatched accounts shown first with Bootstrap modals (create job / link to existing), matched accounts with Unlink
    • Settings > Integrations: Cove section with test connection (AJAX) and manual import trigger
    • Navbar: "Cove Accounts" link for admin/operator when cove_enabled
  • Cove API key findings (from test script + N-able support):
    • Visa is returned at top level of login response, not inside result
    • Settings per account are a list of single-key dicts [{"D09F00":"5"}, ...] — flatten with flat.update(item)
    • EnumerateAccountStatistics params must use lowercase query key and RecordsCount (not RecordCount)
    • Login params must use lowercase username/password
    • D02/D03 are legacy; use D10/D11 or D09 (Total) instead

2026-02-19

  • Added 3CX Update parser support: threecx.py now recognizes subject 3CX Notification: Update Successful - <host> and stores it as informational with:
    • backup_software = 3CX
    • backup_type = Update
    • overall_status = Success
  • 3CX informational schedule behavior:
    • 3CX / Update and 3CX / SSL Certificate are excluded from schedule inference in routes_shared.py (no Expected/Missed generation).
  • Run Checks visibility scope (3CX-only):
    • Run Checks now hides only non-backup 3CX informational jobs (Update, SSL Certificate).
    • Other backup software/types remain visible and unchanged.
  • Fixed remark visibility mismatch:
    • /api/job-runs/<run_id>/alerts now loads remarks from both:
      1. remark_job_runs (explicit run links),
      2. remark_scopes (active job-scoped remarks),
    • with duplicate prevention by remark ID.
    • This resolves cases where the remark indicator appeared but remarks were not shown in Run Checks modal or Job Details modal.

2026-02-13

  • Fixed missed runs ticket propagation: Added link_open_internal_tickets_to_run calls in _ensure_missed_runs_for_job (routes_run_checks.py) after creating both weekly and monthly missed JobRun records. Previously only email-based runs got ticket linking, causing missed runs to not show internal tickets or Autotask tickets. Required db.session.flush() before linking to ensure run.id is available.
  • Fixed checkbox auto-selection: Added autocomplete="off" to all checkboxes on Inbox and Run Checks pages. Prevents browser from automatically re-selecting checkboxes after page reload following delete actions.

2026-02-12

  • Fixed Run Checks modal ticket display: Implemented two-source display logic (ticket_job_runs + ticket_scopes). Previously only showed tickets after they were resolved (when ticket_job_runs entry was created). Now shows tickets immediately upon creation via scope query.
  • Fixed copy button in Edge: Moved clipboard functions inside IIFE scope for proper closure access (Edge is stricter than Firefox about scope resolution).

2026-02-10

  • Added screenshot support to Feedback system: Multiple file upload, inline display, two-stage delete (soft delete for audit trail, permanent delete for cleanup).
  • Completed transition to link-based ticket system: All pages now use JOIN queries, no date-based logic. Added cross-browser copy ticket functionality with three-tier fallback mechanism to both Run Checks and Job Details pages.