diff --git a/docs/technical-notes-codex.md b/docs/technical-notes-codex.md index 172222c..d1aedb2 100644 --- a/docs/technical-notes-codex.md +++ b/docs/technical-notes-codex.md @@ -74,8 +74,35 @@ File: `containers/backupchecks/src/backend/app/models.py` - Logging: - `AuditLog` (legacy alias `AdminLog`). - Domain: - - `Customer`, `Job`, `Override`, plus extensive run/ticket logic in other modules/routes. - - Core entities from system knowledge include `JobRun`, `MailMessage`, ticket/remark link tables, and feedback tables. + - `Customer`, `Job`, `JobRun`, `Override` + - `MailMessage`, `MailObject` + - `Ticket`, `TicketScope`, `TicketJobRun` + - `Remark`, `RemarkScope`, `RemarkJobRun` + - `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment` + +### Foreign Key Relationships & Deletion Order +Critical deletion order to avoid constraint violations: +1. Clean auxiliary tables (ticket_job_runs, remark_job_runs, scopes, overrides) +2. Unlink mails from jobs (UPDATE mail_messages SET job_id = NULL) +3. Delete mail_objects +4. Delete jobs (cascades to job_runs) +5. Delete mails + +### Key Model Fields +**MailMessage model:** +- `from_address` (NOT `sender`!) - sender email +- `subject` - email subject +- `text_body` - plain text content +- `html_body` - HTML content +- `received_at` - timestamp +- `location` - inbox/processed/deleted +- `job_id` - link to Job (nullable) + +**Job model:** +- `customer_id` - FK to Customer +- `job_name` - parsed from email +- `backup_software` - e.g., "Veeam", "Synology" +- `backup_type` - e.g., "Backup Job", "Active Backup" ## Parser Architecture - Folder: `containers/backupchecks/src/backend/app/parsers/` @@ -89,28 +116,143 @@ File: `containers/backupchecks/src/backend/app/models.py` - Practical rule: - extend patterns by adding, not replacing (backward compatibility). +### Parser Types +**Informational Parsers:** +- DSM Updates, Account Protection, Firmware Updates +- Set appropriate backup_type (e.g., "Updates", "Firmware Update") +- Do NOT participate in schedule learning +- Still visible in Run Checks for awareness + +**Regular Parsers:** +- Backup jobs (Veeam, Synology Active Backup, NAKIVO, etc.) +- Participate in schedule learning (daily/weekly/monthly detection) +- Generate missed runs when expected runs don't occur + +**Example: Synology Updates Parser (synology.py)** +- Handles multiple update notification types under same job: + - DSM automatic update cancelled + - Packages out-of-date + - Combined notifications (DSM + packages) +- Detection patterns: + - DSM: "Automatische DSM-update", "DSM-update op", "automatic DSM update" + - Packages: "Packages on", "out-of-date", "Package Center" +- Hostname extraction from multiple patterns +- Returns: backup_type "Updates", job_name "Synology Automatic Update" + ## Ticketing and Autotask (Critical Rules) -- Two ticket tracks: - - internal tickets (`tickets`, `ticket_scopes`, `ticket_job_runs`). - - Autotask ticket fields on `job_runs` + synchronization with internal ticket records. -- Propagation to new runs is handled by `link_open_internal_tickets_to_run`. -- This propagation must be called for: - - email-based run creation (import flow), - - missed-run generation in `routes_run_checks.py`. -- Display logic: - - link-based via explicit JOIN queries. - - resolved tickets (`resolved_at` set) must no longer be linked to new runs. - - historical links remain visible for audit trail. -- Anti-patterns (do not use): - - date-based resolved logic like `resolved_at >= run_date`. + +### Two Ticket Types +1. **Internal Tickets** (tickets table) + - Created manually or via Autotask integration + - Stored in `tickets` table with `ticket_code` (e.g., "T20250123.0001") + - Linked to runs via `ticket_job_runs` many-to-many table + - Scoped to jobs via `ticket_scopes` table + - Have `resolved_at` field for resolution tracking + - **Auto-propagation**: Automatically linked to new runs via `link_open_internal_tickets_to_run` + +2. **Autotask Tickets** (job_runs columns) + - Created via Run Checks modal → "Create Autotask Ticket" + - Stored directly in JobRun columns: `autotask_ticket_id`, `autotask_ticket_number`, etc. + - When created, also creates matching internal ticket for legacy UI compatibility + - Have `autotask_ticket_deleted_at` field for deletion tracking + - Resolution tracked via matching internal ticket's `resolved_at` field + - **Auto-propagation**: Linked to new runs via two-strategy approach + +### Ticket Propagation to New Runs +When a new JobRun is created (via email import OR missed run generation), `link_open_internal_tickets_to_run` ensures: + +**Strategy 1: Internal ticket linking** +- Query finds tickets where: `COALESCE(ts.resolved_at, t.resolved_at) IS NULL` +- Creates `ticket_job_runs` links automatically +- Tickets remain visible until explicitly resolved +- **NO date-based logic** - resolved = immediately hidden from new runs + +**Strategy 2: Autotask ticket propagation (independent)** +1. Check if internal ticket code exists → find matching Autotask run → copy ticket info +2. If no match, directly search for most recent Autotask ticket on job where: + - `autotask_ticket_deleted_at IS NULL` (not deleted in PSA) + - Internal ticket `resolved_at IS NULL` (not resolved in PSA) +3. Copy `autotask_ticket_id`, `autotask_ticket_number`, `created_at`, `created_by_user_id` to new run + +### Where Ticket Linking is Called +`link_open_internal_tickets_to_run` is invoked in three locations: +1. **Email-based runs**: `routes_inbox.py` and `mail_importer.py` - after creating JobRun from parsed email +2. **Missed runs**: `routes_run_checks.py` in `_ensure_missed_runs_for_job` - after creating missed JobRun records + - Weekly schedule: After creating weekly missed run (with flush to get run.id) + - Monthly schedule: After creating monthly missed run (with flush to get run.id) + - **Critical**: Without this call, missed runs don't get ticket propagation! + +### Display Logic - Link-Based System +All pages use **explicit link-based queries** (no date-based logic): + +**Job Details Page:** +- **Two sources** for ticket display: + 1. Direct links (`ticket_job_runs WHERE job_run_id = X`) → always show (audit trail) + 2. Active window (`ticket_scopes WHERE job_id = Y AND resolved_at IS NULL`) → only unresolved +- Result: Old runs keep their ticket references, new runs don't get resolved tickets + +**Run Checks Main Page (Indicators 🎫):** +- Query: `ticket_scopes JOIN tickets WHERE job_id = X AND resolved_at IS NULL` +- Only shows indicator if unresolved tickets exist for the job + +**Run Checks Popup Modal:** +- API: `/api/job-runs//alerts` +- **Two-source ticket display**: + 1. Direct links: `tickets JOIN ticket_job_runs WHERE job_run_id = X` + 2. Job-level scope: `tickets JOIN ticket_scopes WHERE job_id = Y AND resolved_at IS NULL AND active_from_date <= run_date` +- Prevents duplicates by tracking seen ticket IDs +- Shows newly created tickets immediately (via scope) without waiting for resolve action +- Same for remarks: `remarks JOIN remark_job_runs WHERE job_run_id = X` + +### Resolved vs Deleted +- **Resolved**: Ticket completed in Autotask (tracked in internal `tickets.resolved_at`) + - Stops propagating to new runs + - Ticket still exists in PSA + - Synced via PSA polling +- **Deleted**: Ticket removed from Autotask (tracked in `job_runs.autotask_ticket_deleted_at`) + - Also stops propagating + - Ticket no longer exists in PSA + - Rare operation + +### Critical Rules +- ❌ **NEVER** use date-based resolved logic: `resolved_at >= run_date` OR `active_from_date <= run_date` +- ✅ Only show tickets that are ACTUALLY LINKED via `ticket_job_runs` table +- ✅ Resolved tickets stop linking immediately when resolved +- ✅ Old links preserved for audit trail (visible on old runs) +- ✅ All queries must use explicit JOIN to link tables +- ✅ Consistency: All pages use same "resolved = NULL" logic +- ✅ **CRITICAL**: Preserve description field during Autotask updates - must include "description" in optional_fields list ## UI and UX Notes -- Navbar is fixed-top with dynamic main container padding correction. -- Status badges use semantic color coding (success/warning/error/override/reviewed). -- Ticket copy button uses a three-level fallback: - - Clipboard API, - - `execCommand('copy')`, - - `prompt()` fallback. + +### Navbar +- Fixed-top positioning +- Collapses on mobile (hamburger menu) +- Dynamic padding adjustment via JavaScript (measures navbar height, adjusts main content padding-top) +- Role-based menu items (Admin sees more than Operator/Viewer) + +### Status Badges +- Success: Green +- Warning: Yellow/Orange +- Failed/Error: Red +- Override applied: Blue badge +- Reviewed: Checkmark indicator + +### Ticket Copy Functionality +- Copy button (⧉) available on both Run Checks and Job Details pages +- Allows quick copying of ticket numbers to clipboard +- Cross-browser compatible with three-tier fallback mechanism: + 1. **Modern Clipboard API**: `navigator.clipboard.writeText()` - works in modern browsers with HTTPS + 2. **Legacy execCommand**: `document.execCommand('copy')` - fallback for older browsers and Edge + 3. **Prompt fallback**: `window.prompt()` - last resort if clipboard access fails +- Visual feedback: button changes to ✓ checkmark for 800ms after successful copy +- Implementation uses hidden textarea for execCommand method to ensure compatibility +- No user interaction required in modern browsers (direct copy) + +### Checkbox Behavior +- All checkboxes on Inbox and Run Checks pages use `autocomplete="off"` +- Prevents browser from auto-selecting checkboxes after page reload +- Fixes issue where deleting items would cause same number of new items to be selected ## Feedback Module with Screenshots - Models: `FeedbackItem`, `FeedbackVote`, `FeedbackReply`, `FeedbackAttachment`. @@ -163,3 +305,17 @@ File: `build-and-push.sh` - Run Checks routes: `containers/backupchecks/src/backend/app/main/routes_run_checks.py` - Compose stack: `deploy/backupchecks-stack.yml` - Build script: `build-and-push.sh` + +## Recent Changes + +### 2026-02-13 +- **Fixed missed runs ticket propagation**: Added `link_open_internal_tickets_to_run` calls in `_ensure_missed_runs_for_job` (routes_run_checks.py) after creating both weekly and monthly missed JobRun records. Previously only email-based runs got ticket linking, causing missed runs to not show internal tickets or Autotask tickets. Required `db.session.flush()` before linking to ensure run.id is available. +- **Fixed checkbox auto-selection**: Added `autocomplete="off"` to all checkboxes on Inbox and Run Checks pages. Prevents browser from automatically re-selecting checkboxes after page reload following delete actions. + +### 2026-02-12 +- **Fixed Run Checks modal ticket display**: Implemented two-source display logic (ticket_job_runs + ticket_scopes). Previously only showed tickets after they were resolved (when ticket_job_runs entry was created). Now shows tickets immediately upon creation via scope query. +- **Fixed copy button in Edge**: Moved clipboard functions inside IIFE scope for proper closure access (Edge is stricter than Firefox about scope resolution). + +### 2026-02-10 +- **Added screenshot support to Feedback system**: Multiple file upload, inline display, two-stage delete (soft delete for audit trail, permanent delete for cleanup). +- **Completed transition to link-based ticket system**: All pages now use JOIN queries, no date-based logic. Added cross-browser copy ticket functionality with three-tier fallback mechanism to both Run Checks and Job Details pages.