## v0.1.25 This release focuses on parser improvements and maintenance enhancements, adding support for new notification types across Synology and Veeam backup systems while improving system usability with orphaned job cleanup and test email generation features. ### Parser Enhancements **Synology Parsers:** - **Monthly Drive Health Reports**: New parser for Synology NAS drive health notifications - Supports both Dutch ("Maandelijks schijfintegriteitsrapport", "Gezond") and English ("Monthly Drive Health Report", "Healthy") variants - Automatic status detection: Healthy/Gezond/No problem detected → Success, otherwise → Warning - Extracts hostname from subject or body pattern (Van/From NAS-HOSTNAME) - Backup type: "Health Report", Job name: "Monthly Drive Health" - Informational only (excluded from schedule learning and reporting logic) - Registry entry added (order 237) for /parsers page visibility - **DSM Update Notifications - Extended Coverage**: Added support for additional DSM update notification variants - New patterns: "belangrijke DSM-update", "kritieke oplossingen", "wordt automatisch geïnstalleerd", "is beschikbaar op" - Now recognizes 4 different notification types under same job: 1. Automatic update cancelled 2. Packages out-of-date warnings 3. New update available announcements 4. Automatic installation scheduled notifications - All patterns added to existing lists maintaining full backward compatibility - **Active Backup for Business - Skipped Tasks**: Extended parser to recognize skipped/ignored backup tasks - Detects Dutch ("genegeerd") and English ("skipped", "ignored") status indicators - Status mapping: Skipped/Ignored → Warning with "Skipped" message - Common scenario: Backup skipped because previous backup still running **Veeam Parsers:** - **Job Not Started Errors**: New detection for "Job did not start on schedule" error notifications - Recognizes VBO365 and other Veeam backup types that send plain text error notifications - Extracts backup type from subject (e.g., "Veeam Backup for Microsoft 365") - Extracts job name from subject after colon (e.g., "Backup MDS at Work") - Reads error message from plain text body (handles base64 UTF-16 encoding) - Sets overall_status to "Error" for failed-to-start jobs - Example messages: "Proxy server was offline at the time the job was scheduled to run." ### Maintenance Improvements **Orphaned Jobs Cleanup:** - Added "Cleanup orphaned jobs" option in Settings → Maintenance - Removes jobs without valid customer links (useful when customers are deleted) - Permanently deletes job records along with all associated emails and job runs - "Preview orphaned jobs" button shows detailed list before deletion - Displays job information with email and run counts - Safety verification step to prevent accidental deletion **Test Email Generation:** - Added "Generate test emails" feature in Settings → Maintenance - Three separate buttons to create fixed test email sets for parser testing: - Success emails (3 emails with success status) - Warning emails (3 emails with warning status) - Error emails (3 emails with error status) - Each set contains exactly 3 Veeam Backup Job emails with: - Same job name "Test-Backup-Job" for consistency - Different dates, objects, and statuses - Reproducible testing scenarios - Proper status flow testing (success → warning → error progression) ### Data Privacy **Parser Registry Cleanup:** - Replaced real customer names in parser registry examples with generic placeholders - Affected parsers: NTFS Auditing, QNAP Firmware Update, NAKIVO - Example format now uses: NAS-HOSTNAME, SERVER-HOSTNAME, VM-HOSTNAME, example.local - Ensures no customer information in codebase or version control **Autotask Integration:** - Removed customer name from Autotask ticket title for concise display - Format changed from "[Backupchecks] Customer - Job Name - Status" to "[Backupchecks] Job Name - Status" - Reduces redundancy (customer already visible in ticket company field) ### Bug Fixes **User Interface:** - Fixed responsive navbar overlapping page content on smaller screens - Implemented dynamic padding adjustment using JavaScript - Measures actual navbar height on page load, window resize, and navbar collapse toggle - Automatically adjusts main content padding-top to prevent overlap - Debounced resize handler for performance ## v0.1.24 ### Bug Fixes **Autotask Integration:** - Fixed Autotask ticket description being cleared when resolving tickets - Root cause: The `update_ticket_resolution_safe` function was performing a GET to retrieve the ticket, then a PUT to update the resolution field, but the `description` field was not included in the PUT payload - Impact: When clicking "Resolve" on an Autotask ticket, the description would be set to NULL by the Autotask API - Solution: Added `description` to the `optional_fields` list in `update_ticket_resolution_safe` so the original description is preserved from the GET response and included in the PUT request - Location: `containers/backupchecks/src/backend/app/integrations/autotask/client.py` line 672 ## v0.1.23 This comprehensive release introduces a complete built-in documentation system with 33 pages covering all features, enhanced audit logging for compliance and troubleshooting, timezone-aware datetime display throughout the application, and numerous Autotask PSA integration improvements for better usability and workflow efficiency. ### Documentation System – Complete Built-in Wiki **Core Infrastructure:** - New `/documentation` route with dedicated blueprint (doc_bp) accessible via 📖 icon in main navbar - Hierarchical structure with 9 sections and 33 pages covering all application features - Sidebar navigation with collapsible sections, active page highlighting, and sticky positioning during scrolling - Breadcrumb navigation for current location context - Previous/Next pagination buttons for sequential reading - Custom CSS (`documentation.css`) with dark mode support via CSS variables - Callout boxes (info, warning, tip, danger) for highlighting important content - Support for code blocks, tables, images (centered horizontally), and responsive design - Login required but accessible to all user roles - Full content includes comprehensive screenshots, workflow examples, troubleshooting guides, and best practices **Completed Documentation Sections (13 pages with full content):** *Getting Started (3 pages):* - What is BackupChecks: Overview, key features, architecture, supported backup software - First Login: Initial setup, role-based menu visibility, theme/role switching, navigation tips - Quick Start Guide: Complete 5-step setup workflow from mail import to daily review *User Management (3 pages):* - Users & Roles: User management interface, role descriptions (Admin/Operator/Reporter/Viewer), role assignment via checkboxes, password reset, admin protection - Login & Authentication: Login process with captcha (simple math problem), browser compatibility (Firefox recommended), HTTPS context for external access, session management - Profile Settings: Password change functionality, theme selector in navbar (not settings page), role switcher in navbar *Customers & Jobs (4 pages):* - Managing Customers: Customer creation, editing, activation/deactivation, Autotask company mapping with auto-search, CSV export/import with Autotask fields, deletion - Configuring Jobs: Inbox-based approval workflow, customer selection (job name read-only), Mail Parser automatic configuration, Reparse All functionality, job archiving/deletion - Approved Jobs: Jobs list overview, job details with history, status indicators, archive/unarchive workflow, JSON export/import for migration - Job Schedules: Automatic schedule learning from patterns (daily/weekly/monthly), Daily Jobs integration with schedule indicators, requires 10-14 days of data, troubleshooting *Mail & Import (4 pages with comprehensive security guidance):* - Mail Import Setup: Microsoft Graph API configuration, Azure AD app registration with principle of least privilege (start with Mail.Read, add Mail.ReadWrite after testing), Application Access Policy via Exchange PowerShell to restrict to one mailbox, two-step save workflow clarification (save credentials first to enable folder browser, then save complete config), automatic folder browser (no manual path entry), connection testing, troubleshooting authentication and import issues, email retention configuration, advanced multi-mailbox setup - Inbox Management: Inbox overview (shows ONLY unapproved emails, approved disappear immediately), table columns with EML download links, email detail modal with two-column layout (left: metadata/actions/objects, right: email iframe), approve workflow (customer selection only, job name read-only), what happens after approval (auto-approve future emails), re-parsing (only "Re-parse all" button, no individual re-parse), single/bulk delete with soft delete for Admin, downloading .eml files, common workflows for new customer setup and troubleshooting, best practices for verifying parsed job names - Mail Parsing: Parsing workflow (retrieval, preprocessing, parser selection, matching, parsing, storage), viewing parsers on dedicated Parsers page (Admin-only, read-only list), match criteria (sender/subject/body), supported backup software list (Veeam, Synology, NAKIVO, Boxafe, Panel3, QNAP, Syncovery, 3CX, RDrive, NTFS Auditing), parser execution order, re-parsing emails via "Re-parse all" only, parsing workflow examples, troubleshooting (contact support for parser issues - users cannot enable/disable/modify parsers), parser limitations, best practices - Auto-Import Configuration: Auto-import overview, enabling/disabling toggle, import interval (5/15/30/60 minutes), batch size explanation, detailed workflow (timer → authenticate → fetch → parse → auto-approve → move → log → persist), auto-approval logic and conditions, manual import trigger (batch size 1-50), settings summary table, monitoring via audit log and inbox, troubleshooting auto-import issues, best practices for high-volume environments, EML retention configuration *Backup Review (5 pages with detailed workflows):* - Approving Backups: Complete backup review lifecycle with 7 stages (Email Import & Parsing → Inbox Approval → Automatic Processing → Daily Monitoring → Run Review → Issue Tracking → Mark as Reviewed), stage-by-stage workflow descriptions with timestamps, complete workflow example from Day 1 new customer onboarding through Day 15 issue resolved, role-based workflows (Operator: review/create tickets/remarks/overrides/mark reviewed; Admin: plus view reviewed and unmark and delete; Viewer: read-only Daily Jobs), performance tips (filters, review by exception with overrides, batch approve, global overrides for common warnings, tickets for workload tracking, archive resolved tickets), best practices (review daily, approve inbox quickly, triage by status, use overrides for recurring warnings, create tickets for customer action, use remarks for temporary notes, always check objects, document in comments, resolve promptly, monitor schedule inference) - Daily Jobs View: Schedule inference from historical patterns (weekly/monthly detection requires 10-14 days data), status tracking (Success/Warning/Failed/Missed/Expected badges with color coding), missing job detection, override indicators with blue badges showing treated-as-success status, ticket (🎫) and remark (💬) indicators, date selection with timezone support, job table columns (customer, backup software/type, job name, schedule, last run, status, reviewed checkmark Admin-only), schedule explanation (daily/weekly/monthly patterns, irregular/-), viewing job details via Run Checks modal, reviewing jobs (mark as reviewed for warnings/failures, successful runs also need review), common workflows (morning review, checking previous days, identifying schedule changes), best practices (review daily, focus on failures first, watch for missed jobs, use overrides for known issues, create tickets for recurring issues), troubleshooting (job not appearing, schedule shows irregular, missed status issues), clarified Daily Jobs is for viewing schedules and Run Checks is primary operational tool - Run Checks Modal: THE primary daily operational tool (goal: completely empty page), detailed review interface for investigating backup runs (opens from Daily Jobs or Job History), Admin/Operator access only (not Viewer), modal layout (job header, run list left showing unreviewed by default with Admin toggle for reviewed, run details right with status/objects/email/Autotask), review actions (Mark as reviewed acknowledges issues and makes all runs for job disappear together, Unmark reviewed Admin-only per-job action), linking tickets/remarks from modal, backup objects table (name/type/status/message, only if parser extracted them), email body rendering (HTML with formatting, plain text preformatted, sandboxed iframe), Autotask fields (ticket number with "Open in Autotask" direct link button, status, resolved origin, resolved at, deletion info), common workflows (investigating failures, comparing multiple runs, reviewing warnings with overrides, troubleshooting missed), best practices (always check objects, read error messages carefully, review full email body, create tickets for recurring issues, use remarks for temporary issues, compare historical runs), troubleshooting (no objects shown, email body blank, can't mark as reviewed, run list empty), corrected review mechanism from per-run to per-JOB (when marking job as reviewed, ALL runs within that job marked simultaneously), bulk review via multiple job checkboxes (not run checkboxes), blue badge for override-applied runs - Overrides & Exceptions: Override rules for automatically handling known issues and expected warnings, two levels (global: by backup software/type affecting multiple jobs/customers; object-level: specific job or object checked first), match criteria (status: any/success/warning/failed; error text: contains/exact/starts_with/ends_with modes), treat as success action (enabled by default, displays blue badge with override indicator instead of original status for visual differentiation - green=genuine success, blue=override applied, yellow/red=unexpected problems), time windows (From date optional for retroactive application to existing runs, Until date optional for permanent/temporary overrides), creating overrides (form with level/software/type/job/object/status/error match/time window/comment fields, retroactive application to unreviewed runs), managing overrides (table showing level/scope/time/active/comment, Edit to modify, Disable/Enable toggle, Delete Admin-only), override evaluation order (object-level first → job-level → global, first match wins), common scenarios with examples (Veeam retry warnings, planned maintenance windows, known VM issue, global NAKIVO replication warnings), best practices (start specific then generalize, always document with comments/ticket numbers, use time windows for temporary issues, review periodically, be specific with error text, test before global, disable don't delete when unsure), troubleshooting (override not applied - check active/time window/match criteria; override too broad - check level and error match; multiple conflicts - most specific wins) - Remarks & Tickets: Two documentation mechanisms (tickets: external followup requiring tracking in PSA; remarks: internal notes/known issues/temporary problems), accessing via Tickets page with two tabs (Tickets/Remarks, filtering by status/customer/backup software/type/search), ticket properties (ticket code auto-generated format, description, active from date, start date, resolved at, scopes defining affected jobs), creating tickets (from Run Checks modal or Tickets page, auto-linked to job run with scopes auto-populated, manual creation without PSA integration mentioned), ticket scopes (customer/backup software/type/job/run levels for automatic indicators), viewing ticket details (status, code, dates, scopes list, linked runs last 20), resolving tickets (marks with timestamp, remains visible with ✅, 🎫 removed from displays, Admin/Operator only), linking tickets to additional runs, remark properties (body freeform text, start date, resolved at, scopes), creating/viewing/resolving remarks (similar to tickets but simpler), filtering options, tickets vs remarks comparison table, indicators in Daily Jobs and Job History (🎫 active tickets, 💬 active remarks), common workflows (creating ticket for recurring failure, adding remark for planned maintenance, resolving after customer action, reviewing historical tickets), best practices (create tickets promptly, use descriptive codes referencing PSA, set accurate active from dates, resolve promptly, use remarks for temporary issues, link to multiple runs, review monthly, use remarks for documentation), Autotask integration section corrected to clarify tickets are manually created (not automatically on failures), manual ticket creation and PSA synchronization only, troubleshooting (indicator not showing - check active status/date/scope; cannot create - check role permissions; scope affecting wrong jobs - review breadth), ticket editing currently disabled (resolve old and create new) **Placeholder Pages (20 pages with basic structure for future content):** - Reports (4 pages): Creating Reports, Relative Periods, Scheduling, Exporting Data - Autotask Integration (4 pages): Setup, Company Mapping, Creating Tickets, Ticket Management - Settings (6 pages): General, Mail, Autotask, Reporting, User Management, Maintenance - Troubleshooting (3 pages): Common Issues, FAQ, Support Contact **Documentation Accuracy Corrections:** - Fixed critical inaccuracies based on user feedback throughout Backup Review section - Corrected Run Checks from "modal" to "primary daily operational tool" with goal of completely empty page - Emphasized ALL runs must be reviewed regardless of status (including successful runs to prove accountability) - Corrected review mechanism from per-run to per-JOB (marking one job reviews all its runs simultaneously) - Corrected bulk review from "select multiple run checkboxes" to "select multiple job checkboxes" to reflect per-job mechanism - Updated override status indicators to show blue badges (not green) for visual differentiation from genuine successes - Fixed Daily Jobs purpose from "primary operational dashboard" to "schedule viewing tool" with callout that it may be deprecated in future - Removed non-existent features: individual email re-parse in modal, parser enable/disable UI, AI-powered parsing, Inbox filters section - Fixed various UI descriptions: EML column shows download link (not checkmark), customer selection workflow (job name read-only), email detail modal two-column layout, "Re-parse all" button location (inbox page only), parser management (read-only list on separate Parsers page), two-step save workflow for Mail Import (same button clicked twice), Application Access Policy security requirement - Corrected login authentication: added captcha requirement (simple math problem), clarified HTTPS context only for external access (not internal IP deployments) - Fixed user management: checkboxes for role assignment (not comma-separated), admin protection, password reset section, removed non-existent email field - Streamlined Profile Settings: only password change in User Settings page, theme/role selectors in navbar (not settings page), removed non-existent notification preferences and session information sections - Enhanced Mail Import Setup with comprehensive security best practices: principle of least privilege (start Mail.Read only), Application Access Policy to restrict to one mailbox, add Mail.ReadWrite only after testing ### Audit Logging Enhancements **System Renaming and Semantic Clarity:** - Renamed AdminLog to AuditLog for better semantic clarity across codebase - Updated model name: AdminLog → AuditLog (backwards compatible alias maintained) - Updated table name: admin_logs → audit_logs (automatic migration) - Updated function: log_admin_event() → log_audit_event() (alias provided for compatibility) - Updated UI labels: "Admin activity" → "System Audit Log" in logging page header - Better reflects purpose as comprehensive audit trail for both user and system events **Expanded Event Coverage for Compliance:** - **Settings Changes**: Now logs all changes to General, Mail, and Autotask settings - Tracks which settings changed with old value → new value comparison - Event types: `settings_general`, `settings_mail`, `settings_autotask` - Excludes sensitive data (passwords are never logged) - Example logged fields: ui_timezone, require_daily_dashboard_visit, is_sandbox_environment, graph_mailbox, autotask_enabled, autotask_environment, ticket defaults - **Export Operations**: Logs when users export data with full traceability - **Customers export** (event type: `export_customers`): CSV format, record count - **Jobs export** (event type: `export_jobs`): JSON schema version (approved_jobs_export_v1), customer count, job count - **Import Operations**: Logs when users import data with detailed operation counts - **Customers import** (event type: `import_customers`): CSV format, created count, updated count, skipped count - **Jobs import** (event type: `import_jobs`): JSON schema version, customer operations (created/updated/skipped), job operations (created/updated/skipped) - All logging uses structured event_type for filtering and analysis - Detailed JSON in details field for complete audit trail - Maintains 7-day retention policy for performance - No performance impact (async logging) - Helps with compliance audits, security monitoring, and troubleshooting configuration issues ### Timezone-Aware Datetime Display **Consistent Datetime Handling Across Application:** - Added `local_datetime` Jinja2 template filter to convert UTC datetimes to configured UI timezone - All datetime fields now automatically display in configured timezone (Settings > General > UI Timezone) - Database values remain stored in UTC for consistency and reliability - **Affected Pages**: Customers (Autotask last sync time), Settings (reference data last sync), Feedback (created/updated/resolved timestamps), Logging (event timestamps), Overrides (from/until dates), Archived Jobs (archived at), Tickets (active from/start/resolved dates), Remarks (start/resolved dates), Inbox (received timestamps), Job Detail (run timestamps), Admin Mail (received timestamps) - Custom format support via strftime parameter (e.g., `|local_datetime('%d-%m-%Y %H:%M')`) - Enhanced filter to handle both datetime objects and string datetime values for flexibility - Graceful fallback to UTC display if timezone conversion fails - Default timezone: Europe/Amsterdam (configurable via SystemSettings.ui_timezone) - Improves operator experience by showing times in local context ### Autotask PSA Integration Improvements **Usability Enhancements:** - Added "Open in Autotask" button in Run Checks modal next to ticket number for direct navigation to tickets in Autotask PSA - Opens in new tab with proper URL format including workspace and ticket ID parameters - Button only appears when ticket is linked (autotask_ticket_id exists) and Autotask integration is enabled - Styled as small outline button to maintain compact layout - Eliminates manual ticket lookup in Autotask - Added collapsible text functionality (ellipsis-field) to "Overall remark" and "Remark" fields in Run Checks modal - Prevents long text from pushing action buttons off-screen or hiding content - Click to expand/collapse long text - Improves modal usability with lengthy error messages - Auto-load Autotask reference data when opening Settings page - Automatically fetches queues, ticket sources, statuses, and priorities on page load - Only triggers when Autotask is enabled, credentials are configured, and cache is empty - Eliminates need to manually click "Refresh reference data" before selecting defaults - Displays info message with loaded data counts (e.g., "Loaded 5 queues, 3 sources, 8 statuses, 4 priorities") - Improves first-time configuration workflow - Renamed "Refresh" button to "Search" in Link existing Autotask ticket modal for better clarity - Button performs a search operation, not a refresh operation - Reduces user confusion during ticket linking workflow **Settings Organization and Validation:** - Reorganized Autotask settings into two separate forms with dedicated save buttons for better UX - **Autotask Settings** form (connection, environment, API credentials, tracking identifier) with "Save Autotask Settings" button inside card - **Ticket Defaults** form (queue, source, status, priority) with "Save Ticket Defaults" button inside card - Clear visual separation improves user experience and prevents accidental incomplete saves - All fields marked as required with red asterisks (*) - HTML5 validation prevents saving incomplete configurations - Protected default Ticket Status value against accidental clearing - Only updates when a valid status is selected from dropdown - Only allows clearing if reference data is loaded (dropdown has options) - Preserves existing value if dropdown is empty to prevent data loss - Fixes issue where "Create Autotask ticket" failed due to missing default status after saving settings with empty dropdown **Data Portability and Migration Support:** - Extended **Customer CSV export/import** to include Autotask company mappings - Export now includes `autotask_company_id` and `autotask_company_name` columns - Import reads Autotask mapping fields and applies them to existing or new customers - Backwards compatible - old CSV files without Autotask columns still work correctly - Import resets `autotask_mapping_status` and `autotask_last_sync_at` to allow resynchronization after import - Extended **Jobs JSON export/import** to include Autotask fields in customers array - Export includes Autotask fields (`autotask_company_id`, `autotask_company_name`) in customers array - Import processes customers array first, applying Autotask mappings before creating jobs - Schema remains `approved_jobs_export_v1` for backwards compatibility - Import message now shows both created and updated customer counts - Enables preservation of Autotask company mappings during system reset/migration workflows - Critical for disaster recovery and environment promotion (dev → test → production) ### Environment Identification and Safety **Sandbox/Development Environment Indicator:** - Added visual banner system to distinguish non-production environments and prevent accidental actions - New `is_sandbox_environment` boolean column in `SystemSettings` model (defaults to `False` for production safety) - Database migration `migrate_system_settings_sandbox_environment()` for automatic schema update (idempotent, safe to run multiple times) - Settings UI with new "Environment" card in Settings > General (placed after Navigation section) - Toggle switch with clear description: "Mark this as a Sandbox/Development environment" - Help text explains visual banner behavior - CSS styling via `sandbox.css`: - Diagonal red banner in top-left corner displaying "SANDBOX" in uppercase - Position: top-left corner, rotated 45 degrees with letter-spacing for visibility - Color: Bootstrap danger red (#dc3545) with white text and box-shadow for depth - Z-index: 9999 (always on top, visible across all pages) - `pointer-events: none` - banner itself is non-interactive, elements behind it remain clickable - Banner conditionally displayed only when setting is enabled - Banner placed in base template directly after `` tag, before navbar - Helps prevent accidental production-like actions in test/development systems (e.g., sending tickets to real customers) ### Job Visibility and Filtering Improvements **Active Job Filtering:** - Jobs from archived jobs and inactive customers no longer appear in Daily Jobs, Run Checks, and Jobs list pages - Added customer active status filtering to all job list queries in backend - Jobs now filtered where customer is NULL or active=True alongside existing archived=False filter - Prevents showing jobs with "-" status from deleted customers or archived jobs - Reduces clutter in operational views and improves focus on active jobs - Improves query performance by reducing result set size ### Repository Management and Security **Protected Directories:** - Added `.gitignore` file to protect confidential `.claude` directory - Directory contains AI assistant context, memory files, and user-specific configuration - Prevents accidental commits of sensitive information to version control - Protects user privacy and prevents repository pollution with local-only data ### Bug Fixes and Stability Improvements **Backend Fixes:** - Fixed missing import for `_log_admin_event` (now `_log_audit_event`) in routes_customers - Resolved crash when performing customer operations with audit logging - Enhanced `local_datetime` filter to handle string datetime values in addition to datetime objects - Prevents template rendering errors when datetime is stored as string - Automatically converts string to datetime before timezone conversion **UI and CSS Fixes:** - Fixed callout box formatting throughout documentation - Corrected CSS specificity to only make first `` tag in callout boxes block-level - Prevents nested strong tags from causing layout issues - Applied fixes to Quick Start, First Login, and all other documentation pages - Centered all documentation images horizontally (display: block, margin: 20px auto) **Documentation Content Corrections:** - Removed fabricated and non-existent features from documentation - Removed "AI-powered parsing" future enhancement (not in planning, completely fabricated) - Removed "Inbox Filters" section (not in planning) - Removed "Enabling and Disabling Parsers" section (feature doesn't exist - users cannot manage parsers) - Removed individual email re-parse from modal (only "Re-parse all" on inbox page) - Corrected workflow descriptions to match actual UI implementation - Fixed customer selection workflow (job name is read-only during approval, cannot be edited) - Fixed email detail modal layout (two-column: left metadata/actions/objects, right email iframe) - Fixed parser viewing location (separate Parsers page, Admin-only, not in Settings) - Fixed Mail Import setup workflow (two-step save: credentials first, then complete config) - Updated status badge colors and descriptions to match actual behavior - Blue badges indicate override-applied runs (not green) - Green badges only for genuine successes - Expected status badge color corrections - Fixed per-job review mechanism documentation - Clarified marking one job as reviewed marks ALL runs for that job - Corrected bulk review from "select run checkboxes" to "select job checkboxes" - Updated "Unmark Reviewed" to reflect per-job unmarking --- ## v0.1.22 This major release introduces comprehensive Autotask PSA integration, enabling seamless ticket management, customer company mapping, and automated ticket lifecycle handling directly from Backupchecks. The integration includes extensive settings configuration, robust API client implementation, intelligent ticket linking across job runs, and conditional ticket status updates based on time entries. ### Autotask Integration Core Features **Settings and Configuration:** - Complete Autotask integration settings in Settings → Integrations - Environment selection (Sandbox/Production) with automatic zone discovery - API authentication with fallback support for different tenant configurations - Tracking identifier (Integration Code) configuration for ticket attribution - Connection testing and diagnostics - Reference data synchronization (queues, sources, priorities, statuses) - Configurable ticket defaults (queue, source, status, priority) - Autotask integration and automatic mail import can now be properly disabled after being enabled (fixed unchecked checkbox processing) **Customer Company Mapping:** - Explicit Autotask company mapping for customers using ID-based linkage - Company search with auto-suggestions when opening unmapped customers - Automatically populates search box with customer name and displays matching Autotask companies - Mapping status tracking (ok/renamed/missing/invalid) - Bulk mapping refresh for all customers - Clear search boxes when opening modals for better user experience **Ticket Creation and Management:** - Create Autotask tickets directly from Run Checks page - Automatic ticket number assignment and storage - Link existing Autotask tickets to job runs - Cross-company ticket search for overarching infrastructure issues (search by ticket number finds tickets across all companies) - Ticket propagation to all active runs of the same job - Internal ticket registration for legacy compatibility (Tickets, Tickets/Remarks, Job Details) - Real-time ticket status polling and updates - Deleted ticket detection and audit tracking (deletion date/time and deleted-by resource information) **Ticket Resolution and Status Management:** - Conditional ticket status updates based on time entries: - Tickets without time entries: automatically closed (status 5 - Complete) - Tickets with time entries: remain open for time tracking continuation - Dynamic confirmation messages indicating closure behavior based on time entry presence - Safe resolution updates preserving stabilizing fields (issueType, subIssueType, source) - Resolution field mirroring from internal ticket notes - Ticket notes created via `/Tickets/{id}/Notes` endpoint with timezone-aware timestamps - Deleted ticket handling with complete audit trail **Technical Implementation:** - Full-featured Autotask REST API client (`integrations/autotask/client.py`) - Zone information discovery and endpoint resolution - Robust authentication handling with header-based fallback for sandbox environments - Picklist-based reference data retrieval (queues, sources, priorities, statuses) - Entity metadata parsing with tenant-specific field detection - Database migrations for Autotask linkage fields across SystemSettings, Customer, JobRun, and Ticket models - Ticketing utilities for internal/external ticket synchronization - Comprehensive API contract documentation (`docs/autotask_rest_api.md`) - Functional design living document for integration architecture ### User Interface Improvements - Search boxes now clear automatically when opening modals (Run Checks Link existing, Customer mapping) - Auto-search for similar company names when mapping unmapped customers - Cross-company ticket search when using ticket numbers (e.g., "T20260205.0001") - Dynamic confirmation messages for ticket resolution based on time entries - Improved visibility of Autotask ticket information in Run Checks - Status labels displayed instead of numeric codes in ticket lists - "Deleted in PSA" status display with deletion audit information - "Resolved by PSA (Autotask)" differentiation from Backupchecks-driven resolution ### Bug Fixes and Stability - Fixed Autotask REST API base URL casing (ATServicesRest/V1.0) - Fixed reference data retrieval using correct picklist endpoints - Fixed authentication fallback for sandbox-specific behavior - Fixed company name display from nested API responses - Fixed ticket ID normalization and response unwrapping (itemId handling) - Fixed TicketJobRun linkage for legacy ticket behavior - Fixed unchecked checkbox processing for enable/disable toggles (Autotask integration, automatic mail import) - Fixed ticket resolution updates to preserve exact field values from GET response - Fixed picklist field detection for tenant-specific metadata - Fixed migration stability with idempotent column checks - Fixed settings page crash with local helper functions - Fixed Run Checks modal stacking and Bootstrap 4/5 compatibility - Fixed JavaScript errors (renderModal → renderRun) - Fixed indentation errors preventing application startup - Fixed ticket propagation to ensure all active runs receive ticket linkage - Fixed polling to use read-only operations without state mutation ### Documentation - Added comprehensive Autotask REST API contract documentation (`docs/autotask_rest_api.md`) - Created functional design living document for integration architecture - Documented ticket lifecycle, status management, and time entry considerations - Added changelog tracking for Claude Code changes (`docs/changelog-claude.md`) --- ## v0.1.21 This release focuses on improving correctness, consistency, and access control across core application workflows, with particular attention to changelog rendering, browser-specific mail readability, Run Checks visibility, role-based access restrictions, override flexibility, and VSPC object linking reliability. The goal is to ensure predictable behavior, clearer diagnostics, and safer administration across both day-to-day operations and complex multi-entity reports. ### Changelog Rendering and Documentation Accuracy - Updated the Changelog route to render remote Markdown content instead of plain text. - Enabled full Markdown parsing so headings, lists, links, and code blocks are displayed correctly. - Ensured the changelog always fetches the latest version directly from the source repository at request time. - Removed legacy plain-text rendering to prevent loss of structure and formatting. ### Mail Rendering and Browser Compatibility - Forced a light color scheme for embedded mail content to prevent Microsoft Edge from applying automatic dark mode styling. - Added explicit `color-scheme` and `forced-color-adjust` rules so original mail CSS is respected. - Ensured consistent mail readability across Edge and Firefox. - Applied these fixes consistently across Inbox, Deleted Inbox, Job Details, Run Checks, Daily Jobs, and Admin All Mail views. ### Run Checks Visibility and Consistency - Added support for displaying the overall remark (overall_message) directly on the Run Checks page. - Ensured consistency between Run Checks and Job Details, where the overall remark was already available. - Improved operator visibility of high-level run context without requiring navigation to job details. ### Initial Setup and User Existence Safeguards - Fixed an incorrect redirect to the “Initial admin setup” page when users already exist. - Changed setup detection logic from “admin user exists” to “any user exists”. - Ensured existing environments always show the login page instead of allowing a new initial admin to be created. - Prevented direct access to the initial setup route when at least one user is present. ### Role-Based Access Control and Menu Restrictions - Restricted the Reporter role to only access Dashboard, Reports, Changelog, and Feedback. - Updated menu rendering to fully hide unauthorized menu items for Reporter users. - Adjusted route access to ensure Feedback pages remain accessible for the Reporter role. - Improved overall consistency between visible navigation and backend access rules. ### Override Matching Flexibility and Maintainability - Added configurable error text matching modes for overrides: contains, exact, starts with, and ends with. - Updated override evaluation logic to apply the selected match mode across run remarks and object error messages. - Extended the overrides UI with a match type selector and improved edit support for existing overrides. - Added a database migration to create and backfill the `overrides.match_error_mode` field for existing records. ### Job Deletion Stability - Fixed an error that occurred during job deletion. - Corrected backend deletion logic to prevent runtime exceptions. - Ensured related records are handled safely to avoid constraint or reference errors during removal. ### VSPC Object Linking and Normalization - Fixed VSPC company name normalization so detection and object prefixing behave consistently. - Ensured filtered object persistence respects the UNIQUE(customer_id, object_name) constraint. - Correctly update `last_seen` timestamps for existing objects. - Added automatic object persistence routing for VSPC per-company runs, ensuring objects are linked to the correct customer and job. - Improved auto-approval for VSPC Active Alarms summaries with per-company run creation and case-insensitive object matching. - Added best-effort retroactive processing to automatically link older inbox messages once company mappings are approved. ### VSPC Normalization Bug Fixes and Backward Compatibility - Removed duplicate definitions of VSPC Active Alarms company extraction logic that caused inconsistent normalization. - Ensured a single, consistent normalization path is used when creating jobs and linking objects. - Improved object linking so real objects (e.g. HV01, USB Disk) are reliably associated with their jobs. - Restored automatic re-linking for both new and historical VSPC mails. - Added backward-compatible matching to prevent existing VSPC jobs from breaking due to earlier inconsistent company naming. --- ## v0.1.20 This release delivers a comprehensive set of improvements focused on parser correctness, data consistency, and clearer operator workflows across Inbox handling, Run Checks, and administrative tooling. The main goal of these changes is to ensure that backup notifications are parsed reliably, presented consistently, and handled through predictable and auditable workflows, even for complex or multi-entity reports. ### Mail Parsing and Data Integrity - Fixed Veeam Backup for Microsoft 365 parsing where the overall summary message was not consistently stored. - Improved extraction of overall detail messages so permission and role warnings are reliably captured. - Ensured the extracted overall message is always available across Job Details, Run Checks, and reporting views. - Added decoding of HTML entities in parsed object fields (name, type, status, error message) before storage, ensuring characters such as ampersands are displayed correctly. - Improved robustness of parsing logic to prevent partial or misleading data from being stored when mails contain mixed or malformed content. ### Object Classification and Sorting - Updated object list sorting to improve readability and prioritization. - Objects are now grouped by severity in a fixed order: Errors first, then Warnings, followed by all other statuses. - Within each severity group, objects are sorted alphabetically (A–Z). - Applied the same sorting logic consistently across Inbox, Job Details, Run Checks, Daily Jobs, and the Admin All Mail view. - Improved overall run status determination by reliably deriving the worst detected object state. ### Parsers Overview and Maintainability - Refactored the Parsers overview page to use the central parser registry instead of a static, hardcoded list. - All available parsers are now displayed automatically, ensuring the page stays in sync as parsers are added or removed. - Removed hardcoded parser definitions from templates to improve long-term maintainability. - Fixed a startup crash in the parsers route caused by an invalid absolute import by switching to a package-relative import. - Prevented Gunicorn worker boot failures and Bad Gateway errors during application initialization. ### User Management and Feedback Handling - Added support for editing user roles directly from the User Management interface. - Implemented backend logic to update existing role assignments without requiring user deletion. - Ensured role changes are applied immediately and reflected correctly in permissions and access control. - Updated feedback listings to show only Open items by default. - Ensured Resolved items are always sorted to the bottom when viewing all feedback. - Preserved existing filtering, searching, and user-controlled sorting behavior. ### UI Improvements and Usability Enhancements - Introduced reusable ellipsis handling for long detail fields to prevent layout overlap. - Added click-to-expand behavior for truncated fields, with double-click support to expand and select all text. - Added automatic tooltips showing the full value when a field is truncated. - Removed the redundant “Objects” heading above objects tables to reduce visual clutter. - Applied truncation and expansion behavior consistently across Inbox, Deleted Mail, Run Checks, Daily Jobs, Job Detail views, and Admin All Mail. - Reset expanded ellipsis fields when Bootstrap modals or offcanvas components are opened or closed to prevent state leakage. - Fixed layout issues where the Objects table could overlap mail content in the Run Checks popup. ### Veeam Cloud Connect and VSPC Parsing - Improved the Veeam Cloud Connect Report parser by combining User and Repository Name into a single object identifier. - Excluded “TOTAL” rows from object processing. - Correctly classified red rows as Errors and yellow/orange rows as Warnings. - Ensured overall status is set to Error when one or more objects are in error state. - Added support for Veeam Service Provider Console daily alarm summary emails. - Implemented per-company object aggregation and derived overall status from the worst detected state. - Improved detection of VSPC Active Alarms emails to prevent incorrect fallback to other Veeam parsers. - Fixed a SyntaxError in the VSPC parser that caused application startup failures. ### VSPC Company Mapping Workflow - Introduced a dedicated company-mapping popup for VSPC Active Alarms summary reports. - Enabled manual mapping of companies found in mails to existing customers. - Implemented per-company job and run creation using the format “Active alarms summary | ”. - Disabled the standard approval flow for this report type and replaced it with a dedicated mapping workflow. - Required all detected companies to be mapped before full approval, while still allowing partial approvals. - Prevented duplicate run creation on repeated approvals. - Improved visibility and usability of the mapping popup with scroll support for large company lists. - Ensured only alarms belonging to the selected company are attached to the corresponding run. ### NTFS Auditing and Synology ABB Enhancements - Added full parser support for NTFS Auditing reports. - Improved hostname and FQDN extraction from subject lines, supporting multiple subject formats and prefixes. - Ensured consistent job name generation as “ file audits”. - Set overall status to Warning when detected change counts are greater than zero. - Improved Synology Active Backup for Business parsing to detect partially completed jobs as Warning. - Added support for localized completion messages and subject variants. - Improved per-device object extraction and ensured specific device statuses take precedence over generic listings. ### Workflow Simplification and Cleanup - Removed the “Mark success (override)” button from the Run Checks popup. - Prevented creation of unintended overrides when marking individual runs as successful. - Simplified override handling so Run Checks actions no longer affect override administration. - Ensured firmware update notifications (QNAP) are treated as informational warnings and excluded from missing-run detection and reporting. --- ## v0.1.19 This release delivers a broad set of improvements focused on reliability, transparency, and operational control across mail processing, administrative auditing, and Run Checks workflows. The changes aim to make message handling more robust, provide better insight for administrators, and give operators clearer and more flexible control when reviewing backup runs. ### Mail Import Reliability and Data Integrity - Updated the mail import flow so messages are only moved to the processed folder after a successful database store and commit. - Prevented Graph emails from being moved when parsing, storing, or committing data fails, ensuring no messages are lost due to partial failures. - Added explicit commit and rollback handling to guarantee database consistency before any mail state changes occur. - Improved logging around import, commit, and rollback failures to make skipped or retried mails easier to trace and troubleshoot. ### Administrative Mail Auditing and Visibility - Introduced an admin-only “All Mail” audit page that provides a complete overview of all received mail messages. - Implemented pagination with a fixed page size of 50 items to ensure consistent performance and predictable navigation. - Added always-visible search filters that can be combined using AND logic, including From, Subject, Backup, Type, Job name, and a received date/time range. - Added an “Only unlinked” filter to quickly identify messages that are not associated with any job. - Reused the existing Inbox message detail modal to allow consistent inspection of messages from the All Mail page. - Added a dedicated navigation entry so administrators can easily access the All Mail audit view. - Fixed modal opening behavior in the All Mail page to fully align click handling and popups with the Inbox implementation. ### Inbox and Mail Body Rendering Improvements - Treated whitespace-only email bodies as empty during import so HTML report attachments can be extracted and displayed correctly. - Added legacy fallback logic in the Inbox message detail API to extract the first HTML attachment from stored EML files when bodies are empty or invalid. - Improved iframe rendering in the Inbox so full HTML documents (commonly used for report attachments) are rendered directly instead of being wrapped. - Added detection for “effectively empty” HTML bodies, such as empty Graph-generated HTML skeletons. - Ensured that both newly imported and already-stored messages can dynamically fall back to EML attachment extraction without requiring a reset. ### Run Checks Usability and Control - Added a copy-to-clipboard icon next to ticket numbers in the Run Checks popup to quickly copy only the ticket code. - Prevented accidental selection of appended status text when copying ticket numbers. - Introduced a manual “Success (override)” action that allows Operators and Admins to mark a run as successful even if it originally failed or produced warnings. - Implemented backend support to persist the override state without modifying the original run data. - Updated UI indicators so overridden runs are clearly shown with a blue success status. - Ensured overrides apply only to the selected run and do not affect historical or related runs. - Improved Run Checks mail rendering by falling back to text bodies when HTML bodies are missing, matching Inbox and All Mail behavior. - Added support for extracting HTML content from stored EML files when both HTML and text bodies are unavailable. - Ensured plain-text emails are safely rendered using preformatted HTML to preserve readability. ### Customer, Ticket, and Scope Cleanup - Updated customer deletion logic to allow removal of customers even when tickets or remarks are linked. - Added explicit cleanup of related TicketScope and RemarkScope records prior to customer deletion. - Ensured jobs linked to a deleted customer are safely unassigned to prevent foreign key constraint errors. - Eliminated deletion failures caused by lingering ticket or remark relationships. ### Parser Enhancements and Informational Messages - Added parser support for 3CX SSL Certificate notification emails. - Classified these messages as Backup: 3CX with Type: SSL Certificate. - Parsed and displayed certificate information in the Run Checks popup. - Stored these messages as successful runs so certificate status can be tracked over time. - Added detection for Synology DSM automatic update cancellation messages in both Dutch and English. - Classified Synology Updates messages as informational and set their overall status to Warning. - Excluded Synology Updates informational messages from scheduling logic and reporting output. ### UI Layout and Status Accuracy Improvements - Moved the Details section above the email body in both Inbox and Job Details popups to improve readability. - Avoided long detail texts being constrained to narrow side columns. - Adjusted missed run detection to include a ±1 hour grace window around scheduled run times. - Prevented runs that arrive shortly after the scheduled time from being temporarily marked as Missed. - Ensured the Missed status is only applied after the full grace window has elapsed, reducing false alerts in Run Checks and Daily Jobs views. --- ## v0.1.18 This release focuses on improving ticket reuse, scoping, and visibility across jobs, runs, and history views to ensure consistent and flexible ticket handling. ### Ticket Linking and Reuse - Updated ticket linking logic to allow the same existing ticket number to be associated with multiple jobs and job runs. - Prevented duplicate ticket creation errors when reusing an existing ticket code. - Ensured existing tickets are consistently reused and linked instead of being rejected when already present in the system. ### Ticket Scope and Resolution - Fixed missing ticket number display in job and run popups by always creating or reusing a ticket scope when linking an existing ticket to a job. - Updated ticket resolution logic to support per-job resolution when resolving tickets from a job or run context. - Ensured resolving a ticket from the central Tickets view resolves the ticket globally and closes all associated job scopes. - Updated ticket active status determination to be based on open job scopes, allowing the same ticket number to remain active for other jobs when applicable. ### Job History – Tickets and Remarks - Added a Tickets and Remarks section to the Job History mail popup. - Aligned ticket handling in Job History with the existing Run Checks popup behavior. - Enabled viewing of active and resolved tickets and remarks per job run. - Added support for creating new tickets and remarks directly from the Job History popup. - Enabled resolving tickets and remarks directly from the Job History popup. - Ensured tickets and remarks are correctly scoped to the selected run (run_id). --- ## v0.1.17 ### Release Summary This release focuses on improving job normalization, ticket and remark handling, UI usability, and the robustness of run and object detection across the platform. ### Job normalization and aggregation - Veeam job names are now normalized to prevent duplicates: - Jobs with “(Combined)” and “(Full)” suffixes are merged with their base job names. - Ensures accurate aggregation, reporting, and statistics for Veeam Backup and Microsoft 365 jobs. - Added support for archiving inactive jobs while keeping all historical runs fully included in reports. ### Inbox and bulk operations - Introduced multi-select inbox functionality for Operator and Admin roles. - Added a bulk “Delete selected” action with validation, counters, and admin audit logging. ### Jobs UI and navigation - Restored row-click navigation on the Jobs page. - Moved the Archive action from the Jobs table to the Job Details page for consistency. - Improved layout and behavior of job run popups, ensuring objects are visible, correctly rendered, and consistently sorted. ### Tickets and remarks - Ticket creation now always uses a user-provided ticket code with strict format and uniqueness validation. - Editing of tickets and remarks has been fully disabled; items must be resolved and recreated instead. - Removed ticket description fields from creation and detail views to prevent inconsistent data. - Fixed backend indentation errors that previously caused startup and Bad Gateway failures. ### Customer deletion stability - Fixed foreign key constraint issues when deleting customers. - Customer deletion now safely unlinks jobs while preserving historical job and run data. - Enforced cascading deletes where appropriate to prevent integrity errors. ### Feedback handling - Users can now reply directly to Feedback items while they are in the “Open” state. - Replies are stored for audit and history tracking. ### Veeam parser improvements - Configuration Backup parser now correctly captures multi-line overall warning messages. - Improved date parsing to support formats without leading zeros. - Microsoft 365 parser now always persists overall warning/info messages, even on successful runs. ### Run checks and missed run detection - Improved weekly and monthly schedule inference to reduce false positives. - Monthly jobs are now detected and marked as missed on the correct expected date. - Added fallback mechanisms for loading run objects in Run Checks to support legacy and transitional data. Overall, this release significantly improves data consistency, usability, and reliability across jobs, tickets, inbox management, and run analysis. --- ## v0.1.16 This release significantly expands and stabilizes the reporting functionality, focusing on configurability, correctness, and richer output formats. ### Key Highlights - Reports are now **job-aggregated** instead of object-level, making them suitable for high-level status reviews. - Full **report lifecycle management** was added, including secure deletion and reliable refresh behavior. - Introduced **advanced reporting foundations** with configurable report definitions (columns, filters, layout, charts). - Added support for **multiple export formats**: CSV, HTML, and PDF, including graphical HTML previews and basic PDF charts. - Implemented extensive **column selection** across Summary, Snapshot, and Jobs views, with drag-and-drop ordering and persistence. - Added **job-level aggregated metrics**, including per-job success rates and charts over the selected period. - Improved **filtering** to exclude informational (non-backup) jobs and allow selection by backup software and type. - Ensured **success rate and total runs calculations** are correct and consistently based only on selected run statuses. - Added **Customer** as a selectable column and improved multi-customer report clarity. - Introduced configurable **HTML/PDF content scope** (Customers, Jobs, or both). - Fixed numerous **stability issues** (loading states, NameErrors, missing imports, endpoint collisions, JS errors). - Improved HTML report layout, table rendering, column labeling, sorting logic, and visual consistency. - Cleaned up column metadata, removed deprecated/duplicate options, and added migration logic to fix existing configurations. --- ## v0.1.15 This release focused on improving operational clarity and usability by strengthening dashboard guidance, introducing a solid reporting foundation, and significantly refining the Run Checks review workflow for daily accountability. ### Dashboard & User Guidance - Added a comprehensive explanatory section to the Dashboard describing how Backupchecks collects, normalizes, and evaluates backup results. - Clarified dashboard counters, job statuses, overrides, and operational workflows. - Introduced clear guidance emphasizing the goal of keeping the Run Checks page empty or near-empty as proof of daily review and accountability. - Positioned Backupchecks explicitly as a monitoring and validation layer that complements existing backup software. ### Navigation & Usability - Implemented automatic redirection to the Dashboard on a user’s first visit of each day, improving daily visibility of system status. - Refactored the Settings area into clearly separated sections (General, Imports, Users, etc.) for better usability and structure. ### Dashboard News - Added a per-user Dashboard News section with read/unread tracking. - Enabled full admin management of news items (create, edit, delete, pin, severity, publish window). - Provided admin insight into read statistics and the ability to reset read status. ### Reporting – Foundation - Introduced the initial reporting framework focused on raw, immutable data for auditability. - Added object-based reporting across multiple jobs with aggregated success, warning, failed, missed, and override metrics. - Prepared the data model for CSV export, future PDF generation, scheduling, and email delivery. - Enabled reporting access for admin, operator, and reporter roles in phase 1. ### Reports UI & Generation - Added an initial Reports UI to define, generate, and manage reports. - Enabled raw data preview directly in the UI with pagination support. - Implemented backend endpoints to generate and retrieve report data on demand. - Replaced the “New report” modal with a dedicated Create Report page. - Added month-based date selection, quick-select options, and multi-customer reporting support. - Enabled actual report generation while keeping scheduling as a non-functional placeholder. ### Run Checks – Multi-Select Improvements - Added Shift-click multi-selection for efficient bulk review. - Iteratively fixed edge cases including: - Off-by-one range selection issues - Checkbox and row highlight desynchronization - Last-row selection bugs - Browser text selection interference - Improved “Select all” checkbox behavior: - Clearing selection when indeterminate - Correctly resetting from the indeterminate state - Ensuring consistent mouse and keyboard interaction ### Overall Result - Improved clarity, accountability, and daily workflow guidance across the application. - Established a solid, extensible reporting foundation. - Significantly enhanced usability and reliability of the Run Checks review process. --- ## v0.1.14 ### Daily Jobs - Introduced a consistent, case-insensitive multi-level sort order for the Daily Jobs overview: Customer → Backup Software → Backup Type → Job Name. - Fixed backend ordering issues to ensure server-side data no longer overrides the intended sort logic. - Ensured sorting is applied before serialization so the UI always reflects the correct order. - Improved predictability and readability of job listings across environments. ### Veeam Backup for Microsoft 365 - Improved parsing of overall warning messages to correctly extract and display permission- and role-related issues. - Added support for combined permission and role warnings in M365 reports. - Ensured detailed permission warnings take precedence over generic “X of X objects processed” messages. - Fixed incorrect overall message selection and filtered out misleading banner fragments. - Resolved an indentation error in the parser that caused backend startup failures, restoring stability. ### Overrides – Configuration and Matching - Replaced free-text inputs with dropdowns for Backup Software and Backup Type in Overrides, including: - Alphabetical sorting - Preselection of existing values - A global option at the top of each dropdown - Fixed PostgreSQL compatibility issues by replacing DISTINCT queries with GROUP BY while preserving case-insensitive sorting. - Ensured Overrides endpoints no longer crash due to invalid query constructions. ### Overrides – Application, Editing, and Deletion - Made newly created overrides apply immediately and retroactively to all unreviewed runs by default. - Added full support for editing existing overrides and reapplying changes to unreviewed runs. - Restricted override deletion to Admin users and ensured proper reprocessing after removal. - Fixed datetime handling in override edit flows so unchanged values are preserved and NULL constraint violations are avoided. - Ensured Admin users always see delete actions by consistently passing permission flags to the UI. ### Overrides – Matching Logic Improvements - Extended override matching to use persisted run_object_links joined with customer_objects instead of legacy or non-existent relationships. - Improved global override matching by resolving backup software and type from MailMessage data when missing on jobs. - Added support for matching against object-level error messages as well as run-level remarks. - Ensured all override matching remains case-insensitive and consistent across run-level and object-level evaluations. ### Overrides – UI Indicators and Reporting - Introduced a blue status indicator for runs and jobs where overrides are applied. - Updated status labels to display “Success (override)” for clearer distinction without changing canonical stored statuses. - Added persistent override reporting metadata to job runs, including applied override ID, level, and reason. - Ensured dashboards, Daily Jobs, Run Checks, and popups correctly propagate and display override-based success states. - Fixed multiple UI rendering issues so overridden runs are no longer misclassified as warnings or missed jobs. ### Daily Jobs Popups - Fixed popup loading failures and backend unpacking errors related to override handling. - Ensured popup details consistently load correct run data. - Aligned popup override detection and status coloring with Run Checks and Daily Jobs overviews. --- ## v0.1.13 This release focuses on improving visibility and consistency of Tickets and Remarks across Run Checks and Job Details, alongside several UI fixes and backend stability improvements. ### Highlights - Added clear visual indicators for active Tickets and Remarks in the Run Checks overview. - Enhanced Job Details and Job History to display actual ticket numbers and related remark messages, both in tables and popups. - Improved navigation consistency by adding direct “Job page” links for Tickets and Remarks. ### Improvements - Job History popups now reliably show associated ticket numbers and remark content. - Backend job history data is enriched to support consistent UI rendering. - Missed-run detection now includes a ±1 hour tolerance window and respects the configured UI timezone. - Run Checks UI is simplified by hiding last-reviewed columns (data is still retained in the backend). ### Fixes - Resolved a backend indentation issue that caused Gunicorn startup failures. - Made frontend parsing of ticket/remark data more robust against malformed or unexpected payloads. - Fixed JSON encoding issues in HTML data attributes to prevent popup rendering errors. ### Changelog Update - Simplified the changelog by removing “Current Version” and “In testing” sections. - The changelog now only shows completed changes. --- ## v0.1.12 ### Dashboard & UI - Corrected dashboard counters so **Expected**, **Missed**, and **Success (override)** statuses are shown accurately. - Added dedicated counters for Expected and Success (override). - Fixed layout issues on the Inbox dashboard tiles and improved label wrapping. - Added safe timezone fallbacks to prevent incorrect status aggregation. - Restored missing status icons and symbols across Dashboard and Daily Jobs views. - Cleaned up Job Details UI by removing redundant columns and clarifying schedule display. - Extended Job History with weekday labels and review metadata (Admin-only visibility). ### Stability & Reliability - Fixed a Gunicorn startup crash caused by incorrect Python indentation. - Improved migration robustness for soft-delete columns to prevent startup 502 errors on busy databases. - Prevented duplicate or unintended regeneration of reviewed “Missed” runs. ### Inbox & Mail Handling - Introduced soft-delete for Inbox messages with full Admin restore capability. - Added an Admin-only “Deleted mails” page with audit details (who/when). - Added popup previews for deleted mails without requiring restore. - Improved HTML mail handling by extracting content from HTML attachments when the body is empty. - Added an Admin maintenance action to backfill HTML bodies from existing attachments. ### Feedback & Settings - Changed Feedback behavior so resolved items remain visible until explicitly deleted. - Restricted feedback deletion to Admin users only. - Added a User Settings page allowing users to change their own password securely. ### Backup Parser Enhancements - Improved Veeam parsing: - Fixed Health Check Summary parsing. - Normalized job names by stripping “(Retry)”. - Added and refined License Key parsing with correct status detection and exclusions from scheduling logic. - Added and expanded Synology support: - Active Backup for Business (NL/EN). - R-Sync (NL/EN). - Account Protection notifications (informational, no scheduling). - Added new parsers: - R-Drive Image. - Syncovery. - Ensured correct handling of objects, statuses, and scheduling exclusions where applicable. ### Changelog - Removed the “Planned” section from the Changelog. - Future planning is now handled exclusively via the Feedback page. --- ## v0.1.11 ### Stability & Bug Fixes - Fixed multiple page crashes caused by missing imports after refactoring (Jobs, Feedback, Run Checks, Inbox, Daily Jobs). - Resolved Jinja2 template errors and SQL/runtime issues related to timezone handling. - Improved robustness by explicitly importing shared helpers to prevent NameError exceptions. ### Run Checks & Review Workflow - Introduced a new **Run Checks** page to review job runs independently from Daily Jobs. - Displays all unreviewed runs with no time-based filtering. - Supports bulk review actions and per-job review via popups. - Added admin-only features: show reviewed runs, unmark reviewed runs, reviewer metadata, and full audit logging. - Enhanced popups to group runs per job, include missed runs, and show ticket/remark indicators. - Added per-job and per-popup status summaries using visual indicators only. ### UI & Visual Consistency - Unified all job and run status indicators to a single shape differentiated by color. - Added a clear status legend to the Dashboard, including the new **Expected** state. - Removed textual status labels across Daily Jobs and Run Checks for a cleaner UI. - Improved table layouts and widened content areas for better use of 1080p screens. - Ensured consistent indicator rendering across all pages. ### Timezone & Display Improvements - Added a configurable timezone setting in Settings. - Updated all frontend date/time rendering to use the configured timezone instead of UTC. - Fixed offset issues and restored missing timestamps across multiple pages. ### Missed Runs Logic - Refined missed run detection to rely only on historically received mail reports. - Prevented synthetic or never-run schedules from generating false missed runs. ### Settings & Maintenance - Stabilized **Delete all jobs** by adding schema-tolerant cleanup of all related foreign key references. - Refactored the Settings page layout using accordions and cards for improved clarity. - Improved alignment and usability of import/export and user management sections. ### Roles & Access Control - Added support for multiple roles per user with an active role switcher. - Fixed role-based menu rendering and ensured permissions are evaluated against the active role. - Ensured role switching consistently redirects to the Dashboard. ### Theme & UX Fixes - Fixed manual theme switching (Light/Dark/Auto) and ensured user preferences persist. - Corrected Inbox EML download functionality by restoring the missing shared import. Overall, this release significantly improves stability, review workflows, visual consistency, timezone correctness, and administrative reliability, while refining the operator experience and access control model. --- ## v0.1.10 ### Performance & Stability - Reworked **Re-parse all** to process inbox messages in controlled batches, preventing gateway and Gunicorn timeouts on large inboxes. - Added execution time guards to stop processing before proxy limits are reached. - Optimized job-matching queries and disabled session autoflush during batch operations to reduce database load. - Ensured auto-approval and persistence logic only finalize after a full, successful re-parse cycle. - Restored stable backend startup by fixing decorator ordering issues that caused 502 errors. ### Job Matching & Parsing - Fixed approved job imports to persist `from_address`, ensuring correct matching during re-parse. - Improved Veeam Backup Job parsing: - Extracted and stored multi-line warnings/errors and object-level details with preserved line breaks. - Ignored VM summary lines (e.g., “X of X VMs processed”) for overall status detection. - Prevented incorrect overall warnings when issues are object-level only. - Fixed regressions to ensure backup objects are consistently detected, stored, and displayed across all views. ### UI & UX Improvements - Added EML download support for Job Details and Daily Jobs, with automatic availability handling and proper 404s when missing. - Improved rendering to preserve line breaks (pre-wrap) in remarks, overall messages, and object details. - Reduced visual clutter by moving overall status/messages out of tables and into context-specific popups. - Standardized changelog version display by removing date suffixes. - Reordered main navigation for better consistency. ### Daily Jobs & Status Accuracy - Clarified Daily Jobs status logic: - Introduced **Expected** for backups not yet due. - Reserved **Missed** only for jobs past their final expected run time. - Added last remark excerpts and ensured object details are visible in Daily Jobs popups. ### Tickets, Remarks & Overrides - Introduced run-date–scoped ticket activity with `active_from_date`, ensuring accurate historical and current visibility. - Implemented identical scoping for remarks, preserving visibility across runs even after resolution. - Fixed resolve actions to redirect properly in the UI while keeping JSON responses for API/AJAX. - Improved override handling so changes apply immediately to existing job runs with correct priority resolution. ### New Features - Added a **Feedback** board with per-user upvoting, admin moderation (resolve/reopen, soft delete), database migrations, and navigation entry. ### Navigation - Updated menu order to: Inbox, Customers, Jobs, Daily Jobs, Tickets, Overrides, Reports, Settings, Logging, Changelog, Feedback. --- ## v0.1.9 ### Changelog System Improvements - Added and maintained multiple **Completed** changelog entries (v0.1.2 through v0.1.8) with correct release dates. - Ensured all existing Completed, Testing, and Planned changelog entries were preserved without loss. - Migrated the Completed changelog from markdown-based content to a **structured, non-markdown format** aligned with the Planned section. - Simplified changelog rendering logic to use explicit section titles and bullet handling instead of full markdown parsing. - Standardized formatting across all versions for long-term maintainability and consistent UI rendering. ### Bug Fixes & Stability - Fixed multiple backend **Python syntax and runtime errors** related to changelog definitions (missing commas, indentation issues, invalid list entries). - Resolved rendering issues where markdown content was displayed as plain text or collapsed incorrectly. - Restored application startup stability by fixing missing imports (`re`, `html`) and indentation errors in changelog-related routes. ### Refactoring & Maintainability - Refactored a large `routes.py` file into multiple smaller route modules. - Introduced a shared routes module for common imports, helpers, and access control. - Fixed NameError issues after refactoring by explicitly importing underscored helper functions that are not included via wildcard imports. - Ensured all split route modules retained full functional parity with the original implementation. ### Release Management Updates - Moved versions through **Testing → Completed** states correctly: - v0.1.7 marked as Completed (2025-12-23). - v0.1.8 added as Completed (2025-12-24) and restored as Current Version. - Testing advanced to v0.1.9. - Updated v0.1.8 release notes to document consistent job-matching and auto-approval behavior across all mail processing flows. - Verified no regressions in changelog structure or rendering after updates. Overall, v20251225 focused on **hardening the changelog system**, improving backend stability, cleaning up technical debt in routing, and ensuring consistent, reliable release tracking across the application. :contentReference[oaicite:0]{index=0} --- ## v0.1.8 ### Overview This release focuses on making job matching and auto-approval behavior fully consistent across manual inbox actions, automatic mail imports, and the “Re-parse all” process. It also fixes a critical backend startup issue introduced in the re-parse logic. ### Key Changes - Introduced a single, shared job-matching helper based on a full unique key: - From address - Backup software - Backup type - Job name - Updated manual inbox approval to reuse existing jobs when the unique key matches, instead of relying on customer-only matching. - Aligned inbox “Re-parse all” auto-approve logic with the same shared matching behavior. - Fixed automatic mail import auto-approve so it correctly: - Creates a JobRun - Marks the mail as approved - Moves the mail to history when a matching job exists ### Re-parse All Improvements - Auto-approve is now executed during “Re-parse all”, not only on initial mail import. - After re-parsing, all successfully parsed mails without a linked job are re-evaluated against existing jobs using the full unique key. - When a matching active job with auto-approve enabled is found: - The mail is automatically approved - The mail is linked to the job - The mail is moved to history - A corresponding job run is created and shown in Job History ### Fixes - Resolved an issue where “Re-parse all” previously only updated parse metadata and skipped auto-approve logic, causing historical mails not to appear in job history. - Fixed a SyntaxError in the re-parse auto-approve logic that caused backend startup failures (Bad Gateway). - Corrected try/except structure and indentation to ensure re-parse auto-approve runs safely per mail without breaking the overall process. ### Result - Job matching and auto-approval behavior is now consistent across all mail processing flows. - Historical mails are correctly linked to jobs and visible in job history. - Backend stability during startup and re-parse operations is restored. --- ## v0.1.7 ### Key Features - Introduced **export and import functionality for approved jobs**, allowing approved job definitions to be migrated between clean installations via JSON. - Import process automatically creates missing customers and updates existing jobs based on a unique job identity to prevent duplicates. ### Versioning & Changelog - Promoted version **v0.1.7** from *Testing* to *Completed*. - Introduced **v0.1.8** as the new *Testing* release. - Updated the changelog structure and testing notes to reflect active export/import functionality. ### Parser Enhancements - **Boxafe** - Improved parsing for Shared Drives, Domain Accounts (Email, Contact, Drive, Calendar), and handling of *Warning* statuses. - Corrected object detection logic, ensuring no false objects are created when jobs contain no object data. - Removed object parsing for Shared Drives backups entirely. - **Synology Hyper Backup** - Added full support for Dutch notification emails. - Improved status detection for Dutch phrasing. - Confirmed that no objects are parsed for Hyper Backup jobs. - **Veeam** - Added support for **Scale-out Backup Repository (SOBR)** notifications, including storage capacity metrics suitable for logging and graphing. - Added support for **Veeam Health Check** reports with correct object filtering to exclude summary-only entries. ### Job Approval & Auto-Approval Logic - Significantly refined approved job matching logic to prevent cross-customer approvals. - Improved auto-approve behavior during **Re-parse all**, Inbox reprocessing, and Graph imports. - Enhanced resilience against case differences, whitespace, unicode characters, and hidden formatting issues. - Iteratively refined normalization strategy: - Ultimately simplified matching to rely primarily on a normalized **From address**, while ensuring parser-consistent values for Backup, Type, and Job Name. - Ensured deterministic behavior during reprocessing by preventing mutation of message data. ### Stability Fixes - Fixed multiple crashes and approval errors caused by undefined or incorrect job name handling. - Resolved issues with duplicate job records (including `NULL` customer IDs) that previously blocked auto-approval. - Ensured consistent JobRun creation and mail linking during automatic imports and re-parsing. ### Notes - Due to changes in approval matching logic, **previously approved jobs are expected to be recreated** after these updates. --- ## v0.1.6 ### Fixed - Corrected auto-approve logic to ensure it is properly applied during automatic mail imports. - Prevented the **Re-parse all** action from re-processing emails that were already approved. - Ensured approved status is always respected and never overwritten during re-parsing or automatic imports. - Fixed multiple Jinja2 `TemplateSyntaxError` issues in the base layout that caused 500 Internal Server Errors. - Restored correct rendering of all pages affected by template errors, including Dashboard, Parsers, and Changelog. - Resolved Changelog page rendering issues by fixing dictionary access in templates and avoiding conflicts with built-in methods. ### Added - Introduced a centralized **Changelog** page containing: - Active production version - Testing version - Planned / Todo items - Completed changes - Added the Changelog entry point to the main navigation. - Applied a clear versioning convention, e.g. `v0.1.7 (v20251222-03)` for testing releases. - Marked version `0.1.6` as the active production release. ### Planned - Export and import of jobs to allow restoring approved jobs after a clean installation. - Always register “New license key is not available” as an error. - Support for a scale-out backup repository Cloud Connect Immutable parser. - Ability to attach EML files to Daily Jobs and Job Details. - Fix for Light/Dark theme switching so users can properly change themes. - Restrict ticket creation and editing to Operator and Admin roles only. ### Known Bugs - Emails that were previously approved remain in the Inbox instead of being removed, even though they appear auto-approved and linked to Jobs. --- ## v0.1.5 ### Overview This release focuses on restoring Microsoft Graph functionality, improving application reliability, and introducing a robust reset mechanism to allow a clean restart of the application state. ### Key Fixes - Restored Microsoft Graph folder retrieval by fixing an incorrect import that caused a `ModuleNotFoundError`. - Resolved failures in the automatic mail importer caused by `signal`-based timeout handling by replacing it with a thread-safe mechanism. - Fixed backend startup crashes and Bad Gateway errors related to the automatic mail importer. - Implemented missing backend logic required for automatic imports to function correctly. ### New Features - Added an **Application Reset** option in the Settings page. - Introduced a confirmation step to prevent accidental resets. ### Improvements & Changes - Implemented full backend support for a complete application reset. - Reset now clears all application data, including: - Approved and pending jobs - Imported and processed emails - Daily job runs - Logs - User-defined settings (system defaults are preserved) - Ensured database cleanup runs in the correct order to respect foreign key constraints. - Aligned automatic mail import logic with the existing manual import flow for consistent behavior. - Applied the automatic import cutoff date directly via a Microsoft Graph `$filter`, leaving older emails untouched in the inbox. ### Result - Graph API functionality is fully restored. - Automatic mail import runs reliably on its configured schedule. - The application can now be safely reset to a clean, fresh-install state when needed. --- ## v0.1.4 This release focuses on improving backend stability, database reliability, and consistency in object parsing and mail handling. ### Key Changes - Database migrations for tickets and remarks were stabilized by running each migration in its own transaction scope, preventing closed-connection errors during startup. - Backend startup issues causing Gunicorn failures and 502 Bad Gateway errors were resolved. - The title field was fully removed from tickets and remarks, simplifying both backend validation and UI forms to use only descriptive content. - Manual mail imports were aligned with the **Re-parse all** behavior, ensuring immediate and consistent object detection. - Object visibility on the Daily Jobs page was corrected for previously approved jobs. - Manual imports were hardened against Microsoft Graph timeouts by adding DNS preflight checks and safely skipping message moves when Graph is unreachable. ### Improvements - Eliminated `ResourceClosedError` exceptions during backend boot. - Increased reliability of migrations and overall application startup. - Ensured object parsing is consistently re-evaluated on every job run, with correct detection of added or removed objects. - Prevented internal server errors and Gunicorn worker timeouts caused by long-running external Graph operations. Overall, v0.1.4 significantly improves robustness, consistency, and fault tolerance across database migrations, job parsing, and manual mail imports. --- ## v0.1.3 ### Logging & Stability - Fixed logging persistence so log entries are consistently stored in the database. - Resolved cases where certain log events were not stored due to object lifecycle handling. - Improved reliability of log creation during background/asynchronous processes. - Corrected log retrieval so stored logs are properly fetched and shown in the web UI. - Added pagination to the logging overview (20 entries per page). - Extended the logging view to show all available log fields and fixed missing columns in the UI. ### UI & Table Layout Improvements - Improved the logging page usability by placing pagination controls at the top and keeping them available at the bottom. - Increased logging table width to better fit a 1080p layout. - Fixed column layout so all columns remain in consistent positions regardless of content length. - Updated status styling to use colored text only within the status column (Success, Warning, Error/Failed, Missed), including clear differentiation for overrides. - Fixed JavaScript errors in the Daily Jobs popup that prevented rendering. ### Jobs & Daily Jobs Enhancements - Standardized default sorting for both Jobs and Daily Jobs tables (Customer → Backup → Type → Job name). - Persisted the Daily Jobs start date setting in the database and ensured it reloads correctly in the Settings UI. - Corrected missed-status calculation to start from the configured Daily Jobs start date. - Improved Daily Jobs table readability: - Moved the number of runs into a dedicated Runs column. - Prevented layout shifting caused by variable text in the Last result column. - Restored the original Runs visual representation and adjusted placement for better readability. - Reduced the Last result column width so only status text is shown and the Runs column remains visible. ### Parsing & Data Normalization - Stripped retry suffixes like “(Retry 1)”, “(Retry 2)”, etc. from job names so retries don’t create separate job identities. - Extended the NAKIVO parser to support VMware Replication job emails: - Detects job type (Backup vs Replication) based on email content. - Improves replication job name parsing. - Extracts VM names from the Objects/Virtual Machines section. - Maps overall job status correctly for replication reports. ### Tickets & Remarks (New Feature Area) - Added database schema for globally unique, persistent tickets linked to job runs for long-term reporting: - New tables: tickets, ticket_scopes, ticket_job_runs. - Ticket codes are globally unique using format TYYYYMMDD.NNNN. - Tickets require at least one customer scope. - Added database schema for remarks with scoped attachment and persistent linkage to job runs: - New tables: remarks, remark_scopes, remark_job_runs. - Implemented a new Tickets page with tabbed navigation (Tickets / Remarks): - Overviews with filtering. - Detail views showing scopes, linked job runs, and actions. - Added indicators in Daily Jobs to show active tickets and/or remarks. - Added ticket/remark management in the job run popup (create, edit, resolve). - Introduced consistent icons for tickets, remarks, actions, and status indicators. - Added backend API endpoints for listing/creating/updating/resolving/linking tickets and remarks, plus an endpoint to retrieve all alerts for a specific job run. --- ## v0.1.2 This release focuses on improved parser support, more robust data cleanup, and a fully reworked in-app logging and object persistence system. ### Parser & support - Extended the Synology Hyper Backup mail parser with proper recognition of Strato HiDrive backups. - Added support for parsing job names from the “Backup Task:” field for Strato HiDrive. - Correct handling of successful runs without listed objects. - Added a Strato HiDrive example to the parser templates for validation and reference. ### Administration & cleanup - Introduced an admin-only action to delete all jobs in a single operation. - Ensured related run mails are moved back to the Inbox when jobs are deleted. - Fixed foreign key constraint issues by enforcing the correct deletion order: - run_object_links first - job_object_links next - then job runs and jobs - Stabilized the “Delete all jobs” action to fully clean up all related data. ### Logging (reworked) - Moved logging away from container/stdout logging to in-app logging. - Introduced AdminLog-based logging for: - Mail import - Auto-approval - Manual job approval - Job deletion - Added detailed logging per imported and auto-approved email. - Added summary logging at the end of each mail import run. - Ensured all relevant events are logged exclusively via the AdminLog table and visible on the Logging page. ### Object persistence - Restored persistence of parsed objects after manual approval of inbox mails. - Restored persistence of parsed objects during auto-approval (reparse-all). - Ensured objects from approved mails are: - Upserted into `customer_objects` - Linked to jobs via `job_object_links` (with first/last seen tracking) - Linked to runs via `run_object_links` (with status and error details) - Added centralized helper logic to ensure consistent object persistence. - Added an admin-only maintenance action to backfill missing object links for already approved runs. - Object persistence failures no longer block mail approval. - Daily Jobs and Run detail views correctly display objects again for both new and historical runs after backfilling.