From cdbb45fcfbb1e6398021dbe856ea378f5dfb5253 Mon Sep 17 00:00:00 2001 From: Ivo Oskamp Date: Mon, 9 Feb 2026 09:28:13 +0100 Subject: [PATCH] Update changelog with v0.1.23 release notes Consolidated all changes from 2026-02-06 through 2026-02-08: - Documentation System (33 pages, 13 with full content) - Audit Logging enhancements (settings, export, import) - Timezone-aware datetime display across all pages - Autotask improvements (direct links, auto-load, collapsible text) - Environment identification (sandbox banner) - Job visibility filtering (archived/inactive) - Multiple bug fixes and documentation corrections Co-Authored-By: Claude Sonnet 4.5 --- docs/changelog-claude.md | 5 + docs/changelog.md | 235 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 240 insertions(+) diff --git a/docs/changelog-claude.md b/docs/changelog-claude.md index a8cc825..48373b9 100644 --- a/docs/changelog-claude.md +++ b/docs/changelog-claude.md @@ -2,6 +2,11 @@ This file documents all changes made to this project via Claude Code. +## [2026-02-09] + +### Changed +- Updated `docs/changelog.md` with comprehensive v0.1.23 release notes consolidating all changes from 2026-02-06 through 2026-02-08 (Documentation System, Audit Logging, Timezone-Aware Display, Autotask Improvements, Environment Identification, Bug Fixes) + ## [2026-02-08] ### Changed diff --git a/docs/changelog.md b/docs/changelog.md index 44cfe9c..00af03a 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -1,3 +1,238 @@ +## v0.1.23 + +This comprehensive release introduces a complete built-in documentation system with 33 pages covering all features, enhanced audit logging for compliance and troubleshooting, timezone-aware datetime display throughout the application, and numerous Autotask PSA integration improvements for better usability and workflow efficiency. + +### Documentation System – Complete Built-in Wiki + +**Core Infrastructure:** +- New `/documentation` route with dedicated blueprint (doc_bp) accessible via 📖 icon in main navbar +- Hierarchical structure with 9 sections and 33 pages covering all application features +- Sidebar navigation with collapsible sections, active page highlighting, and sticky positioning during scrolling +- Breadcrumb navigation for current location context +- Previous/Next pagination buttons for sequential reading +- Custom CSS (`documentation.css`) with dark mode support via CSS variables +- Callout boxes (info, warning, tip, danger) for highlighting important content +- Support for code blocks, tables, images (centered horizontally), and responsive design +- Login required but accessible to all user roles +- Full content includes comprehensive screenshots, workflow examples, troubleshooting guides, and best practices + +**Completed Documentation Sections (13 pages with full content):** + +*Getting Started (3 pages):* +- What is BackupChecks: Overview, key features, architecture, supported backup software +- First Login: Initial setup, role-based menu visibility, theme/role switching, navigation tips +- Quick Start Guide: Complete 5-step setup workflow from mail import to daily review + +*User Management (3 pages):* +- Users & Roles: User management interface, role descriptions (Admin/Operator/Reporter/Viewer), role assignment via checkboxes, password reset, admin protection +- Login & Authentication: Login process with captcha (simple math problem), browser compatibility (Firefox recommended), HTTPS context for external access, session management +- Profile Settings: Password change functionality, theme selector in navbar (not settings page), role switcher in navbar + +*Customers & Jobs (4 pages):* +- Managing Customers: Customer creation, editing, activation/deactivation, Autotask company mapping with auto-search, CSV export/import with Autotask fields, deletion +- Configuring Jobs: Inbox-based approval workflow, customer selection (job name read-only), Mail Parser automatic configuration, Reparse All functionality, job archiving/deletion +- Approved Jobs: Jobs list overview, job details with history, status indicators, archive/unarchive workflow, JSON export/import for migration +- Job Schedules: Automatic schedule learning from patterns (daily/weekly/monthly), Daily Jobs integration with schedule indicators, requires 10-14 days of data, troubleshooting + +*Mail & Import (4 pages with comprehensive security guidance):* +- Mail Import Setup: Microsoft Graph API configuration, Azure AD app registration with principle of least privilege (start with Mail.Read, add Mail.ReadWrite after testing), Application Access Policy via Exchange PowerShell to restrict to one mailbox, two-step save workflow clarification (save credentials first to enable folder browser, then save complete config), automatic folder browser (no manual path entry), connection testing, troubleshooting authentication and import issues, email retention configuration, advanced multi-mailbox setup +- Inbox Management: Inbox overview (shows ONLY unapproved emails, approved disappear immediately), table columns with EML download links, email detail modal with two-column layout (left: metadata/actions/objects, right: email iframe), approve workflow (customer selection only, job name read-only), what happens after approval (auto-approve future emails), re-parsing (only "Re-parse all" button, no individual re-parse), single/bulk delete with soft delete for Admin, downloading .eml files, common workflows for new customer setup and troubleshooting, best practices for verifying parsed job names +- Mail Parsing: Parsing workflow (retrieval, preprocessing, parser selection, matching, parsing, storage), viewing parsers on dedicated Parsers page (Admin-only, read-only list), match criteria (sender/subject/body), supported backup software list (Veeam, Synology, NAKIVO, Boxafe, Panel3, QNAP, Syncovery, 3CX, RDrive, NTFS Auditing), parser execution order, re-parsing emails via "Re-parse all" only, parsing workflow examples, troubleshooting (contact support for parser issues - users cannot enable/disable/modify parsers), parser limitations, best practices +- Auto-Import Configuration: Auto-import overview, enabling/disabling toggle, import interval (5/15/30/60 minutes), batch size explanation, detailed workflow (timer → authenticate → fetch → parse → auto-approve → move → log → persist), auto-approval logic and conditions, manual import trigger (batch size 1-50), settings summary table, monitoring via audit log and inbox, troubleshooting auto-import issues, best practices for high-volume environments, EML retention configuration + +*Backup Review (5 pages with detailed workflows):* +- Approving Backups: Complete backup review lifecycle with 7 stages (Email Import & Parsing → Inbox Approval → Automatic Processing → Daily Monitoring → Run Review → Issue Tracking → Mark as Reviewed), stage-by-stage workflow descriptions with timestamps, complete workflow example from Day 1 new customer onboarding through Day 15 issue resolved, role-based workflows (Operator: review/create tickets/remarks/overrides/mark reviewed; Admin: plus view reviewed and unmark and delete; Viewer: read-only Daily Jobs), performance tips (filters, review by exception with overrides, batch approve, global overrides for common warnings, tickets for workload tracking, archive resolved tickets), best practices (review daily, approve inbox quickly, triage by status, use overrides for recurring warnings, create tickets for customer action, use remarks for temporary notes, always check objects, document in comments, resolve promptly, monitor schedule inference) +- Daily Jobs View: Schedule inference from historical patterns (weekly/monthly detection requires 10-14 days data), status tracking (Success/Warning/Failed/Missed/Expected badges with color coding), missing job detection, override indicators with blue badges showing treated-as-success status, ticket (🎫) and remark (💬) indicators, date selection with timezone support, job table columns (customer, backup software/type, job name, schedule, last run, status, reviewed checkmark Admin-only), schedule explanation (daily/weekly/monthly patterns, irregular/-), viewing job details via Run Checks modal, reviewing jobs (mark as reviewed for warnings/failures, successful runs also need review), common workflows (morning review, checking previous days, identifying schedule changes), best practices (review daily, focus on failures first, watch for missed jobs, use overrides for known issues, create tickets for recurring issues), troubleshooting (job not appearing, schedule shows irregular, missed status issues), clarified Daily Jobs is for viewing schedules and Run Checks is primary operational tool +- Run Checks Modal: THE primary daily operational tool (goal: completely empty page), detailed review interface for investigating backup runs (opens from Daily Jobs or Job History), Admin/Operator access only (not Viewer), modal layout (job header, run list left showing unreviewed by default with Admin toggle for reviewed, run details right with status/objects/email/Autotask), review actions (Mark as reviewed acknowledges issues and makes all runs for job disappear together, Unmark reviewed Admin-only per-job action), linking tickets/remarks from modal, backup objects table (name/type/status/message, only if parser extracted them), email body rendering (HTML with formatting, plain text preformatted, sandboxed iframe), Autotask fields (ticket number with "Open in Autotask" direct link button, status, resolved origin, resolved at, deletion info), common workflows (investigating failures, comparing multiple runs, reviewing warnings with overrides, troubleshooting missed), best practices (always check objects, read error messages carefully, review full email body, create tickets for recurring issues, use remarks for temporary issues, compare historical runs), troubleshooting (no objects shown, email body blank, can't mark as reviewed, run list empty), corrected review mechanism from per-run to per-JOB (when marking job as reviewed, ALL runs within that job marked simultaneously), bulk review via multiple job checkboxes (not run checkboxes), blue badge for override-applied runs +- Overrides & Exceptions: Override rules for automatically handling known issues and expected warnings, two levels (global: by backup software/type affecting multiple jobs/customers; object-level: specific job or object checked first), match criteria (status: any/success/warning/failed; error text: contains/exact/starts_with/ends_with modes), treat as success action (enabled by default, displays blue badge with override indicator instead of original status for visual differentiation - green=genuine success, blue=override applied, yellow/red=unexpected problems), time windows (From date optional for retroactive application to existing runs, Until date optional for permanent/temporary overrides), creating overrides (form with level/software/type/job/object/status/error match/time window/comment fields, retroactive application to unreviewed runs), managing overrides (table showing level/scope/time/active/comment, Edit to modify, Disable/Enable toggle, Delete Admin-only), override evaluation order (object-level first → job-level → global, first match wins), common scenarios with examples (Veeam retry warnings, planned maintenance windows, known VM issue, global NAKIVO replication warnings), best practices (start specific then generalize, always document with comments/ticket numbers, use time windows for temporary issues, review periodically, be specific with error text, test before global, disable don't delete when unsure), troubleshooting (override not applied - check active/time window/match criteria; override too broad - check level and error match; multiple conflicts - most specific wins) +- Remarks & Tickets: Two documentation mechanisms (tickets: external followup requiring tracking in PSA; remarks: internal notes/known issues/temporary problems), accessing via Tickets page with two tabs (Tickets/Remarks, filtering by status/customer/backup software/type/search), ticket properties (ticket code auto-generated format, description, active from date, start date, resolved at, scopes defining affected jobs), creating tickets (from Run Checks modal or Tickets page, auto-linked to job run with scopes auto-populated, manual creation without PSA integration mentioned), ticket scopes (customer/backup software/type/job/run levels for automatic indicators), viewing ticket details (status, code, dates, scopes list, linked runs last 20), resolving tickets (marks with timestamp, remains visible with ✅, 🎫 removed from displays, Admin/Operator only), linking tickets to additional runs, remark properties (body freeform text, start date, resolved at, scopes), creating/viewing/resolving remarks (similar to tickets but simpler), filtering options, tickets vs remarks comparison table, indicators in Daily Jobs and Job History (🎫 active tickets, 💬 active remarks), common workflows (creating ticket for recurring failure, adding remark for planned maintenance, resolving after customer action, reviewing historical tickets), best practices (create tickets promptly, use descriptive codes referencing PSA, set accurate active from dates, resolve promptly, use remarks for temporary issues, link to multiple runs, review monthly, use remarks for documentation), Autotask integration section corrected to clarify tickets are manually created (not automatically on failures), manual ticket creation and PSA synchronization only, troubleshooting (indicator not showing - check active status/date/scope; cannot create - check role permissions; scope affecting wrong jobs - review breadth), ticket editing currently disabled (resolve old and create new) + +**Placeholder Pages (20 pages with basic structure for future content):** +- Reports (4 pages): Creating Reports, Relative Periods, Scheduling, Exporting Data +- Autotask Integration (4 pages): Setup, Company Mapping, Creating Tickets, Ticket Management +- Settings (6 pages): General, Mail, Autotask, Reporting, User Management, Maintenance +- Troubleshooting (3 pages): Common Issues, FAQ, Support Contact + +**Documentation Accuracy Corrections:** +- Fixed critical inaccuracies based on user feedback throughout Backup Review section +- Corrected Run Checks from "modal" to "primary daily operational tool" with goal of completely empty page +- Emphasized ALL runs must be reviewed regardless of status (including successful runs to prove accountability) +- Corrected review mechanism from per-run to per-JOB (marking one job reviews all its runs simultaneously) +- Corrected bulk review from "select multiple run checkboxes" to "select multiple job checkboxes" to reflect per-job mechanism +- Updated override status indicators to show blue badges (not green) for visual differentiation from genuine successes +- Fixed Daily Jobs purpose from "primary operational dashboard" to "schedule viewing tool" with callout that it may be deprecated in future +- Removed non-existent features: individual email re-parse in modal, parser enable/disable UI, AI-powered parsing, Inbox filters section +- Fixed various UI descriptions: EML column shows download link (not checkmark), customer selection workflow (job name read-only), email detail modal two-column layout, "Re-parse all" button location (inbox page only), parser management (read-only list on separate Parsers page), two-step save workflow for Mail Import (same button clicked twice), Application Access Policy security requirement +- Corrected login authentication: added captcha requirement (simple math problem), clarified HTTPS context only for external access (not internal IP deployments) +- Fixed user management: checkboxes for role assignment (not comma-separated), admin protection, password reset section, removed non-existent email field +- Streamlined Profile Settings: only password change in User Settings page, theme/role selectors in navbar (not settings page), removed non-existent notification preferences and session information sections +- Enhanced Mail Import Setup with comprehensive security best practices: principle of least privilege (start Mail.Read only), Application Access Policy to restrict to one mailbox, add Mail.ReadWrite only after testing + +### Audit Logging Enhancements + +**System Renaming and Semantic Clarity:** +- Renamed AdminLog to AuditLog for better semantic clarity across codebase +- Updated model name: AdminLog → AuditLog (backwards compatible alias maintained) +- Updated table name: admin_logs → audit_logs (automatic migration) +- Updated function: log_admin_event() → log_audit_event() (alias provided for compatibility) +- Updated UI labels: "Admin activity" → "System Audit Log" in logging page header +- Better reflects purpose as comprehensive audit trail for both user and system events + +**Expanded Event Coverage for Compliance:** +- **Settings Changes**: Now logs all changes to General, Mail, and Autotask settings + - Tracks which settings changed with old value → new value comparison + - Event types: `settings_general`, `settings_mail`, `settings_autotask` + - Excludes sensitive data (passwords are never logged) + - Example logged fields: ui_timezone, require_daily_dashboard_visit, is_sandbox_environment, graph_mailbox, autotask_enabled, autotask_environment, ticket defaults +- **Export Operations**: Logs when users export data with full traceability + - **Customers export** (event type: `export_customers`): CSV format, record count + - **Jobs export** (event type: `export_jobs`): JSON schema version (approved_jobs_export_v1), customer count, job count +- **Import Operations**: Logs when users import data with detailed operation counts + - **Customers import** (event type: `import_customers`): CSV format, created count, updated count, skipped count + - **Jobs import** (event type: `import_jobs`): JSON schema version, customer operations (created/updated/skipped), job operations (created/updated/skipped) +- All logging uses structured event_type for filtering and analysis +- Detailed JSON in details field for complete audit trail +- Maintains 7-day retention policy for performance +- No performance impact (async logging) +- Helps with compliance audits, security monitoring, and troubleshooting configuration issues + +### Timezone-Aware Datetime Display + +**Consistent Datetime Handling Across Application:** +- Added `local_datetime` Jinja2 template filter to convert UTC datetimes to configured UI timezone +- All datetime fields now automatically display in configured timezone (Settings > General > UI Timezone) +- Database values remain stored in UTC for consistency and reliability +- **Affected Pages**: Customers (Autotask last sync time), Settings (reference data last sync), Feedback (created/updated/resolved timestamps), Logging (event timestamps), Overrides (from/until dates), Archived Jobs (archived at), Tickets (active from/start/resolved dates), Remarks (start/resolved dates), Inbox (received timestamps), Job Detail (run timestamps), Admin Mail (received timestamps) +- Custom format support via strftime parameter (e.g., `|local_datetime('%d-%m-%Y %H:%M')`) +- Enhanced filter to handle both datetime objects and string datetime values for flexibility +- Graceful fallback to UTC display if timezone conversion fails +- Default timezone: Europe/Amsterdam (configurable via SystemSettings.ui_timezone) +- Improves operator experience by showing times in local context + +### Autotask PSA Integration Improvements + +**Usability Enhancements:** +- Added "Open in Autotask" button in Run Checks modal next to ticket number for direct navigation to tickets in Autotask PSA + - Opens in new tab with proper URL format including workspace and ticket ID parameters + - Button only appears when ticket is linked (autotask_ticket_id exists) and Autotask integration is enabled + - Styled as small outline button to maintain compact layout + - Eliminates manual ticket lookup in Autotask +- Added collapsible text functionality (ellipsis-field) to "Overall remark" and "Remark" fields in Run Checks modal + - Prevents long text from pushing action buttons off-screen or hiding content + - Click to expand/collapse long text + - Improves modal usability with lengthy error messages +- Auto-load Autotask reference data when opening Settings page + - Automatically fetches queues, ticket sources, statuses, and priorities on page load + - Only triggers when Autotask is enabled, credentials are configured, and cache is empty + - Eliminates need to manually click "Refresh reference data" before selecting defaults + - Displays info message with loaded data counts (e.g., "Loaded 5 queues, 3 sources, 8 statuses, 4 priorities") + - Improves first-time configuration workflow +- Renamed "Refresh" button to "Search" in Link existing Autotask ticket modal for better clarity + - Button performs a search operation, not a refresh operation + - Reduces user confusion during ticket linking workflow + +**Settings Organization and Validation:** +- Reorganized Autotask settings into two separate forms with dedicated save buttons for better UX + - **Autotask Settings** form (connection, environment, API credentials, tracking identifier) with "Save Autotask Settings" button inside card + - **Ticket Defaults** form (queue, source, status, priority) with "Save Ticket Defaults" button inside card + - Clear visual separation improves user experience and prevents accidental incomplete saves +- All fields marked as required with red asterisks (*) +- HTML5 validation prevents saving incomplete configurations +- Protected default Ticket Status value against accidental clearing + - Only updates when a valid status is selected from dropdown + - Only allows clearing if reference data is loaded (dropdown has options) + - Preserves existing value if dropdown is empty to prevent data loss + - Fixes issue where "Create Autotask ticket" failed due to missing default status after saving settings with empty dropdown + +**Data Portability and Migration Support:** +- Extended **Customer CSV export/import** to include Autotask company mappings + - Export now includes `autotask_company_id` and `autotask_company_name` columns + - Import reads Autotask mapping fields and applies them to existing or new customers + - Backwards compatible - old CSV files without Autotask columns still work correctly + - Import resets `autotask_mapping_status` and `autotask_last_sync_at` to allow resynchronization after import +- Extended **Jobs JSON export/import** to include Autotask fields in customers array + - Export includes Autotask fields (`autotask_company_id`, `autotask_company_name`) in customers array + - Import processes customers array first, applying Autotask mappings before creating jobs + - Schema remains `approved_jobs_export_v1` for backwards compatibility + - Import message now shows both created and updated customer counts +- Enables preservation of Autotask company mappings during system reset/migration workflows +- Critical for disaster recovery and environment promotion (dev → test → production) + +### Environment Identification and Safety + +**Sandbox/Development Environment Indicator:** +- Added visual banner system to distinguish non-production environments and prevent accidental actions +- New `is_sandbox_environment` boolean column in `SystemSettings` model (defaults to `False` for production safety) +- Database migration `migrate_system_settings_sandbox_environment()` for automatic schema update (idempotent, safe to run multiple times) +- Settings UI with new "Environment" card in Settings > General (placed after Navigation section) + - Toggle switch with clear description: "Mark this as a Sandbox/Development environment" + - Help text explains visual banner behavior +- CSS styling via `sandbox.css`: + - Diagonal red banner in top-left corner displaying "SANDBOX" in uppercase + - Position: top-left corner, rotated 45 degrees with letter-spacing for visibility + - Color: Bootstrap danger red (#dc3545) with white text and box-shadow for depth + - Z-index: 9999 (always on top, visible across all pages) + - `pointer-events: none` - banner itself is non-interactive, elements behind it remain clickable +- Banner conditionally displayed only when setting is enabled +- Banner placed in base template directly after `` tag, before navbar +- Helps prevent accidental production-like actions in test/development systems (e.g., sending tickets to real customers) + +### Job Visibility and Filtering Improvements + +**Active Job Filtering:** +- Jobs from archived jobs and inactive customers no longer appear in Daily Jobs, Run Checks, and Jobs list pages +- Added customer active status filtering to all job list queries in backend +- Jobs now filtered where customer is NULL or active=True alongside existing archived=False filter +- Prevents showing jobs with "-" status from deleted customers or archived jobs +- Reduces clutter in operational views and improves focus on active jobs +- Improves query performance by reducing result set size + +### Repository Management and Security + +**Protected Directories:** +- Added `.gitignore` file to protect confidential `.claude` directory +- Directory contains AI assistant context, memory files, and user-specific configuration +- Prevents accidental commits of sensitive information to version control +- Protects user privacy and prevents repository pollution with local-only data + +### Bug Fixes and Stability Improvements + +**Backend Fixes:** +- Fixed missing import for `_log_admin_event` (now `_log_audit_event`) in routes_customers + - Resolved crash when performing customer operations with audit logging +- Enhanced `local_datetime` filter to handle string datetime values in addition to datetime objects + - Prevents template rendering errors when datetime is stored as string + - Automatically converts string to datetime before timezone conversion + +**UI and CSS Fixes:** +- Fixed callout box formatting throughout documentation + - Corrected CSS specificity to only make first `` tag in callout boxes block-level + - Prevents nested strong tags from causing layout issues + - Applied fixes to Quick Start, First Login, and all other documentation pages +- Centered all documentation images horizontally (display: block, margin: 20px auto) + +**Documentation Content Corrections:** +- Removed fabricated and non-existent features from documentation + - Removed "AI-powered parsing" future enhancement (not in planning, completely fabricated) + - Removed "Inbox Filters" section (not in planning) + - Removed "Enabling and Disabling Parsers" section (feature doesn't exist - users cannot manage parsers) + - Removed individual email re-parse from modal (only "Re-parse all" on inbox page) +- Corrected workflow descriptions to match actual UI implementation + - Fixed customer selection workflow (job name is read-only during approval, cannot be edited) + - Fixed email detail modal layout (two-column: left metadata/actions/objects, right email iframe) + - Fixed parser viewing location (separate Parsers page, Admin-only, not in Settings) + - Fixed Mail Import setup workflow (two-step save: credentials first, then complete config) +- Updated status badge colors and descriptions to match actual behavior + - Blue badges indicate override-applied runs (not green) + - Green badges only for genuine successes + - Expected status badge color corrections +- Fixed per-job review mechanism documentation + - Clarified marking one job as reviewed marks ALL runs for that job + - Corrected bulk review from "select run checkboxes" to "select job checkboxes" + - Updated "Unmark Reviewed" to reflect per-job unmarking + +--- + ## v0.1.22 This major release introduces comprehensive Autotask PSA integration, enabling seamless ticket management, customer company mapping, and automated ticket lifecycle handling directly from Backupchecks. The integration includes extensive settings configuration, robust API client implementation, intelligent ticket linking across job runs, and conditional ticket status updates based on time entries.