backupchecks/docs/changelog-claude.md

# Changelog - Claude Code

This file documents all changes made to this project via Claude Code.

## [2026-02-08]

### Added
- Added comprehensive documentation system for user onboarding and reference:
  - **Documentation Blueprint**: New `/documentation` route with dedicated blueprint (doc_bp)
  - **Navigation Structure**: Hierarchical documentation with 9 sections and 33 pages covering all features
    - Getting Started (3 pages): What is BackupChecks, First Login, Quick Start
    - User Management (3 pages): Users & Roles, Authentication, Profile Settings
    - Customers & Jobs (4 pages): Managing Customers, Configuring Jobs, Approved Jobs, Job Schedules
    - Mail & Import (4 pages): Setup, Inbox Management, Mail Parsing, Auto-Import
    - Backup Review (5 pages): Approving Backups, Daily Jobs, Run Checks Modal, Overrides, Remarks & Tickets
    - Reports (4 pages): Creating Reports, Relative Periods, Scheduling, Exporting Data
    - Autotask Integration (4 pages): Setup, Company Mapping, Creating Tickets, Ticket Management
    - Settings (6 pages): General, Mail, Autotask, Reporting, User Management, Maintenance
    - Troubleshooting (3 pages): Common Issues, FAQ, Support Contact
  - **UI Components**:
    - Sidebar navigation with collapsible sections and active page highlighting
    - Breadcrumb navigation for current location context
    - Previous/Next pagination buttons for sequential reading
    - Documentation menu item in main navbar (📖 icon) visible to all authenticated users
  - **Styling**:
    - Custom CSS with support for dark mode
    - Callout boxes (info, warning, tip, danger) for highlighting important content
    - Code blocks, tables, and image support
    - Responsive design for mobile and desktop
  - **Access Control**: Login required (@login_required) - accessible to all user roles
  - **Current Status**: Core infrastructure complete, getting-started section (3 pages) and users section (3 pages) completed with comprehensive content
  - **Placeholder Pages**: Remaining 27 pages created with basic structure for future content
  - Full content for remaining sections will be added incrementally in future updates

### Changed
- Enhanced documentation accuracy in Users section:
  - **Login & Authentication page**: Added captcha requirement explanation (simple math problem), noted future plan to make it optional via Settings
  - **Browser Security section**: Clarified HTTPS context - only relevant for external access via domain name, not for internal IP-based deployments (typical use case)
  - **Browser Compatibility**: Marked Mozilla Firefox as "Recommended - Most tested" browser for optimal experience
  - **Profile Settings page**: Streamlined to reflect actual functionality - only password change available in User Settings page, theme and role selectors located in navigation bar

### Removed
- Removed non-existent features from Profile Settings documentation:
  - Notification Preferences section (no plans for implementation)
  - Session Information section (not displayed in User Settings)
  - Redundant theme/role configuration sections (these are in navbar, not settings page)

## [2026-02-07]

### Changed
- Renamed AdminLog to AuditLog for better semantic clarity:
  - **Model**: AdminLog → AuditLog (backwards compatible alias maintained)
  - **Table**: admin_logs → audit_logs (automatic migration)
  - **Function**: log_admin_event() → log_audit_event() (alias provided)
  - Better reflects purpose as comprehensive audit trail for both user and system events
- Updated UI labels to reflect audit log semantics:
  - Changed "Admin activity" to "System Audit Log" in logging page header

### Added
- Expanded audit logging for critical operations:
  - **Settings Changes**: Now logs all changes to General, Mail, and Autotask settings
    - Tracks which settings changed with old value → new value comparison
    - Event types: `settings_general`, `settings_mail`, `settings_autotask`
    - Excludes sensitive data (passwords are never logged)
    - Example logged fields: ui_timezone, require_daily_dashboard_visit, is_sandbox_environment, graph_mailbox, autotask_enabled
  - **Export Operations**: Logs when users export data
    - **Customers export** (event type: `export_customers`): CSV format, record count
    - **Jobs export** (event type: `export_jobs`): JSON schema version, customer/job counts
  - **Import Operations**: Logs when users import data
    - **Customers import** (event type: `import_customers`): CSV format, created/updated/skipped counts
    - **Jobs import** (event type: `import_jobs`): JSON schema version, all operation counts (customers and jobs)
  - All logging uses structured event_type for filtering and includes detailed JSON in details field
  - Maintains 7-day retention policy
  - No performance impact (async logging)
  - Helps with compliance, troubleshooting, and security audits

## [2026-02-06]

### Added
- Added Sandbox/Development environment indicator feature:
  - **Database Model**: Added `is_sandbox_environment` boolean column to `SystemSettings` model (defaults to `False` for production safety)
  - **Database Migration**: Created `migrate_system_settings_sandbox_environment()` function for automatic schema update on deployment (idempotent, safe to run multiple times)
  - **Backend Routes**:
    - Extended `routes_settings.py` to process the new checkbox setting in the General tab
    - Updated `routes_shared.py` context processor to inject `system_settings` into all template contexts
  - **Settings UI**:
    - Added new "Environment" card in Settings > General (placed after Navigation section)
    - Toggle switch with clear English description: "Mark this as a Sandbox/Development environment"
    - Help text explains visual banner behavior
  - **CSS Styling**:
    - Created `sandbox.css` with diagonal red banner styling
    - Position: top-left corner, rotated 45 degrees
    - Color: Bootstrap danger red (#dc3545) with white text
    - Z-index: 9999 (always on top)
    - `pointer-events: none` - banner itself is non-interactive, elements behind it remain clickable
    - Box-shadow for depth effect
  - **Base Template**:
    - Banner conditionally displayed only when setting is enabled
    - CSS include added to `<head>` section
    - Banner placed directly after `<body>` tag, before navbar
  - The banner displays "SANDBOX" in uppercase with letter-spacing for clear visibility across all pages
- Auto-load Autotask reference data on Settings page load:
  - Automatically fetches queues, ticket sources, statuses, and priorities when opening Settings
  - Only triggers when Autotask is enabled, credentials are configured, and cache is empty
  - Eliminates need to manually click "Refresh reference data" before selecting defaults
  - Displays info message with loaded data counts
- Extended export/import functionality to include Autotask company mappings:
  - **Customer Export/Import** (CSV at /customers/export):
    - Export now includes `autotask_company_id` and `autotask_company_name` columns
    - Import reads Autotask mapping fields and applies them to existing or new customers
    - Backwards compatible - old CSV files without Autotask columns still work
    - Import resets `autotask_mapping_status` and `autotask_last_sync_at` to allow resynchronization
  - **Jobs Export/Import** (JSON at Settings > Maintenance):
    - Export now includes Autotask fields in customers array (`autotask_company_id`, `autotask_company_name`)
    - Import processes customers array first, applying Autotask mappings before creating jobs
    - Schema remains `approved_jobs_export_v1` for backwards compatibility
    - Import message now shows both created and updated customer counts
  - Enables preservation of Autotask company mappings during system reset/migration workflows
- Implemented timezone-aware datetime display across all pages:
  - **Template Filter**: Added `local_datetime` Jinja2 filter to convert UTC datetimes to UI timezone
  - **Automatic Conversion**: All datetime fields now automatically display in configured timezone (Settings > General > UI Timezone)
  - **Database**: All datetime values remain stored in UTC for consistency
  - **Affected Pages**: Customers (Autotask sync time), Settings (reference data sync), Feedback, Logging, Overrides, Archived Jobs, Tickets, Remarks, Inbox, Job Detail, Admin Mail
  - **Custom Format Support**: Filter accepts strftime format parameter (e.g., `|local_datetime('%d-%m-%Y %H:%M')`)
  - **Graceful Fallback**: Falls back to UTC display if timezone conversion fails
  - **Default Timezone**: Europe/Amsterdam (configurable via SystemSettings.ui_timezone)

### Changed
- Renamed "Refresh" button to "Search" in Link existing Autotask ticket modal for better clarity (the button performs a search operation)
- Added ellipsis-field functionality to "Overall remark" and "Remark" fields in Run Checks modal to prevent long text from hiding action buttons (click to expand/collapse)
- Enhanced Autotask ticket display in Run Checks modal:
  - Added "Open in Autotask" button next to ticket number for direct navigation to ticket in Autotask PSA
  - Button only appears when ticket is linked (autotask_ticket_id exists) and Autotask integration is enabled
  - Opens in new tab with proper URL format including workspace and ticket ID parameters
  - Styled as small outline button to maintain compact layout
- Merged all feature branches from v20260203-01 through v20260205-13 into main branch
  - Consolidated 29 feature branches spanning three development cycles
  - Used final state from v20260205-13-changelog-python-structure to preserve all functionality
  - Successfully integrated Autotask PSA integration, changelog restructuring, and UI improvements
  - All features from individual branches now available in main
- Reorganized Autotask settings into two separate forms with dedicated save buttons:
  - **Autotask Settings** form with "Save Autotask Settings" button inside the card
  - **Ticket Defaults** form with "Save Ticket Defaults" button inside the card
  - All fields marked as required with red asterisks (*)
  - HTML5 validation prevents saving incomplete configurations
  - Clear visual separation improves UX and prevents accidental saves

### Fixed
- Jobs from archived jobs and inactive customers no longer appear on Daily Jobs, Run Checks, and Jobs list pages
  - Added customer active status filtering to all job list queries
  - Jobs now filtered where customer is NULL or active=True alongside existing archived=False filter
  - Prevents showing jobs with "-" status from deleted customers or archived jobs
- Default Ticket Status dropdown no longer appears empty on first visit to Settings page
- Default Ticket Status value is now protected against accidental clearing:
  - Only updates when a valid status is selected
  - Only allows clearing if reference data is loaded (dropdown has options)
  - Preserves existing value if dropdown is empty (prevents data loss)
  - Fixes issue where "Create Autotask ticket" failed due to missing default status after saving settings with empty dropdown

### Removed
- Cleaned up 66 merged feature branches from repository (both local and remote):
  - Removed all v20260113-* branches (8 branches)
  - Removed all v20260115-* branches (17 branches)
  - Removed all v20260116-* branches (12 branches)
  - Removed all v20260119-* branches (18 branches)
  - Removed all v20260120-* branches (11 branches)
  - Repository now contains only main branch and current working branches (v20260206-*)

## [2026-02-05]

### Added
- Redesigned changelog system to use Python-based structure instead of Markdown:
  - Created `app/changelog.py` with structured changelog data (21 versions from v0.1.22 to v0.1.2)
  - Each version contains: version number, date, summary, and structured sections
  - Sections include: title, type (feature/improvement/fixed/documentation), and list of changes
  - Removed Gitea dependency for changelog rendering - now fully self-contained
  - No external dependencies, faster loading, always available
- New changelog.html template with modern design:
  - Sidebar navigation with all versions for quick jumping between releases
  - Sticky sidebar that remains visible during scrolling
  - Bootstrap cards for each version with gradient blue headers
  - Color-coded type badges for sections:
    - Green gradient: Feature
    - Blue gradient: Improvement
    - Red gradient: Fixed
    - Purple gradient: Documentation
  - Responsive design (sidebar hidden on mobile devices)
  - Summary section with blue left border highlight
  - Click-to-expand sections with smooth animations
- Created `static/css/changelog.css` with comprehensive styling:
  - Modern gradients for badges and headers
  - Dark mode support via CSS variables
  - Hover effects on navigation links and list items
  - Smooth scrolling to version anchors
  - Compact spacing optimizations (reduced padding, margins, font sizes)
  - CSS specificity enhancements with !important flags to override Bootstrap defaults
- Added `{% block head %}` to base.html template to allow pages to inject custom CSS

### Changed
- Updated `routes_changelog.py` to load data from `changelog.py` instead of fetching from Gitea
- Simplified changelog route - removed markdown parsing and external HTTP requests
- Removed dependency on `markdown` library for changelog rendering
- Template now receives structured Python data instead of HTML string

### Fixed
- Fixed module import path in routes_changelog.py (changed from `from app.changelog` to `from ..changelog`)
- Fixed dictionary key conflict - renamed `items` to `changes` to avoid collision with dict.items() method
- Added missing `{% block head %}` in base.html that prevented custom CSS from loading

### Technical Details
- Changelog data structure uses dictionaries with keys: version, date, summary, sections
- Sections contain: title, type, subsections (optional), changes
- Subsections contain: subtitle, changes
- All list items use "changes" key instead of "items" to avoid Python reserved method conflicts
- CSS uses !important flags and increased specificity (.changelog-nav .changelog-nav-link) to override Bootstrap
- Compact spacing achieved with: 0.15rem padding, 0.15rem margins, 0.85rem/0.7rem font sizes, 1.1/1.0 line heights

### Added
- Autotask customer mapping now auto-searches for similar company names when opening unmapped customers:
  - Automatically populates search box with customer name
  - Displays matching Autotask companies as suggestions
  - Speeds up mapping process by eliminating manual search for most customers
- Autotask "Link existing ticket" now supports cross-company ticket search:
  - Added `query_tickets_by_number()` to search tickets by number across all companies
  - When searching with a ticket number (e.g., "T20260205.0001"), results include:
    - Tickets from the customer's company (primary results)
    - Matching tickets from other companies (for overarching issues)
  - Enables linking tickets for multi-company infrastructure issues

### Changed
- Autotask resolve confirmation and note messages now correctly indicate ticket closure status:
  - Frontend confirmation dialog explains conditional closure based on time entries
  - Backend route checks time entries before creating note and generates dynamic message:
    - "ticket will be closed in Autotask" when no time entries exist
    - "ticket remains open in Autotask due to existing time entries" when time entries exist
  - Route docstring updated to reflect conditional status update behaviour

### Added
- Autotask conditional ticket status update based on time entries (API contract section 9):
  - `query_time_entries_by_ticket_id()` - Query time entries for a ticket via POST /TimeEntries/query
  - `update_ticket_resolution_safe()` now checks for time entries and conditionally sets status:
    - If NO time entries exist: sets status to 5 (Complete) with completedDate and resolvedDateTime
    - If time entries exist: keeps current status unchanged (ticket remains open)

### Fixed
- Automatic mail import can now be disabled in Settings after being enabled (fixed unchecked checkbox not being processed)
- Autotask "Link existing" search box now clears when opening the modal instead of retaining previous search text
- Autotask customer mapping search box now clears when opening the edit modal instead of retaining previous search text
- Autotask ticket resolution update now correctly preserves exact field values from GET response in PUT payload.
  The `issueType`, `subIssueType`, and `source` fields are copied with their exact values (including null)
  from the GET response, as required by Autotask API. Previously these fields were being skipped or modified.

### Added
- Restored Autotask PSA integration from branch `v20260203-13-autotask-resolution-item-wrapper`:
  - `integrations/autotask/client.py` - Autotask REST API client with full support for:
    - Zone information discovery
    - Ticket CRUD operations (create, get, update)
    - Ticket notes via `/Tickets/{id}/Notes` endpoint
    - Safe resolution updates preserving stabilizing fields
    - Query support for companies, tickets, time entries, deleted ticket logs
    - Reference data retrieval (queues, ticket sources, priorities, statuses)
  - `ticketing_utils.py` - Utilities for internal ticket management and Autotask linkage
  - Database migrations for Autotask fields:
    - `SystemSettings`: Autotask connection settings, defaults, and cached reference data
    - `Customer`: Autotask company mapping fields
    - `JobRun`: Autotask ticket linkage and deletion tracking fields
    - `Ticket`: Resolution origin tracking
  - Settings UI for Autotask configuration (connection test, reference data sync)
  - Run Checks integration for Autotask ticket creation
  - Customers page with Autotask company mapping
  - Documentation files for Autotask integration design and implementation
- Added `docs/autotask_rest_api.md` - Validated Autotask REST API contract based on Postman testing

## [2026-02-04]

### Added
- `docs/changelog-claude.md` - Changelog file for tracking changes made via Claude Code
- Setting to enable/disable daily dashboard redirect requirement (Settings > General > Navigation)
  - New `require_daily_dashboard_visit` column in `SystemSettings` model
  - Migration in `migrations.py` to add the column
  - Toggle in Settings General page under new "Navigation" card
  - Default value is OFF (disabled) - users can navigate directly to any page

### Changed
- Converted changelog to English (all project documentation must be in English)
- Documented branch naming convention and build workflow in Claude memory
- Filled README.md with comprehensive project documentation based on source code analysis

### Performance
- Added database indexes migration (`migrations.py`) for frequently queried foreign key columns:
  - `JobRun`: indexes on `job_id`, `job_id+run_at`, `job_id+reviewed_at`, `mail_message_id`
  - `MailMessage`: indexes on `job_id`, `location`, `job_id+location`
  - `MailObject`: index on `mail_message_id`
  - `TicketScope`: indexes on `ticket_id`, `job_id`
  - `RemarkScope`: indexes on `remark_id`, `job_id`
- Fixed N+1 query in `_recompute_override_flags_for_runs()` - batch loads all jobs instead of per-run queries
- Optimized Daily Jobs page with batch queries:
  - Batch load all today's runs for all jobs in single query
  - Batch infer weekly schedules for all jobs (was per-job query)
  - Batch infer monthly schedules for jobs without weekly schedule
  - Batch load ticket/remark indicators for all jobs