novela/docs/TECHNICAL.md

Novela 2.0 - Technical Status (Develop)

Scope

This document describes the current technical status of the develop codebase. It is the primary technical reference for the current implementation.

Architecture

• Stack: FastAPI, Jinja2 templates, plain JavaScript, PostgreSQL 16, Docker.
• Startup lifecycle (main.py):
  1. init_pool()
  2. run_migrations()
  3. start_backup_scheduler()
  4. mount routers
• Shutdown lifecycle:
  1. stop_backup_scheduler()
  2. close_pool()
• Source-of-truth rule: files on disk are authoritative, the database is an index/cache.

Router Status

routers/library.py

• GET /library
• GET /api/library
• POST /library/rescan
• POST /library/import (EPUB/PDF/CBR/CBZ)
• DELETE /library/file/{filename}
• GET /library/cover/{filename}
• GET /library/cover-cached/{filename}
• POST /library/cover/{filename} (EPUB)
• POST /library/want-to-read/{filename}
• POST /library/archive/{filename}
• POST /library/new/mark-reviewed (bulk needs_review=false)
• GET /home
• GET /api/home
• GET /stats
• GET /api/stats
• GET /library/list (compat)

GET /api/library runs in fast-path mode by default (DB-only, no full disk rescan). For a forced sync: GET /api/library?rescan=true or POST /library/rescan. include_file_info=true is optional for file size/mtime enrichment.

/api/home returns:

• continue_reading
• shorts_unread
• novels_unread
• shorts_read
• novels_read

/api/stats returns totals plus chart/history data for stats.html:

• reads_by_month, reads_by_dow, reads_by_hour
• genre_counts, publisher_counts, fav_genre, fav_publisher
• top_books, history

Home sections exclude series books via:

• COALESCE(series, '') = ''
• filename NOT LIKE '%/Series/%'

Home read sections are ordered oldest-first:

• shorts_read: ORDER BY MAX(read_at) ASC
• novels_read: ORDER BY MAX(read_at) ASC

routers/reader.py

• EPUB serving/chapters/images
• Reader page + book detail
• Metadata patch (PATCH /library/book/{filename})
• Progress read/write/delete
• Mark-as-read
• PDF render endpoint
• CBR/CBZ page endpoint
• Genres endpoint

routers/editor.py

• Editor page
• Chapter get/save
• Chapter add
• Chapter delete

routers/grabber.py

• Grabber page + convert/debug flows
• SSE events
• Credential management for scraper sites
• Credentials manager UI (/credentials-manager)

routers/backup.py

• GET /backup
• GET/POST/DELETE /api/backup/credentials
• GET /api/backup/health
• GET /api/backup/status
• GET /api/backup/history
• POST /api/backup/run

Backup & Security

• Dropbox token is stored encrypted-at-rest in credentials (site='dropbox').
• Dropbox backup root is stored encrypted in credentials (site='dropbox_backup_root').
• Retention (snapshots to keep) is stored encrypted in credentials (site='dropbox_backup_retention').
• Backup schedule (enabled + interval_hours) is stored encrypted in credentials (site='dropbox_backup_schedule').
• Encryption uses NOVELA_MASTER_KEY (Fernet).

Implementation details:

• Versioned backups with deduplication:
  • file objects in Dropbox: library_objects/{sha256_prefix}/{sha256}
  • snapshots in Dropbox: library_snapshots/snapshot-YYYYMMDD-HHMMSS.json
• Each run creates a new snapshot version and uploads only missing objects.
• Retention removes older snapshots above the configured limit.
• Orphan object pruning removes objects no longer referenced by retained snapshots.
• Local manifest cache (config/backup_manifest.json) speeds up change detection.
• Database backup is done via pg_dump to Dropbox postgres/.
• POST /api/backup/run always starts a background task and returns immediately.
• Scheduler runs in the background (start_backup_scheduler) and triggers on interval when enabled.
• Concurrency guard: only one backup can run at a time.
• After container restart/crash, stale running logs are auto-marked as interrupted/error.

Environment

stack/novela.env should include at least:

• POSTGRES_DB
• POSTGRES_USER
• POSTGRES_PASSWORD
• NOVELA_MASTER_KEY
• CONFIG_DIR

Dropbox settings are managed via the web UI on /backup.

UI Notes

• Library import accepts EPUB/PDF/CBR/CBZ.
• Home supports the same import formats.
• Home includes search.
• Home header/dropzone alignment matches Library (search top-right, dropzone below).
• New view supports Grid and List mode.
• Bulk selection + Remove from New works only in List mode.
• List mode has a column visibility filter with columns:
  • Publisher
  • Author
  • Series
  • Volume
  • Title
  • Has cover
  • Updated
  • Genres
  • Sub-genres
  • Tags
  • Status
• List mode supports multi-select with Shift+click range selection on checkboxes.
• Grid mode shows no selection checkboxes or bulk actions.
• Backup page supports:
  • manual run and dry-run
  • Dropbox root settings
  • snapshot retention count
  • scheduled backup (on/off + interval in hours)
  • status + history overview

Known Conventions

• Book deletion flow: delete file, prune empty directories, then DELETE FROM library (cascade removes child rows).
• Cover strategy:
  • EPUB: cover from file + cache
  • PDF/CBR: thumbnail via cover cache

Performance Notes

• Library load is optimized for large datasets:
  • list_library_json() uses pre-aggregation for reading_sessions.
  • has_cached_cover is provided directly via SQL join instead of full cache fetch.
• Additional migration indexes:
  • idx_library_sort_coalesce
  • idx_library_needs_review
  • idx_library_archived
  • idx_reading_sessions_filename_readat
  • idx_book_tags_filename_tag