novela/docs/TECHNICAL.md

Novela 2.0 - Technical Status (Develop)

Scope

This document describes the current technical status of the develop codebase. It is the primary technical reference for the current implementation.

Architecture

• Stack: FastAPI, Jinja2 templates, plain JavaScript, PostgreSQL 16, Docker.
• Startup lifecycle (main.py):
  1. init_pool()
  2. run_migrations()
  3. start_backup_scheduler()
  4. mount routers
• Shutdown lifecycle:
  1. stop_backup_scheduler()
  2. close_pool()
• Source-of-truth rule: files on disk are authoritative, the database is an index/cache.

Router Status

routers/library.py

• GET /library
• GET /api/library
• POST /library/rescan
• POST /library/import (EPUB/PDF/CBR/CBZ)
• DELETE /library/file/{filename}
• GET /library/cover/{filename}
• GET /library/cover-cached/{filename}
• POST /library/cover/{filename} (EPUB)
• POST /library/want-to-read/{filename}
• POST /library/archive/{filename}
• POST /library/new/mark-reviewed (bulk needs_review=false)
• POST /library/rating/{filename} (set/clear star rating, body: {"rating": 0-5})
• GET /home
• GET /api/home
• GET /stats
• GET /api/stats
• GET /library/list (compat)

GET /api/library runs in fast-path mode by default (DB-only, no full disk rescan). For a forced sync: GET /api/library?rescan=true or POST /library/rescan. include_file_info=true is optional for file size/mtime enrichment.

/api/home returns:

• continue_reading
• shorts_unread
• novels_unread
• shorts_read
• novels_read

/api/stats returns totals plus chart/history data for stats.html:

• reads_by_month, reads_by_dow, reads_by_hour
• genre_counts, publisher_counts, fav_genre, fav_publisher
• top_books, history

Home sections exclude series books via:

• COALESCE(series, '') = ''
• filename NOT LIKE '%/Series/%'

Home read sections are ordered oldest-first:

• shorts_read: ORDER BY MAX(read_at) ASC
• novels_read: ORDER BY MAX(read_at) ASC

routers/reader.py

• EPUB serving/chapters/images
• Reader page + book detail
• Metadata patch (PATCH /library/book/{filename}): updates DB for all formats; writes to file only for EPUB
• Progress read/write/delete
• Mark-as-read
• Star rating (POST /library/rating/{filename}): validates 0–5, writes to file (EPUB OPF / CBZ ComicInfo.xml) and DB; DB-only for CBR/PDF
• PDF render endpoint (GET /library/pdf/{filename}?page=N&dpi=150) — returns page as PNG
• PDF info endpoint (GET /api/pdf/info/{filename}) — returns {"page_count": N}
• CBR/CBZ page endpoint
• Genres endpoint

routers/editor.py

• Editor page
• Chapter get/save
• Chapter add
• Chapter delete

routers/grabber.py

• Grabber page + convert/debug flows
• SSE events
• Credential management for scraper sites
• Credentials manager UI (/credentials-manager)

routers/backup.py

• GET /backup
• GET/POST/DELETE /api/backup/credentials
• GET /api/backup/health
• GET /api/backup/status
• GET /api/backup/history
• POST /api/backup/run

Backup & Security

• Dropbox token is stored encrypted-at-rest in credentials (site='dropbox').
• Dropbox backup root is stored encrypted in credentials (site='dropbox_backup_root').
• Retention (snapshots to keep) is stored encrypted in credentials (site='dropbox_backup_retention').
• Backup schedule (enabled + interval_hours) is stored encrypted in credentials (site='dropbox_backup_schedule').
• Encryption uses NOVELA_MASTER_KEY (Fernet).

Implementation details:

• Versioned backups with deduplication:
  • file objects in Dropbox: library_objects/{sha256_prefix}/{sha256}
  • snapshots in Dropbox: library_snapshots/snapshot-YYYYMMDD-HHMMSS.json
• Each run creates a new snapshot version and uploads only missing objects.
• Retention removes older snapshots above the configured limit.
• Orphan object pruning removes objects no longer referenced by retained snapshots.
• Local manifest cache (config/backup_manifest.json) speeds up change detection.
• Database backup is done via pg_dump to Dropbox postgres/.
• POST /api/backup/run always starts a background task and returns immediately.
• Scheduler runs in the background (start_backup_scheduler) and triggers on interval when enabled.
• Concurrency guard: only one backup can run at a time.
• After container restart/crash, stale running logs are auto-marked as interrupted/error.

Environment

stack/novela.env should include at least:

• POSTGRES_DB
• POSTGRES_USER
• POSTGRES_PASSWORD
• NOVELA_MASTER_KEY
• CONFIG_DIR

Dropbox settings are managed via the web UI on /backup.

UI Notes

• Library import accepts EPUB/PDF/CBR/CBZ.
• Home supports the same import formats.
• Home includes search.
• Home header/dropzone alignment matches Library (search top-right, dropzone below).
• New view supports Grid and List mode.
• Bulk selection + Remove from New works only in List mode.
• List mode has a column visibility filter with columns:
  • Publisher
  • Author
  • Series
  • Volume
  • Title
  • Has cover
  • Updated
  • Genres
  • Sub-genres
  • Tags
  • Status
• List mode supports multi-select with Shift+click range selection on checkboxes.
• Grid mode shows no selection checkboxes or bulk actions.
• All books view supports Grid and List mode (same columns as New, no selection/bulk actions).
  • View mode persisted in localStorage as novela.all.viewMode.
  • Column visibility persisted in localStorage as novela.all.visibleColumns.
• Star ratings (1–5) are shown under the cover in all grid views (Library, Home):
  • Display-only in grid cards (no click handler, prevents accidental taps).
  • Interactive in Book Detail (1.1rem, clickable; clicking the active star clears the rating).
  • Amber color: filled #c8a03a, unfilled rgba(200, 160, 58, 0.25).
• Reader has a text colour setting in the hamburger menu:
  • 5 presets from #e8e2d9 (bright) to #938d86 (dim), persisted in localStorage as reader-text-colour.
  • Hamburger and back-link are visually separated with margin-left: 1rem on .header-back.
• Backup page supports:
  • manual run and dry-run
  • Dropbox root settings
  • snapshot retention count
  • scheduled backup (on/off + interval in hours)
  • status + history overview
• Reader supports EPUB and PDF:
  • EPUB: chapter-text rendering (existing flow)
  • PDF: page-image rendering via /library/pdf/{filename}?page=N; page count fetched from /api/pdf/info/{filename}; progress tracked per page; keyboard/button navigation identical to EPUB
  • reader.html branches on FORMAT variable injected by the server
• Edit EPUB button in Book Detail is only shown for .epub files.

Known Bugs Fixed

• renderGenreView and renderSearchResults in library.js referenced b.genres (non-existent field on the book object). All tag data lives in b.tags as {tag, tag_type} objects; the correct helpers are bookGenres(), bookSubgenres(), bookPlainTags().
• PillInput in book.js did not handle comma as a delimiter and did not flush pending input on save. Fixed with comma keydown handler and flush() called in saveEdit().
• PATCH /library/book/{filename} failed for PDFs: _sync_epub_metadata tried to open the PDF as a ZIP, throwing an exception that aborted the entire save (including the DB update). Fixed by only calling _sync_epub_metadata when ext == ".epub".
• _make_rel_path in reader.py lacked the format prefix (epub/, pdf/, comics/) used by common.make_rel_path, causing files to be moved outside their format directory on metadata save. Fixed by aligning the path logic: EPUB → epub/{publisher}/{author}/…, PDF → pdf/{author}/{title}.pdf, CBR/CBZ → comics/{author}/{title}{ext}.
• PDF reader showed infinite loading: reader.html always called /library/chapters/{filename} (EPUB-only) and tried to render chapter text. PDF reader now fetches page count and renders page images.

Known Conventions

• Book deletion flow: delete file, prune empty directories, then DELETE FROM library (cascade removes child rows).
• Cover strategy:
  • EPUB: cover from file + cache
  • PDF/CBR: thumbnail via cover cache
• Rating storage:
  • EPUB: <meta name="novela:rating" content="N"/> in OPF
  • CBZ: <NovelaRating>N</NovelaRating> in ComicInfo.xml inside the ZIP
  • CBR/PDF: DB only
  • upsert_book uses CASE WHEN EXCLUDED.rating > 0 THEN EXCLUDED.rating ELSE library.rating END to restore rating from file without overwriting existing DB value

Performance Notes

• Library load is optimized for large datasets:
  • list_library_json() uses pre-aggregation for reading_sessions.
  • has_cached_cover is provided directly via SQL join instead of full cache fetch.
• Additional migration indexes:
  • idx_library_sort_coalesce
  • idx_library_needs_review
  • idx_library_archived
  • idx_reading_sessions_filename_readat
  • idx_book_tags_filename_tag