novela/docs/TECHNICAL.md

# Novela 2.0 - Technical Status (Develop)

## Scope
This document describes the current technical status of the `develop` codebase.
It is the primary technical reference for the current implementation.

## Architecture
- Stack: FastAPI, Jinja2 templates, plain JavaScript, PostgreSQL 16, Docker.
- Startup lifecycle (`main.py`):
  1. `init_pool()`
  2. `run_migrations()`
  3. `start_backup_scheduler()`
  4. mount routers
- Shutdown lifecycle:
  1. `stop_backup_scheduler()`
  2. `close_pool()`
- Source-of-truth rule: files on disk are authoritative, the database is an index/cache.

## Router Status

### `routers/library.py`
- `GET /library`
- `GET /api/library`
- `POST /library/rescan`
- `POST /library/import` (EPUB/PDF/CBR/CBZ)
- `DELETE /library/file/{filename}`
- `GET /library/cover/{filename}`
- `GET /library/cover-cached/{filename}`
- `POST /library/cover/{filename}` (EPUB)
- `POST /library/want-to-read/{filename}`
- `POST /library/archive/{filename}`
- `POST /library/new/mark-reviewed` (bulk `needs_review=false`)
- `GET /home`
- `GET /api/home`
- `GET /stats`
- `GET /api/stats`
- `GET /library/list` (compat)

`GET /api/library` runs in fast-path mode by default (DB-only, no full disk rescan).
For a forced sync: `GET /api/library?rescan=true` or `POST /library/rescan`.
`include_file_info=true` is optional for file size/mtime enrichment.

`/api/home` returns:
- `continue_reading`
- `shorts_unread`
- `novels_unread`
- `shorts_read`
- `novels_read`

`/api/stats` returns totals plus chart/history data for `stats.html`:
- `reads_by_month`, `reads_by_dow`, `reads_by_hour`
- `genre_counts`, `publisher_counts`, `fav_genre`, `fav_publisher`
- `top_books`, `history`

Home sections exclude series books via:
- `COALESCE(series, '') = ''`
- `filename NOT LIKE '%/Series/%'`

Home read sections are ordered oldest-first:
- `shorts_read`: `ORDER BY MAX(read_at) ASC`
- `novels_read`: `ORDER BY MAX(read_at) ASC`

### `routers/reader.py`
- EPUB serving/chapters/images
- Reader page + book detail
- Metadata patch (`PATCH /library/book/{filename}`)
- Progress read/write/delete
- Mark-as-read
- PDF render endpoint
- CBR/CBZ page endpoint
- Genres endpoint

### `routers/editor.py`
- Editor page
- Chapter get/save
- Chapter add
- Chapter delete

### `routers/grabber.py`
- Grabber page + convert/debug flows
- SSE events
- Credential management for scraper sites
- Credentials manager UI (`/credentials-manager`)

### `routers/backup.py`
- `GET /backup`
- `GET/POST/DELETE /api/backup/credentials`
- `GET /api/backup/health`
- `GET /api/backup/status`
- `GET /api/backup/history`
- `POST /api/backup/run`

## Backup & Security
- Dropbox token is stored encrypted-at-rest in `credentials` (`site='dropbox'`).
- Dropbox backup root is stored encrypted in `credentials` (`site='dropbox_backup_root'`).
- Retention (`snapshots to keep`) is stored encrypted in `credentials` (`site='dropbox_backup_retention'`).
- Backup schedule (`enabled` + `interval_hours`) is stored encrypted in `credentials` (`site='dropbox_backup_schedule'`).
- Encryption uses `NOVELA_MASTER_KEY` (Fernet).

Implementation details:
- Versioned backups with deduplication:
  - file objects in Dropbox: `library_objects/{sha256_prefix}/{sha256}`
  - snapshots in Dropbox: `library_snapshots/snapshot-YYYYMMDD-HHMMSS.json`
- Each run creates a new snapshot version and uploads only missing objects.
- Retention removes older snapshots above the configured limit.
- Orphan object pruning removes objects no longer referenced by retained snapshots.
- Local manifest cache (`config/backup_manifest.json`) speeds up change detection.
- Database backup is done via `pg_dump` to Dropbox `postgres/`.
- `POST /api/backup/run` always starts a background task and returns immediately.
- Scheduler runs in the background (`start_backup_scheduler`) and triggers on interval when enabled.
- Concurrency guard: only one backup can run at a time.
- After container restart/crash, stale `running` logs are auto-marked as interrupted/error.

## Environment
`stack/novela.env` should include at least:
- `POSTGRES_DB`
- `POSTGRES_USER`
- `POSTGRES_PASSWORD`
- `NOVELA_MASTER_KEY`
- `CONFIG_DIR`

Dropbox settings are managed via the web UI on `/backup`.

## UI Notes
- Library import accepts EPUB/PDF/CBR/CBZ.
- Home supports the same import formats.
- Home includes search.
- Home header/dropzone alignment matches Library (search top-right, dropzone below).
- `New` view supports `Grid` and `List` mode.
- Bulk selection + `Remove from New` works only in `List` mode.
- `List` mode has a column visibility filter with columns:
  - Publisher
  - Author
  - Series
  - Volume
  - Title
  - Has cover
  - Updated
  - Genres
  - Sub-genres
  - Tags
  - Status
- `List` mode supports multi-select with `Shift+click` range selection on checkboxes.
- `Grid` mode shows no selection checkboxes or bulk actions.
- Backup page supports:
  - manual run and dry-run
  - Dropbox root settings
  - snapshot retention count
  - scheduled backup (on/off + interval in hours)
  - status + history overview

## Known Conventions
- Book deletion flow: delete file, prune empty directories, then `DELETE FROM library` (cascade removes child rows).
- Cover strategy:
  - EPUB: cover from file + cache
  - PDF/CBR: thumbnail via cover cache

## Performance Notes
- Library load is optimized for large datasets:
  - `list_library_json()` uses pre-aggregation for `reading_sessions`.
  - `has_cached_cover` is provided directly via SQL join instead of full cache fetch.
- Additional migration indexes:
  - `idx_library_sort_coalesce`
  - `idx_library_needs_review`
  - `idx_library_archived`
  - `idx_reading_sessions_filename_readat`
  - `idx_book_tags_filename_tag`