diff --git a/README.md b/README.md new file mode 100644 index 0000000..4df5538 --- /dev/null +++ b/README.md @@ -0,0 +1,49 @@ +# Novela + +Novela is a self-hosted web application for managing and reading a personal digital library. +It supports EPUB, PDF, and CBR/CBZ, with metadata editing, reading progress tracking, and Dropbox backups. + +## What Novela Provides +- Library import and indexing for EPUB/PDF/CBR/CBZ +- Home dashboard with continue reading and unread/read sections +- Reader support for EPUB, PDF, and comics (CBR/CBZ) +- Metadata editing (title, author, publisher, series, volume, tags, genres) +- `New` review workflow with list/grid view, column toggles, and bulk actions +- Reading analytics/statistics dashboard +- Dropbox backup with: + - versioned snapshots + - object deduplication + - retention policy + - scheduled background runs + +## Tech Stack +- FastAPI +- Jinja2 templates + vanilla JavaScript +- PostgreSQL 16 +- Docker / Docker Compose style deployment + +## Repository Layout +- `containers/novela/` - application code (routers, templates, static assets, migrations) +- `stack/` - deployment stack files and environment configuration +- `docs/` - technical status and changelog documentation +- `build-and-push.sh` - helper script for container build/push + +## Quick Start (Development) +1. Configure environment values in `stack/novela.env`. +2. Start the stack using your preferred Docker workflow (Compose/Portainer stack). +3. Open the app and complete backup credential setup on `/backup` if needed. + +## Key Environment Variables +- `POSTGRES_DB` +- `POSTGRES_USER` +- `POSTGRES_PASSWORD` +- `NOVELA_MASTER_KEY` +- `CONFIG_DIR` +- `LIBRARY_DIR` (optional override) + +## Documentation +- Technical status: `docs/TECHNICAL.md` +- Develop changelog: `docs/changelog-develop.md` + +## Status +This repository is actively evolving. The develop documentation reflects implemented behavior in the current codebase. diff --git a/docs/TECHNICAL.md b/docs/TECHNICAL.md index 6281a15..55fd169 100644 --- a/docs/TECHNICAL.md +++ b/docs/TECHNICAL.md @@ -1,20 +1,20 @@ # Novela 2.0 - Technical Status (Develop) ## Scope -Dit document beschrijft de actuele technische status van de `develop` codebase. -Dit document is de primaire technische documentatie voor de huidige codebase. +This document describes the current technical status of the `develop` codebase. +It is the primary technical reference for the current implementation. ## Architecture -- Stack: FastAPI, Jinja2 templates, plain JS, PostgreSQL 16, Docker. +- Stack: FastAPI, Jinja2 templates, plain JavaScript, PostgreSQL 16, Docker. - Startup lifecycle (`main.py`): 1. `init_pool()` 2. `run_migrations()` 3. `start_backup_scheduler()` - 4. routers mounten + 4. mount routers - Shutdown lifecycle: 1. `stop_backup_scheduler()` 2. `close_pool()` -- Source-of-truth regel: bestand op schijf leidend, database als index/cache. +- Source-of-truth rule: files on disk are authoritative, the database is an index/cache. ## Router Status @@ -36,33 +36,33 @@ Dit document is de primaire technische documentatie voor de huidige codebase. - `GET /api/stats` - `GET /library/list` (compat) -`GET /api/library` draait standaard in fast-path (DB-only, geen full disk rescan). -Voor geforceerde sync: `GET /api/library?rescan=true` of `POST /library/rescan`. -`include_file_info=true` is optioneel voor bestandsgrootte/mtime verrijking. +`GET /api/library` runs in fast-path mode by default (DB-only, no full disk rescan). +For a forced sync: `GET /api/library?rescan=true` or `POST /library/rescan`. +`include_file_info=true` is optional for file size/mtime enrichment. -`/api/home` levert: +`/api/home` returns: - `continue_reading` - `shorts_unread` - `novels_unread` - `shorts_read` - `novels_read` -`/api/stats` levert naast totals ook chart- en history-data voor `stats.html`: +`/api/stats` returns totals plus chart/history data for `stats.html`: - `reads_by_month`, `reads_by_dow`, `reads_by_hour` - `genre_counts`, `publisher_counts`, `fav_genre`, `fav_publisher` - `top_books`, `history` -Home-secties filteren series uit met: +Home sections exclude series books via: - `COALESCE(series, '') = ''` - `filename NOT LIKE '%/Series/%'` -Read-secties op Home zijn gesorteerd op oudste eerst: +Home read sections are ordered oldest-first: - `shorts_read`: `ORDER BY MAX(read_at) ASC` - `novels_read`: `ORDER BY MAX(read_at) ASC` ### `routers/reader.py` -- Epub serving/chapters/images -- Reader pagina + book detail +- EPUB serving/chapters/images +- Reader page + book detail - Metadata patch (`PATCH /library/book/{filename}`) - Progress read/write/delete - Mark-as-read @@ -71,15 +71,15 @@ Read-secties op Home zijn gesorteerd op oudste eerst: - Genres endpoint ### `routers/editor.py` -- Editor pagina +- Editor page - Chapter get/save - Chapter add - Chapter delete ### `routers/grabber.py` -- Grabber pagina + convert/debug flows +- Grabber page + convert/debug flows - SSE events -- Credentials beheer voor scraper-sites +- Credential management for scraper sites - Credentials manager UI (`/credentials-manager`) ### `routers/backup.py` @@ -91,44 +91,44 @@ Read-secties op Home zijn gesorteerd op oudste eerst: - `POST /api/backup/run` ## Backup & Security -- Dropbox token encrypted-at-rest in `credentials` (`site='dropbox'`). -- Dropbox backup root encrypted opgeslagen in `credentials` (`site='dropbox_backup_root'`). -- Retentie (`snapshots to keep`) encrypted opgeslagen in `credentials` (`site='dropbox_backup_retention'`). -- Backup schedule (`enabled` + `interval_hours`) encrypted opgeslagen in `credentials` (`site='dropbox_backup_schedule'`). -- Encryptie via `NOVELA_MASTER_KEY` (Fernet). +- Dropbox token is stored encrypted-at-rest in `credentials` (`site='dropbox'`). +- Dropbox backup root is stored encrypted in `credentials` (`site='dropbox_backup_root'`). +- Retention (`snapshots to keep`) is stored encrypted in `credentials` (`site='dropbox_backup_retention'`). +- Backup schedule (`enabled` + `interval_hours`) is stored encrypted in `credentials` (`site='dropbox_backup_schedule'`). +- Encryption uses `NOVELA_MASTER_KEY` (Fernet). -Implementatie: -- Versie-gebaseerde backups met deduplicatie: - - bestandsobjecten in Dropbox: `library_objects/{sha256_prefix}/{sha256}` +Implementation details: +- Versioned backups with deduplication: + - file objects in Dropbox: `library_objects/{sha256_prefix}/{sha256}` - snapshots in Dropbox: `library_snapshots/snapshot-YYYYMMDD-HHMMSS.json` -- Elke run maakt een nieuwe snapshot (versie) en uploadt alleen ontbrekende objecten. -- Retentie verwijdert oude snapshots boven de ingestelde limiet. -- Orphan object-prune verwijdert objecten die niet meer door retained snapshots worden gerefereerd. -- Lokale manifestcache (`config/backup_manifest.json`) versnelt changeddetectie. -- Database backup via `pg_dump` naar Dropbox `postgres/`. -- `POST /api/backup/run` start altijd als background task en geeft direct status terug. -- Scheduler draait in achtergrond (`start_backup_scheduler`) en triggert op interval als backup aanstaat. -- Concurrentierestrictie: slechts 1 backup tegelijk. -- Bij container restart/crash worden stale `running` logs automatisch gemarkeerd als interrupted/error. +- Each run creates a new snapshot version and uploads only missing objects. +- Retention removes older snapshots above the configured limit. +- Orphan object pruning removes objects no longer referenced by retained snapshots. +- Local manifest cache (`config/backup_manifest.json`) speeds up change detection. +- Database backup is done via `pg_dump` to Dropbox `postgres/`. +- `POST /api/backup/run` always starts a background task and returns immediately. +- Scheduler runs in the background (`start_backup_scheduler`) and triggers on interval when enabled. +- Concurrency guard: only one backup can run at a time. +- After container restart/crash, stale `running` logs are auto-marked as interrupted/error. ## Environment -`stack/novela.env` bevat nu minimaal: +`stack/novela.env` should include at least: - `POSTGRES_DB` - `POSTGRES_USER` - `POSTGRES_PASSWORD` - `NOVELA_MASTER_KEY` - `CONFIG_DIR` -Dropbox settings verlopen via webinterface op `/backup`. +Dropbox settings are managed via the web UI on `/backup`. ## UI Notes -- Library import accepteert: EPUB/PDF/CBR/CBZ. -- Home heeft dezelfde importmogelijkheden. -- Home heeft zoekfunctionaliteit. -- Home header/dropzone uitlijning gelijkgetrokken met Library (zoek rechtsboven, dropzone eronder). -- `New` view ondersteunt `Grid` en `List` mode. -- Bulk selectie + `Remove from New` werkt alleen in `List` mode. -- `List` mode heeft kolomfilter (aan/uit) met kolommen: +- Library import accepts EPUB/PDF/CBR/CBZ. +- Home supports the same import formats. +- Home includes search. +- Home header/dropzone alignment matches Library (search top-right, dropzone below). +- `New` view supports `Grid` and `List` mode. +- Bulk selection + `Remove from New` works only in `List` mode. +- `List` mode has a column visibility filter with columns: - Publisher - Author - Series @@ -140,27 +140,26 @@ Dropbox settings verlopen via webinterface op `/backup`. - Sub-genres - Tags - Status -- `List` mode ondersteunt multi-select met `Shift+klik` range-select op checkboxes. -- `Grid` mode toont geen selectie-checkboxes of bulkacties. -- Backup pagina ondersteunt: - - handmatige run en dry-run - - instellingen voor Dropbox root - - retentie-aantal snapshots - - geplande backup (aan/uit + interval in uren) - - status + history overzicht +- `List` mode supports multi-select with `Shift+click` range selection on checkboxes. +- `Grid` mode shows no selection checkboxes or bulk actions. +- Backup page supports: + - manual run and dry-run + - Dropbox root settings + - snapshot retention count + - scheduled backup (on/off + interval in hours) + - status + history overview ## Known Conventions -- Verwijderen boek: bestand verwijderen, lege mappen prunen, daarna `DELETE FROM library` (cascade op child-tabellen). -- Cover strategie: - - EPUB: cover uit bestand + cache +- Book deletion flow: delete file, prune empty directories, then `DELETE FROM library` (cascade removes child rows). +- Cover strategy: + - EPUB: cover from file + cache - PDF/CBR: thumbnail via cover cache - ## Performance Notes -- Library-load geoptimaliseerd voor grote datasets: - - `list_library_json()` gebruikt pre-aggregatie voor `reading_sessions`. - - `has_cached_cover` komt direct uit SQL join i.p.v. losse volledige cache-fetch. -- Nieuwe migrations-indexen: +- Library load is optimized for large datasets: + - `list_library_json()` uses pre-aggregation for `reading_sessions`. + - `has_cached_cover` is provided directly via SQL join instead of full cache fetch. +- Additional migration indexes: - `idx_library_sort_coalesce` - `idx_library_needs_review` - `idx_library_archived` diff --git a/docs/changelog-develop.md b/docs/changelog-develop.md index 08680a0..6bd2e65 100644 --- a/docs/changelog-develop.md +++ b/docs/changelog-develop.md @@ -1,47 +1,47 @@ -# Changelog Develop +# Develop Changelog -Dit bestand houdt wijzigingen op de `develop` lijn bij. -`changelog.md` wordt later gebruikt voor release-samenvattingen. +This file tracks changes on the `develop` line. +`changelog.md` can later be used for release summaries. ## 2026-03-22 -- Blueprint en technische documentatie toegevoegd in `docs/`. -- Router-splitsing en bootstrapstructuur afgerond (`main.py`, routers, migrations, db pool). -- Media support uitgebreid naar EPUB/PDF/CBR/CBZ in import- en scanflow. -- Home UI uitgebreid met: - - import-dropzone voor EPUB/PDF/CBR/CBZ - - zoekfunctie - - uitlijning gelijk aan Library (zoek rechtsboven, dropzone eronder) -- Library UI importteksten en drag/drop filtering bijgewerkt voor multi-format. -- Library `New` view uitgebreid: +- Added blueprint/technical documentation structure in `docs/`. +- Completed router split and bootstrap structure (`main.py`, routers, migrations, DB pool). +- Expanded media support to EPUB/PDF/CBR/CBZ in import and scan flows. +- Expanded Home UI with: + - import dropzone for EPUB/PDF/CBR/CBZ + - search functionality + - alignment matching Library (search top-right, dropzone below) +- Updated Library import texts and drag/drop filtering for multi-format support. +- Expanded Library `New` view with: - `Grid`/`List` toggle - - kolomfilter in `List` + - column visibility filter in `List` - multi-select + bulk `Remove from New` - - selectie alleen in `List` mode - - `Shift+klik` range-select op checkboxes -- Nieuwe route toegevoegd: `POST /library/new/mark-reviewed` (bulk `needs_review=false`). -- Library performance verbeterd: - - `/api/library` fast-path (geen full rescan per page-load) - - optionele `rescan=true`/`include_file_info=true` - - SQL-optimalisatie in `list_library_json()` - - extra DB-indexen voor schaal -- `/api/home` hersteld naar volledige dataset-output: + - selection only in `List` mode + - `Shift+click` range selection on checkboxes +- Added route: `POST /library/new/mark-reviewed` (bulk set `needs_review=false`). +- Improved library performance: + - `/api/library` fast-path (no full rescan on every page load) + - optional `rescan=true` / `include_file_info=true` + - SQL optimizations in `list_library_json()` + - additional DB indexes for scale +- Restored `/api/home` full dataset output: - `continue_reading` - `shorts_unread` - `novels_unread` - `shorts_read` - `novels_read` -- Home-sectiefilters expliciet zonder serieboeken gezet. -- Home read-volgorde gecorrigeerd: in `shorts_read` en `novels_read` staat de oudste bovenaan (`ORDER BY MAX(read_at) ASC`). -- Statistics pagina hersteld: `/api/stats` levert weer volledige payload voor charts, favorieten, topboeken en reading history. -- Backup verbeterd: - - Dropbox token encrypted opgeslagen in DB - - Dropbox backup root instelbaar via webinterface en encrypted in DB - - versie-gebaseerde snapshots + object-store deduplicatie in Dropbox (`library_snapshots` / `library_objects`) - - instelbare snapshot-retentie (`snapshots to keep`) via backup settings - - object prune op basis van retained snapshots - - geplande backup (enable + interval in uren) - - backup runs als background process zodat navigeren op site door kan lopen - - herstel op stale running state na restart/crash (oude running logs markeren als interrupted/error) - - dry-run ondersteuning op nieuwe flow -- Docker image aangepast met `postgresql-client` voor `pg_dump`. -- Meerdere test builds uitgevoerd en gepusht naar `gitea.oskamp.info/ivooskamp/novela:dev`. +- Explicitly filtered series books out of Home sections. +- Corrected Home read ordering: `shorts_read` and `novels_read` now show oldest first (`ORDER BY MAX(read_at) ASC`). +- Restored Statistics page by returning the full `/api/stats` payload required for charts, favorites, top books, and reading history. +- Improved backup implementation: + - Dropbox token stored encrypted in DB + - Dropbox backup root configurable via web UI and stored encrypted in DB + - versioned snapshots + object-store deduplication in Dropbox (`library_snapshots` / `library_objects`) + - configurable snapshot retention (`snapshots to keep`) via backup settings + - object pruning based on retained snapshots + - scheduled backup (enable + interval in hours) + - backup runs as a background process, so page navigation does not block execution + - stale running state recovery after restart/crash (old running logs marked interrupted/error) + - dry-run support in the new flow +- Updated Docker image with `postgresql-client` for `pg_dump`. +- Multiple test builds pushed to `gitea.oskamp.info/ivooskamp/novela:dev`.