novela/docs/changelog-develop.md
Ivo Oskamp e4d2e2c636 DB-stored books, full-text search, backup restore, and AO3 scraper
- DB-stored books (Fase 1–6): chapters and images stored in PostgreSQL; grabber writes to DB, EPUB→DB conversion, DB→EPUB export, FTS search page (/search)
- Chapter editor: Monaco editor supports DB-stored books; inline title editing
- Grabber: DB/EPUB storage toggle on Convert page
- Backup: restore from Dropbox snapshot (browse snapshots, restore individual or selected files)
- AO3 scraper: initial implementation
- Changelog: v0.1.2 and v0.1.3 entries added to changelog.py and changelog.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-03 15:13:08 +02:00

369 lines
32 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Develop Changelog
## 2026-04-03 (3)
- DB chapter editor: Monaco-based editor now supports DB-stored books
- `GET /library/editor/{filename}` handles `db/…` filenames; `is_db` flag passed to template
- `GET /api/edit/chapter/{index}/{filename}` and `POST …`: DB branches query/update `book_chapters` directly; save calls `upsert_chapter` (updates `content_tsv` too)
- `POST /api/edit/chapter/add/{filename}` and `DELETE …`: DB branches insert/delete with `chapter_index` shift via `UPDATE … SET chapter_index = chapter_index ± 1`
- Title editing: header chapter-name replaced with a text input for DB books; `pendingTitles` map preserves unsaved titles across chapter switches (parallel to `pendingContent`); title-only dirty chapters correctly saved in Save All
- `insertBreak`: scene-break image path is `/static/break.png` for DB books (vs `../Images/break.png` for EPUB)
- Fix: `editor.focus()` called after content load so Monaco receives keyboard focus immediately
- Fix: `header-chapter` "Loading…" text suppressed for DB books where that element is hidden
- `book.html`: "Edit chapters" button shown for `storage_type = 'db'` books
- Search: chapter titles now included in FTS
- `upsert_chapter` prepends title to the plain-text input for `to_tsvector`: `title + " " + stripped_html`
- `GET /api/search`: added `OR LOWER(bc.title) LIKE LOWER('%…%')` fallback for chapters whose title matches but content doesn't
- Startup migration `migrate_rebuild_chapter_tsv_with_title()` rebuilds existing `content_tsv` values to include titles
- Grabber: added DB/EPUB storage toggle on the Convert page
- UI toggle above Convert button ("Save as: DB | EPUB file"); `storageMode` JS variable sent in POST body
- `POST /convert`: reads `storage_mode` from body; stored in job as `'db'` or `'epub'`
- `_run_scrape`: EPUB path builds chapters via `make_chapter_xhtml`, calls `make_epub`, writes file, calls `upsert_book(storage_type='file')`; DB path unchanged
- `done` SSE event includes `storage_type`; `conversion.js` updates the download button label/action accordingly
- EPUB → DB conversion: fixed double chapter title
- `_epub_body_inner` strips the first `<h1>`/`<h2>`/`<h3>` heading from each chapter body before storing; the editor prepends its own heading, so storing the EPUB heading too caused it to appear twice
- Fix for `NavigableString` crash: `getattr(child, "name", None) is None` used instead of `hasattr(child, "name")``NavigableString` has `name = None` but no `decompose()` method
- Sidebar: Search link styling fixed
- Stray `<li>Search</li>` moved inside the Library `<ul class="sidebar-nav">` (was outside, causing incorrect HTML structure)
- `sidebar.css`: added `a:visited { color: var(--text-dim) }` and `a.active:visited { color: var(--accent) }` to prevent the browser's default purple visited color
## 2026-04-03 (2)
- DB-stored books (Fase 46): EPUB→DB conversion, DB→EPUB export, full-text search
- **Fase 4** — EPUB-to-DB conversion: `POST /api/library/convert-to-db/{filename}` converts an existing on-disk EPUB to a DB-stored book; extracts chapters via `_epub_body_inner` (rewrites img src to imagestore URLs), migrates all child rows (book_tags, reading_progress, reading_sessions, bookmarks, library_cover_cache) to the new `db/…` filename using INSERT→UPDATE→DELETE to respect FK constraints, then deletes the EPUB file
- **Fase 5** — DB→EPUB export: `GET /api/library/export-epub/{filename}` builds and streams an EPUB from DB content; `_rewrite_db_images_for_epub` rewrites `/library/db-images/…` URLs back to `OEBPS/Images/…` paths, deduplicating by sha256; `Content-Disposition: attachment` response
- **Fase 6** — Full-text search: new `routers/search.py` with `GET /search` (page) and `GET /api/search?q=…` (FTS over `book_chapters.content_tsv` via `plainto_tsquery('simple', q)`, `ts_headline` for snippets, `ts_rank` for ordering, LIMIT 30, excludes archived); new `templates/search.html` with highlight (`<mark>`), "Read here" link (`?bm_ch=N&bm_scroll=0`), and "Book detail" link; Search entry added to sidebar
- `book.html`: DB books show "Export EPUB" instead of "Download"; "Edit EPUB" and "Convert to DB" buttons only shown for `.epub` files; delete modal text differs for DB vs file books
- `PATCH /library/book/{filename}`: DB book branch added — skips file move, recomputes synthetic `db/…` filename via `make_rel_path`, applies same FK-safe rename pattern, updates `book_chapters` and `bookmarks` in addition to standard child tables
## 2026-04-03 (1)
- DB-stored books (Fase 13): grabber now stores scraped books in PostgreSQL instead of EPUB files on disk
- New `book_chapters` table: `filename FK, chapter_index, title, content TEXT, content_tsv TSVECTOR`; GIN index on `content_tsv` for future FTS
- New `book_images` table: `sha256 PK, ext, media_type, size_bytes`; content-addressed imagestore at `library/images/{sha2}/{sha256}{ext}`
- New `storage_type VARCHAR(10) DEFAULT 'file'` column on `library`; DB-stored books use `'db'`
- New utilities in `common.py`: `is_db_filename`, `write_image_file`, `store_db_image`, `html_to_plain`, `upsert_chapter`, `ensure_unique_db_filename`; `make_rel_path` now handles `media_type="db"` → synthetic `db/{pub}/{auth}/...` filename
- `upsert_book` and `list_library_json` updated to include `storage_type`
- Grabber: `_run_scrape` stores chapters in `book_chapters`, chapter images in imagestore (absolute `/library/db-images/` URLs embedded in HTML), cover in `library_cover_cache`; no EPUB file written
- New `GET /library/db-images/{path:path}` endpoint serves imagestore files
- Reader: `GET /library/chapters/` and `GET /library/chapter/` have DB branches for `storage_type='db'` books (query `book_chapters` directly)
- Reader page (`/library/read/`), book detail page, mark-read, and rating endpoints all handle DB filenames (no file existence required)
- Cover endpoints (`/library/cover/`, `/library/cover-cached/`) serve DB books from `library_cover_cache`
## 2026-04-02 (1)
- Added Restore functionality to the Backup page
- New `GET /api/backup/snapshots` endpoint: lists available Dropbox snapshots (name + date parsed from filename, no downloads needed)
- New `GET /api/backup/snapshots/{snapshot_name}/files` endpoint: loads a snapshot from Dropbox and returns all files with path, size, sha256, and whether the file currently exists locally
- New `POST /api/backup/restore` endpoint: downloads file objects from Dropbox, writes to disk, and re-indexes via `scan_media` + `upsert_book`; returns per-file result with errors
- New "Restore" card on the backup page: snapshot dropdown (auto-loaded on page open), file list with filter/search, per-file "Restore" button, multi-select + "Restore selected", on-disk indicator, inline status feedback
- After restore, the file list refreshes to reflect updated on-disk state
## 2026-03-29 (10)
- Duplicates: fixed `updateCounts` crashing with a TypeError (`g.books.length` → `g.length`); the crash prevented `renderGrid` from running, so the duplicates view never rendered and the counter was stale
## 2026-03-29 (9)
- Duplicates: volume-aware duplicate detection — a book is only a duplicate when title + author + volume all match
- `_duplicateGroups` in `library.js` (Duplicates view): key now includes `series_index` when > 0, so different volumes of the same series are no longer grouped as duplicates
- `preload` in `grabber.py` (Grabber): when the scraper returns a `series_index_hint`, the DB lookup only flags a match when title + author + volume all match; falls back to title + author when no volume is known — same logic as the bulk importer
## 2026-03-29 (8)
- Shared code: eliminated all remaining duplication across templates and JS files
- CSS custom properties: extracted single `:root { }` block into `static/theme.css`; removed duplicate inline `:root` from all 15 templates and from `library.css`, `book.css``editor.css` keeps editor-specific vars (`--danger`, `--header-h`, `--panel-w`), `reader.html` keeps page-specific vars (`--header-h`, `--footer-h`, `--content-w`), `backup.html` keeps (`--ok`, `--warn`, `--err`)
- Cover helpers: moved `strHash`, `COVER_PALETTES`, `makePlaceholderCover`, `wrapText`, `truncate` from `library.js` and `book.js` into `books.js`; removed from `home.html` and `following.html`
- HTML escape: `esc()` added to `books.js`; removed from `library.js`, `editor.js`, and all 8 templates that defined it inline
- SSE/EventSource: extracted shared `connectConversionStream(job_id)` and `addLog()` into new `static/conversion.js`; both `index.html` and `grabber.html` now call the shared function (removed ~70 duplicate lines)
## 2026-03-29 (7)
- Search: extracted shared book helpers and search logic into `static/books.js`
- `_filenameBase`, `bookTitle`, `bookAuthor`, `tagValuesByType`, `bookGenres`, `bookSubgenres`, `bookPlainTags`, `filterBooks`, `setupSearchInput` moved from `library.js` and `home.html` to a single shared file
- Both pages now use identical search behaviour: Enter to search, × to clear
- Both pages now search across title, author, publisher, genre, sub-genre, and tag
## 2026-03-29 (6)
- Search: changed from search-as-you-type (250 ms debounce) to Enter-to-search on both Library and Home — prevents iPad keyboard from locking up on large collections
## 2026-03-29 (5)
- Accent colour updated to match logo orange (`#ffa20e`); secondary accent updated to `#ffb840`
- Applied across all CSS files, templates, and inline styles
## 2026-03-29 (4)
- Branding: added logo, favicon, and Apple touch icon to all pages
- Static assets: `logo.png` (sidebar), `favicon.ico` (16×16), `favicon-32.png` (32×32), `favicon-256.png` (256×256), `apple-touch-icon.png` (180×180 with `#0f0e0c` background)
- Favicon `<link>` tags added to all 15 templates
- Sidebar: image logo (`logo.png`) placed next to the existing "No**vela**" wordmark using flexbox
- `apple-touch-icon.png` uses dark `#0f0e0c` background — renders as a native-looking iOS home screen icon
## 2026-03-29 (3)
- Dockerfile: replaced `unrar-free` with proprietary `unrar` (RARLAB v6.2.6) from Debian non-free — fixes "Failed to read enough data" errors on RAR archives using newer compression methods
## 2026-03-29 (2)
- CBR reader: detect archive format via magic bytes instead of file extension — `.cbr` files that are actually ZIP or 7-zip archives now open correctly; added `py7zr` dependency for 7-zip support
## 2026-03-29 (1)
- Bulk Import: duplicate check now volume-aware — books with the same title+author but a different volume number (e.g. recoloured reprints) are no longer flagged as duplicates; `volume` is included in the API call and matched against `series_index` in the DB
## 2026-03-28 (11)
- Bulk Import: duplicate detection against existing library
- New `POST /api/bulk-check-duplicates` endpoint: case-insensitive title+author+volume match, single SQL query with OR conditions for all pairs
- Duplicate rows highlighted in red in the preview table; a skip checkbox appears per duplicate row
- Stats bar shows "X duplicates · Skip all · Import all" action buttons
- Duplicate rows are skipped by default; user can toggle individual rows or use bulk actions
- `startImport()` filters out skipped rows before batching; skipped files appear in the result summary as "Duplicate skipped"
- If all rows are skipped (nothing to import), result shown immediately without sending any request
## 2026-03-28 (10)
- After "Remove from New" succeeds, the library is now reloaded from the server (`loadLibrary()`) so the New list updates immediately without a manual refresh
## 2026-03-28 (9)
- Bulk delete is now batched: new `POST /library/bulk-delete` endpoint accepts a JSON list of filenames, deletes files and removes DB rows in one query per batch; JS sends 20 files per batch with a progress bar in the confirmation dialog
## 2026-03-28 (8)
- Disk usage warning in sidebar: `GET /api/disk` returns partition usage for the library directory; sidebar polls every 60 s and shows a warning bar (amber ≥ 85% or < 2 GB free, red 95% or < 500 MB free) above the backup status bar
## 2026-03-28 (7)
- Fixed `mark-reviewed` JS: UI (allBooks update + renderGrid) now only runs after confirmed server success; `catch` no longer fires a false error after a successful response where renderGrid throws
## 2026-03-28 (6)
- Bulk Import: rewrote pattern editor from token-pills to free-text `%placeholder%` syntax
- Pattern input: free-text field; placeholders: `%series%` `%volume%` `%title%` `%year%` `%month%` `%day%` `%author%` `%publisher%` `%ignore%`
- Colored chip row: click to insert at cursor position; drag onto input also supported
- Colored live preview below the pattern input; regex-based parser replaces delimiter+token logic
- Shared metadata now wins over filename-parsed values (previously filename won)
- All UI text translated from Dutch to English
## 2026-03-28 (5)
- Status badge (top-right cover) and want-to-read star (top-left cover) now use dark fill `rgba(15,14,12,0.82)` + `box-shadow: 0 0 0 2px #0f0e0c` ring always readable regardless of cover colour
## 2026-03-28 (4)
- Added `Temporary Hold` status; renamed `Hiatus` `Long-Term Hold`
- Startup migration `migrate_rename_hiatus()` converts existing DB records automatically
- Status colours: Complete=green, Ongoing=blue, Temporary Hold=amber, Long-Term Hold=orange
- `statusBadgeHtml()` helper in `library.js` replaces three identical inline badge blocks
- Grabber: `Temporary-Hold` (gayauthors.org) now maps to `Temporary Hold`; `Long-Term Hold` passes through unchanged
- Status dropdowns updated in Book Detail and Bulk Import
## 2026-03-28 (3)
- Added Bulk Import page (`/bulk-import`) for batch importing CBR/CBZ/EPUB/PDF files with filename-based metadata parsing
- Configurable token pattern: Serie, Volume, Titel, Jaar, Auteur, Uitgever, Negeer (in any order)
- Configurable delimiter (default ` - `); year token right-anchored when last so title absorbs overflow segments
- Live test-parse box: type any filename and see how it parses before selecting files
- Shared metadata card: author, publisher, status, genres, tags applied to all files (overridden by pattern tokens or manual edits)
- Preview table: all parsed fields editable via contenteditable cells; warning indicator for rows with fewer segments than tokens; sorted by filename
- Batch upload in groups of 5 with progress bar (N / total files processed)
- Result summary with list of skipped files and reasons
- New `routers/bulk_import.py`: `GET /bulk-import`, `POST /library/bulk-import`
- Sidebar link under Tools between Book Builder and Credentials
This file tracks changes on the `develop` line.
`changelog.md` can later be used for release summaries.
## 2026-03-28 (2)
- Performance: library page now loads instantly for large collections (1000+ books)
- `IntersectionObserver` defers both cover image loading and placeholder canvas drawing until cards enter the viewport eliminates hundreds of upfront canvas ops that blocked the initial render
- `ETag` caching on `/library/list`: server returns `304 Not Modified` when nothing changed, client skips JSON parse and re-download
- Single DOM pass in `renderBooksGrid`, `renderDuplicatesView`, `renderSeriesDetail`: canvas and img set up via `card.querySelector` immediately after `innerHTML`, removing a second iteration with `document.getElementById` per card
- `book_tags` joined via `json_agg` in the main `list_library_json()` query, eliminating a separate `SELECT * FROM book_tags` query and Python merge loop
- `loadLibrary` now shows an error message instead of staying stuck on "Loading…" when the fetch or render fails
## 2026-03-28 (1)
- Added Following page (`/following`): track external author URLs outside Library and Tools
- New `authors` table: `name` (unique), `url`, `created_at`, `updated_at`
- New `routers/following.py`: `GET /following` page, `GET /api/following` (all authors + URL + book count + last added), `POST /api/following/{name}` (set/clear URL)
- Sidebar: new Following section between Library and Tools; counter shows number of followed authors
- Following page: two tabs Following (authors with URL) and All Authors; inline URL editing with Enter/Escape keyboard support; Visit button opens external URL in a new tab; author name links to library author view
- Added Incomplete view to Library (`#incomplete`): shows all non-archived books where `publication_status ≠ Complete`; sidebar counter included; entry placed after New in the Library section
## 2026-03-27 (1)
- Convert page: duplicate warning shown after loading metadata when a book with the same title+author already exists in the library; warning includes a link to the existing book; user can still proceed with conversion
- Library: added Duplicates section to sidebar (between Rated and Statistics); counter shows total number of books that are part of a duplicate group (same title+author, case-insensitive); Duplicates view groups books by title+author with a subheading per group
- Fixed Duplicates view not loading covers: card renderer now uses the same canvas + two-pass img/`makePlaceholderCover` pattern as `renderBooksGrid`
## 2026-03-26 (2)
- Fixed Book Builder page showing white background: `library.css` added to `builder.html` to load `:root` CSS variables and dark `body` background; all CSS variable references in `builder.css` aligned with library theme names (`--text`, `--surface`, `--surface2`, `--text-dim`, `--border`, `--accent`, `--sidebar`)
## 2026-03-26 (1)
- Added Book Builder: create EPUB books manually from scratch via a WYSIWYG editor
- New `builder_drafts` table (`id UUID`, `title`, `author`, `publisher`, `source_url`, `chapters JSONB`)
- `build_epub()` in `epub.py`: builds a standards-compliant EPUB 2.0 ZIP from title/author/publisher/chapters; embeds inline CSS and `break.png` if present
- `normalize_wysiwyg_html()` in `xhtml.py`: converts contenteditable HTML to EPUB-safe XHTML; handles scene-breaks, `<blockquote class="author-note">`, inline formatting, and `<hr>` break image
- `routers/builder.py`: draft CRUD, chapter CRUD (`GET/POST/PUT/DELETE`), normalize preview endpoint, publish endpoint (normalizes all chapters builds EPUB writes to `library/epub/` upserts to DB deletes draft redirects to book detail)
- `templates/builder.html` + `static/builder.js` + `static/builder.css`: index page (new draft form + draft list) and editor (chapter panel, contenteditable pane, toolbar with bold/italic/underline/blockquote/author-note/scene-break/normalize, autosave every 30 s, Ctrl+S, publish button)
- Book Builder link added to sidebar Tools section (between Convert and Credentials)
- `.author-note` blockquote style added to `static/epub-style.css`
## 2026-03-25 (20)
- Added Rated section to library sidebar: shows all non-archived books with `rating > 0`, sorted by rating descending then title alphabetically; badge displays total count; navigable via `#rated` URL hash
## 2026-03-25 (19)
- Fixed series index 0 not displaying in series slot view, grid cards, list volume column, and book detail:
- Series slot labels now show `#0` for all slots when the series has at least one positively-indexed sibling (consistent label height across all cards)
- Grid card and list volume column use the same cross-book heuristic (`indexedSeriesSet`) to show `[0]` where appropriate
- Book detail page queries whether any sibling in the same series has `series_index > 0` (`series_is_indexed`) and shows `[0]` accordingly
## 2026-03-25 (18)
- Added autocomplete for Author, Publisher, and Series in the book edit panel: typing shows a filtered dropdown of existing values from the library; selecting a value fills the field (same dropdown styling as genres/tags)
- Status field now defaults to "Complete" when opening the edit panel for a book that has no status set yet
## 2026-03-25 (17)
- Fixed CBR reader showing only first page: `cbr_page_count` was missing from the `cbr` import in `reader.py`; `/api/cbr/info/` was returning an error, causing `page_count` to fall back to 1 and the Next button to remain disabled
## 2026-03-25 (16)
- Fixed CBR/CBZ reader stuck on loading: added `/api/cbr/info/{filename}` endpoint returning `{page_count}`, and added a CBR/CBZ branch in `reader.html` that mirrors the PDF paged reader (page images served via `/library/cbr/{filename}/{page}`)
## 2026-03-25 (15)
- Added series volume 0 and letter suffix support (e.g. "21a", "21b"):
- New `series_suffix VARCHAR(10)` column added to `library` table via `migrate_series_suffix`
- `series_index` lower bound changed from 1 to 0 throughout (index 0 = special/prequel edition)
- Volume field in the book editor changed from `type="number"` to `type="text"` accepts "0", "1", "21a", etc.
- Server parses the combined volume string into `series_index` (INTEGER) + `series_suffix` (VARCHAR) via `parse_volume_str`
- File naming includes suffix: `021a - Title.epub`
- `novela:series_suffix` meta tag written to/read from EPUB OPF for persistence across rescans
- Series detail view: books sorted by `(series_index, series_suffix)`; slot labels show "21a"; index 0 shown as slot when any sibling has index > 0
- Grid cards, list view Volume column, author/publisher sort all include suffix
## 2026-03-25 (14)
- Added multi-select and bulk delete to `All books` List view:
- Checkbox column added to the list table (List mode only; Grid mode unchanged)
- Select all / Clear all / Clear selection controls in the controls bar
- `Shift+click` range selection on checkboxes
- `Delete selected` button (red, disabled when nothing is selected) triggers a confirmation dialog; confirmed deletions remove files from disk and database, then reload the library
## 2026-03-25 (13)
- Fixed EPUB cover replacement reverting to original after upload:
- `GET /library/cover/{filename}` for EPUBs now checks the DB cover cache first (populated on upload) before falling back to extracting from the EPUB file; the cache is also warmed on cache-miss so subsequent requests are fast.
- `add_cover_to_epub` fully rewritten: locates the OPF via `META-INF/container.xml`, finds the existing cover image in the OPF manifest, removes it from the ZIP, and writes the new cover in the same directory — works for any EPUB directory structure (`OEBPS/`, `EPUB/`, etc.).
## 2026-03-25 (12)
- Fixed chapter images not loading in EPUBs that use a non-standard root directory (e.g. `EPUB/` instead of `OEBPS/`): image paths are now passed as full ZIP paths instead of stripping the root segment, and the image endpoint no longer hardcodes an `OEBPS/` prefix. Case-insensitive fallback retained for EPUBs with mismatched image folder casing.
## 2026-03-25 (11)
- Fixed reader progress bar jumping to 100% immediately when navigating to the second chapter in short EPUBs (2 chapters): changed progress formula denominator from `total - 1` to `total` so 100% is only reached after fully scrolling through the last chapter. Same fix applied to PDF page progress.
## 2026-03-25 (10)
- Added bookmark feature:
- `bookmarks` table in DB (`migrate_create_bookmarks`): `filename`, `chapter_index`, `scroll_frac`, `chapter_title`, `note`, `created_at`
- New API endpoints: `GET/POST /library/bookmarks/{filename}`, `PATCH/DELETE /library/bookmarks/{id}`, `GET /api/bookmarks`
- Bookmark button in reader header (orange, bookmark icon): opens modal with optional note field
- Bookmark modal closes on Escape, backdrop click, or Cancel
- Reader supports `?bm_ch=N&bm_scroll=F` URL params to jump directly to bookmarked position (overrides saved progress)
- Library sidebar: Bookmarks section in Library nav with live count badge
- Library `#bookmarks` view: card list showing cover, book title, author, chapter, note, date, Go-to and Delete buttons
## 2026-03-22
- Added blueprint/technical documentation structure in `docs/`.
- Completed router split and bootstrap structure (`main.py`, routers, migrations, DB pool).
- Expanded media support to EPUB/PDF/CBR/CBZ in import and scan flows.
- Expanded Home UI with:
- import dropzone for EPUB/PDF/CBR/CBZ
- search functionality
- alignment matching Library (search top-right, dropzone below)
- Updated Library import texts and drag/drop filtering for multi-format support.
- Expanded Library `New` view with:
- `Grid`/`List` toggle
- column visibility filter in `List`
- multi-select + bulk `Remove from New`
- selection only in `List` mode
- `Shift+click` range selection on checkboxes
- Added route: `POST /library/new/mark-reviewed` (bulk set `needs_review=false`).
- Improved library performance:
- `/api/library` fast-path (no full rescan on every page load)
- optional `rescan=true` / `include_file_info=true`
- SQL optimizations in `list_library_json()`
- additional DB indexes for scale
- Restored `/api/home` full dataset output:
- `continue_reading`
- `shorts_unread`
- `novels_unread`
- `shorts_read`
- `novels_read`
- Explicitly filtered series books out of Home sections.
- Corrected Home read ordering: `shorts_read` and `novels_read` now show oldest first (`ORDER BY MAX(read_at) ASC`).
- Restored Statistics page by returning the full `/api/stats` payload required for charts, favorites, top books, and reading history.
- Improved backup implementation:
- Dropbox token stored encrypted in DB
- Dropbox backup root configurable via web UI and stored encrypted in DB
- versioned snapshots + object-store deduplication in Dropbox (`library_snapshots` / `library_objects`)
- configurable snapshot retention (`snapshots to keep`) via backup settings
- object pruning based on retained snapshots
- scheduled backup (enable + interval in hours)
- backup runs as a background process, so page navigation does not block execution
- stale running state recovery after restart/crash (old running logs marked interrupted/error)
- dry-run support in the new flow
- Updated Docker image with `postgresql-client` for `pg_dump`.
- Multiple test builds pushed to `gitea.oskamp.info/ivooskamp/novela:dev`.
## 2026-03-25 (9)
- Fixed backup progress not visible immediately after clicking Run Live Backup: sidebar `loadBackupProgress()` is now called directly from the backup page after a successful start
- Fixed backup status timestamp showing wrong relative time (e.g. "1h ago" for a recent backup): server UTC timestamps without timezone suffix are now correctly interpreted as UTC in the browser
- Fixed backup status bar using browser-default purple link color: now uses `var(--accent)` orange consistent with the rest of the UI
- Added password manager ignore attributes to App Key and App Secret fields: `data-1p-ignore` (1Password), `data-lpignore="true"` (LastPass), `data-form-type="other"` (1Password v8)
## 2026-03-25 (8)
- Added Dropbox OAuth2 refresh token support (no redirect URI needed):
- New endpoints: `POST /api/backup/oauth/prepare` (generates auth URL) and `POST /api/backup/oauth/exchange` (exchanges code for refresh token)
- `_dbx()` now prefers refresh token mode (app key + secret + refresh token) with automatic renewal; falls back to legacy access token for backwards compatibility
- App key and secret stored encrypted in `credentials` table (`dropbox_app_key`, `dropbox_app_secret`)
- `DELETE /api/backup/credentials` now also cleans up `dropbox_app_key` and `dropbox_app_secret`
- Backup page updated with two-step OAuth flow UI (app key/secret → auth URL → paste code)
- Settings status shows `• refresh token` or `• legacy token` indicator
## 2026-03-25 (7)
- Added backup progress tracking: `GET /api/backup/progress` returns `{running, done, total, phase}`; sidebar shows live file count (e.g. `312 / 652 · uploading`) polling every 3 s while running
## 2026-03-25 (6)
- Fixed `GET /api/backup/health` missing `schedule_enabled` and `schedule_interval_hours` in response (values were read but not returned; Health card showed "disabled" even when schedule was on)
- Added backup status indicator in sidebar above Rescan library button: coloured dot + status text (OK/running/failed/never), links to `/backup`, auto-refreshes every 8 s while running, shows time-ago for last successful backup
## 2026-03-25 (5)
- One-time path migration: all 571 existing library files moved to correct format-prefixed structure (`epub/`, `pdf/`, `comics/`) and all DB references updated (`migrate_paths.py --execute`)
- Empty old publisher root directories pruned after migration
- Recovery script (`recover_decock049.py`) written; confirmed file was added after last Dropbox backup so not recoverable — to be re-imported manually
## 2026-03-25 (4)
- Added {publisher} directory to PDF and CBR/CBZ paths: `pdf/{publisher}/{author}/{title}.pdf`, `comics/{publisher}/{author}/{title}.cbr/cbz` — consistent with EPUB structure
## 2026-03-25 (3)
- Fixed CBZ extension in import: `common.make_rel_path` always generated `.cbr` for CBZ files; now accepts `ext` parameter; `library.py` passes actual suffix so CBZ files land at `comics/{author}/{title}.cbz`
- Added missing `GET /download/{filename}` endpoint (referenced in book.html but was 404)
- TECHNICAL.md fully rewritten: added File Storage Paths section, complete endpoint lists for all routers including settings.py, corrected path documentation
## 2026-03-25 (2)
- Fixed PDF metadata editing (PATCH /library/book):
- `_sync_epub_metadata` is now only called for `.epub` files; PDFs update DB only
- `_make_rel_path` now includes the format prefix matching import: EPUB → `epub/{publisher}/{author}/…`, PDF → `pdf/{author}/{title}.pdf`, CBR/CBZ → `comics/{author}/{title}{ext}`; previously files were moved outside their format directory on metadata save
- Fixed PDF reader (infinite loading screen):
- `reader_page` now passes `format` (epub/pdf/cbr/cbz) to `reader.html`
- Added `GET /api/pdf/info/{filename}` endpoint returning `{"page_count": N}`
- `reader.html` branches on `FORMAT`: PDFs render page images from `/library/pdf/{filename}?page=N`, EPUB flow unchanged
- PDF progress tracked per page; keyboard and button navigation work identically to EPUB
- `Edit EPUB` button in Book Detail hidden for non-EPUB files
## 2026-03-23
- Added `All books` Grid/List toggle in Library:
- same columns as `New` view (Publisher, Author, Series, Volume, Title, Has cover, Updated, Genres, Sub-genres, Tags, Status)
- column visibility filter in `List` mode
- no selection checkboxes or bulk actions
- view mode and column visibility persisted separately in `localStorage` (`novela.all.viewMode`, `novela.all.visibleColumns`)
- Added 15 star rating for books:
- stored in the database (`rating SMALLINT DEFAULT 0`)
- written to EPUB OPF as `<meta name="novela:rating" content="N"/>` on rating change
- written to CBZ `ComicInfo.xml` as `<NovelaRating>N</NovelaRating>` on rating change
- CBR and PDF are DB-only (file format constraints)
- rating is recovered from file metadata during rescan/DB rebuild (`upsert_book` preserves file rating over default 0)
- stars displayed under the cover (outside `.book-info`) in all grid views (Library, Home)
- stars in grid cards are display-only (no click) to prevent accidental taps while scrolling
- stars in Book Detail are interactive (larger, 1.1rem), clicking same star removes rating
- star color: amber (`#c8a03a`) for filled, `rgba(200, 160, 58, 0.25)` for unfilled — consistent across Library, Home, and Book Detail
- Added text colour setting to reader hamburger menu:
- 5 warm-tone presets from bright (`#e8e2d9`) to dim (`#938d86`)
- active preset shown with accent-coloured ring
- choice persisted in `localStorage` (`reader-text-colour`) and restored on next open
- Increased spacing between hamburger button and back link in reader header (`margin-left: 1rem`) to prevent accidental taps
- Removed `Cover Missing` auto-tag:
- tag is no longer added on import, rescan, or grabber download
- `ensure_cover_missing_tag()` removed from `common.py`, `library.py`, and `grabber.py`
- startup migration removes all existing `Cover Missing` tags from the database
- Fixed Tags/Genres/Sub-Genres not saving in book edit panel on desktop:
- `,` (comma) now acts as a confirmation key alongside Enter in `PillInput`
- `flush()` added to `PillInput`: any text still in the input field is auto-confirmed when Save is clicked
- Fixed tag/genre search and tag-pill navigation being broken:
- `renderGenreView` was filtering on `b.genres` (non-existent field); now uses `bookGenres()`, `bookSubgenres()`, `bookPlainTags()`
- `renderSearchResults` had the same bug; search now covers title, author, genres, sub-genres, and tags