From b43366723c57745e8e6ccaa51bc41e0599638fec Mon Sep 17 00:00:00 2001 From: Ivo Oskamp Date: Sun, 29 Mar 2026 14:20:25 +0200 Subject: [PATCH] Add Bulk Import, Following, Incomplete, status overhaul, performance, and CBR fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Bulk Import page: filename pattern parsing, shared metadata, duplicate detection (volume-aware), batch upload with progress - Following page: track external author URLs; authors table; sidebar counter - Incomplete view: non-archived books with publication_status ≠ Complete - Status: added Temporary Hold, renamed Hiatus → Long-Term Hold; statusBadgeHtml() helper - Status/want-to-read badges: dark fill + ring for readability on any cover colour - Disk usage warning in sidebar (amber/red thresholds) - Bulk delete batched via POST /library/bulk-delete - CBR: magic bytes format detection + py7zr 7-zip support; unrar → proprietary unrar v6 - Performance: IntersectionObserver lazy covers, ETag 304, single DOM pass, json_agg tags - Duplicate detection in library and Convert page warning - All books Grid/List toggle; star ratings; reader text colour presets; bookmarks - Docs: TECHNICAL.md and changelog updated Co-Authored-By: Claude Sonnet 4.6 --- README.md | 77 +- containers/novela/Dockerfile | 5 +- containers/novela/cbr.py | 36 +- containers/novela/main.py | 2 + containers/novela/migrations.py | 5 + containers/novela/requirements.txt | 1 + containers/novela/routers/__init__.py | 2 + containers/novela/routers/bulk_import.py | 145 +++ containers/novela/routers/common.py | 5 + containers/novela/routers/grabber.py | 2 +- containers/novela/routers/library.py | 108 +++ containers/novela/routers/reader.py | 5 + containers/novela/static/book.css | 7 +- containers/novela/static/library.css | 14 +- containers/novela/static/library.js | 103 +- containers/novela/static/sidebar.css | 19 + containers/novela/templates/_sidebar.html | 38 + containers/novela/templates/book.html | 5 +- containers/novela/templates/bulk_import.html | 953 +++++++++++++++++++ containers/novela/templates/library.html | 8 +- docs/TECHNICAL.md | 22 +- docs/changelog-develop.md | 61 ++ 22 files changed, 1533 insertions(+), 90 deletions(-) create mode 100644 containers/novela/routers/bulk_import.py create mode 100644 containers/novela/templates/bulk_import.html diff --git a/README.md b/README.md index 30bb811..ccff0b7 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,65 @@ # Novela Novela is a self-hosted web application for managing and reading a personal digital library. -It supports EPUB, PDF, and CBR/CBZ, with metadata editing, reading progress tracking, and Dropbox backups. +It supports EPUB, PDF, and CBR/CBZ, with metadata editing, reading progress tracking, a web scraper/converter, and Dropbox backups. ## What Novela Provides -- Library import and indexing for EPUB/PDF/CBR/CBZ -- Home dashboard with continue reading and unread/read sections -- Reader support for EPUB, PDF, and comics (CBR/CBZ) -- Metadata editing (title, author, publisher, series, volume, tags, genres) -- `New` review workflow with list/grid view, column toggles, and bulk actions -- Reading analytics/statistics dashboard -- Dropbox backup with: - - versioned snapshots - - object deduplication - - retention policy - - scheduled background runs + +### Library +- Import and indexing for EPUB, PDF, CBR/CBZ +- Drag-and-drop import from library page or home page +- Cover extraction and caching (EPUB, PDF first page, CBR/CBZ first page); manual cover upload for EPUB +- Metadata editing: title, author, publisher, series, volume, tags, genres, sub-genres, star rating, publication status +- Publication statuses: Complete, Ongoing, Temporary Hold, Long-Term Hold +- Want-to-read flag and archived flag +- 1–5 star ratings (stored in EPUB OPF / CBZ ComicInfo.xml / DB) +- Download individual files + +### Views and Navigation +- Home dashboard: continue reading, unread shorts/novels, read shorts/novels +- Library grid/list views with search (title, author, genre) +- Series view: grouped by series with volume order +- Duplicates view: groups books with matching title+author +- Incomplete view: books not marked Complete +- New view: recently imported books awaiting review; bulk "Remove from New" action +- Following page: track external URLs per author + +### Bulk Import +- `/bulk-import` page: import multiple files at once with filename-based metadata parsing +- Free-text `%placeholder%` pattern editor (e.g. `%series% - %volume% - %title% - %year%`) +- Available placeholders: `%series%`, `%volume%`, `%title%`, `%year%`, `%month%`, `%day%`, `%author%`, `%publisher%`, `%ignore%` +- Colored chips: click to insert at cursor or drag onto the pattern input +- Shared metadata fields (author, publisher, status, genres, tags) override filename-parsed values +- Preview table with editable cells before importing +- Duplicate detection: checks title+author against existing library; duplicate rows highlighted, skipped by default +- Batched upload (5 files per request) with progress bar +- Batched bulk delete (20 files per request) with progress bar + +### Reader +- EPUB reader: chapter navigation, configurable content width and text colour, bookmarks +- PDF reader: page-by-page rendering +- CBR/CBZ reader: page-by-page image rendering +- Reading progress saved per book (resume where you left off) +- Bookmarks with notes; listed in library sidebar section + +### EPUB Tools +- EPUB editor (`/library/editor/{filename}`): edit chapter HTML in the browser +- Book Builder (`/builder`): create EPUB books from scratch with a WYSIWYG editor; publish directly to library +- Web Grabber (`/grabber`): scrape and convert web fiction to EPUB; site credentials management +- Converter (`/convert`): convert a URL to EPUB; duplicate detection before conversion + +### Analytics +- Statistics dashboard: reads by month / day / hour, genre and publisher breakdowns, top books, read history + +### Backups +- Dropbox backup: versioned snapshots with object-level deduplication and retention policy +- OAuth2 flow (preferred) or legacy access token +- Scheduled background backups (configurable interval) +- Live progress indicator during backup runs +- PostgreSQL dump included in each backup + +### Sidebar +- Disk usage warning: amber ≥ 85% or < 2 GB free; red ≥ 95% or < 500 MB free ## Tech Stack - FastAPI @@ -23,10 +68,10 @@ It supports EPUB, PDF, and CBR/CBZ, with metadata editing, reading progress trac - Docker / Docker Compose style deployment ## Repository Layout -- `containers/novela/` - application code (routers, templates, static assets, migrations) -- `stack/` - deployment stack files and environment configuration -- `docs/` - technical status and changelog documentation -- `build-and-push.sh` - helper script for container build/push +- `containers/novela/` — application code (routers, templates, static assets, migrations) +- `stack/` — deployment stack files and environment configuration +- `docs/` — technical status and changelog documentation +- `build-and-push.sh` — helper script for container build/push ## Quick Start (Development) 1. Configure environment values in `stack/novela.env`. diff --git a/containers/novela/Dockerfile b/containers/novela/Dockerfile index 61835d2..ddf227e 100644 --- a/containers/novela/Dockerfile +++ b/containers/novela/Dockerfile @@ -2,10 +2,11 @@ FROM python:3.12-slim WORKDIR /app -RUN apt-get update && apt-get install -y --no-install-recommends \ +RUN echo "deb http://deb.debian.org/debian bookworm non-free" >> /etc/apt/sources.list \ + && apt-get update && apt-get install -y --no-install-recommends \ build-essential \ libmagic1 \ - unrar-free \ + unrar \ postgresql-client \ && rm -rf /var/lib/apt/lists/* diff --git a/containers/novela/cbr.py b/containers/novela/cbr.py index 621e7a6..f425d12 100644 --- a/containers/novela/cbr.py +++ b/containers/novela/cbr.py @@ -2,21 +2,39 @@ from io import BytesIO from pathlib import Path import zipfile +import py7zr import rarfile from PIL import Image, ImageOps SUPPORTED_IMG = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp"} +_MAGIC_RAR = b"Rar!\x1a\x07" +_MAGIC_ZIP = b"PK" +_MAGIC_7Z = b"7z\xbc\xaf\x27\x1c" -def _is_cbz(path: Path) -> bool: - return path.suffix.lower() == ".cbz" +def _detect_format(path: Path) -> str: + """Detect archive format by magic bytes, ignoring file extension.""" + with open(path, "rb") as f: + header = f.read(8) + if header[:6] == _MAGIC_RAR: + return "rar" + if header[:2] == _MAGIC_ZIP: + return "zip" + if header[:6] == _MAGIC_7Z: + return "7z" + # Fallback: trust the extension + return "zip" if path.suffix.lower() == ".cbz" else "rar" def cbr_page_list(path: Path) -> list[str]: - if _is_cbz(path): + fmt = _detect_format(path) + if fmt == "zip": with zipfile.ZipFile(path) as zf: names = [n for n in zf.namelist() if Path(n).suffix.lower() in SUPPORTED_IMG] + elif fmt == "7z": + with py7zr.SevenZipFile(path, mode="r") as zf: + names = [n for n in zf.getnames() if Path(n).suffix.lower() in SUPPORTED_IMG] else: with rarfile.RarFile(path) as rf: names = [n for n in rf.namelist() if Path(n).suffix.lower() in SUPPORTED_IMG] @@ -42,11 +60,17 @@ def cbr_get_page(path: Path, page_num: int) -> tuple[bytes, str]: "bmp": "image/bmp", }.get(ext, "image/jpeg") - if _is_cbz(path): + fmt = _detect_format(path) + if fmt == "zip": with zipfile.ZipFile(path) as zf: return zf.read(name), mime - with rarfile.RarFile(path) as rf: - return rf.read(name), mime + elif fmt == "7z": + with py7zr.SevenZipFile(path, mode="r") as zf: + data = zf.read(targets=[name]) + return data[name].read(), mime + else: + with rarfile.RarFile(path) as rf: + return rf.read(name), mime def cbr_cover_thumb(path: Path) -> bytes: diff --git a/containers/novela/main.py b/containers/novela/main.py index e57425e..36c7645 100644 --- a/containers/novela/main.py +++ b/containers/novela/main.py @@ -10,6 +10,7 @@ from routers.backup import start_backup_scheduler, stop_backup_scheduler from routers import ( backup_router, builder_router, + bulk_import_router, editor_router, following_router, grabber_router, @@ -41,6 +42,7 @@ app.include_router(grabber_router) app.include_router(settings_router) app.include_router(backup_router) app.include_router(builder_router) +app.include_router(bulk_import_router) app.include_router(following_router) diff --git a/containers/novela/migrations.py b/containers/novela/migrations.py index 3c517a4..5fd144f 100644 --- a/containers/novela/migrations.py +++ b/containers/novela/migrations.py @@ -292,6 +292,10 @@ def migrate_create_authors() -> None: ) +def migrate_rename_hiatus() -> None: + _exec("UPDATE library SET publication_status = 'Long-Term Hold' WHERE publication_status = 'Hiatus'") + + def run_migrations() -> None: migrate_create_library() migrate_create_book_tags() @@ -309,3 +313,4 @@ def run_migrations() -> None: migrate_series_suffix() migrate_create_builder_drafts() migrate_create_authors() + migrate_rename_hiatus() diff --git a/containers/novela/requirements.txt b/containers/novela/requirements.txt index 85697da..29b440b 100644 --- a/containers/novela/requirements.txt +++ b/containers/novela/requirements.txt @@ -9,6 +9,7 @@ jinja2==3.1.4 Pillow==11.0.0 pymupdf==1.24.0 rarfile==4.2 +py7zr==0.22.0 dropbox==12.0.2 apscheduler==3.10.4 cryptography==44.0.1 diff --git a/containers/novela/routers/__init__.py b/containers/novela/routers/__init__.py index 9715803..7a90ec2 100644 --- a/containers/novela/routers/__init__.py +++ b/containers/novela/routers/__init__.py @@ -1,5 +1,6 @@ from routers.backup import router as backup_router from routers.builder import router as builder_router +from routers.bulk_import import router as bulk_import_router from routers.editor import router as editor_router from routers.following import router as following_router from routers.grabber import router as grabber_router @@ -15,5 +16,6 @@ __all__ = [ "backup_router", "settings_router", "builder_router", + "bulk_import_router", "following_router", ] diff --git a/containers/novela/routers/bulk_import.py b/containers/novela/routers/bulk_import.py new file mode 100644 index 0000000..c8ce7fd --- /dev/null +++ b/containers/novela/routers/bulk_import.py @@ -0,0 +1,145 @@ +import json +import uuid +from pathlib import Path + +from fastapi import APIRouter, File, Form, Request, UploadFile +from fastapi.responses import HTMLResponse, JSONResponse +from fastapi.templating import Jinja2Templates + +from cbr import cbr_page_count +from db import get_db_conn +from routers.common import ( + LIBRARY_DIR, + ensure_cover_cache_for_book, + ensure_unique_rel_path, + make_rel_path, + media_type_from_suffix, + parse_volume_str, + upsert_book, +) + +templates = Jinja2Templates(directory="templates") +router = APIRouter() + + +@router.get("/bulk-import", response_class=HTMLResponse) +async def bulk_import_page(request: Request): + return templates.TemplateResponse(request, "bulk_import.html", {"active": "bulk_import"}) + + +@router.post("/library/bulk-import") +async def library_bulk_import( + files: list[UploadFile] = File(...), + rows: str = Form(...), + shared: str = Form("{}"), +): + try: + rows_data = json.loads(rows) + shared_data = json.loads(shared) + except Exception: + return JSONResponse({"ok": False, "error": "Invalid JSON"}, status_code=400) + + rows_by_name = {r["original_filename"]: r for r in rows_data} + shared_author = shared_data.get("author", "") + shared_publisher = shared_data.get("publisher", "") + shared_status = shared_data.get("status", "") + shared_genres = [t.strip() for t in shared_data.get("genres", "").split(",") if t.strip()] + shared_tags = [t.strip() for t in shared_data.get("tags", "").split(",") if t.strip()] + + imported: list[str] = [] + skipped: list[dict] = [] + + for upload in files: + try: + name = upload.filename or "upload.bin" + suffix = Path(name).suffix.lower() + if suffix not in {".epub", ".pdf", ".cbr", ".cbz"}: + skipped.append({"file": name, "reason": "Unsupported file type"}) + continue + + mt = media_type_from_suffix(Path(name)) + if not mt: + skipped.append({"file": name, "reason": "Could not detect media type"}) + continue + + data = await upload.read() + if not data: + skipped.append({"file": name, "reason": "Empty upload"}) + continue + + row = rows_by_name.get(name, {}) + + title = (row.get("title") or "").strip() or Path(name).stem + author = (row.get("author") or "").strip() or shared_author + publisher = (row.get("publisher") or "").strip() or shared_publisher + series = (row.get("series") or "").strip() or shared_data.get("series", "") + series_index, series_suffix = parse_volume_str(row.get("volume") or "") + status = (row.get("status") or "").strip() or shared_status + + year = str(row.get("year") or "").strip() + publish_date: str | None = f"{year}-01-01" if year.isdigit() and len(year) == 4 else None + + row_genres = [t.strip() for t in (row.get("genres") or "").split(",") if t.strip()] + row_tags = [t.strip() for t in (row.get("tags") or "").split(",") if t.strip()] + genres = row_genres if row_genres else shared_genres + plain_tags = row_tags if row_tags else shared_tags + + tmp = LIBRARY_DIR / f".import-{uuid.uuid4().hex}{suffix}" + tmp.parent.mkdir(parents=True, exist_ok=True) + tmp.write_bytes(data) + + rel = ensure_unique_rel_path( + make_rel_path( + media_type=mt, + publisher=publisher, + author=author, + title=title, + series=series, + series_index=series_index, + series_suffix=series_suffix, + ext=suffix, + ) + ) + dest = LIBRARY_DIR / rel + dest.parent.mkdir(parents=True, exist_ok=True) + tmp.replace(dest) + + rel_name = rel.as_posix() + + has_cover = False + try: + if mt == "cbr": + has_cover = cbr_page_count(dest) > 0 + except Exception: + pass + + meta = { + "media_type": mt, + "title": title, + "author": author, + "publisher": publisher, + "series": series, + "series_index": series_index, + "series_suffix": series_suffix, + "publication_status": status, + "publish_date": publish_date, + "has_cover": has_cover, + "needs_review": True, + "description": "", + "source_url": "", + } + tag_tuples = [(g, "genre") for g in genres] + [(t, "tag") for t in plain_tags] + + with get_db_conn() as conn: + with conn: + upsert_book(conn, rel_name, meta, tag_tuples) + ensure_cover_cache_for_book(conn, rel_name, dest, mt) + + imported.append(rel_name) + + except Exception as e: + skipped.append({"file": upload.filename or "upload", "reason": str(e)}) + finally: + await upload.close() + + return {"ok": True, "imported": imported, "skipped": skipped} diff --git a/containers/novela/routers/common.py b/containers/novela/routers/common.py index bf7e3cb..b782195 100644 --- a/containers/novela/routers/common.py +++ b/containers/novela/routers/common.py @@ -100,6 +100,11 @@ def make_rel_path(*, media_type: str, publisher: str, author: str, title: str, s pub = clean_segment(publisher, "Unknown Publisher", 80) auth = clean_segment(author, "Unknown", 80) ttl = clean_segment(title, "Untitled", 140) + series_name = clean_segment(series, "", 80) + if series_name: + idx = coerce_series_index(series_index) + sfx = re.sub(r"[^a-z]", "", (series_suffix or "").lower())[:5] + return Path("comics") / pub / auth / "Series" / series_name / f"{idx:03d}{sfx} - {ttl}{comics_ext}" return Path("comics") / pub / auth / f"{ttl}{comics_ext}" diff --git a/containers/novela/routers/grabber.py b/containers/novela/routers/grabber.py index ca54113..ef96d63 100644 --- a/containers/novela/routers/grabber.py +++ b/containers/novela/routers/grabber.py @@ -293,7 +293,7 @@ async def _run_scrape(job_id: str, url: str, username: str, password: str, send) tags = list(book.get("tags", [])) if len(book["chapters"]) < 4 and "Shorts" not in tags: tags.append("Shorts") - status_map = {"Long-Term Hold": "Hiatus"} + status_map = {"Temporary-Hold": "Temporary Hold"} pub_status = status_map.get(book.get("publication_status", ""), book.get("publication_status", "")) series = book.get("series", "") diff --git a/containers/novela/routers/library.py b/containers/novela/routers/library.py index b24ca3f..e0ca3ee 100644 --- a/containers/novela/routers/library.py +++ b/containers/novela/routers/library.py @@ -1,4 +1,5 @@ import base64 +import shutil import uuid from datetime import datetime, timezone from pathlib import Path @@ -142,6 +143,7 @@ async def library_import(files: list[UploadFile] = File(...)): title=meta.get("title") or Path(name).stem, series=meta.get("series", ""), series_index=meta.get("series_index", 0), + series_suffix=meta.get("series_suffix", ""), ext=suffix, ) ) @@ -190,6 +192,45 @@ async def library_delete(filename: str): return {"ok": True} +@router.post("/library/bulk-delete") +async def library_bulk_delete(request: Request): + body = await request.json() + filenames = body.get("filenames", []) + if not isinstance(filenames, list): + return JSONResponse({"error": "filenames must be a list"}, status_code=400) + + deleted: list[str] = [] + skipped: list[str] = [] + + for filename in filenames: + if not isinstance(filename, str): + continue + full = resolve_library_path(filename) + if full is None: + skipped.append(filename) + continue + try: + if full.exists(): + parent = full.parent + full.unlink() + prune_empty_dirs(parent) + deleted.append(filename) + except Exception: + skipped.append(filename) + + if deleted: + placeholders = ", ".join(["%s"] * len(deleted)) + with get_db_conn() as conn: + with conn: + with conn.cursor() as cur: + cur.execute( + f"DELETE FROM library WHERE filename IN ({placeholders})", + tuple(deleted), + ) + + return {"ok": True, "deleted": len(deleted), "skipped": skipped} + + @router.get("/library/cover-cached/{filename:path}") async def library_cover_cached(filename: str): full = resolve_library_path(filename) @@ -573,6 +614,73 @@ async def stats_page(request: Request): return templates.TemplateResponse(request, "stats.html", {"active": "stats"}) +@router.get("/api/disk") +async def api_disk(): + usage = shutil.disk_usage(str(LIBRARY_DIR)) + pct_used = round(usage.used / usage.total * 100, 1) if usage.total > 0 else 0 + return { + "total": usage.total, + "used": usage.used, + "free": usage.free, + "pct_used": pct_used, + } + + +@router.post("/api/bulk-check-duplicates") +async def bulk_check_duplicates(request: Request): + body = await request.json() + items = body.get("items", []) + if not items or not isinstance(items, list): + return {"duplicates": []} + + parsed = [] + for item in items: + title = item.get("title", "").strip().lower() + author = item.get("author", "").strip().lower() + vol_str = item.get("volume", "").strip() + try: + vol_int = int(vol_str) if vol_str else None + except ValueError: + vol_int = None + parsed.append((title, author, vol_int)) + + # Fetch all DB rows matching any (title, author) pair + title_author_pairs = list({(t, a) for t, a, _ in parsed if t}) + if not title_author_pairs: + return {"duplicates": [False] * len(items)} + + conditions = " OR ".join( + "(LOWER(TRIM(title)) = %s AND LOWER(TRIM(author)) = %s)" for _ in title_author_pairs + ) + params = [v for pair in title_author_pairs for v in pair] + with get_db_conn() as conn: + with conn.cursor() as cur: + cur.execute( + f"SELECT LOWER(TRIM(title)), LOWER(TRIM(author)), series_index" + f" FROM library WHERE {conditions}", + params, + ) + rows = cur.fetchall() + + # (title, author, series_index) for volume-aware lookup + existing_with_vol = {(r[0] or "", r[1] or "", r[2]) for r in rows} + # (title, author) for volume-less lookup + existing_title_author = {(r[0] or "", r[1] or "") for r in rows} + + duplicates = [] + for title, author, vol_int in parsed: + if not title: + duplicates.append(False) + elif vol_int is not None: + # Volume known: only a duplicate when title+author+volume all match + duplicates.append((title, author, vol_int) in existing_with_vol) + else: + # No volume: duplicate if any title+author match exists + duplicates.append((title, author) in existing_title_author) + + return {"duplicates": duplicates} + + @router.get("/api/stats") async def api_stats(): with get_db_conn() as conn: diff --git a/containers/novela/routers/reader.py b/containers/novela/routers/reader.py index d3c17df..e0bb980 100644 --- a/containers/novela/routers/reader.py +++ b/containers/novela/routers/reader.py @@ -428,6 +428,11 @@ def _make_rel_path( # .cbr / .cbz pub = _clean_segment(publisher, "Unknown Publisher", 80) + series_name = _clean_segment(series, "", 80) + if series_name: + idx = _coerce_series_index(series_index) + sfx = re.sub(r"[^a-z]", "", (series_suffix or "").lower())[:5] + return Path("comics") / pub / auth / "Series" / series_name / f"{idx:03d}{sfx} - {ttl}{ext}" return Path("comics") / pub / auth / f"{ttl}{ext}" diff --git a/containers/novela/static/book.css b/containers/novela/static/book.css index 9c8b19a..ff5242c 100644 --- a/containers/novela/static/book.css +++ b/containers/novela/static/book.css @@ -123,9 +123,10 @@ a.tag-pill:hover { color: var(--accent2); border-color: var(--accent); } display: inline-block; font-family: var(--mono); font-size: 0.62rem; padding: 0.15rem 0.5rem; border-radius: 3px; } -.status-complete { background: rgba(107,170,107,0.12); color: var(--success); border: 1px solid rgba(107,170,107,0.25); } -.status-ongoing { background: rgba(200,160,58,0.12); color: var(--warning); border: 1px solid rgba(200,160,58,0.25); } -.status-hiatus { background: rgba(200,160,58,0.12); color: var(--warning); border: 1px solid rgba(200,160,58,0.25); } +.status-complete { background: rgba(107,170,107,0.12); color: #6baa6b; border: 1px solid rgba(107,170,107,0.25); } +.status-ongoing { background: rgba(74,144,184,0.12); color: #4a90b8; border: 1px solid rgba(74,144,184,0.25); } +.status-temporary-hold { background: rgba(200,160,58,0.12); color: #c8a03a; border: 1px solid rgba(200,160,58,0.25); } +.status-long-term-hold { background: rgba(200,120,58,0.12); color: #c8783a; border: 1px solid rgba(200,120,58,0.25); } /* Progress */ .progress-section { margin-bottom: 1.25rem; } diff --git a/containers/novela/static/library.css b/containers/novela/static/library.css index 67662aa..9074228 100644 --- a/containers/novela/static/library.css +++ b/containers/novela/static/library.css @@ -151,10 +151,13 @@ html, body { align-items: center; justify-content: center; z-index: 2; + background: rgba(15,14,12,0.82); + box-shadow: 0 0 0 2px #0f0e0c; } -.badge-complete { background: rgba(107,170,107,0.18); color: var(--success); } -.badge-ongoing { background: rgba(200,160,58,0.18); color: var(--warning); } -.badge-hiatus { background: rgba(200,160,58,0.18); color: var(--warning); } +.badge-complete { color: #6baa6b; } +.badge-ongoing { color: #4a90b8; } +.badge-temporary-hold { color: #c8a03a; } +.badge-long-term-hold { color: #c8783a; } /* Star: want-to-read top-left */ .btn-star { @@ -164,7 +167,7 @@ html, body { width: 22px; height: 22px; border: none; - background: rgba(15,14,12,0.6); + background: rgba(15,14,12,0.82); border-radius: 50%; display: flex; align-items: center; @@ -174,8 +177,9 @@ html, body { transition: color 0.15s, background 0.15s; padding: 0; z-index: 2; + box-shadow: 0 0 0 2px #0f0e0c; } -.btn-star:hover { color: var(--warning); background: rgba(15,14,12,0.8); } +.btn-star:hover { color: var(--warning); background: rgba(15,14,12,0.82); } .btn-star.starred { color: var(--warning); } /* Book info below cover */ diff --git a/containers/novela/static/library.js b/containers/novela/static/library.js index d789294..71e3a09 100644 --- a/containers/novela/static/library.js +++ b/containers/novela/static/library.js @@ -1,5 +1,19 @@ /* ── Novela — Library page script ─────────────────────────────────────── */ +function statusBadgeHtml(publicationStatus) { + const st = (publicationStatus || '').toLowerCase(); + if (st === 'complete') { + return `
`; + } else if (st === 'ongoing') { + return `
`; + } else if (st === 'temporary hold') { + return `
`; + } else if (st === 'long-term hold') { + return `
`; + } + return ''; +} + let allBooks = []; let currentView = 'all'; let currentParam = null; @@ -460,6 +474,7 @@ async function markSelectedNewAsReviewed(books) { const btn = document.getElementById('btn-mark-reviewed'); if (btn) btn.disabled = true; + let succeeded = false; try { const resp = await fetch('/library/new/mark-reviewed', { method: 'POST', @@ -469,21 +484,19 @@ async function markSelectedNewAsReviewed(books) { const result = await resp.json(); if (!resp.ok || result.error) { alert(result.error || 'Could not mark books as reviewed.'); - return; + } else { + succeeded = true; } - - const selectedSet = new Set(selected); - allBooks.forEach(b => { - if (selectedSet.has(b.filename)) b.needs_review = false; - }); - selected.forEach(f => newSelectedFilenames.delete(f)); - updateCounts(); - renderGrid(); } catch { alert('Could not mark books as reviewed.'); } finally { if (btn) btn.disabled = false; } + + if (succeeded) { + selected.forEach(f => newSelectedFilenames.delete(f)); + await loadLibrary(); + } } function tagValuesByType(book, type) { @@ -919,14 +932,38 @@ function closeBulkDeleteDialog() { async function confirmBulkDelete() { const filenames = [...allSelectedFilenames]; if (!filenames.length) return; - const btn = document.getElementById('bulk-delete-btn'); + + const btn = document.getElementById('bulk-delete-btn'); + const actions = document.getElementById('bulk-delete-actions'); + const progress = document.getElementById('bulk-delete-progress'); + const bar = document.getElementById('bulk-delete-bar'); + const status = document.getElementById('bulk-delete-status'); + if (btn) btn.disabled = true; - for (const filename of filenames) { + if (actions) actions.querySelector('.btn-cancel').disabled = true; + if (progress) progress.style.display = 'block'; + + const BATCH = 20; + const total = filenames.length; + let done = 0; + + for (let i = 0; i < filenames.length; i += BATCH) { + const batch = filenames.slice(i, i + BATCH); try { - await fetch(`/library/file/${encodeURIComponent(filename)}`, { method: 'DELETE' }); + await fetch('/library/bulk-delete', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ filenames: batch }), + }); } catch {} + done = Math.min(i + BATCH, total); + if (bar) bar.style.width = Math.round((done / total) * 100) + '%'; + if (status) status.textContent = `${done} / ${total} deleted…`; } + closeBulkDeleteDialog(); + if (progress) progress.style.display = 'none'; + if (bar) bar.style.width = '0%'; allSelectedFilenames.clear(); allLastToggledIndex = null; await loadLibrary(); @@ -984,21 +1021,7 @@ function renderBooksGrid(books) { card.className = 'book-card'; card.id = `card-${cssId(b.filename)}`; - const st = (b.publication_status || '').toLowerCase(); - let statusBadge = ''; - if (st === 'complete') { - statusBadge = `
- -
`; - } else if (st === 'ongoing') { - statusBadge = `
- -
`; - } else if (st === 'hiatus') { - statusBadge = `
- -
`; - } + const statusBadge = statusBadgeHtml(b.publication_status); const starClass = b.want_to_read ? 'btn-star starred' : 'btn-star'; const seriesVol = seriesVolLabel(b, idxSeries); @@ -1237,15 +1260,7 @@ function renderSeriesDetail(seriesName) { const title = bookTitle(b); const cid = cssId(b.filename); - const st = (b.publication_status || '').toLowerCase(); - let statusBadge = ''; - if (st === 'complete') { - statusBadge = `
`; - } else if (st === 'ongoing') { - statusBadge = `
`; - } else if (st === 'hiatus') { - statusBadge = `
`; - } + const statusBadge = statusBadgeHtml(b.publication_status); const bookCard = document.createElement('div'); bookCard.className = 'book-card'; @@ -1572,21 +1587,7 @@ function renderDuplicatesView() { card.className = 'book-card'; card.id = `card-${cssId(b.filename)}`; - const st = (b.publication_status || '').toLowerCase(); - let statusBadge = ''; - if (st === 'complete') { - statusBadge = `
- -
`; - } else if (st === 'ongoing') { - statusBadge = `
- -
`; - } else if (st === 'hiatus') { - statusBadge = `
- -
`; - } + const statusBadge = statusBadgeHtml(b.publication_status); const starClass = b.want_to_read ? 'btn-star starred' : 'btn-star'; diff --git a/containers/novela/static/sidebar.css b/containers/novela/static/sidebar.css index e726e08..58dd390 100644 --- a/containers/novela/static/sidebar.css +++ b/containers/novela/static/sidebar.css @@ -89,6 +89,25 @@ html { .sidebar-bottom { margin-top: auto; } +.disk-warning { + display: flex; + align-items: center; + gap: 0.4rem; + padding: 0.4rem 0.6rem; + border-radius: var(--radius); + font-family: var(--mono); + font-size: 0.7rem; + margin-bottom: 0.5rem; + background: rgba(200,160,58,0.12); + border: 1px solid rgba(200,160,58,0.3); + color: #c8a03a; +} +.disk-warning.critical { + background: rgba(200,90,58,0.12); + border-color: rgba(200,90,58,0.3); + color: #c85a3a; +} + .btn-rescan { display: flex; align-items: center; diff --git a/containers/novela/templates/_sidebar.html b/containers/novela/templates/_sidebar.html index 34ad2e4..c136ded 100644 --- a/containers/novela/templates/_sidebar.html +++ b/containers/novela/templates/_sidebar.html @@ -199,6 +199,16 @@ Book Builder +
  • + + + + + + + Bulk Import + +
  • @@ -237,6 +247,10 @@
    + Backup… @@ -423,8 +437,32 @@ } catch (_) {} } + async function checkDiskUsage() { + try { + const r = await fetch('/api/disk'); + if (!r.ok) return; + const d = await r.json(); + const el = document.getElementById('disk-warning'); + const txt = document.getElementById('disk-warning-text'); + if (!el || !txt) return; + const gb = d.free / (1024 ** 3); + const critical = d.pct_used >= 95 || gb < 0.5; + const warning = d.pct_used >= 85 || gb < 2; + if (critical || warning) { + const freeStr = gb < 1 ? (d.free / (1024 ** 2)).toFixed(0) + ' MB' : gb.toFixed(1) + ' GB'; + txt.textContent = `Storage ${d.pct_used}% used · ${freeStr} free`; + el.className = 'disk-warning' + (critical ? ' critical' : ''); + el.style.display = 'flex'; + } else { + el.style.display = 'none'; + } + } catch (_) {} + } + refreshLibraryCounts(); refreshBookmarkCount(); refreshFollowingCount(); loadBackupStatus(); + checkDiskUsage(); + setInterval(checkDiskUsage, 60_000); diff --git a/containers/novela/templates/book.html b/containers/novela/templates/book.html index b362a69..aeffe59 100644 --- a/containers/novela/templates/book.html +++ b/containers/novela/templates/book.html @@ -65,7 +65,7 @@ Status {% set st = publication_status | lower %} - + {{ publication_status }} @@ -252,7 +252,8 @@ - + +
    diff --git a/containers/novela/templates/bulk_import.html b/containers/novela/templates/bulk_import.html new file mode 100644 index 0000000..97e6940 --- /dev/null +++ b/containers/novela/templates/bulk_import.html @@ -0,0 +1,953 @@ + + + + + + Novela – Bulk Import + + + + + + + +{% include "_sidebar.html" %} + +
    + + +
    +
    Filename Pattern
    + + + + + +
    + +
    + + + +
    +
    + + +
    +
    Shared Metadata
    +

    Applies to all files. Filled-in fields override values parsed from the pattern.

    + +
    +
    + +
    + + +
    +
    +
    + +
    + + +
    +
    +
    + +
    +
    + +
    + + +
    +
    +
    + + +
    +
    + + +
    +
    + + + +
    + + +
    +
    Select Files
    +
    + +
    + + + + + + Click or drop files here — CBR, CBZ, EPUB, PDF +
    +
    +
    +
    + + + + + + + +
    + + + + diff --git a/containers/novela/templates/library.html b/containers/novela/templates/library.html index aea557d..1985eee 100644 --- a/containers/novela/templates/library.html +++ b/containers/novela/templates/library.html @@ -60,7 +60,13 @@
    Delete books

    Delete selected book(s)?
    Files will be permanently removed from disk. This cannot be undone.

    -
    + +
    diff --git a/docs/TECHNICAL.md b/docs/TECHNICAL.md index c04084f..8185954 100644 --- a/docs/TECHNICAL.md +++ b/docs/TECHNICAL.md @@ -28,8 +28,10 @@ All files are stored under `library/` (relative to the app working directory, ma | EPUB (no series) | `library/epub/{publisher}/{author}/Stories/{title}.epub` | | EPUB (series) | `library/epub/{publisher}/{author}/Series/{series}/{idx:03d} - {title}.epub` | | PDF | `library/pdf/{publisher}/{author}/{title}.pdf` | -| CBR | `library/comics/{publisher}/{author}/{title}.cbr` | -| CBZ | `library/comics/{publisher}/{author}/{title}.cbz` | +| CBR (no series) | `library/comics/{publisher}/{author}/{title}.cbr` | +| CBR (series) | `library/comics/{publisher}/{author}/Series/{series}/{idx:03d} - {title}.cbr` | +| CBZ (no series) | `library/comics/{publisher}/{author}/{title}.cbz` | +| CBZ (series) | `library/comics/{publisher}/{author}/Series/{series}/{idx:03d} - {title}.cbz` | - Segments are sanitised: special chars stripped, max lengths applied (publisher/author 80, title 140, series 80). - Series index is zero-padded to 3 digits (`001`, `002`, …), clamped to 1–999. @@ -68,11 +70,14 @@ All files are stored under `library/` (relative to the app working directory, ma - `POST /library/want-to-read/{filename}` — toggle want-to-read flag - `POST /library/archive/{filename}` — toggle archived flag - `POST /library/new/mark-reviewed` — bulk set `needs_review=false` +- `POST /library/bulk-delete` — delete multiple files; accepts `{"filenames": [...]}`, removes files from disk and DB in one query per batch; returns `{ok, deleted, skipped}` - `POST /library/rating/{filename}` — set/clear star rating `{"rating": 0-5}` - `GET /home` — home page - `GET /api/home` — home data JSON - `GET /stats` — statistics page - `GET /api/stats` — statistics data JSON +- `GET /api/disk` — partition usage for the library directory: `{total, used, free, pct_used}` +- `POST /api/bulk-check-duplicates` — accepts `{"items": [{title, author, volume}, ...]}`, returns `{"duplicates": [bool, ...]}` — when `volume` is a number, requires title+author+series_index to all match; when volume is absent, matches on title+author only - `GET /library/list` — compat alias `GET /api/library` runs in fast-path mode by default (DB-only, no full disk rescan). @@ -123,6 +128,12 @@ Home read sections are ordered oldest-first: - `DELETE /library/bookmarks/{id}` — delete bookmark - `GET /api/bookmarks` — all bookmarks across all books (includes `book_title`, `book_author`) +### `routers/bulk_import.py` +- `GET /bulk-import` — Bulk Import page +- `POST /library/bulk-import` — import files with pre-parsed metadata; accepts multipart `files[]`, `rows` (JSON array of per-file metadata), `shared` (JSON with author/publisher/status/genres/tags applied to all files) + +Filename parsing is done client-side in `bulk_import.html`. The page uses a free-text `%placeholder%` pattern (e.g. `%series% - %volume% - %title% - %year%`). Available placeholders: `%series%` `%volume%` `%title%` `%year%` `%month%` `%day%` `%author%` `%publisher%` `%ignore%`. Colored chips can be clicked (insert at cursor) or dragged onto the input. Pattern is converted to a regex at parse time. Shared metadata fields override filename-parsed values. Files are uploaded in batches of 5 with a progress bar. + ### `routers/editor.py` - `GET /library/editor/{filename}` — EPUB chapter editor page - `GET /api/edit/chapter/{index}/{filename}` — get chapter HTML @@ -252,6 +263,11 @@ Dropbox settings are managed via the web UI on `/backup`. - Column visibility persisted in `localStorage` as `novela.all.visibleColumns`. - `List` mode has a checkbox column, column visibility filter, and multi-select with `Shift+click` range selection. - `List` mode has a `Delete selected` bulk action: confirms then calls `DELETE /library/file/{filename}` for each selected book. +- Publication status values: `Complete`, `Ongoing`, `Temporary Hold`, `Long-Term Hold` (blank = unknown). `Hiatus` was renamed to `Long-Term Hold` via startup migration `migrate_rename_hiatus()`. +- Status badges (top-right of grid card cover): circular icon, dark fill `rgba(15,14,12,0.82)` + `box-shadow: 0 0 0 2px #0f0e0c` ring for visibility on any cover colour. Icon colour per status: Complete=green `#6baa6b`, Ongoing=blue `#4a90b8`, Temporary Hold=amber `#c8a03a`, Long-Term Hold=orange `#c8783a`. `statusBadgeHtml()` in `library.js` is the single source for badge HTML across all grid views. +- Want-to-read star (top-left of grid card cover): same dark fill + ring as status badges. +- Status pills in Book Detail (`book.css`): `status-complete`, `status-ongoing`, `status-temporary-hold`, `status-long-term-hold` — same colour scheme as badges. +- Grabber status mapping (`grabber.py`): `Temporary-Hold` (gayauthors.org) → `Temporary Hold`; `Long-Term Hold` passes through unchanged. - Star ratings (1–5) shown under the cover in all grid views: - Display-only in grid cards (no click, prevents accidental taps while scrolling). - Interactive in Book Detail (1.1rem, clickable; clicking the active star clears the rating). @@ -269,7 +285,7 @@ Dropbox settings are managed via the web UI on `/backup`. - Bookmarks: saved per book via `POST /library/bookmarks/{filename}`; shown in Library sidebar section; navigated via `?bm_ch=N&bm_scroll=F` URL params on reader page. - Convert page: after loading metadata, if a book with the same title+author already exists in the library, a warning banner is shown (with a link to the existing book); user can still proceed with conversion. Check is done server-side in `/preload` response (`already_exists`, `existing_books`). - Duplicates view (`#duplicates`): groups non-archived books by `(title, author)` (case-insensitive); shows only groups with ≥ 2 copies; counter in sidebar shows total number of duplicate books. Detection is entirely client-side from the existing library data. -- Incomplete view (`#incomplete`): shows all non-archived books where `publication_status` is not `Complete` (Ongoing, Hiatus, or blank); sidebar counter included. +- Incomplete view (`#incomplete`): shows all non-archived books where `publication_status` is not `Complete` (Ongoing, Temporary Hold, Long-Term Hold, or blank); sidebar counter included. - Following page (`/following`): dedicated page in its own sidebar section between Library and Tools; shows all library authors with their external URL; two tabs — Following (authors with URL set) and All Authors; inline URL editing with keyboard support (Enter = save, Escape = cancel); clicking Visit opens the external URL in a new tab. Author URLs are stored in the `authors` table. Sidebar counter shows number of followed authors. - Book Builder (`/builder`): create EPUB books from scratch; drafts stored in `builder_drafts` (JSONB chapters); contenteditable editor with toolbar (bold/italic/underline/blockquote/author-note/scene-break/normalize); autosave every 30 s + Ctrl+S; publish normalizes HTML via `normalize_wysiwyg_html()` and builds EPUB via `build_epub()`. diff --git a/docs/changelog-develop.md b/docs/changelog-develop.md index 26a616d..fcdc5f5 100644 --- a/docs/changelog-develop.md +++ b/docs/changelog-develop.md @@ -1,5 +1,66 @@ # Develop Changelog +## 2026-03-29 (14) +- Dockerfile: replaced `unrar-free` with proprietary `unrar` (RARLAB v6.2.6) from Debian non-free — fixes "Failed to read enough data" errors on RAR archives using newer compression methods + +## 2026-03-29 (13) +- CBR reader: detect archive format via magic bytes instead of file extension — `.cbr` files that are actually ZIP or 7-zip archives now open correctly; added `py7zr` dependency for 7-zip support + +## 2026-03-29 (12) +- Bulk Import: duplicate check now volume-aware — books with the same title+author but a different volume number (e.g. recoloured reprints) are no longer flagged as duplicates; `volume` is included in the API call and matched against `series_index` in the DB + +## 2026-03-28 (11) +- Bulk Import: duplicate detection against existing library + - New `POST /api/bulk-check-duplicates` endpoint: case-insensitive title+author+volume match, single SQL query with OR conditions for all pairs + - Duplicate rows highlighted in red in the preview table; a skip checkbox appears per duplicate row + - Stats bar shows "X duplicates · Skip all · Import all" action buttons + - Duplicate rows are skipped by default; user can toggle individual rows or use bulk actions + - `startImport()` filters out skipped rows before batching; skipped files appear in the result summary as "Duplicate – skipped" + - If all rows are skipped (nothing to import), result shown immediately without sending any request + +## 2026-03-28 (10) +- After "Remove from New" succeeds, the library is now reloaded from the server (`loadLibrary()`) so the New list updates immediately without a manual refresh + +## 2026-03-28 (9) +- Bulk delete is now batched: new `POST /library/bulk-delete` endpoint accepts a JSON list of filenames, deletes files and removes DB rows in one query per batch; JS sends 20 files per batch with a progress bar in the confirmation dialog + +## 2026-03-28 (8) +- Disk usage warning in sidebar: `GET /api/disk` returns partition usage for the library directory; sidebar polls every 60 s and shows a warning bar (amber ≥ 85% or < 2 GB free, red ≥ 95% or < 500 MB free) above the backup status bar + +## 2026-03-28 (7) +- Fixed `mark-reviewed` JS: UI (allBooks update + renderGrid) now only runs after confirmed server success; `catch` no longer fires a false error after a successful response where renderGrid throws + +## 2026-03-28 (6) +- Bulk Import: rewrote pattern editor from token-pills to free-text `%placeholder%` syntax + - Pattern input: free-text field; placeholders: `%series%` `%volume%` `%title%` `%year%` `%month%` `%day%` `%author%` `%publisher%` `%ignore%` + - Colored chip row: click to insert at cursor position; drag onto input also supported + - Colored live preview below the pattern input; regex-based parser replaces delimiter+token logic + - Shared metadata now wins over filename-parsed values (previously filename won) + - All UI text translated from Dutch to English + +## 2026-03-28 (5) +- Status badge (top-right cover) and want-to-read star (top-left cover) now use dark fill `rgba(15,14,12,0.82)` + `box-shadow: 0 0 0 2px #0f0e0c` ring — always readable regardless of cover colour + +## 2026-03-28 (4) +- Added `Temporary Hold` status; renamed `Hiatus` → `Long-Term Hold` + - Startup migration `migrate_rename_hiatus()` converts existing DB records automatically + - Status colours: Complete=green, Ongoing=blue, Temporary Hold=amber, Long-Term Hold=orange + - `statusBadgeHtml()` helper in `library.js` replaces three identical inline badge blocks + - Grabber: `Temporary-Hold` (gayauthors.org) now maps to `Temporary Hold`; `Long-Term Hold` passes through unchanged + - Status dropdowns updated in Book Detail and Bulk Import + +## 2026-03-28 (3) +- Added Bulk Import page (`/bulk-import`) for batch importing CBR/CBZ/EPUB/PDF files with filename-based metadata parsing + - Configurable token pattern: Serie, Volume, Titel, Jaar, Auteur, Uitgever, Negeer (in any order) + - Configurable delimiter (default ` - `); year token right-anchored when last so title absorbs overflow segments + - Live test-parse box: type any filename and see how it parses before selecting files + - Shared metadata card: author, publisher, status, genres, tags applied to all files (overridden by pattern tokens or manual edits) + - Preview table: all parsed fields editable via contenteditable cells; warning indicator for rows with fewer segments than tokens; sorted by filename + - Batch upload in groups of 5 with progress bar (N / total files processed) + - Result summary with list of skipped files and reasons + - New `routers/bulk_import.py`: `GET /bulk-import`, `POST /library/bulk-import` + - Sidebar link under Tools between Book Builder and Credentials + This file tracks changes on the `develop` line. `changelog.md` can later be used for release summaries.