ivooskamp/novela
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
ced5b25dbe
novela/docs/BLUEPRINT.md

421 lines
11 KiB
Markdown
Raw Blame History

# Novela 2.0 - Blauwdruk
> Vervangt repository `story-grabber`. Nieuwe repo: **Novela**.
> Stack: FastAPI · Jinja2 · plain JS · PostgreSQL 16 · Docker / Portainer
---
## 1. Doelstelling
Novela 2.0 is een volledig zelfgehoste media-bibliotheek en e-reader voor epub, pdf en cbr/cbz.
Het vervangt Kavita (library), Calibre (metadata), en Sigil (epub editor) in een web-applicatie.
Kernprincipe: **de database is de snelle index, het bestand is de bron van waarheid.**
Elke schrijfactie raakt altijd beide: eerst het bestand, dan de database. Lezen gaat altijd via de database.
---
## 2. Wat behouden blijft uit v1
| Module | Bestand | Toelichting |
|---|---|---|
| EPUB bouw | `epub.py` | `make_epub`, `make_chapter_xhtml`, `add_cover_to_epub` |
| EPUB lezen/schrijven | `epub.py` | `read_epub_file`, `write_epub_file` |
| XHTML conversie | `xhtml.py` | `element_to_xhtml`, `is_break_element`, `configure_break_patterns` |
| Scrapers | `scrapers/` | base, awesomedude, gayauthors, plugin-patroon blijft |
| SSE job streaming | `main.py` | `JOBS` dict + `/events/{job_id}` `StreamingResponse` |
| Migrations patroon | `migrations.py` | idempotente `CREATE IF NOT EXISTS`, `run_migrations()` bij startup |
| Cover cache | DB tabel | `library_cover_cache`, WebP thumbnails 300x450 |
| Reading progress | DB tabel | CFI voor epub, paginanummer voor pdf/cbr |
| Reading sessions | DB tabel | leesgeschiedenis per boek |
| Break patterns | DB tabel | regex + css_class patronen voor scene-breaks |
---
## 3. Projectstructuur
```text
novela/
├── containers/
│ └── novela/
│ ├── main.py
│ ├── migrations.py
│ ├── db.py
│ ├── epub.py
│ ├── xhtml.py
│ ├── pdf.py
│ ├── cbr.py
│ ├── routers/
│ │ ├── __init__.py
│ │ ├── library.py
│ │ ├── reader.py
│ │ ├── editor.py
│ │ ├── grabber.py
│ │ ├── backup.py
│ │ └── settings.py
│ ├── scrapers/
│ ├── static/
│ ├── templates/
│ ├── requirements.txt
│ └── Dockerfile
├── stack/
│ ├── stack.yml
│ └── novela.env
└── docs/
├── BLUEPRINT.md
└── TECHNICAL.md
```
---
## 4. Bibliotheek op schijf
`output/` wordt `library/`.
```text
library/
├── epub/
│ └── {Publisher}/
│ └── {Author}/
│ ├── Stories/
│ │ └── {Titel}.epub
│ └── Series/
│ └── {Serienaam}/
│ └── {001 - Titel}.epub
├── pdf/
│ └── {Author}/
│ └── {Titel}.pdf
├── comics/
│ └── {Author of Serienaam}/
│ └── {001 - Titel}.cbr
└── covers/
```
Naamgeving-regels:
- Ongeldige tekens weg: `< > : " / \\ | ? *` en control chars
- Max 80 tekens per map-segment, 140 voor bestandsnaam
- Bij conflict: `Titel (2).epub`, `Titel (3).epub`, enz.
Hernoemen na metadata-bewerking:
- Bestand verplaatsen op schijf
- DB-verwijzingen updaten: `library`, `book_tags`, `reading_progress`, `reading_sessions`, `library_cover_cache`
- Lege mappen opruimen
---
## 5. Database schema
### 5.1 `library`
```sql
CREATE TABLE library (
id SERIAL PRIMARY KEY,
filename VARCHAR(600) UNIQUE NOT NULL,
media_type VARCHAR(10) NOT NULL DEFAULT 'epub',
title VARCHAR(500),
author VARCHAR(255),
publisher VARCHAR(255),
series VARCHAR(500),
series_index INTEGER DEFAULT 0,
publication_status VARCHAR(100),
has_cover BOOLEAN DEFAULT FALSE,
description TEXT DEFAULT '',
source_url VARCHAR(1000),
publish_date DATE,
archived BOOLEAN DEFAULT FALSE,
want_to_read BOOLEAN DEFAULT FALSE,
needs_review BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
```
### 5.2 `book_tags`
```sql
CREATE TABLE book_tags (
id SERIAL PRIMARY KEY,
filename VARCHAR(600) NOT NULL REFERENCES library(filename) ON DELETE CASCADE,
tag VARCHAR(255) NOT NULL,
tag_type VARCHAR(20) NOT NULL,
UNIQUE (filename, tag, tag_type)
);
CREATE INDEX idx_book_tags_filename ON book_tags (filename);
```
`tag_type`:
- `genre`
- `subgenre`
- `tag`
- `subject`
### 5.3 `reading_progress`
```sql
CREATE TABLE reading_progress (
id SERIAL PRIMARY KEY,
filename VARCHAR(600) UNIQUE NOT NULL REFERENCES library(filename) ON DELETE CASCADE,
cfi TEXT,
page INTEGER,
progress INTEGER DEFAULT 0,
updated_at TIMESTAMP DEFAULT NOW()
);
```
### 5.4 `reading_sessions`
```sql
CREATE TABLE reading_sessions (
id SERIAL PRIMARY KEY,
filename VARCHAR(600) NOT NULL REFERENCES library(filename) ON DELETE CASCADE,
read_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_reading_sessions_filename ON reading_sessions (filename);
```
### 5.5 `library_cover_cache`
```sql
CREATE TABLE library_cover_cache (
filename VARCHAR(600) PRIMARY KEY REFERENCES library(filename) ON DELETE CASCADE,
mime_type VARCHAR(100) NOT NULL,
thumb_webp BYTEA NOT NULL,
updated_at TIMESTAMP DEFAULT NOW()
);
```
### 5.6 `credentials`
```sql
CREATE TABLE credentials (
id SERIAL PRIMARY KEY,
site VARCHAR(255) UNIQUE NOT NULL,
username VARCHAR(255) NOT NULL,
password VARCHAR(255) NOT NULL,
updated_at TIMESTAMP DEFAULT NOW()
);
```
### 5.7 `break_patterns`
```sql
CREATE TABLE break_patterns (
id SERIAL PRIMARY KEY,
pattern_type VARCHAR(20) NOT NULL,
pattern TEXT NOT NULL,
enabled BOOLEAN DEFAULT TRUE,
is_default BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE (pattern_type, pattern)
);
```
### 5.8 `backup_log`
```sql
CREATE TABLE backup_log (
id SERIAL PRIMARY KEY,
status VARCHAR(20) NOT NULL,
files_count INTEGER,
size_bytes BIGINT,
error_msg TEXT,
started_at TIMESTAMP DEFAULT NOW(),
finished_at TIMESTAMP
);
```
---
## 6. Schrijfprincipe: bestand en database synchroon
Volgorde per bewerking:
1. Bewerk bestand op schijf
2. Update database
3. Retourneer succes
Nooit alleen DB updaten zonder bestand.
---
## 7. Coverstrategie
Opslaan:
- EPUB cover in bestand (`OEBPS/Images/cover.{ext}`)
- Thumbnail als `300x450` WebP in `library_cover_cache`
Ontbrekende cover:
- Als geen cover: voeg tag `Cover Missing` toe
- UI upload schrijft cover in EPUB en cache
Opvragen:
- Primair: `/library/cover-cached/{filename}`
- Fallback: `/library/cover/{filename}`
PDF en CBR:
- PDF: eerste pagina als thumbnail
- CBR/CBZ: eerste afbeelding als thumbnail
---
## 8. Verwijder-flow
`DELETE /library/file/{filename}`:
1. Verwijder bestand
2. Prune lege mappen
3. Delete uit `library` (cascade verwijdert gerelateerde tabellen)
---
## 9. Router-overzicht
### 9.1 `routers/library.py`
- `GET /library`
- `GET /api/library`
- `POST /library/rescan`
- `POST /library/import`
- `DELETE /library/file/{filename}`
- `GET /library/cover/{filename}`
- `GET /library/cover-cached/{filename}`
- `POST /library/cover/{filename}`
- `POST /library/want-to-read/{filename}`
- `POST /library/archive/{filename}`
- `GET /home`
- `GET /api/home`
- `GET /stats`
- `GET /api/stats`
### 9.2 `routers/reader.py`
- `GET /library/read/{filename}`
- `GET /library/book/{filename}`
- `PATCH /library/book/{filename}`
- `GET /library/epub/{filename}`
- `GET /library/chapters/{filename}`
- `GET /library/chapter/{index}/{filename}`
- `GET /library/chapter-img/{path}`
- `GET /library/pdf/{filename}`
- `GET /library/cbr/{filename}/{page}`
- `GET /library/progress/{filename}`
- `POST /library/progress/{filename}`
- `DELETE /library/progress/{filename}`
- `POST /library/mark-read/{filename}`
- `GET /api/genres`
### 9.3 `routers/editor.py`
- `GET /library/editor/{filename}`
- `GET /api/edit/chapter/{index}/{filename}`
- `POST /api/edit/chapter/{index}/{filename}`
- `POST /api/edit/chapter/add/{filename}`
- `DELETE /api/edit/chapter/{index}/{filename}`
### 9.4 `routers/grabber.py`
- `GET /grabber`
- `POST /preload`
- `POST /convert`
- `GET /events/{job_id}`
- `GET /debug`
- `POST /debug/run`
- `GET /credentials`
- `POST /credentials`
- `DELETE /credentials/{site}`
### 9.5 `routers/backup.py`
- `GET /backup`
- `GET /api/backup/status`
- `POST /api/backup/run`
- `GET /api/backup/history`
### 9.6 `routers/settings.py`
- `GET /settings`
- `GET /api/break-patterns`
- `POST /api/break-patterns`
- `PATCH /api/break-patterns/{id}`
- `DELETE /api/break-patterns/{id}`
- `DELETE /api/reading-history`
---
## 10. Nieuwe modules
### 10.1 `db.py`
Gedeelde psycopg2 connection pool (`init_pool`, `get_conn`, `release_conn`).
### 10.2 `pdf.py`
PyMuPDF rendering (`pdf_render_page`), page count en cover thumb.
### 10.3 `cbr.py`
RAR/ZIP paginalijst, page extract en cover thumb.
---
## 11. Cover-flow per mediatype
| Actie | EPUB | PDF | CBR/CBZ |
|---|---|---|---|
| Cover import | Uit OPF/Images | Eerste pagina render | Eerste image uit archief |
| Thumbnail | Pillow -> WebP | PyMuPDF + Pillow -> WebP | Pillow -> WebP |
| Opslag | EPUB + cache | cache | cache |
| Cover vervangen | Ja | Nee | Nee |
| Geen cover | `Cover Missing` tag | `Cover Missing` tag | `Cover Missing` tag |
---
## 12. Database-opzet
- Start met schone v2 database
- Geen migratiepad vanuit v1 data
- `run_migrations()` op startup
- `CREATE TABLE IF NOT EXISTS` overal idempotent
---
## 13. Docker stack
Zie [`stack/stack.yml`](../stack/stack.yml).
Belangrijk:
- App container expose `8099 -> 8000`
- PostgreSQL 16
- Adminer op `8098`
- `NOVELA_MASTER_KEY` in `stack/novela.env` en doorgifte in `stack/stack.yml` voor encrypted credentials
---
## 14. Requirements
Zie [`containers/novela/requirements.txt`](../containers/novela/requirements.txt).
---
## 15. Bestanden klaarzetten
Bron: `/docker/develop/story-grabber/containers/story-grabber`.
Doel: `/docker/develop/novela/containers/novela`.
Overnemen:
- `epub.py`
- `xhtml.py`
- `scrapers/*`
- `static/*`
- `templates/*`
Nieuw schrijven:
- `main.py`, `db.py`, `pdf.py`, `cbr.py`, `migrations.py`
- `routers/*`
---
## 16. Bouw-volgorde
1. `db.py`
2. `migrations.py`
3. `main.py`
4. `routers/library.py`
5. `routers/reader.py`
6. `routers/editor.py`
7. `routers/grabber.py`
8. `routers/settings.py`
9. `pdf.py` + reader uitbreiding
10. `cbr.py` + reader uitbreiding
11. `routers/backup.py`
12. `routers/library.py` uitbreiden voor pdf/cbr import
Reference in New Issue View Git Blame Copy Permalink