clearview/docs/changelog-develop.md

# Changelog - Develop

This file documents changes on the develop branch of this project.

## [2026-04-13]

### Added
- **Site filter in Job Details** — dropdown in the Selected Job Details panel to filter Targets and Deviations tables by site URL (client-side, no extra API call).
- **Excel export** — `GET /api/scan-jobs/{id}/export` endpoint (optional `?site_url=` filter) returns a `.xlsx` file with two sheets:
  - _Targets_: URL, status, attempts, error, timestamps.
  - _Deviations_: Site URL, relative Object URL, Object Type, Principal, Link Risk (colour-coded), Resolved Members, Role, Delta — sorted by Site URL → Object URL → Principal.
- **Hierarchical deduplication** — after scanning a target, deviations are post-processed to suppress child-level entries already covered by a parent (library/folder). Prevents result explosion on large sites with deeply inherited permissions. No additional API calls.
- **SharingLinks classification and colour coding** — SharePoint sharing-link principals are parsed and displayed with a risk badge in the Deviations table:
  - `Anonymous*` → Critical (red)
  - `Flexible` → High (orange)
  - `Organization*` → Low (blue)
  - `Direct*` → Low (green)
- **Resolve Sharing Links** — post-scan action in the Job Details panel. Fetches the actual member list of sharing-link groups via `/_api/web/sitegroups/getbyname/users`. Stored in new `permission_deviations.resolved_members` column. Anonymous links produce an empty member list (shown as `(public link)`). New endpoint: `POST /api/scan-jobs/{id}/resolve-sharing-links`.
- **Role name normalisation** — common Dutch SharePoint role names (e.g. "Volledig beheer", "Bijdragen") are translated to their English equivalents at scan time before being stored.
- **`openpyxl` dependency** added to `requirements.txt`.
- **Favicon** replaced with a dedicated icon (blue rounded square with eye/keyhole symbol) instead of the concept design SVG.

### Changed
- `SCAN_TARGET_TIMEOUT_SEC` default raised from 180 s to 3600 s (1 hour) to accommodate large sites with tens of thousands of files.
- `permission_deviations` table extended with `resolved_members TEXT` column (auto-migrated on startup).
- Object URL in the Deviations table and Excel export is now shown relative to the site URL (site URL prefix stripped).
- Principal display in the Deviations table strips the SharePoint claim prefix (e.g. `i:0#.f|membership|`) and shows only the email/name; full value visible on hover.
- Site URL in the Deviations table is abbreviated to the last path segment with full URL on hover.
- Deviations table uses `table-layout: fixed` with column widths sized to fit on a 1080p display.
- `docs/TECHNICAL.md` and `README.md` updated to reflect all new functionality.

## [2026-04-13]

### Added
- Certificate-based authentication for SharePoint app-only access:
  - Clearview generates a self-signed RSA-2048 certificate per tenant (no external CA required).
  - New endpoint `POST /api/tenants/{id}/generate-certificate` stores the private key and returns the public cert.
  - Public certificate downloadable as a `.cer` file from the UI, named after the tenant.
  - Scanner uses MSAL with certificate when available; client secret remains as fallback.
  - Resolves SharePoint error "Unsupported app only token" when using client secret authentication.
  - `TenantProfile` extended with `cert_private_key`, `cert_thumbprint`, and `cert_expires_at`.
  - Tenant table shows auth method (cert with expiry date or secret).
- Client secret is now optional when creating a tenant profile (can be omitted when a certificate will be used).
- Job deletion: `DELETE /api/scan-jobs/{id}` endpoint added (not allowed for queued or running jobs).
  - Delete button per job in the UI; cascades to targets and deviations.

### Fixed
- SharePoint REST API error when fetching list items: removed `$filter=HasUniqueRoleAssignments eq true` as SharePoint does not support this field as an OData filter. The check is now performed client-side.

## [2026-04-13]

### Added
- Multi-tenant support: Clearview now manages multiple customer tenants from a single instance.
  - New `TenantProfile` data model (`tenant_profiles` table) for storing customer credentials.
  - `ScanJob` linked to a tenant profile via `tenant_profile_id` FK.
  - API endpoints for tenant profile management: `GET/POST /api/tenants`, `DELETE /api/tenants/{id}`.
  - `GET /api/scan-jobs` supports filtering by `tenant_profile_id`.
- UI fully redesigned for multi-tenant use:
  - New **Tenants** panel with a table of configured customers, Add/Delete actions, and a Scan shortcut per tenant.
  - Onboarding flow (Connect Microsoft / manual instructions) moved into the Add Tenant form.
  - Scan form uses a tenant profile dropdown; manual credentials only shown as a fallback option.
  - Jobs table extended with a **Tenant** column and a tenant filter dropdown.
  - Hero stats now show: Tenants / Jobs / Active Jobs.
- XSS escaping added for all user-supplied data rendered in the jobs and deviations tables.

### Changed
- `TECHNICAL.md` updated with multi-tenant model documentation, tenant profile API, and redesigned onboarding flow.

## [2026-04-13]

### Added
- Initial repository structure.
- `containers/` directory added with the `clearview` starter service.
- `build-and-push.sh` added for container build and push.
- `docs/TECHNICAL.md` added.
- `docs/changelog-develop.md` added.
- `version.txt` added with initial version.
- `.last-branch` added for branch tracking in the build script.

## [2026-04-13]

### Added
- FastAPI backend integrated into the `clearview` container (single-container app runtime).
- PostgreSQL-backed scan job model (`scan_jobs`, `scan_targets`, `permission_deviations`).
- Background scan worker with queue processing, retries, and per-target timeout controls.
- API endpoints for manual URL scan creation, CSV import, job listing, and job detail retrieval.
- CSV parsing support for Microsoft Sites export format with URL normalization and de-duplication.
- Default-site skip rules for tenant root and app catalog paths.
- Frontend replaced with production-oriented scan UI:
  - Manual URL submission
  - CSV upload
  - Job status overview
  - Target-level result view
  - Deviation table view
- Stack configuration extended with scan worker runtime environment settings.

### Changed
- `containers/clearview/Dockerfile` switched from static nginx hosting to Python FastAPI runtime.

### Added
- Real SharePoint scan implementation for app-only authentication mode (`SHAREPOINT_SCAN_MODE=sharepoint_app_only`):
  - OAuth2 client credentials token acquisition via Microsoft Entra ID.
  - Site root permission baseline loading through SharePoint REST `roleassignments`.
  - Document library, folder, and file traversal with unique-permission detection (`HasUniqueRoleAssignments`).
  - Deviation persistence only for rights not present on site root (`delta_type=added`).
  - HTTP retry/backoff and throttle handling (429/503), plus list-level scan caps.
- Scanner HTTP retry/backoff and list-limit controls added in backend configuration.

### Changed
- Authentication flow updated to universal multi-tenant style:
  - Azure credentials are now supplied per scan job from the web UI/API payload.
  - No Azure tenant/client/secret dependency in stack `.env`.
- Added UI and technical documentation guidance for one-time Entra app setup and required SharePoint permission (`Sites.FullControl.All` + admin consent).

### Added
- Automated onboarding endpoint `POST /api/onboarding/create-scan-app`:
  - Creates a dedicated scan app in Entra for the connected tenant.
  - Configures SharePoint app permission `Sites.FullControl.All`.
  - Creates service principal and assigns app role consent.
  - Generates and returns a new client secret.
- Microsoft connect/admin-consent flow endpoints:
  - `GET /api/onboarding/microsoft/connect-url`
  - `GET /api/onboarding/microsoft/callback`
- UI onboarding flow updated:
  - `Connect Microsoft` button for admin consent redirect
  - Callback handling to capture tenant id
  - Automatic scan-app creation without manual bootstrap app input