clearview/docs/changelog-develop.md

Changelog - Develop

This file documents changes on the develop branch of this project.

[2026-04-28]

Changed

• Excel export sheet name + columns adapt to scan type — second sheet is now named Mailbox Permissions for mailbox jobs, Group Memberships for Entra-group jobs, Root Permissions for SharePoint-root jobs, and Deviations for the original SharePoint deviation scan. Column sets are tailored per type so headers like "Object URL" / "Link Risk" / "Delta" no longer appear on exports where they don't apply. Targets sheet first column label switches between Site URL / Mailbox / Group based on the job.

Added

• Entra Group Scan — new scan type entra_groups dedicated to enumerating Microsoft 365 / Azure AD group memberships. New scanners/entra.py resolves a target (Object ID, mail, or display name) via Microsoft Graph and stores one deviation per user with role Member or Owner (with (via group > nested-group) chain when expanded recursively). Group classification (Microsoft 365 / Security / Mail-enabled Security / Distribution) is stored in permission_type. New helper entra.list_all_groups for the "All groups in tenant" option. New CSV parser parse_entra_groups_csv reads the Object ID column from the Entra portal Groups export. New sidebar route #/scan/entra with three forms (manual IDs, CSV import, all-tenant). New filter option in the Scan Jobs type dropdown. Job Details renders Group / Group Type / User / Role columns for these jobs. Requires Group.Read.All on Microsoft Graph.
• Recursive group expansion via Microsoft Graph — when a SharePoint group member is itself a Microsoft 365 / Azure AD group, the resolver now expands it transitively. New helpers _expand_aad_group_via_graph and _graph_collect in scanners/sharepoint.py call /groups?$filter=mail eq … to look up the group, then /groups/{id}/members and /groups/{id}/owners to enumerate users. Owners are tagged with (owner) in the output. Recursion is depth-limited to 3 with a per-resolve seen set to break cycles. Output format puts nested members in square brackets after the group name, e.g. Pharmacology@contoso.onmicrosoft.com [alice@contoso.com, bob@contoso.com (owner)]. Requires the new Group.Read.All Application permission on Microsoft Graph (added to the onboarding instructions). Without it, group lines remain collapsed and labelled (group, no readable members).
• Resolve SharePoint groups — new "Resolve groups" action on the Job Details panel for SharePoint and SharePoint-root jobs. Expands every SharePoint group principal (Owners / Members / Visitors / custom site groups) to its underlying user list via /_api/web/sitegroups/getbyname/<group>/users and writes the comma-separated members to permission_deviations.resolved_members. Members are rendered below the principal in the Deviations table and included in the Excel export. Azure AD security groups and federated claims (principals starting with c:0… / i:0… or containing |) are skipped — those would need Group.Read.All on Microsoft Graph. New endpoint POST /api/scan-jobs/{id}/resolve-groups, helper sharepoint.is_sharepoint_group_principal().
• SharePoint root-permissions scan mode — new scan_type='sharepoint_root' that lists role assignments on the site root only, without traversing libraries/folders/files. Much faster (~1 HTTP call per target) and useful for an inventory of who has site-level access. New scanner function sharepoint.scan_site_root_permissions. Records are stored with delta_type='root' and object_type='Site'. Selectable on the New SharePoint Scan page via a "Scan mode" dropdown that controls both the manual-URL and CSV-import forms. New filter option in the Scan Jobs type filter. Noise filter _is_noise_principal excludes SharingLinks groups, SHAREPOINT\system/NT AUTHORITY\* accounts, and "Limited Access System Group" entries — these are SharePoint plumbing surfaced at site-root by spotted-item shares and are not part of a meaningful root inventory.
• Tenant primary_domain field — new column on tenant_profiles, exposed in the Add Tenant form (e.g. contoso.onmicrosoft.com). When set, the Mailbox scan page auto-fills the Organization field on tenant selection, and the API falls back to it when organization is omitted on a scan_all_mailboxes request. SharePoint scans are unaffected.
• Expanded mailbox-scan onboarding instructions — new "Enable mailbox scanning" section in the Add Tenant form covers adding the Exchange.ManageAsApp API permission, granting admin consent, assigning the Exchange Administrator Entra role to the service principal, certificate generation/upload, and primary-domain entry. Always visible (independent of automated/manual onboarding mode).
• Scan all mailboxes in a tenant — third option on the Mailbox scan page next to manual UPNs and CSV import. Clearview enumerates every mailbox via Get-EXOMailbox -ResultSize Unlimited and queues one target per mailbox. Requires the tenant's primary domain (e.g. contoso.onmicrosoft.com) and a tenant certificate. New PowerShell script exo_scripts/list-mailboxes.ps1, new Python helper mailbox.list_mailboxes(), new request fields scan_all_mailboxes and organization. Job source type is recorded as tenant_all.

Changed

• Sidebar logo — replaced with a dark-background variant (assets/clearview-logo-dark.svg) so the "view" wordmark stays legible on the dark sidebar (previously rendered in #141413 and was invisible).
• English-only UI — replaced remaining Dutch labels in the application with English equivalents: probe status Nog niet getest/Mislukt → Not tested yet/Failed, button label Testen… → Testing…, error toast Test mislukt: → Test failed:, and probe hints in scanners/sharepoint.py + scanners/mailbox.py. The Dutch→English role-name mapping table in sharepoint.py is unchanged (it normalizes incoming SharePoint role names).
• Mailbox permission scanning — Clearview can now scan Exchange Online mailboxes for delegated access alongside SharePoint sites.
  • Permission categories collected: Full Access (Get-MailboxPermission), Send As (Get-RecipientPermission), Send on Behalf (GrantSendOnBehalfTo mailbox property), and folder delegations on Calendar and Inbox (Get-MailboxFolderPermission).
  • Implementation: pwsh subprocess invoking the ExchangeOnlineManagement module with certificate-based app-only authentication (same tenant profile cert as SharePoint scans).
  • Default principals (NT AUTHORITY\SELF, S-1-5-*, folder Default/Anonymous=None) are filtered out at scan time; only non-default permissions become deviations.
  • Mailbox scans require a tenant certificate plus the Office 365 Exchange Online → Exchange.ManageAsApp API permission and the Exchange Administrator Entra role on the scan app's service principal. Client-secret auth is not supported by Exchange Online.
• Frontend sidebar layout — single-page UI replaced with a fixed left sidebar (200px, dark) and routed pages, mirroring the AlertHub layout convention.
  • Routes via hash-based router: #/dashboard, #/jobs, #/scan/sharepoint, #/scan/mailbox, #/tenants, #/settings. Implementation stays vanilla HTML/JS/CSS (no React introduction).
  • Job Details panel adapts column labels and headers based on scan_type: SharePoint shows Site/Object/Type/Principal/Role/Delta; Mailbox shows Mailbox/Object/Permission Type/Principal/Access Rights. SharingLinks resolution is hidden for mailbox jobs.
  • Jobs list gets a Type column (SharePoint / Mailbox) and a type filter.
• Scanners package — clearview_app/scanner.py split into clearview_app/scanners/{__init__.py, common.py, sharepoint.py, mailbox.py, exo_scripts/}. Public dispatcher scanners.scan(scan_type, target, auth, progress) and scanners.probe(scan_type, target, auth). The original scanner.py remains as a thin compatibility shim re-exporting the SharePoint API.
• Datamodel changes (auto-migrated on startup):
  • scan_jobs.scan_type VARCHAR(32) NOT NULL DEFAULT 'sharepoint'
  • permission_deviations.permission_type VARCHAR(32) — populated by mailbox scans (FullAccess, SendAs, SendOnBehalf, Folder:Calendar, Folder:Inbox)
  • tenant_profiles.cert_public_pem TEXT — public PEM is now stored alongside the private key so the mailbox scanner can build a .pfx for Connect-ExchangeOnline -CertificateFilePath. Existing tenants need to regenerate the certificate before mailbox scanning is available; SharePoint scans keep working with the existing key.
• Mailbox CSV import — parse_mailboxes_csv accepts UserPrincipalName / UPN / Email / Mailbox / Primary SMTP Address columns with case-insensitive matching, dedup, and email-shape validation.
• API additions:
  • POST /api/scan-jobs payload extended with scan_type and mailboxes[] next to the existing site_urls[].
  • POST /api/scan-jobs/import-csv accepts a scan_type form field (sharepoint|mailbox).
  • GET /api/scan-jobs?scan_type=… filter.
  • ScanJobSummary.scan_type and PermissionDeviationItem.permission_type returned.
• Dockerfile now installs Microsoft PowerShell 7 from the official Microsoft repository plus the ExchangeOnlineManagement PowerShell module from PSGallery. Adds ~150 MB to the image.
• Build script migration — replaced the local build-and-push.sh with the shared version from /docker/develop/shared-integrations/tooling/docker-build-and-push/. Reads the version from docs/changelog.md (release-summary file) instead of version.txt.
• docs/changelog.md — new release-summary changelog file used by the new build script. The development log (changelog-develop.md) remains the append-only source of truth for individual changes.

[2026-04-23]

Added

• Connection preflight per scan target — before a target is scanned, a lightweight probe validates that the configured credentials can reach the site and read role assignments (/_api/web + /_api/web/roleassignments?$top=1). Targets that fail preflight are marked failed with a clear reason (401/403/404 hints) instead of attempting the full scan. Fixes the previous silent-failure behaviour when admin consent or the certificate upload was missing in Azure.
• Manual "Test" button — new button in the Targets table in Job Details that re-runs the probe on demand. New endpoint: POST /api/scan-jobs/{id}/targets/{target_id}/test-connection. Blocked while the job is still queued or running.
• Probe status in UI — each target row shows the last probe result (OK / Mislukt / Nog niet getest) with timestamp and error message. Fields persist until the next test, so "last known status" remains visible even after permissions are later revoked.
• scan_targets table extended with last_probe_at, last_probe_ok, last_probe_message (auto-migrated on startup).

[2026-04-13]

Added

• Site filter in Job Details — dropdown in the Selected Job Details panel to filter Targets and Deviations tables by site URL (client-side, no extra API call).
• Excel export — GET /api/scan-jobs/{id}/export endpoint (optional ?site_url= filter) returns a .xlsx file with two sheets:
  • Targets: URL, status, attempts, error, timestamps.
  • Deviations: Site URL, relative Object URL, Object Type, Principal, Link Risk (colour-coded), Resolved Members, Role, Delta — sorted by Site URL → Object URL → Principal.
• Hierarchical deduplication — after scanning a target, deviations are post-processed to suppress child-level entries already covered by a parent (library/folder). Prevents result explosion on large sites with deeply inherited permissions. No additional API calls.
• SharingLinks classification and colour coding — SharePoint sharing-link principals are parsed and displayed with a risk badge in the Deviations table:
  • Anonymous* → Critical (red)
  • Flexible → High (orange)
  • Organization* → Low (blue)
  • Direct* → Low (green)
• Resolve Sharing Links — post-scan action in the Job Details panel. Fetches the actual member list of sharing-link groups via /_api/web/sitegroups/getbyname/users. Stored in new permission_deviations.resolved_members column. Anonymous links produce an empty member list (shown as (public link)). New endpoint: POST /api/scan-jobs/{id}/resolve-sharing-links.
• Role name normalisation — common Dutch SharePoint role names (e.g. "Volledig beheer", "Bijdragen") are translated to their English equivalents at scan time before being stored.
• openpyxl dependency added to requirements.txt.
• Favicon replaced with a dedicated icon (blue rounded square with eye/keyhole symbol) instead of the concept design SVG.

Changed

• SCAN_TARGET_TIMEOUT_SEC default raised from 180 s to 3600 s (1 hour) to accommodate large sites with tens of thousands of files.
• permission_deviations table extended with resolved_members TEXT column (auto-migrated on startup).
• Object URL in the Deviations table and Excel export is now shown relative to the site URL (site URL prefix stripped).
• Principal display in the Deviations table strips the SharePoint claim prefix (e.g. i:0#.f|membership|) and shows only the email/name; full value visible on hover.
• Site URL in the Deviations table is abbreviated to the last path segment with full URL on hover.
• Deviations table uses table-layout: fixed with column widths sized to fit on a 1080p display.
• docs/TECHNICAL.md and README.md updated to reflect all new functionality.

[2026-04-13]

Added

• Certificate-based authentication for SharePoint app-only access:
  • Clearview generates a self-signed RSA-2048 certificate per tenant (no external CA required).
  • New endpoint POST /api/tenants/{id}/generate-certificate stores the private key and returns the public cert.
  • Public certificate downloadable as a .cer file from the UI, named after the tenant.
  • Scanner uses MSAL with certificate when available; client secret remains as fallback.
  • Resolves SharePoint error "Unsupported app only token" when using client secret authentication.
  • TenantProfile extended with cert_private_key, cert_thumbprint, and cert_expires_at.
  • Tenant table shows auth method (cert with expiry date or secret).
• Client secret is now optional when creating a tenant profile (can be omitted when a certificate will be used).
• Job deletion: DELETE /api/scan-jobs/{id} endpoint added (not allowed for queued or running jobs).
  • Delete button per job in the UI; cascades to targets and deviations.

Fixed

• SharePoint REST API error when fetching list items: removed $filter=HasUniqueRoleAssignments eq true as SharePoint does not support this field as an OData filter. The check is now performed client-side.

[2026-04-13]

Added

• Multi-tenant support: Clearview now manages multiple customer tenants from a single instance.
  • New TenantProfile data model (tenant_profiles table) for storing customer credentials.
  • ScanJob linked to a tenant profile via tenant_profile_id FK.
  • API endpoints for tenant profile management: GET/POST /api/tenants, DELETE /api/tenants/{id}.
  • GET /api/scan-jobs supports filtering by tenant_profile_id.
• UI fully redesigned for multi-tenant use:
  • New Tenants panel with a table of configured customers, Add/Delete actions, and a Scan shortcut per tenant.
  • Onboarding flow (Connect Microsoft / manual instructions) moved into the Add Tenant form.
  • Scan form uses a tenant profile dropdown; manual credentials only shown as a fallback option.
  • Jobs table extended with a Tenant column and a tenant filter dropdown.
  • Hero stats now show: Tenants / Jobs / Active Jobs.
• XSS escaping added for all user-supplied data rendered in the jobs and deviations tables.

Changed

• TECHNICAL.md updated with multi-tenant model documentation, tenant profile API, and redesigned onboarding flow.

[2026-04-13]

Added

• Initial repository structure.
• containers/ directory added with the clearview starter service.
• build-and-push.sh added for container build and push.
• docs/TECHNICAL.md added.
• docs/changelog-develop.md added.
• version.txt added with initial version.
• .last-branch added for branch tracking in the build script.

[2026-04-13]

Added

• FastAPI backend integrated into the clearview container (single-container app runtime).
• PostgreSQL-backed scan job model (scan_jobs, scan_targets, permission_deviations).
• Background scan worker with queue processing, retries, and per-target timeout controls.
• API endpoints for manual URL scan creation, CSV import, job listing, and job detail retrieval.
• CSV parsing support for Microsoft Sites export format with URL normalization and de-duplication.
• Default-site skip rules for tenant root and app catalog paths.
• Frontend replaced with production-oriented scan UI:
  • Manual URL submission
  • CSV upload
  • Job status overview
  • Target-level result view
  • Deviation table view
• Stack configuration extended with scan worker runtime environment settings.

Changed

• containers/clearview/Dockerfile switched from static nginx hosting to Python FastAPI runtime.

Added

• Real SharePoint scan implementation for app-only authentication mode (SHAREPOINT_SCAN_MODE=sharepoint_app_only):
  • OAuth2 client credentials token acquisition via Microsoft Entra ID.
  • Site root permission baseline loading through SharePoint REST roleassignments.
  • Document library, folder, and file traversal with unique-permission detection (HasUniqueRoleAssignments).
  • Deviation persistence only for rights not present on site root (delta_type=added).
  • HTTP retry/backoff and throttle handling (429/503), plus list-level scan caps.
• Scanner HTTP retry/backoff and list-limit controls added in backend configuration.

Changed

• Authentication flow updated to universal multi-tenant style:
  • Azure credentials are now supplied per scan job from the web UI/API payload.
  • No Azure tenant/client/secret dependency in stack .env.
• Added UI and technical documentation guidance for one-time Entra app setup and required SharePoint permission (Sites.FullControl.All + admin consent).

Added

• Automated onboarding endpoint POST /api/onboarding/create-scan-app:
  • Creates a dedicated scan app in Entra for the connected tenant.
  • Configures SharePoint app permission Sites.FullControl.All.
  • Creates service principal and assigns app role consent.
  • Generates and returns a new client secret.
• Microsoft connect/admin-consent flow endpoints:
  • GET /api/onboarding/microsoft/connect-url
  • GET /api/onboarding/microsoft/callback
• UI onboarding flow updated:
  • Connect Microsoft button for admin consent redirect
  • Callback handling to capture tenant id
  • Automatic scan-app creation without manual bootstrap app input