{% extends "documentation/base.html" %}

{% block doc_content %}
<h1>Auto-Import Configuration</h1>

<p class="lead">
  Configure automatic email import from your Microsoft Graph mailbox to run on a schedule.
</p>

<h2>Overview</h2>

<p>BackupChecks can automatically import backup report emails from your mailbox at regular intervals. This eliminates the need to manually trigger imports and ensures new backup reports are processed promptly.</p>

<p>Auto-import features:</p>

<ul>
  <li><strong>Scheduled imports:</strong> Automatically fetch new emails at configured intervals</li>
  <li><strong>Auto-approval:</strong> Emails matching existing jobs are automatically approved</li>
  <li><strong>Background processing:</strong> Runs in a background thread without blocking the application</li>
  <li><strong>Error handling:</strong> Logs failures and retries on the next interval</li>
  <li><strong>Manual override:</strong> Trigger imports manually when needed</li>
</ul>

<h2>Automatic Import</h2>

<p>Automatic import runs continuously in the background and fetches new emails based on a configured interval.</p>

<h3>Enabling Auto-Import</h3>

<p>To enable automatic import:</p>

<ol>
  <li>Navigate to <strong>Settings</strong> → <strong>Imports</strong> tab</li>
  <li>Locate the <strong>Automatic mail import</strong> section</li>
  <li>Check the <strong>Enable automatic import</strong> checkbox</li>
  <li>Set the <strong>Import interval</strong> in minutes (default: 15 minutes)</li>
  <li>Click <strong>Save settings</strong> at the bottom of the page</li>
</ol>

<div class="doc-callout doc-callout-info">
  <strong>💡 First Run:</strong><br>
  After enabling auto-import, the first import runs immediately. Subsequent imports run at the configured interval.
</div>

<h3>Import Interval</h3>

<p>The import interval determines how often BackupChecks checks for new emails:</p>

<table>
  <thead>
    <tr>
      <th>Interval</th>
      <th>Use Case</th>
      <th>Recommendation</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>5 minutes</td>
      <td>High-frequency backups, real-time monitoring</td>
      <td>May increase server load, use only if necessary</td>
    </tr>
    <tr>
      <td>15 minutes</td>
      <td>Standard setup (default)</td>
      <td><strong>Recommended</strong> for most organizations</td>
    </tr>
    <tr>
      <td>30 minutes</td>
      <td>Low-priority backups, small teams</td>
      <td>Good balance between timeliness and load</td>
    </tr>
    <tr>
      <td>60 minutes</td>
      <td>Batch processing, overnight backups only</td>
      <td>Suitable for organizations with scheduled backup windows</td>
    </tr>
  </tbody>
</table>

<div class="doc-callout doc-callout-tip">
  <strong>💡 Performance Consideration:</strong><br>
  Shorter intervals provide faster email processing but increase server load and Microsoft Graph API usage. 15 minutes is recommended as a good balance for most organizations.
</div>

<h3>Batch Size</h3>

<p>Auto-import fetches a fixed number of emails per run to prevent overwhelming the system:</p>

<ul>
  <li><strong>Automatic import:</strong> Always fetches <strong>10 messages</strong> per run</li>
  <li>If more than 10 new emails exist, they will be fetched on subsequent runs</li>
  <li>This prevents long-running imports that could block other operations</li>
</ul>

<h3>How Auto-Import Works</h3>

<p>Here's what happens during each automatic import cycle:</p>

<ol>
  <li><strong>Timer triggers:</strong> The configured interval elapses (e.g., 15 minutes)</li>
  <li><strong>Check settings:</strong> Verify that auto-import is still enabled</li>
  <li><strong>Authenticate:</strong> Obtain access token from Microsoft Graph</li>
  <li><strong>Fetch emails:</strong> Retrieve up to 10 new emails from the incoming folder</li>
  <li><strong>Parse emails:</strong> Run parsers to extract backup information</li>
  <li><strong>Auto-approve:</strong> If the email matches an existing approved job, automatically approve it</li>
  <li><strong>Move emails:</strong> Move processed emails to the processed folder (if configured)</li>
  <li><strong>Log results:</strong> Record import statistics in the audit log</li>
  <li><strong>Persist objects:</strong> Store backup objects for auto-approved runs</li>
  <li><strong>Wait:</strong> Sleep until the next interval</li>
</ol>

<h3>Auto-Approval</h3>

<p>During automatic import, BackupChecks can automatically approve emails that match previously approved jobs. This eliminates the need to manually approve recurring backup reports.</p>

<h4>How Auto-Approval Works</h4>

<p>An email is automatically approved if:</p>

<ol>
  <li>The email is <strong>successfully parsed</strong> (backup software, type, and job name extracted)</li>
  <li>A <strong>matching job</strong> already exists in the database (based on sender, job name, backup software)</li>
  <li>The job was <strong>previously approved</strong> by a user</li>
  <li>The match is <strong>unique</strong> across all customers (no ambiguity)</li>
</ol>

<p>When auto-approved:</p>

<ul>
  <li>A new <code>JobRun</code> is created and linked to the existing job</li>
  <li>Backup objects are extracted and stored</li>
  <li>The email is marked as approved and moved out of the inbox</li>
  <li>The run appears in Daily Jobs, Run Checks, and job history</li>
</ul>

<div class="doc-callout doc-callout-info">
  <strong>💡 First Approval is Manual:</strong><br>
  The <strong>first email</strong> for a new backup job must always be manually approved from the inbox. After that, future emails for the same job are automatically approved.
</div>

<h4>When Auto-Approval Fails</h4>

<p>If auto-approval fails, the email remains in the inbox and requires manual approval:</p>

<ul>
  <li><strong>Parsing failed:</strong> Email format not recognized by any parser</li>
  <li><strong>No matching job:</strong> This is the first email for this backup job</li>
  <li><strong>Ambiguous match:</strong> Multiple jobs match the same criteria (different customers with identical job names)</li>
  <li><strong>Job renamed:</strong> The backup software renamed the job, breaking the match</li>
</ul>

<h2>Manual Import</h2>

<p>You can manually trigger an import at any time without waiting for the automatic import interval.</p>

<h3>Triggering Manual Import</h3>

<ol>
  <li>Navigate to <strong>Settings</strong> → <strong>Imports</strong> tab</li>
  <li>Scroll to the <strong>Manual mail import</strong> section</li>
  <li>Optionally adjust the <strong>Number of items</strong> (default: 50)</li>
  <li>Click the <strong>Import now</strong> button</li>
  <li>BackupChecks will immediately fetch emails from the mailbox</li>
  <li>You'll see a success message showing how many emails were imported</li>
</ol>

<h3>Manual Import Batch Size</h3>

<p>Manual import allows you to configure how many emails to fetch:</p>

<ul>
  <li><strong>Default:</strong> 50 emails per manual import</li>
  <li><strong>Range:</strong> 1 to 50 emails</li>
  <li><strong>Use case:</strong> Fetch a large batch when setting up a new customer or after a long period without imports</li>
</ul>

<p>The default batch size can be changed in <strong>Settings</strong> → <strong>Imports</strong> → <strong>Manual import batch size</strong>.</p>

<div class="doc-callout doc-callout-tip">
  <strong>💡 When to Use Manual Import:</strong><br>
  Manual import is useful when:
  <ul>
    <li>Testing mail configuration for the first time</li>
    <li>Setting up a new customer and you need their emails imported immediately</li>
    <li>Auto-import is disabled but you need to import specific emails</li>
    <li>You want to fetch a larger batch than the automatic 10-message limit</li>
  </ul>
</div>

<h2>Import Settings Summary</h2>

<p>Here's a complete reference of all mail import settings:</p>

<table>
  <thead>
    <tr>
      <th>Setting</th>
      <th>Location</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Enable automatic import</strong></td>
      <td>Settings → Imports</td>
      <td>Disabled</td>
      <td>Turn automatic email import on or off</td>
    </tr>
    <tr>
      <td><strong>Import interval</strong></td>
      <td>Settings → Imports</td>
      <td>15 minutes</td>
      <td>How often to check for new emails (automatic import only)</td>
    </tr>
    <tr>
      <td><strong>Manual import batch size</strong></td>
      <td>Settings → Imports</td>
      <td>50 messages</td>
      <td>Default number of emails to fetch during manual import</td>
    </tr>
    <tr>
      <td><strong>EML retention period</strong></td>
      <td>Settings → Imports</td>
      <td>7 days</td>
      <td>How long to store raw .eml files (0, 7, or 14 days)</td>
    </tr>
  </tbody>
</table>

<h2>Monitoring Import Activity</h2>

<p>You can monitor import activity through several interfaces:</p>

<h3>Audit Log</h3>

<p>Every import operation is logged in the audit log:</p>

<ol>
  <li>Navigate to <strong>Settings</strong> → <strong>Maintenance</strong> → <strong>Admin events</strong></li>
  <li>Look for events like:
    <ul>
      <li><code>mail_import_auto</code> - Automatic import completed</li>
      <li><code>mail_import_manual</code> - Manual import completed</li>
      <li><code>mail_import_auto_error</code> - Automatic import failed</li>
    </ul>
  </li>
  <li>Event messages show statistics: <code>fetched=10, new=3, auto_approved=7, errors=0</code></li>
</ol>

<h3>Inbox Page</h3>

<p>The inbox shows all imported emails:</p>

<ul>
  <li>Check the <strong>Parsed</strong> column to see when emails were imported and parsed</li>
  <li>Emails with blank <strong>Backup</strong> or <strong>Job name</strong> were not successfully parsed</li>
  <li>Approved emails move out of the inbox automatically</li>
</ul>

<h3>Jobs and Daily Jobs</h3>

<p>Auto-approved emails appear as new runs in:</p>

<ul>
  <li><strong>Jobs</strong> page - Shows all backup jobs with their latest run status</li>
  <li><strong>Daily Jobs</strong> page - Shows backup runs for today, grouped by customer</li>
  <li><strong>Run Checks</strong> page - Shows all runs awaiting review</li>
</ul>

<h2>Troubleshooting</h2>

<h3>Auto-Import Not Running</h3>

<p>If auto-import is enabled but emails are not being imported:</p>

<ol>
  <li>Verify that <strong>Enable automatic import</strong> is checked in Settings → Imports</li>
  <li>Check the audit log for <code>mail_import_auto_error</code> events</li>
  <li>Verify Microsoft Graph credentials are still valid (client secret may have expired)</li>
  <li>Ensure the BackupChecks application server is running (the background thread starts on app startup)</li>
  <li>Check system logs for thread errors or crashes</li>
</ol>

<h3>Emails Not Auto-Approved</h3>

<p>If emails are imported but not auto-approved:</p>

<ol>
  <li>Check the <strong>Inbox</strong> - emails may be parsed but not matched to existing jobs</li>
  <li>Verify the <strong>first email</strong> for this job was manually approved (auto-approval only works for subsequent emails)</li>
  <li>Ensure the job name, backup software, and sender match exactly between emails</li>
  <li>Check for duplicate jobs across different customers (auto-approval requires a unique match)</li>
</ol>

<h3>Import Errors</h3>

<p>If imports fail with errors:</p>

<ul>
  <li><strong>Authentication error:</strong> Verify Microsoft Graph credentials (tenant ID, client ID, client secret)</li>
  <li><strong>Folder not found:</strong> Check that incoming/processed folder paths are correct</li>
  <li><strong>Timeout error:</strong> Network issues or Microsoft Graph API unavailability - will retry on next interval</li>
  <li><strong>Disk space error:</strong> Free disk space below 2 GB - mail import is blocked until space is freed</li>
</ul>

<h2>Best Practices</h2>

<ul>
  <li><strong>Enable auto-import:</strong> Use automatic import for hands-off email processing</li>
  <li><strong>Use 15-minute interval:</strong> Recommended for most setups (balances timeliness and load)</li>
  <li><strong>Monitor audit logs:</strong> Regularly check for import errors</li>
  <li><strong>Approve first emails manually:</strong> Always manually approve the first email for a new job to enable auto-approval</li>
  <li><strong>Keep credentials fresh:</strong> Set calendar reminders to renew Microsoft Graph client secrets before expiration</li>
  <li><strong>Test manual import first:</strong> Before enabling auto-import, test manual import to verify configuration</li>
  <li><strong>Use dedicated mailbox:</strong> Don't mix backup reports with personal email</li>
</ul>

<h2>Advanced Configuration</h2>

<h3>Disabling Auto-Import Temporarily</h3>

<p>To temporarily stop automatic imports without losing configuration:</p>

<ol>
  <li>Navigate to <strong>Settings</strong> → <strong>Imports</strong></li>
  <li>Uncheck <strong>Enable automatic import</strong></li>
  <li>Click <strong>Save settings</strong></li>
  <li>The background thread will stop fetching emails but settings are preserved</li>
  <li>Re-enable later by checking the box again</li>
</ol>

<h3>EML Storage Retention</h3>

<p>Control how long raw .eml files are stored in the database:</p>

<ul>
  <li><strong>0 days:</strong> Don't store .eml files (saves database space, but troubleshooting is harder)</li>
  <li><strong>7 days:</strong> Default - good balance between troubleshooting capability and storage</li>
  <li><strong>14 days:</strong> Extended retention for environments with complex parsing issues</li>
</ul>

<p>Older .eml files are automatically purged based on this retention period.</p>

<h3>High-Volume Environments</h3>

<p>For organizations with hundreds of backup jobs:</p>

<ul>
  <li>Consider <strong>5-minute intervals</strong> to process emails more quickly</li>
  <li>Ensure server resources (CPU, RAM, disk) are sufficient for frequent imports</li>
  <li>Monitor database size growth - consider reducing EML retention to 0 days</li>
  <li>Use multiple BackupChecks instances with separate mailboxes to distribute load</li>
</ul>

<h2>Next Steps</h2>

<ul>
  <li><a href="{{ url_for('documentation.page', section='mail-import', page='setup') }}">Mail Import Setup</a> - Configure Microsoft Graph API integration</li>
  <li><a href="{{ url_for('documentation.page', section='mail-import', page='inbox-management') }}">Inbox Management</a> - Review and approve imported emails</li>
  <li><a href="{{ url_for('documentation.page', section='backup-review', page='daily-jobs') }}">Daily Jobs View</a> - Monitor auto-approved backup runs</li>
</ul>

{% endblock %}