Batch Status vs Step Status - Implementation Guide

Overview

This guide documents the new dual-status tracking system for IAM Employee Sync jobs. We now differentiate between:

Batch Status: High-level lifecycle of the entire sync batch
Step Status: Individual step execution state within the batch

This separation provides better observability, clearer error tracking, and improved user experience.

Status Types

Batch Status (High-Level Lifecycle)

Located in: AutoMate.API.Features.IAM.Sync.Models.BatchStatus

public static class BatchStatus
{
    public const string Pending = "Pending";           // Initial state
    public const string Running = "Running";           // Batch is executing
    public const string Completed = "Completed";       // All steps succeeded
    public const string Failed = "Failed";             // Unrecoverable error occurred
    public const string Canceled = "Canceled";         // Graceful shutdown
    public const string PartialSuccess = "PartialSuccess"; // Some failures, but continued
}

Usage:

Set once at batch start (first step)
Set once at batch end (last step)
Set on fatal/unrecoverable errors only

Step Status (Individual Step Execution)

Located in: AutoMate.API.Features.IAM.Sync.Models.StepStatus

public static class StepStatus
{
    public const string Running = "Running";
    public const string Success = "Success";
    public const string Retry = "Retry";
    public const string Failed = "Failed";
    public const string Skipped = "Skipped";
    public const string WaitingForUsers = "WaitingForUsers";
    public const string DeltaReady = "DeltaReady";
    public const string Applied = "Applied";
    public const string NothingToPatch = "NothingToPatch";
    public const string PartialFailure = "PartialFailure";
}

Format: {StepName}:{Status}

Examples:

GetHRData:Running
TranslateHRData:Success
SetInitialPassword:WaitingForUsers

Database Schema

The IAM_Employee_Sync_AuditLogs table now includes:

-- Existing column (step-level details)
[SyncStatus] NVARCHAR(255) -- e.g., "GetHRData:Running"

-- New column (batch-level lifecycle)
[BatchStatus] NVARCHAR(255) NOT NULL DEFAULT 'Pending' -- e.g., "Running"

Migration: 065_Add_BatchStatus_To_AuditLogs.cs

Implementation Patterns

Pattern 1: First Step Job (Initialize Batch)

Example: IamEmployeePreConfigurationCheckJob

private async Task InitializeAuditAsync(Guid batchId, string pipelineTag, CancellationToken ct)
{
    await syncAuditLogger.LoadBatchAsync(batchId, ct);

    syncAuditLogger.SetType("IAM-EmployeeSync");
    syncAuditLogger.SetUser("system");

    // Initialize batch status (THIS IS THE FIRST STEP)
    syncAuditLogger.SetBatchStatus(BatchStatus.Running);

    // Set step status
    syncAuditLogger.SetStatus("PreCheck:Running");

    syncAuditLogger.Append("Batch started: IAM Employee Sync");
    await syncAuditLogger.SaveAsync(ct);
}

// On success
private async Task HandleSuccessAsync(...)
{
    syncAuditLogger.SetStatus("PreCheck:Success");
    // Batch status remains Running (more steps to follow)
    await syncAuditLogger.SaveAsync(ct);

    // Schedule next steps...
}

// On failure
private async Task HandleFailureAsync(...)
{
    syncAuditLogger.SetStatus("PreCheck:Failed");
    // Mark batch as failed (unrecoverable)
    syncAuditLogger.SetBatchStatus(BatchStatus.Failed);
    await syncAuditLogger.SaveAsync(ct);
}

Key Points:

✅ Call SetBatchStatus(BatchStatus.Running) to initialize
✅ Load batch context with LoadBatchAsync()
✅ Set both batch and step status
✅ On fatal error: set BatchStatus.Failed

Pattern 2: Middle Step Job (Continue Batch)

Example: IamEmployeeGetHrDataJob, IamEmployeeReadLogsJob

private async Task InitializeAuditAsync(Guid batchGuid, string primaryBatchId, CancellationToken ct)
{
    // Load existing batch context
    await syncAuditLogger.LoadBatchAsync(batchGuid, ct);

    syncAuditLogger.SetType("IAM-EmployeeSync");
    syncAuditLogger.SetUser("system");

    // Set step status ONLY (batch status remains Running from previous step)
    syncAuditLogger.SetStatus("GetHRData:Running");

    await syncAuditLogger.SaveAsync(ct);
}

// On success
private async Task PersistHrDataAsync(...)
{
    // Step completed successfully
    syncAuditLogger.SetStatus("GetHRData:Success");
    // Batch status remains Running (more steps to follow)
    await syncAuditLogger.SaveAsync(ct);
}

// On transient error (will retry)
catch (Exception e)
{
    syncAuditLogger.SetStatus("GetHRData:Retry");
    // Batch status remains Running (will retry)
    await syncAuditLogger.SaveAsync(ct);
    throw; // Hangfire will retry
}

// On fatal error
private bool ValidateConnectors(...)
{
    if (sourceConnector == null)
    {
        syncAuditLogger.SetStatus("GetHRData:Failed");
        // Mark batch as failed (cannot proceed)
        syncAuditLogger.SetBatchStatus(BatchStatus.Failed);
        await syncAuditLogger.SaveAsync(ct);
        return false;
    }
}

Key Points:

✅ Load batch context with LoadBatchAsync()
✅ Set step status only (batch status inherited)
✅ On success: step = Success, batch = Running (continues)
✅ On retry: step = Retry, batch = Running
✅ On fatal error: step = Failed, batch = Failed

Pattern 3: Last Step Job (Finalize Batch)

Example: IamEmployeeTriggerWorkflowsJob

private async Task InitializeAuditAsync(string batchId, CancellationToken ct)
{
    if (Guid.TryParse(batchId, out var batchGuid))
    {
        await syncAuditLogger.LoadBatchAsync(batchGuid, ct);
    }

    syncAuditLogger.SetType("IAM-EmployeeSync");
    syncAuditLogger.SetUser("system");
    syncAuditLogger.SetStatus("TriggerWorkflows:Running");

    await syncAuditLogger.SaveAsync(ct);
}

// On success (FINALIZE THE BATCH)
private async Task FinalizeAuditSuccessAsync(...)
{
    // Update step status
    syncAuditLogger.SetStatus("TriggerWorkflows:Success");

    // Mark batch as Completed (THIS IS THE LAST STEP)
    syncAuditLogger.SetBatchStatus(BatchStatus.Completed);

    syncAuditLogger.Append("✓ IAM Employee Sync batch completed successfully");
    await syncAuditLogger.SaveAsync(ct);
}

// On error
private async Task FinalizeAuditFailedAsync(...)
{
    syncAuditLogger.SetStatus("TriggerWorkflows:Failed");
    syncAuditLogger.SetBatchStatus(BatchStatus.Failed);
    await syncAuditLogger.SaveAsync(ct);
}

// On cancellation
private async Task FinalizeAuditCanceledAsync(...)
{
    syncAuditLogger.SetStatus("TriggerWorkflows:Canceled");
    syncAuditLogger.SetBatchStatus(BatchStatus.Canceled);
    await syncAuditLogger.SaveAsync(ct);
}

Key Points:

✅ Call SetBatchStatus(BatchStatus.Completed) on success
✅ This finalizes the entire batch
✅ Handle cancellation with BatchStatus.Canceled
✅ Handle errors with BatchStatus.Failed

Pattern 4: Standalone Batch (No Multi-Step Pipeline)

Example: ValidationRunService

public async IAsyncEnumerable<ValidationProgressEvent> RunAsync(...)
{
    // Initialize batch
    audit.SetBatchStatus(BatchStatus.Running);
    audit.Append("Start validation run");

    try
    {
        await foreach (var e in employeeService.GetMatchesAndChangesAsync(...))
        {
            if (e.Step == "completed")
            {
                audit.SetBatchStatus(BatchStatus.Completed);
                audit.Append("Validation run completed");
            }
            else if (e.Step == "error")
            {
                audit.SetBatchStatus(BatchStatus.Failed);
            }
            else if (e.Step == "canceled")
            {
                audit.SetBatchStatus(BatchStatus.Canceled);
            }

            yield return e;
        }
    }
    finally
    {
        // Ensure status is persisted
        await audit.SaveAsync(CancellationToken.None);
    }
}

Key Points:

✅ Use batch status for entire operation
✅ No step status needed (no multi-step pipeline)
✅ Update batch status based on completion state

Decision Tree: When to Update BatchStatus

❌ DO NOT Update BatchStatus

✅ Step completed successfully → Step = Success, Batch = Running
✅ Transient error (will retry) → Step = Retry, Batch = Running
✅ Graceful skip → Step = Skipped, Batch = Running
✅ Waiting for resources → Step = WaitingForUsers, Batch = Running

✅ DO Update BatchStatus

✅ First step starting → Batch = Running
✅ Last step completed → Batch = Completed
✅ Fatal/unrecoverable error → Batch = Failed
✅ Graceful shutdown → Batch = Canceled
✅ Configuration error → Batch = Failed
✅ Missing required data → Batch = Failed

Complete IAM Employee Sync Flow

┌────────────────────────────────────────────────────────────────┐
│ Batch: IAM Employee Sync                                       │
│ BatchStatus: Pending → Running → Completed                     │
└────────────────────────────────────────────────────────────────┘

Step 1: PreConfigurationCheck
├─ Initialize: BatchStatus = Running ✨
├─ StepStatus: PreCheck:Running
├─ On Success: PreCheck:Success (batch continues)
└─ On Failure: PreCheck:Failed + BatchStatus.Failed

Step 2: GetHRData
├─ StepStatus: GetHRData:Running
├─ On Success: GetHRData:Success (batch continues)
├─ On Retry: GetHRData:Retry (batch continues)
└─ On Fatal Error: GetHRData:Failed + BatchStatus.Failed

Step 3: TranslateHRData
├─ StepStatus: TranslateHRData:Running
├─ On Success: TranslateHRData:Success (batch continues)
└─ On Fatal Error: TranslateHRData:Failed + BatchStatus.Failed

Step 4: SendToEntraProvisioning
├─ StepStatus: SendToEntraProvisioning:Running
├─ On Success: SendToEntraProvisioning:Success (batch continues)
└─ On Error: Retry or Failed + BatchStatus.Failed

Step 5: ReadLogs
├─ StepStatus: ReadLogs:Running
├─ On Success: ReadLogs:Success (batch continues)
└─ On Error: ReadLogs:Failed + BatchStatus.Failed

Step 6: SetInitialPassword
├─ StepStatus: SetInitialPassword:Running
├─ On Success: SetInitialPassword:Success (batch continues)
├─ On Waiting: SetInitialPassword:WaitingForUsers (retry)
└─ On Error: SetInitialPassword:Failed + BatchStatus.Failed

Step 7: UpdateFields
├─ StepStatus: UpdateFields:Running
├─ On Success: UpdateFields:Success (batch continues)
├─ On Skip: UpdateFields:Skipped (batch continues)
└─ On Error: UpdateFields:Failed + BatchStatus.Failed

Step 8: TriggerWorkflows (FINAL)
├─ StepStatus: TriggerWorkflows:Running
├─ On Success: TriggerWorkflows:Success + BatchStatus.Completed ✨
├─ On Cancel: TriggerWorkflows:Canceled + BatchStatus.Canceled
└─ On Error: TriggerWorkflows:Failed + BatchStatus.Failed

Updated Interfaces and Services

ISyncAuditLogger

public interface ISyncAuditLogger
{
    void Append(string message);
    void SetUser(string user);
    void SetType(string type);
    void SetStatus(string status);           // Step-level status
    void SetBatchStatus(string batchStatus); // ← NEW: Batch-level status
    void SetOperationId(string operationId);
    void SetBatchId(Guid? batchId);
    Task LoadBatchAsync(Guid batchId, CancellationToken cancellationToken = default);
    Task FinalizeAsync(CancellationToken cancellationToken = default);
    Task SaveAsync(CancellationToken cancellationToken = default);
}

ISyncAuditLogWriter

public interface ISyncAuditLogWriter
{
    // Batch-level status methods
    Task BeginBatchAsync(string batchType, ISyncAuditLogger audit, CancellationToken ct);
    Task EndBatchSuccessAsync(ISyncAuditLogger audit, CancellationToken ct);
    Task EndBatchPartialSuccessAsync(ISyncAuditLogger audit, CancellationToken ct);
    Task EndBatchFailedAsync(ISyncAuditLogger audit, CancellationToken ct);
    Task EndBatchCanceledAsync(ISyncAuditLogger audit, CancellationToken ct);

    // Step-level status methods (format: StepName:Status)
    Task BeginStepAsync(string stepName, string? pipelineTag, ISyncAuditLogger audit, CancellationToken ct);
    Task EndStepSuccessAsync(string stepName, ISyncAuditLogger audit, CancellationToken ct);
    Task EndStepRetryAsync(string stepName, ISyncAuditLogger audit, CancellationToken ct);
    Task EndStepFailedAsync(string stepName, ISyncAuditLogger audit, CancellationToken ct);

    // ... other logging methods
}

Common Mistakes to Avoid

❌ Don't set BatchStatus on every step

// WRONG - Don't do this in middle steps
private async Task HandleSuccess(...)
{
    syncAuditLogger.SetStatus("GetHRData:Success");
    syncAuditLogger.SetBatchStatus(BatchStatus.Running); // ❌ UNNECESSARY
    await syncAuditLogger.SaveAsync(ct);
}

// CORRECT
private async Task HandleSuccess(...)
{
    syncAuditLogger.SetStatus("GetHRData:Success");
    // Batch status inherited from previous step ✅
    await syncAuditLogger.SaveAsync(ct);
}

❌ Don't forget to initialize batch on first step

// WRONG - First step without batch initialization
private async Task InitializeAuditAsync(...)
{
    syncAuditLogger.SetStatus("PreCheck:Running"); // ❌ Batch not initialized
    await syncAuditLogger.SaveAsync(ct);
}

// CORRECT
private async Task InitializeAuditAsync(...)
{
    syncAuditLogger.SetBatchStatus(BatchStatus.Running); // ✅ Initialize batch
    syncAuditLogger.SetStatus("PreCheck:Running");
    await syncAuditLogger.SaveAsync(ct);
}

❌ Don't forget to finalize batch on last step

// WRONG - Last step without batch finalization
private async Task FinalizeSuccess(...)
{
    syncAuditLogger.SetStatus("TriggerWorkflows:Success"); // ❌ Batch not finalized
    await syncAuditLogger.SaveAsync(ct);
}

// CORRECT
private async Task FinalizeSuccess(...)
{
    syncAuditLogger.SetStatus("TriggerWorkflows:Success");
    syncAuditLogger.SetBatchStatus(BatchStatus.Completed); // ✅ Finalize batch
    await syncAuditLogger.SaveAsync(ct);
}

❌ Don't use old status format

// WRONG - Old format
syncAuditLogger.SetStatus("Running:GetHRData-Running"); // ❌ Old format
syncAuditLogger.SetStatus("Finished:GetHRData-OK");     // ❌ Old format

// CORRECT - New format
syncAuditLogger.SetStatus("GetHRData:Running");         // ✅ New format
syncAuditLogger.SetStatus("GetHRData:Success");         // ✅ New format

Querying the Database

Get all running batches

SELECT * FROM IAM_Employee_Sync_AuditLogs
WHERE BatchStatus = 'Running'
  AND IsFinalized = 0
ORDER BY SyncTimestamp DESC;

Get batches that failed

SELECT * FROM IAM_Employee_Sync_AuditLogs
WHERE BatchStatus = 'Failed'
ORDER BY SyncTimestamp DESC;

Get steps needing retry

SELECT * FROM IAM_Employee_Sync_AuditLogs
WHERE SyncStatus LIKE '%:Retry'
  AND IsFinalized = 0
ORDER BY SyncTimestamp DESC;

Get current step of a running batch

SELECT TOP 1
    BatchId,
    SyncStatus,  -- e.g., "GetHRData:Running"
    BatchStatus, -- e.g., "Running"
    SyncTimestamp
FROM IAM_Employee_Sync_AuditLogs
WHERE BatchId = @BatchId
ORDER BY SyncTimestamp DESC;

Frontend Integration

Display Batch Progress

interface SyncAuditLog {
  batchId: string;
  batchStatus: 'Pending' | 'Running' | 'Completed' | 'Failed' | 'Canceled' | 'PartialSuccess';
  syncStatus: string;  // e.g., "GetHRData:Running"
  syncTimestamp: Date;
  logContent: string;
}

// Parse step from syncStatus
function getCurrentStep(syncStatus: string): { step: string, status: string } {
  const [step, status] = syncStatus.split(':');
  return { step, status };
}

// Example usage
const log = await fetchLatestAuditLog(batchId);
const { step, status } = getCurrentStep(log.syncStatus);

console.log(`Batch: ${log.batchStatus}`);     // "Running"
console.log(`Current Step: ${step}`);          // "GetHRData"
console.log(`Step Status: ${status}`);         // "Running"

Display High-Level Status

// Top-level batch indicator
if (batchStatus === 'Running') {
  showSpinner();
  showMessage(`Processing... (${step})`);
} else if (batchStatus === 'Completed') {
  showSuccess('Sync completed successfully');
} else if (batchStatus === 'Failed') {
  showError(`Sync failed at step: ${step}`);
}

Migration Strategy

For Existing Data

Run a data migration to populate BatchStatus based on SyncStatus:

-- Set BatchStatus based on old SyncStatus format
UPDATE IAM_Employee_Sync_AuditLogs
SET BatchStatus =
  CASE
    WHEN SyncStatus LIKE 'Finished:OK' THEN 'Completed'
    WHEN SyncStatus LIKE 'Finished:FAILED' THEN 'Failed'
    WHEN SyncStatus LIKE 'Finished:Canceled' THEN 'Canceled'
    WHEN SyncStatus LIKE 'Running:' THEN 'Running'
    ELSE 'Pending'
  END
WHERE BatchStatus = 'Pending';

For New Jobs

Follow the patterns in this guide when creating new jobs or updating existing ones.

Benefits of This Approach

✅ Clear Separation: Batch lifecycle vs step execution ✅ Better Observability: Query batches vs steps independently ✅ Improved UX: Show high-level progress + detailed step info ✅ Easier Debugging: Know exactly where failures occur ✅ Retry Logic: Distinguish transient vs fatal errors ✅ Consistent Pattern: All jobs follow same structure ✅ Backward Compatible: Old code continues to work during migration

Reference: Updated Jobs

All jobs in the IAM Employee Sync pipeline have been updated:

✅ IamEmployeePreConfigurationCheckJob (first step)
✅ IamEmployeeGetHrDataJob (middle step)
✅ IamEmployeeTranslateHrDataJob (middle step)
✅ IamEmployeeSendToEntraProvisioningJob (middle step)
✅ IamEmployeeReadLogsJob (middle step)
✅ IamEmployeeSetInitialPasswordJob (middle step)
✅ IamEmployeeUpdateFieldsJob (middle step)
✅ IamEmployeeTriggerWorkflowsJob (last step)
✅ ValidationRunService (standalone)

Questions or Issues?

If you have questions about implementing batch status tracking:

Review the patterns in this guide
Look at existing updated jobs as examples
Check the SyncAuditLogger.cs implementation
Reach out to the team for clarification

Last Updated: February 9, 2026 Migration: 065_Add_BatchStatus_To_AuditLogs.cs

Last modified: 23 March 2026