30. Implementing Background Jobs

Building job classes, error handling, status tracking, logging, and testing strategies for background jobs

About this chapter

Build robust background job implementations with proper error handling, status tracking, logging, and testing strategies for reliable asynchronous processing.

Job class structure: Clear interfaces and dependency injection patterns
Error handling: Try-catch blocks, logging exceptions, and graceful failures
Status tracking: Storing job progress and outcomes in persistent storage
Logging strategy: Detailed logging for debugging and monitoring
Idempotency: Designing jobs that run safely multiple times
Testing approaches: Unit testing jobs independently from Hangfire

Learning outcomes:

Design job classes with clear interfaces and dependencies
Implement comprehensive error handling and logging
Track job status and execution results
Handle retries and failed job scenarios
Ensure jobs can be executed idempotently
Write testable job code independent of Hangfire

30.1 Job Class Fundamentals

Background jobs need structure. Unlike controllers, they don’t have HTTP context. They need:

Clear interface - what inputs they accept
Error handling - what happens if things break
Logging - how you track what happened
Idempotency - can they run multiple times safely?
Status tracking - where’s the job now?

Let’s start with a job that sends a command notification email:

// Jobs/SendCommandNotificationEmailJob.cs
using System;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using CommandAPI.Services;

namespace CommandAPI.Jobs
{
    public interface ISendCommandNotificationEmailJob
    {
        Task ExecuteAsync(int commandId, string recipientEmail, string commandTitle);
    }

    public class SendCommandNotificationEmailJob : ISendCommandNotificationEmailJob
    {
        private readonly IEmailService _emailService;
        private readonly ILogger<SendCommandNotificationEmailJob> _logger;

        public SendCommandNotificationEmailJob(
            IEmailService emailService,
            ILogger<SendCommandNotificationEmailJob> logger)
        {
            _emailService = emailService;
            _logger = logger;
        }

        public async Task ExecuteAsync(int commandId, string recipientEmail, string commandTitle)
        {
            try
            {
                _logger.LogInformation(
                    "Starting SendCommandNotificationEmailJob for Command {CommandId} to {RecipientEmail}",
                    commandId, recipientEmail);

                var subject = $"Command Created: {commandTitle}";
                var body = $@"
                    <h2>New Command Created</h2>
                    <p>Command ID: {commandId}</p>
                    <p>Title: {commandTitle}</p>
                    <p>This is an automated notification.</p>
                ";

                await _emailService.SendEmailAsync(recipientEmail, subject, body);

                _logger.LogInformation(
                    "SendCommandNotificationEmailJob completed successfully for Command {CommandId}",
                    commandId);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex,
                    "SendCommandNotificationEmailJob failed for Command {CommandId}. Exception: {Message}",
                    commandId, ex.Message);
                throw; // Rethrow so Hangfire retries
            }
        }
    }
}

Why interface? Dependency injection. Your job class gets its dependencies injected, just like controllers.

Why throw exceptions? Hangfire sees it, retries automatically based on your policy.

30.2 Enqueueing Jobs from Controllers

Controllers enqueue jobs, then return immediately:

// Controllers/CommandsController.cs
using Hangfire;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using CommandAPI.Data;
using CommandAPI.Dtos;
using CommandAPI.Jobs;
using CommandAPI.Models;
using AutoMapper;

namespace CommandAPI.Controllers
{
    [ApiController]
    [Route("api/[controller]")]
    public class CommandsController : ControllerBase
    {
        private readonly ICommandRepository _repository;
        private readonly IMapper _mapper;
        private readonly IBackgroundJobClient _backgroundJobClient;
        private readonly ILogger<CommandsController> _logger;

        public CommandsController(
            ICommandRepository repository,
            IMapper mapper,
            IBackgroundJobClient backgroundJobClient,
            ILogger<CommandsController> logger)
        {
            _repository = repository;
            _mapper = mapper;
            _backgroundJobClient = backgroundJobClient;
            _logger = logger;
        }

        [HttpPost]
        public async Task<ActionResult<CommandReadDto>> CreateCommand(CommandMutateDto dto)
        {
            var command = _mapper.Map<Command>(dto);
            _repository.CreateCommand(command);
            await _repository.SaveChangesAsync();

            // Enqueue background job - doesn't block the response
            _backgroundJobClient.Enqueue<ISendCommandNotificationEmailJob>(
                job => job.ExecuteAsync(
                    command.Id,
                    "admin@commandapi.com",
                    command.Title));

            _logger.LogInformation("Command {CommandId} created. Notification email job enqueued.", command.Id);

            var readDto = _mapper.Map<CommandReadDto>(command);
            return CreatedAtRoute(nameof(GetCommandById), new { id = command.Id }, readDto);
        }

        [HttpGet("{id}")]
        public async Task<ActionResult<CommandReadDto>> GetCommandById(int id)
        {
            var command = await _repository.GetCommandByIdAsync(id);
            if (command == null)
                return NotFound();

            var readDto = _mapper.Map<CommandReadDto>(command);
            return Ok(readDto);
        }
    }
}

Key points:

IBackgroundJobClient injected from DI container
Enqueue<T> takes a job interface type
Lambda expression passes parameters directly
Method returns immediately—job runs in background
User gets 201 Created while email processes asynchronously

30.3 Job Status Repository

Users want to know: “Is my email sent yet?” You need status tracking:

// Models/JobStatus.cs
using System;

namespace CommandAPI.Models
{
    public class JobStatus
    {
        public int Id { get; set; }
        public string JobId { get; set; } // Hangfire job ID
        public string JobType { get; set; } // e.g., "SendEmail"
        public string Status { get; set; } // "Queued", "Processing", "Completed", "Failed"
        public int? RelatedEntityId { get; set; } // CommandId, PlatformId, etc.
        public string? ErrorMessage { get; set; }
        public DateTime CreatedAt { get; set; }
        public DateTime? CompletedAt { get; set; }
        public int RetryCount { get; set; }
    }
}

// Repositories/IJobStatusRepository.cs
using System.Threading.Tasks;
using CommandAPI.Models;

namespace CommandAPI.Data
{
    public interface IJobStatusRepository
    {
        Task<JobStatus?> GetByJobIdAsync(string jobId);
        Task<JobStatus?> GetByRelatedEntityAsync(int entityId, string jobType);
        void CreateJobStatus(JobStatus status);
        void UpdateJobStatus(JobStatus status);
        Task SaveChangesAsync();
    }
}

// Repositories/PgSqlJobStatusRepository.cs
using System.Linq;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;
using CommandAPI.Data;
using CommandAPI.Models;

namespace CommandAPI.Repositories
{
    public class PgSqlJobStatusRepository : IJobStatusRepository
    {
        private readonly AppDbContext _context;

        public PgSqlJobStatusRepository(AppDbContext context)
        {
            _context = context;
        }

        public async Task<JobStatus?> GetByJobIdAsync(string jobId)
        {
            return await _context.JobStatuses
                .FirstOrDefaultAsync(j => j.JobId == jobId);
        }

        public async Task<JobStatus?> GetByRelatedEntityAsync(int entityId, string jobType)
        {
            return await _context.JobStatuses
                .Where(j => j.RelatedEntityId == entityId && j.JobType == jobType)
                .OrderByDescending(j => j.CreatedAt)
                .FirstOrDefaultAsync();
        }

        public void CreateJobStatus(JobStatus status)
        {
            _context.JobStatuses.Add(status);
        }

        public void UpdateJobStatus(JobStatus status)
        {
            _context.JobStatuses.Update(status);
        }

        public async Task SaveChangesAsync()
        {
            await _context.SaveChangesAsync();
        }
    }
}

30.4 Job with Status Tracking

Now update your job to track status:

// Jobs/SendCommandNotificationEmailJobWithTracking.cs
using System;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using CommandAPI.Services;
using CommandAPI.Data;
using CommandAPI.Models;

namespace CommandAPI.Jobs
{
    public interface ISendCommandNotificationEmailJobWithTracking
    {
        Task ExecuteAsync(int commandId, string recipientEmail, string commandTitle, string hangfireJobId);
    }

    public class SendCommandNotificationEmailJobWithTracking : ISendCommandNotificationEmailJobWithTracking
    {
        private readonly IEmailService _emailService;
        private readonly IJobStatusRepository _jobStatusRepository;
        private readonly ILogger<SendCommandNotificationEmailJobWithTracking> _logger;

        public SendCommandNotificationEmailJobWithTracking(
            IEmailService emailService,
            IJobStatusRepository jobStatusRepository,
            ILogger<SendCommandNotificationEmailJobWithTracking> logger)
        {
            _emailService = emailService;
            _jobStatusRepository = jobStatusRepository;
            _logger = logger;
        }

        public async Task ExecuteAsync(int commandId, string recipientEmail, string commandTitle, string hangfireJobId)
        {
            var jobStatus = new JobStatus
            {
                JobId = hangfireJobId,
                JobType = "SendEmail",
                Status = "Processing",
                RelatedEntityId = commandId,
                CreatedAt = DateTime.UtcNow,
                RetryCount = 0
            };

            try
            {
                _logger.LogInformation(
                    "Processing email job {JobId} for Command {CommandId}",
                    hangfireJobId, commandId);

                _jobStatusRepository.CreateJobStatus(jobStatus);
                await _jobStatusRepository.SaveChangesAsync();

                var subject = $"Command Created: {commandTitle}";
                var body = $"<h2>New Command</h2><p>ID: {commandId}</p>";

                await _emailService.SendEmailAsync(recipientEmail, subject, body);

                // Mark as completed
                jobStatus.Status = "Completed";
                jobStatus.CompletedAt = DateTime.UtcNow;
                _jobStatusRepository.UpdateJobStatus(jobStatus);
                await _jobStatusRepository.SaveChangesAsync();

                _logger.LogInformation("Email job {JobId} completed successfully", hangfireJobId);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Email job {JobId} failed: {Message}", hangfireJobId, ex.Message);

                jobStatus.Status = "Failed";
                jobStatus.ErrorMessage = ex.Message;
                jobStatus.CompletedAt = DateTime.UtcNow;
                _jobStatusRepository.UpdateJobStatus(jobStatus);
                await _jobStatusRepository.SaveChangesAsync();

                throw;
            }
        }
    }
}

30.5 Logging Best Practices (Not Console.WriteLine!)

This is critical. Never do:

// WRONG - Don't do this!
Console.WriteLine($"Job started for command {commandId}");

Why? Console output isn’t captured in production. You can’t search logs later.

// RIGHT - Use ILogger<T>
private readonly ILogger<YourJobClass> _logger;

public YourJobClass(ILogger<YourJobClass> logger)
{
    _logger = logger;
}

// In your job method
_logger.LogInformation("Job started for command {CommandId}", commandId);
_logger.LogError(ex, "Job failed: {ErrorMessage}", ex.Message);
_logger.LogWarning("Retry attempt {RetryNumber} for command {CommandId}", retryCount + 1, commandId);

Why structured logging?

Machine-readable: LogInformation with {CommandId} is parsed into properties
Searchable: Find all failures for command ID 42
Contextual: Track correlation IDs across logs
Provenance: Timestamps, severity levels, exception details

Set up serilog (industry standard):

// Program.cs
using Serilog;

var builder = WebApplication.CreateBuilder(args);

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.File("logs/commandapi-.txt", rollingInterval: RollingInterval.Day)
    .CreateLogger();

builder.Services.AddLogging(config =>
{
    config.ClearProviders();
    config.AddSerilog();
});

var app = builder.Build();
// Rest of setup...

Now job failures are logged to files and can be analyzed later.

30.6 Idempotency and Retries

Background jobs can run multiple times:

Network hiccup during send → Job fails → Hangfire retries → Might send email twice
Server crash → Job restarts → Process runs again
Manual retry → You click “Retry” in dashboard → Job runs again

Your job must be idempotent: running it 5 times produces same result as running it once.

// WRONG - Not idempotent
public async Task ExecuteAsync(int commandId)
{
    var command = await _repository.GetCommandByIdAsync(commandId);
    // Create a new log entry every time
    var logEntry = new CommandLog { CommandId = commandId, Message = "Processed" };
    _repository.CreateLog(logEntry);
    await _repository.SaveChangesAsync();
    // If job runs twice, you have two log entries. Oops.
}

// RIGHT - Idempotent
public async Task ExecuteAsync(int commandId)
{
    var command = await _repository.GetCommandByIdAsync(commandId);
    
    // Check if already processed
    var existing = await _repository.GetCommandLogAsync(commandId, "Processed");
    if (existing != null)
    {
        _logger.LogInformation("Job already processed for command {CommandId}", commandId);
        return;
    }

    // Now safe to create
    var logEntry = new CommandLog { CommandId = commandId, Message = "Processed" };
    _repository.CreateLog(logEntry);
    await _repository.SaveChangesAsync();
}

Pattern: Check before you create/mutate.

30.7 Configuring Retry Policy

Control how Hangfire retries failed jobs:

// Program.cs
using Hangfire;

builder.Services.AddHangfire(config =>
{
    config.UsePostgreSqlStorage(connectionString);
    
    // Global retry policy: 5 attempts with exponential backoff
    config.UseSerializingJobFactory();
});

builder.Services.AddHangfireServer(options =>
{
    options.ServerName = "CommandAPI-Server";
    options.WorkerCount = Environment.ProcessorCount;
});

app.UseHangfireServer();

// For specific jobs, use [AutomaticRetry]
public class SendCommandNotificationEmailJob
{
    // Retry up to 3 times, don't retry on OutOfMemoryException
    [AutomaticRetry(Attempts = 3, 
        OnAttemptsExceeded = AttemptsExceededAction.Delete)]
    public async Task ExecuteAsync(int commandId, string email, string title)
    {
        // Job implementation
    }
}

Best practices:

Fast jobs (< 5 seconds): 3-5 retries
Slow jobs (> 30 seconds): 2-3 retries
External APIs: Higher retry count (network is flaky)
Database-dependent: Lower count (if DB is down, retries won’t help)

30.8 Job Failure Alerting

Hangfire stores failed jobs, but you need to know about them:

// Services/IJobAlertService.cs
using System.Threading.Tasks;

namespace CommandAPI.Services
{
    public interface IJobAlertService
    {
        Task AlertJobFailureAsync(string jobId, string jobType, string errorMessage);
    }
}

// Services/JobAlertService.cs
using System;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;

namespace CommandAPI.Services
{
    public class JobAlertService : IJobAlertService
    {
        private readonly IEmailService _emailService;
        private readonly ILogger<JobAlertService> _logger;

        public JobAlertService(IEmailService emailService, ILogger<JobAlertService> logger)
        {
            _emailService = emailService;
            _logger = logger;
        }

        public async Task AlertJobFailureAsync(string jobId, string jobType, string errorMessage)
        {
            try
            {
                var subject = $"Background Job Failed: {jobType}";
                var body = $@"
                    <h2>Job Failure Alert</h2>
                    <p>Job Type: {jobType}</p>
                    <p>Job ID: {jobId}</p>
                    <p>Error: {errorMessage}</p>
                    <p>Check Hangfire Dashboard: https://yourapi.com/hangfire</p>
                ";

                await _emailService.SendEmailAsync("ops@commandapi.com", subject, body);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Failed to send job failure alert for job {JobId}", jobId);
            }
        }
    }
}

Hook it into your job:

public class SendCommandNotificationEmailJobWithAlert
{
    private readonly IEmailService _emailService;
    private readonly IJobStatusRepository _jobStatusRepository;
    private readonly IJobAlertService _jobAlertService;
    private readonly ILogger<SendCommandNotificationEmailJobWithAlert> _logger;

    public SendCommandNotificationEmailJobWithAlert(
        IEmailService emailService,
        IJobStatusRepository jobStatusRepository,
        IJobAlertService jobAlertService,
        ILogger<SendCommandNotificationEmailJobWithAlert> logger)
    {
        _emailService = emailService;
        _jobStatusRepository = jobStatusRepository;
        _jobAlertService = jobAlertService;
        _logger = logger;
    }

    public async Task ExecuteAsync(int commandId, string email, string title, string jobId)
    {
        try
        {
            // Job implementation
            await _emailService.SendEmailAsync(email, title, "...");
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Job {JobId} failed", jobId);
            await _jobAlertService.AlertJobFailureAsync(jobId, "SendEmail", ex.Message);
            throw;
        }
    }
}

30.9 Unit Testing Background Jobs

Jobs need tests like any other code:

// Tests/Jobs/SendCommandNotificationEmailJobTests.cs
using System;
using System.Threading.Tasks;
using Xunit;
using Moq;
using Microsoft.Extensions.Logging;
using CommandAPI.Jobs;
using CommandAPI.Services;

namespace CommandAPI.Tests.Jobs
{
    public class SendCommandNotificationEmailJobTests
    {
        private readonly Mock<IEmailService> _mockEmailService;
        private readonly Mock<ILogger<SendCommandNotificationEmailJob>> _mockLogger;
        private readonly SendCommandNotificationEmailJob _job;

        public SendCommandNotificationEmailJobTests()
        {
            _mockEmailService = new Mock<IEmailService>();
            _mockLogger = new Mock<ILogger<SendCommandNotificationEmailJob>>();
            _job = new SendCommandNotificationEmailJob(_mockEmailService.Object, _mockLogger.Object);
        }

        [Fact]
        public async Task ExecuteAsync_WithValidEmail_SendsEmail()
        {
            // Arrange
            int commandId = 1;
            string email = "test@example.com";
            string title = "Deploy Application";

            _mockEmailService
                .Setup(x => x.SendEmailAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>()))
                .Returns(Task.CompletedTask);

            // Act
            await _job.ExecuteAsync(commandId, email, title);

            // Assert
            _mockEmailService.Verify(
                x => x.SendEmailAsync(email, It.IsAny<string>(), It.IsAny<string>()),
                Times.Once);
        }

        [Fact]
        public async Task ExecuteAsync_WhenEmailServiceThrows_LogsError()
        {
            // Arrange
            var exception = new InvalidOperationException("SMTP connection failed");
            _mockEmailService
                .Setup(x => x.SendEmailAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>()))
                .ThrowsAsync(exception);

            // Act & Assert
            var ex = await Assert.ThrowsAsync<InvalidOperationException>(
                () => _job.ExecuteAsync(1, "test@example.com", "Test"));

            _mockLogger.Verify(
                x => x.Log(
                    LogLevel.Error,
                    It.IsAny<EventId>(),
                    It.IsAny<It.IsAnyType>(),
                    exception,
                    It.IsAny<Func<It.IsAnyType, Exception?, string>>()),
                Times.Once);
        }

        [Fact]
        public async Task ExecuteAsync_WithEmptyEmail_ThrowsArgumentException()
        {
            // Act & Assert
            await Assert.ThrowsAsync<ArgumentException>(
                () => _job.ExecuteAsync(1, "", "Test"));
        }
    }
}

Test patterns:

Happy path: Job succeeds, email sent, no errors
Error handling: Job fails gracefully, logs error, doesn’t crash
Input validation: Reject bad inputs
Mock dependencies: Don’t send real emails during tests

30.10 Common Job Anti-Patterns to Avoid

Anti-Pattern 1: Blocking HTTP responses

// WRONG
[HttpPost]
public async Task<ActionResult> CreateCommand(CommandMutateDto dto)
{
    var command = _mapper.Map<Command>(dto);
    _repository.CreateCommand(command);
    await _repository.SaveChangesAsync();

    // BLOCKING! User waits for email to send
    await _emailService.SendEmailAsync("admin@commandapi.com", "New command", "...");

    return CreatedAtRoute(nameof(GetCommandById), new { id = command.Id }, readDto);
}

Right approach: Enqueue and return immediately.

Anti-Pattern 2: Shared mutable state

// WRONG
public class MyJob
{
    private static List<int> processedIds = new(); // Shared state!

    public async Task ExecuteAsync(int commandId)
    {
        processedIds.Add(commandId); // Race condition!
        await ProcessCommand(commandId);
    }
}

Jobs run in parallel. Shared state causes bugs.

Anti-Pattern 3: No timeouts

// WRONG
public async Task ExecuteAsync(int commandId)
{
    var result = await _externalApi.GetDataAsync(); // Hangs forever?
    // Process result
}

// RIGHT
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
var result = await _externalApi.GetDataAsync(cts.Token);

30.11 Monitoring Job Performance

Track job metrics:

// Services/IJobMetricsService.cs
using System.Threading.Tasks;

namespace CommandAPI.Services
{
    public interface IJobMetricsService
    {
        Task RecordJobStartAsync(string jobId, string jobType);
        Task RecordJobSuccessAsync(string jobId, long durationMs);
        Task RecordJobFailureAsync(string jobId, string errorType);
    }
}

// Implementation with Application Insights or Prometheus
public class JobMetricsService : IJobMetricsService
{
    private readonly ILogger<JobMetricsService> _logger;

    public async Task RecordJobStartAsync(string jobId, string jobType)
    {
        _logger.LogInformation("Job {JobId} of type {JobType} started", jobId, jobType);
        await Task.CompletedTask;
    }

    public async Task RecordJobSuccessAsync(string jobId, long durationMs)
    {
        _logger.LogInformation("Job {JobId} succeeded in {Duration}ms", jobId, durationMs);
        await Task.CompletedTask;
    }

    public async Task RecordJobFailureAsync(string jobId, string errorType)
    {
        _logger.LogError("Job {JobId} failed with error type {ErrorType}", jobId, errorType);
        await Task.CompletedTask;
    }
}

Use in your job:

public async Task ExecuteAsync(int commandId, string email, string title, string jobId)
{
    var startTime = DateTime.UtcNow;

    try
    {
        await _metricsService.RecordJobStartAsync(jobId, "SendEmail");
        await _emailService.SendEmailAsync(email, title, "...");
        
        var duration = (DateTime.UtcNow - startTime).TotalMilliseconds;
        await _metricsService.RecordJobSuccessAsync(jobId, (long)duration);
    }
    catch (Exception ex)
    {
        await _metricsService.RecordJobFailureAsync(jobId, ex.GetType().Name);
        throw;
    }
}

30.12 What’s Next

Now you have:

✓ Job classes with dependency injection
✓ Status tracking and queries
✓ Proper logging (not Console.WriteLine)
✓ Idempotent job design
✓ Retry policies and alerting
✓ Unit tests
✓ Performance monitoring

Next chapter: Bulk Operations Pattern—when you need to process thousands of items efficiently.