How to Build a PostgreSQL Backed Job Queue in Go

When you build a web application, not every task should happen inside the user’s request.

Some work is slow. Some work can fail. Some work should happen later. Sending emails, resizing images, processing webhooks, generating reports, and retrying third-party APIs are all good examples.

These tasks are usually handled by a background job system.

In this article, we will use a Go project called Swig as a practical example of how a PostgreSQL-backed job queue can work.

The goal is not only to understand Swig. The goal is also to learn Go concepts that show up in real backend systems:

• Interfaces
• Goroutines
• Contexts
• Transactions
• JSON serialization
• Driver abstraction
• Graceful shutdown
• PostgreSQL row locks
• PostgreSQL LISTEN/NOTIFY
• PostgreSQL advisory locks

By the end, you should understand how a background job queue can be built with Go and PostgreSQL, and why PostgreSQL is more than just a place to store rows.

What Is a Job Queue?

A job queue is a system that stores work to be done later.

Your application adds a job to the queue. A worker takes a job from the queue and runs it.

For example, when a user signs up, your application might create the user immediately, then add a job like this:

{
  "kind": "send_welcome_email",
  "payload": {
    "to": "user@example.com",
    "subject": "Welcome!"
  }
}

A background worker later picks up that job and sends the email.

This keeps the user request fast. The signup route does not need to wait for the email provider before returning a response.

A job queue usually needs to answer a few important questions:

• Where are jobs stored?
• How do workers find jobs?
• How do we stop two workers from processing the same job?
• How do we retry failed jobs?
• How do we shut workers down safely?
• How do we keep job creation consistent with application data?

Swig answers those questions with Go and PostgreSQL.

Why Use PostgreSQL for a Queue?

Many job queues use Redis, RabbitMQ, SQS, Kafka, or another dedicated queueing system. Those are all useful tools.

But many applications already depend on PostgreSQL. If your app already has Postgres, you may not want to operate another service just to run background jobs.

PostgreSQL gives us several features that are surprisingly useful for queues:

• Tables for durable job storage
• Transactions for atomic writes
• Row locks for safe concurrent processing
• SKIP LOCKED for letting workers claim different jobs
• LISTEN/NOTIFY for waking workers when new jobs arrive
• Advisory locks for leader election
• JSONB for flexible job payloads

Swig is a Go job queue that uses these PostgreSQL features directly.

The tradeoff is important. A PostgreSQL-backed queue is not trying to replace Kafka for event streaming or RabbitMQ for complex routing. It is trying to make common application background jobs simple, reliable, and easy to operate.

Swig’s Basic Architecture

At a high level, Swig has five parts:

1. A swig_jobs table in PostgreSQL
2. Go workers that process jobs
3. A worker registry that maps job names to worker types
4. A driver layer that supports both pgx and database/sql
5. A leader loop for shared maintenance work

The basic flow looks like this:

1. Your app calls AddJob.
2. Swig serializes the job payload to JSON.
3. Swig inserts a row into swig_jobs.
4. PostgreSQL sends a notification that a job was created.
5. A Go worker wakes up and tries to claim one pending job.
6. PostgreSQL row locks ensure only one worker claims that row.
7. The worker runs the job.
8. Swig marks the job as completed or failed.

The hard parts are concurrency, failure, connection lifecycle, and shutdown. That is where Go and PostgreSQL work together.

Representing Jobs in PostgreSQL

A simplified version of Swig’s job table looks like this:

CREATE TABLE swig_jobs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  kind TEXT NOT NULL,
  queue TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  priority INTEGER NOT NULL DEFAULT 0,
  attempts INTEGER NOT NULL DEFAULT 0,
  max_attempts INTEGER NOT NULL DEFAULT 3,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  scheduled_for TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  instance_id UUID,
  worker_id UUID,
  locked_at TIMESTAMPTZ,
  last_error TEXT,
  last_error_at TIMESTAMPTZ
);

Each row is one job. The important columns are:

• kind: the type of job, such as send_email
• queue: the queue the job belongs to
• payload: the JSON data needed to run the job
• status: whether the job is pending, processing, completed, or failed
• priority: the job’s priority inside the queue
• attempts: how many times the job has been tried
• scheduled_for: when the job is allowed to run
• instance_id / worker_id: which Swig instance and worker claimed the job
• locked_at: when the job was claimed
• last_error: the last error returned by the worker

The table is the source of truth. PostgreSQL notifications can wake workers, but notifications are not the durable queue. The rows in swig_jobs are.

Defining a Worker in Go

In Swig, a worker is a Go type that knows how to process one kind of job.

Here is a simple email worker:

type EmailWorker struct {
    To      string `json:"to"`
    Subject string `json:"subject"`
    Body    string `json:"body"`
}

func (w *EmailWorker) JobName() string {
    return "send_email"
}

func (w *EmailWorker) Process(ctx context.Context) error {
    fmt.Printf("Sending email to %s with subject %s\n", w.To, w.Subject)
    return nil
}

There are two important methods:

• JobName tells Swig what kind of job this worker handles.
• Process contains the actual work.

The struct fields are also the job arguments.

When you enqueue an EmailWorker, Swig serializes the struct into JSON and stores it in PostgreSQL. Later, a worker claims the row, unmarshals the JSON back into a fresh EmailWorker, and calls Process.

Go Concept: Interfaces

Go interfaces describe behavior.

Swig does not need to know the exact concrete type of every worker. It only needs to know that a worker can provide a job name and process a job. Conceptually, that behavior looks like this:

type Worker interface {
    JobName() string
    Process(context.Context) error
}

If a type has those methods, it satisfies the interface with no explicit declaration required.

This is one of the reasons interfaces are so useful in Go. They let you design around behavior instead of inheritance.

Registering Workers Without Sharing State

Swig has a worker registry. The registry maps a job name to a worker type.

registry := workers.NewWorkerRegistry()
registry.RegisterWorker(&EmailWorker{})

Later, when a job row says kind = 'send_email', Swig can look up the registered worker and run it.

There is a subtle concurrency issue here. If the registry stored the exact &EmailWorker{} pointer and reused it for every job, multiple goroutines could unmarshal payloads into the same Go value at the same time. Shared mutable state is where many concurrency bugs come from.

Swig avoids this with a factory approach internally. Registration captures the worker type, and each claimed job gets a fresh worker instance before JSON is unmarshaled.

The API stays simple:

registry.RegisterWorker(&EmailWorker{})

But internally, Swig creates a new EmailWorker for each job. This is a useful Go pattern: keep the public API simple while making the internal lifecycle safer.

Adding a Job

Here is what adding a job looks like from the user side:

err := swigClient.AddJob(ctx, &EmailWorker{
    To:      "user@example.com",
    Subject: "Welcome!",
    Body:    "Thanks for signing up.",
})

Inside Swig, the process is roughly:

argsJSON, err := json.Marshal(workerWithArgs)
if err != nil {
    return err
}

_, err = db.ExecContext(ctx, `
    INSERT INTO swig_jobs (kind, queue, payload, priority, scheduled_for, status)
    VALUES ($1, $2, $3, $4, $5, 'pending')
`, jobName, queue, argsJSON, priority, runAt)

The worker struct becomes JSON. PostgreSQL stores that JSON in a JSONB column. Later, a worker unmarshals it back into a fresh worker instance.

Transactional Job Enqueueing

One of the best reasons to use PostgreSQL for jobs is transactional enqueueing.

Imagine a user signs up. You want to insert the user and queue a welcome email. If those happen separately, you can get inconsistent states. With a transaction, both succeed or both fail:

tx, err := pool.Begin(ctx)
if err != nil {
    return err
}
defer tx.Rollback(ctx)

_, err = tx.Exec(ctx, `
    INSERT INTO users (email) VALUES ($1)
`, email)
if err != nil {
    return err
}

err = swigClient.AddJobWithTx(ctx, tx, &EmailWorker{
    To:      email,
    Subject: "Welcome!",
    Body:    "Thanks for joining.",
})
if err != nil {
    return err
}

return tx.Commit(ctx)

If the transaction rolls back, the user is not created and the job is not queued. If the transaction commits, both are visible. This is much harder to guarantee when your database and queue are separate systems.

Go Concept: Context

Most Swig methods accept context.Context:

func (s *Swig) AddJob(ctx context.Context, worker interface{}) error

Contexts are used for cancellation, timeouts, and deadlines. That matters in a job queue because many operations can block: database queries, waiting for notifications, processing long jobs, graceful shutdown.

For example, you can create a timeout:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

If the database operation takes longer than five seconds, the context cancels it.

In Go services, passing context through your call stack is a standard habit. It gives callers a way to say: “This work is no longer needed.”

Swig uses contexts for worker lifecycles too. When Stop is called, Swig cancels worker contexts so blocked database calls and notification waits can return.

The Hard Part: Multiple Workers

A queue gets interesting when many workers run at the same time. Imagine three workers all asking PostgreSQL for the next pending job. We do not want all three to process the same job.

A naive approach might be:

SELECT id FROM swig_jobs WHERE status = 'pending' LIMIT 1;
-- then the worker updates it:
UPDATE swig_jobs SET status = 'processing' WHERE id = $1;

This has a race condition. Two workers can select the same job before either one updates it.

PostgreSQL Concept: FOR UPDATE SKIP LOCKED

PostgreSQL can lock rows selected inside a transaction.

FOR UPDATE means “lock this row because I plan to update it.” SKIP LOCKED means “if another worker already locked a row, skip it and find another one.”

This is perfect for a queue:

• Worker A locks job 1.
• Worker B skips job 1 and locks job 2.
• Worker C skips jobs 1 and 2 and locks job 3.

No central coordinator is needed.

A helpful Go analogy is sync.Mutex. A Go mutex protects shared memory inside one process. PostgreSQL row locks protect shared rows across many transactions, processes, and machines. SKIP LOCKED plays the same coordination role, but at the database level.

Atomic Job Claiming

Swig uses an atomic update pattern:

UPDATE swig_jobs
SET status = 'processing',
    instance_id = $1,
    worker_id = $2,
    locked_at = NOW(),
    attempts = attempts + 1
WHERE id = (
  SELECT id
  FROM swig_jobs
  WHERE status = 'pending'
    AND scheduled_for <= NOW()
  ORDER BY priority DESC, created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, kind, payload;

This query does several things at once: finds a pending job, skips already-locked jobs, marks it as processing, records which worker claimed it, and returns the job data. The selection and state change happen atomically. Workers never do a separate SELECT and hope the later UPDATE is still safe.

Go Concept: Goroutines

Swig starts worker loops as goroutines:

for i := 0; i < maxWorkers; i++ {
    go s.startWorker(ctx, queueType)
}

Each worker runs independently. PostgreSQL coordinates which job each worker gets.

This is a nice division of responsibility: Go handles concurrency with goroutines, while PostgreSQL handles safe job claiming with locks. This is one of the reasons Go works well for background processing: you can express concurrency directly, while still relying on the database for cross-process coordination.

WaitGroup and Graceful Shutdown

When a service shuts down, it should wait for workers to finish cleanly. Go’s sync.WaitGroup helps:

var wg sync.WaitGroup

wg.Add(1)
go func() {
    defer wg.Done()
    processJobs()
}()

wg.Wait()

Swig uses a WaitGroup to track active worker goroutines. When Stop is called, Swig cancels worker contexts, closes the shutdown signal, waits for workers, cleans up jobs owned by the instance, and releases leader resources.

Swig also uses sync.Once to make shutdown idempotent. Calling Stop more than once should not panic because of a double channel close. Shutdown paths are often where production systems behave differently from happy-path demos.

Waking Workers with LISTEN/NOTIFY

If workers constantly poll the database for jobs, they waste resources when the queue is empty.

PostgreSQL has LISTEN/NOTIFY. A connection can listen on a channel:

LISTEN swig_jobs;

Another session can send a notification:

NOTIFY swig_jobs, '{"id":"job-id"}';

Swig creates a trigger so PostgreSQL sends a notification after a job is inserted:

CREATE TRIGGER swig_jobs_notify_trigger
AFTER INSERT ON swig_jobs
FOR EACH ROW
EXECUTE FUNCTION notify_job_created();

Workers sleep when there is no work and wake when a new job arrives.

But there is an important PostgreSQL detail: LISTEN is session-scoped. A worker must wait for notifications on the same database session that executed LISTEN. If LISTEN happens on one pooled connection but WaitForNotification happens on another, the worker may never receive the notification.

Swig handles this by creating a dedicated listener for each worker. The listener owns one database session; that same session runs LISTEN, waits for notifications, and gets closed during shutdown.

This is a great example of a backend engineering lesson: abstractions like connection pools are useful, but some database features still depend on the lifecycle of a specific connection.

Leader Election with Advisory Locks

Some queue maintenance tasks should only be done by one instance:

• Retrying failed jobs
• Recovering stale processing jobs
• Cleaning old job history

Swig uses PostgreSQL advisory locks for leader election:

SELECT pg_try_advisory_lock($1);

If the result is true, that Swig instance becomes the leader. Advisory locks are application-defined locks managed by PostgreSQL. You can coordinate distributed processes without a separate lock service.

But advisory locks are also session-scoped. If the session ends, PostgreSQL releases the lock. Swig uses a dedicated advisory-lock connection for leadership. When Swig shuts down, it releases the lock from the same session that acquired it.

If an instance does not become leader, it keeps retrying. If the current leader dies and PostgreSQL releases the lock, another Swig instance can acquire leadership. Simple failover without ZooKeeper, etcd, or a separate coordinator.

Go Concept: Driver Abstraction

Swig supports both pgx and database/sql.

pgx is a PostgreSQL-focused driver. database/sql is Go’s standard database abstraction. Instead of writing the queue twice, Swig defines a driver interface:

type Driver interface {
    Exec(ctx context.Context, sql string, args ...interface{}) error
    Query(ctx context.Context, sql string, args ...interface{}) (Rows, error)
    QueryRow(ctx context.Context, sql string, args ...interface{}) Row
    WithTx(ctx context.Context, fn func(tx Transaction) error) error
    NewListener(ctx context.Context, channel string) (Listener, error)
    TryAdvisoryLock(ctx context.Context, lockID int64) (AdvisoryLock, bool, error)
}

The driver interface hides the concrete database library from the queue logic. The core queue code only depends on behavior:

• Can you execute SQL?
• Can you run a transaction?
• Can you create a listener?
• Can you try to acquire an advisory lock?

This is a common Go design: define the behavior your core package needs, write small adapters for concrete dependencies, and keep the core logic independent. Interfaces are especially useful at boundaries: database drivers, HTTP clients, storage backends, and queues are all good places to use them.

Handling Failed Jobs

Jobs can fail. An email provider might be down. A network request might timeout. A third-party API might rate limit you.

When a worker returns an error, Swig records the error and either retries the job or marks it as failed:

UPDATE swig_jobs
SET status = CASE
    WHEN attempts >= max_attempts THEN 'failed'
    ELSE 'pending'
  END,
  last_error = $2,
  last_error_at = NOW()
WHERE id = $1;

Failed jobs can be retried with backoff. Jobs that exceed their maximum attempts stay failed for inspection.

In production you often add more around this: dead letter queues, metrics, dashboards, alerts, manual retry tools. But the foundation is simple: store the attempt count and error state in the job row.

Delivery Semantics: Be Careful with “Exactly Once”

It is tempting to say a job queue processes jobs exactly once. In distributed systems, that is a dangerous claim.

Swig prevents multiple workers from claiming the same pending job at the same time. But imagine this:

1. Worker sends an email.
2. Worker crashes before marking the job completed.
3. The job is retried.
4. The email might be sent again.

The safer language is: Swig provides atomic claiming and at-least-once processing.

Because jobs can be retried, workers should be idempotent. Running the same operation more than once should have the same result as running it once. For example, an email job could use an idempotency key so the email provider does not send duplicates.

This is one of the most important lessons in queue design. The queue can reduce duplicate processing, but workers still need to handle retries safely.

What This Project Teaches About Go

Swig is a useful learning project because it touches many parts of Go backend development.

It teaches interfaces because the database layer supports multiple drivers.

It teaches goroutines because workers run concurrently.

It teaches contexts because database calls, workers, and shutdown need cancellation.

It teaches transactions because enqueueing jobs can be atomic with application writes.

It teaches JSON serialization because job arguments need to cross a process boundary.

It teaches error handling because failed jobs need retry behavior.

It teaches lifecycle management because listeners, advisory locks, and worker goroutines all need to be opened and closed carefully.

Most importantly, it teaches that backend systems are not just about the happy path. The interesting work is in concurrency, failure, connection ownership, shutdown, and consistency.

Conclusion

A PostgreSQL-backed queue is not the right answer for every system. If you need massive event streaming, Kafka may be a better fit. If you need complex routing, RabbitMQ may be better. If you need a simple fast queue, Redis may be a good choice.

But for many Go applications, PostgreSQL is already there.

Swig shows how far you can get with a small Go API and a few PostgreSQL features:

• Store jobs in a table.
• Claim jobs with FOR UPDATE SKIP LOCKED.
• Wake workers with dedicated LISTEN/NOTIFY sessions.
• Coordinate leadership with dedicated advisory-lock sessions.
• Use transactions to keep app data and jobs consistent.
• Use goroutines and contexts to manage worker lifecycles.

That combination makes a great project for learning Go, PostgreSQL, and distributed systems fundamentals.