What if PostgreSQL could do much more than store data?
What if it could orchestrate ETL pipelines, schedule jobs, recover from crashes, call cloud APIs, and automate database maintenance, all without Airflow, Temporal, or another orchestration platform?
That’s exactly what Microsoft’s new open-source PostgreSQL extension, pg_durable, is trying to achieve. Instead of treating PostgreSQL as just a database, it turns it into a durable workflow engine capable of executing long-running, fault-tolerant workflows directly inside your database.
For Applications that already running PostgreSQL, this is one of the most interesting database innovations of the year.
Let’s dive in.
Why Traditional PostgreSQL Workflows Become Messy
Imagine your application needs to:
Import customer CSV files every night
Clean invalid records
Generate embeddings
Refresh materialized views
Send notifications
Retry failed API calls
Wait for manual approval before publishing data
Normally you’d need tools like:
Airflow
Temporal
Azure Durable Functions
Cron jobs
Custom worker queues
Redis
RabbitMQ
Your architecture suddenly looks like this:
Application
↓
Queue
↓
Workers
↓
Retry Logic
↓
Scheduler
↓
PostgreSQLNow you have multiple services to monitor, deploy, debug, and scale.
pg_durable flips this model completely.
Instead of moving data outside Postgres for orchestration, the orchestration happens where the data already lives.
What Exactly is pg_durable?
pg_durable is an open-source PostgreSQL extension created by Microsoft that introduces durable SQL workflows.
Think of it as:
Azure Durable Functions, but running directly inside PostgreSQL.
It provides:
Sequential workflows
Parallel execution
Automatic retries
Scheduling
Conditional branching
Crash recovery
Checkpointing
HTTP integrations
And everything is written using SQL.
The Magic Behind “Durable”
Suppose this ETL job runs:
Import CSV
↓
Validate Data
↓
Transform Records
↓
Generate Embeddings
↓
Load Into ProductionWhat happens if PostgreSQL crashes during embedding generation? Traditional SQL functions start over.
pg_durable doesn’t.
It checkpoints every completed step.
After restart:
✔ Import CSV
✔ Validate
✔ Transform
✖ Generate Embeddings (crashed)
Restart...
↓
Resume HereNo duplicated work. No manual recovery. No custom retry logic.
Real-World Use Case 1: Nightly ETL Pipeline
Imagine an e-commerce company.
Every night they:
Download orders
Clean data
Aggregate revenue
Refresh dashboards
Instead of building workers:
SELECT df.start(
'SELECT import_orders()'
~>
'SELECT clean_orders()'
~>
'SELECT aggregate_sales()'
~>
'REFRESH MATERIALIZED VIEW sales_dashboard'
);The workflow is durable. If step three fails:
Step one won’t rerun.
Step two won’t rerun.
Only the failed step retries.
Real-World Use Case 2: Parallel Analytics
Instead of executing reports one by one:
Count Users
Count Orders
Revenue
Inventory
↓
DashboardRun them simultaneously.
SELECT df.start(
'SELECT COUNT(*) FROM users'
&
'SELECT COUNT(*) FROM orders'
&
'SELECT SUM(amount) FROM orders'
~>
'REFRESH MATERIALIZED VIEW dashboard'
);The & operator creates parallel fan-out, while ~> waits for all tasks before continuing.
💡 Enjoying this article?
Every week day, I publish practical, production-ready deep dives covering Web development, System Design, Open source projects, Tech industry trends and AI Engineering and tools.
Real-World Use Case 3: Automated Database Maintenance
Database administrators spend hours doing repetitive tasks:
Vacuum tables
Detect bloated indexes
Archive old records
Reindex tables
Check wraparound risk
Instead of manual scripts:
Check Table Size
↓
Archive Old Data
↓
VACUUM
↓
Analyze
↓
Send ReportEach step becomes recoverable. Even if maintenance stops halfway through, it resumes automatically.
Real-World Use Case 4: AI Data Pipelines
Modern applications constantly process documents.
Example:
Upload PDF
↓
Split Content
↓
Generate Embeddings
↓
Store Vectors
↓
Searchpg_durable supports HTTP calls, making cloud-connected AI workflows possible directly from SQL. Microsoft also uses it as the execution layer for AI pipelines in Azure HorizonDB Preview.
Conditional Workflows
Need approval before publishing?
Validate Data
↓
IF Valid
↓
Publish
Else
↓
RejectInstead of embedding logic inside application code, the workflow itself handles branching using durable SQL constructs.
Scheduling Jobs
Instead of relying on Linux cron:
Every Night
↓
Import CSV
↓
Generate Report
↓
Email Teampg_durable supports scheduled execution with durable state, meaning scheduled workflows survive database restarts without losing progress.
Why Developers Should Care
Most teams already trust PostgreSQL for critical production data. Now the same database can also orchestrate:
Data pipelines
Business workflows
Database maintenance
AI preprocessing
Cloud integrations
Scheduled jobs
That means:
✅ Fewer services
✅ Less infrastructure
✅ Easier debugging
✅ SQL-first development
✅ Built-in durability
pg_durable vs Traditional Workflow Tools
If your workflows are tightly coupled to PostgreSQL, keeping orchestration inside the database can significantly reduce operational complexity.
Is It Meant to Replace Airflow or Temporal?
Not entirely.
If you’re orchestrating hundreds of micro-services across many systems, dedicated workflow engines still have advantages.
But if most of your work already revolves around PostgreSQL, pg_durable can remove an entire layer of infrastructure while giving you durable execution, retries, scheduling, and observability directly in SQL.
Final Thoughts
For years, PostgreSQL has steadily evolved beyond “just a relational database” adding JSON, logical replication, vector search, full-text search, and powerful extensions.
pg_durable is another step in that evolution.
Instead of moving data to an external workflow engine, it brings orchestration to where the data already lives.
For teams building ETL pipelines, AI preprocessing, scheduled jobs, approval flows, or database maintenance, that can mean simpler architectures, fewer moving parts, and workflows that automatically survive crashes.
As databases continue absorbing more application responsibilities, one thing is becoming clear:
The future of PostgreSQL isn’t just storing data, it’s running your data workflows too.
Resources
pg_durableGitHubpg_durableDocumentation
Thank You for Reading!
I hope you found it helpful and informative. If you have any questions or feedback, feel free to leave a comment below. Your support and engagement mean a lot to me.