Microsoft Sentinel Cost Optimization with Staged Routes and Commit Processors

As security data volumes grow, so do the costs of processing and storing them. Microsoft Sentinel and other SIEM platforms charge based on data ingestion, which makes every decision about normalization rules critical and every duplicate log a direct expense.

Enterprise-scale security data pipelines face a persistent problem: data duplication across normalization tiers. As logs move through multiple transformation stages, it’s often impossible to know in advance which version will succeed.

We asked ourselves the same question many teams face: how can duplication be eliminated without adding layers of conditional logic that are hard to maintain? That was a tough challenge, but eventually we created an elegant solution for this problem: Staged Routes and Commit Processors. This pattern will help enterprise customers to optimize their Microsoft Sentinel ingestion costs even further by maintaining a single version of their log in the right table, while maintaining cleaner, more maintainable pipelines.

This isn’t just a feature addition; it’s a fundamental shift in how we approach multi-tier data transformation.

The problem: unknown destination formats during processing

In modern data pipelines, especially for security platforms like Microsoft Sentinel, logs undergo multiple transformation stages. A single syslog message might flow through several normalization attempts:

Syslog – Raw, unstructured log data
CommonSecurityLog (CSL) – Structured CEF/LEEF format
ASIM (Advanced Security Information Model) – Fully normalized, schema-compliant format

Here’s the critical challenge: At the start of processing, you don’t know which normalization level the log will ultimately reach.

Consider this flow:

You receive a syslog message
It might be in CEF format (can transform to CSL)
Or it might not (stays as Syslog)
If it transforms to CSL, it might match an ASIM schema (can transform to ASIM)
Or it might not (stays as CSL)

Without proper handling, you face two bad options:

Option 1: Send everything to all tables (wasteful)

Send raw Syslog → Syslog table
Transform to CSL → Send to CommonSecurityLog table
Transform to ASIM → Send to ASIM table

Result: Same log stored 3 times, 3x ingestion costs

Option 2: Complex conditional logic (unmaintainable)

Before sending Syslog, check if CEF transformation will succeed
Before sending CSL, check if ASIM transformation will succeed
Store previous versions in case next transformation fails
Add rollback logic everywhere

The traditional approach: conditional hell

Let’s see what this looks like without staged routes. You need to check future transformation success before committing current data:

The problems are obvious:

You’re predicting the future (checking if transformations will succeed before they run)
Complex rollback logic in failure handlers
Each reroute must know about all subsequent transformations
Adding a new normalization tier requires updating all existing conditionals
Maintaining vendor-specific transformation rules becomes a nightmare

The solution: staged routes with commit processor

The staged routes pattern solves this elegantly by deferring the routing decision until after all transformations complete. The concept is simple:

Each reroute to the same destination overwrites the previous staged version. Only the final staged version gets committed.

No predictions. No rollbacks. No complex conditionals. Just natural pipeline flow.

How it works

The beauty of simplicity

Each processor is now independent and only checks its own success criteria:

Syslog reroute: “Was I able to parse syslog?” → Stage it
CSL transform: “Can I convert this to CSL?” → Try the transformation
CSL reroute: “Did the CSL transformation succeed?” → Stage it (overwrites Syslog)
ASIM transform: “Can I convert this to ASIM?” → Try the transformation
ASIM reroute: “Did the ASIM transformation succeed?” → Stage it (overwrites CSL)
Commit: “What’s staged for this destination?” → Send it

No processor needs to know about downstream transformations. No rollback logic. No storing previous versions.

Real-world example: Microsoft Sentinel Automation Pack

In our Microsoft Sentinel automation content pack, we’ve implemented this pattern with a simple configuration flag: prioritize_normalized_tables. When enabled, it automatically uses staged routes to eliminate data duplication across Syslog, CommonSecurityLog, and ASIM tables.

Configuration

microsoft sentinel Сost optimization 2.1

Complete pipeline example

Microsoft Sentinel cost optimization in one click – with DataStream automation pack

Staged routes and commit processors solve a fundamental problem in data pipeline architecture: you can’t predict the future, but you can defer decisions until the future becomes the present.

By eliminating the need to predict transformation success, we achieve:

1. Simpler code – No complex conditionals checking downstream transformations

2. Better maintainability – Add new normalization tiers without touching existing logic

3. Cost optimization – Send only the most valuable version of each log

4. Flexibility – Runtime decisions based on actual transformation results

5. Reduced duplication – One log, one destination, highest normalization achieved

Whether you’re managing Microsoft Sentinel ingestion costs, building multi-tier data lakes, or implementing sophisticated routing logic based on data quality, staged routes provide an elegant solution that scales with your needs.

For organizations using Microsoft Sentinel, the DataStream automation pack with built-in prioritize_normalized_tables support makes this pattern immediately accessible. Enable one flag, and your pipelines automatically deduplicate across Syslog, CommonSecurityLog, and ASIM tables while always sending the most normalized version available.

Want to see it in action? Book a live walkthrough or talk to our team to learn more.