Sigil: Iterating on Speed and Scale

Sigil works. It reads Arcana’s enriched bar data, runs it through strategy definitions, and produces signals. That part is solved. But there’s a meaningful difference between “works” and “works fast enough to iterate through hundreds of strategies against years of market data without losing your mind waiting.”

This post is about closing that gap.

The Starting Point

After separating Sigil from Arcana, I had a clean system that could process strategies sequentially. Load the bar data, initialize a strategy, run it forward through the dataset, collect signals, move to the next strategy. Correct, straightforward, and slow.

The initial benchmarks told the story: processing a single strategy against six months of ETH-USD imbalance bars took about 12 seconds. Multiply that by a few hundred strategy configurations, and you’re looking at roughly 40 minutes for a full sweep. Not catastrophic, but slow enough that rapid experimentation was painful. When you want to tweak a parameter and see what happens, waiting 40 minutes kills the feedback loop.

Where the Time Goes

Before optimizing anything, I profiled. The results weren’t what I expected.

Strategy logic itself was fast — the actual computation of signals from bar data was a fraction of the total time. The bottlenecks were in the scaffolding:

1. Data loading and deserialization. Each strategy was independently loading and parsing the bar data from storage. With hundreds of strategies reading the same dataset, this was redundant I/O at massive scale.

2. Strategy initialization overhead. Each strategy object was being fully constructed per run, including parameter validation, indicator pre-computation, and state allocation. For strategies that differ only in a parameter (like a lookback window), this was wasteful.

3. Memory allocation patterns. Intermediate results were being allocated and freed per-bar, per-strategy. The garbage collector was working overtime.

The Optimizations

Shared data loading. The most impactful change was loading bar data once and passing read-only views to each strategy. Bar data is immutable during a strategy run — there’s no reason to deserialize it hundreds of times. This single change cut total processing time by roughly 60%.

Strategy batching by bar type. Strategies that consume the same bar type (e.g., all strategies that run on dollar imbalance bars) now run in sequence against a single loaded dataset before moving to the next bar type. This keeps the working set in cache and avoids thrashing between different data shapes.

Pre-allocated result buffers. Instead of allocating signal arrays per-bar, each strategy gets a pre-allocated buffer sized to the dataset length. Signals are written in place. This dramatically reduced GC pressure and improved cache locality.

Parameter-variant batching. For strategy families that share logic but vary parameters (different EMA windows, different threshold values), the core computation is shared and only the parameter-dependent branches diverge. This avoids redundant indicator computation across variants.

The Results

After optimization, the same full sweep — hundreds of strategies against six months of data — runs in under 4 minutes. That’s roughly a 10x improvement from the initial 40 minutes.

More importantly, the feedback loop is now practical. Tweak a parameter, run a targeted subset of strategies, see results in seconds. Run the full sweep when you want comprehensive validation. The system scales linearly with strategy count, which means adding new strategies doesn’t degrade the experience for existing ones.

Individual strategy processing dropped from ~12 seconds to under 1 second for most configurations. The remaining time is dominated by I/O (loading bar data from storage) and result serialization (writing signals for downstream analysis).

What “Fast Enough” Actually Means

For Sigil, “fast enough” isn’t real-time — that’s Arcana’s job. Sigil processes historical data in batch. The goal is iteration speed: how quickly can I go from “what if I tried this?” to “here’s what happened.”

Under 4 minutes for a full sweep means I can run 15+ complete iterations in an hour. That’s the difference between exploring the strategy space methodically and guessing because the feedback is too slow to experiment.

The next phase is deciding which strategies survive the backtest. That’s a different kind of optimization — not computational speed, but signal quality. The pipeline from Arcana through Sigil is now production-grade. The data engineering is solid. Time to see what the data actually tells us.