Sigil: Born from Arcana
If you’ve been following the Information Bars series, you know the last few weeks have been about getting the data right — building bars that actually respect market microstructure instead of arbitrarily slicing time. That work happened inside Arcana, and it’s been solid. But as I turned my attention to the next phase — running strategies against that enriched data — something became obvious pretty quickly.
Arcana was doing too much.
The Problem with One Big System
When I first started building out the strategy processing layer, it lived inside Arcana as a module. Makes sense on paper: Arcana ingests the data, enriches it into bars, and then processes those bars through trading strategies. One pipeline, one codebase, one deployment.
In practice, this created coupling that made both systems worse. Data ingestion and strategy processing have fundamentally different concerns:
-
Failure modes are different. If the Coinbase WebSocket drops, that’s an ingestion problem. Arcana needs to reconnect, track where it left off, and resume without gaps. If a strategy definition has a bug, that’s a processing problem — it shouldn’t take down the data pipeline.
-
Performance profiles are different. Arcana needs to be lightweight and persistent. It runs 24/7, streaming data, handling backfills, writing to storage. Strategy processing is bursty and compute-heavy — it needs to tear through years of bar data across hundreds of strategy configurations.
-
Release cycles are different. I was iterating on strategy definitions daily, sometimes hourly. Every deploy risked the data pipeline. Every Arcana change risked breaking a strategy run mid-flight.
The Separation
Sigil is now its own system. It reads from Arcana’s output — the persisted bar data (tick bars, volume bars, dollar bars, imbalance bars) — but runs on its own lifecycle, its own process, its own failure boundary.
The architecture is straightforward:
- Arcana handles data ingestion, fault tolerance, bar construction, and persistence. It writes enriched bar data to storage.
- Sigil reads that persisted data and runs it through configurable strategy definitions to produce trading signals.
If Sigil crashes mid-strategy, Arcana doesn’t notice. It keeps ingesting. If Arcana restarts during a backfill, Sigil can still process from whatever data has already been persisted. They don’t share memory, they don’t share processes, and they don’t share deployment risk.
What Changed Architecturally
The main refactor was extracting Sigil’s data access layer so it reads from Arcana’s storage format without needing Arcana’s runtime. Sigil has its own configuration for which bar types to consume, which strategies to run, and how to output signals.
Strategy definitions became first-class objects with their own schema. Each strategy specifies the bar types it needs, the parameters it accepts, and the signal format it produces. This makes it trivial to add, remove, or modify strategies without touching the core processing engine.
The Payoff
The separation was one of those decisions that felt like overhead at first and obvious in hindsight. Within the first week of running Sigil independently:
- I deployed 14 strategy iterations without touching Arcana once.
- Arcana ran uninterrupted for the entire week while I was thrashing on strategy code.
- Testing became dramatically simpler — I could unit test strategies against fixture data without spinning up the full ingestion pipeline.
- Profiling Sigil’s hot paths became clean because the metrics weren’t polluted by Arcana’s I/O patterns.
If you’re building a data system where the ingestion layer and the processing layer have different stability requirements, separate them. It’s more work upfront and it’s worth every line.
Sigil is private software — the strategy implementations are too sensitive to open source. But I’ll continue sharing architecture decisions, performance work, and lessons learned here on the blog. Next up: making Sigil fast enough to iterate through hundreds of strategies without waiting hours for results.