Embarrassingly Parallel: How Sprite Rethinks Forensic Processing at Scale
If you've ever watched a progress bar crawl across your screen while verifying a disk image, you already understand the bottleneck. Traditional forensic tools are constrained by the hardware they run on: I/O throughput, thread counts, and memory, all competing for resources while the UI renders on the same machine. Sprite takes a different approach, starting with a concept from distributed computing: embarrassingly parallel workloads.
An embarrassingly parallel problem is one that can be divided into many independent parts with little or no coordination between them. It sounds like a weakness, though it's actually a superpower when you have the right architecture.
Starting with the Hard Problem
Verification is, by nature, serial. You have to read an image from start to finish, in order, without skipping a byte, and there's no shortcut around that. That's exactly where I started: building a simple EWF disk verifier to prove out the underlying technology.
The approach was to build what I call a JetStream: a small, self-contained pipeline that takes an evidence item, determines its size, provisions a high-performance volume tuned to the workload, pulls the image down from S3, verifies it, and emits the results. Once complete, it tears down all of that infrastructure behind it, leaving no lingering resources or idle machines.
With some careful work on read-ahead buffering and matching the compute profile to the storage I/O, I ended up with throughput that slightly exceeded what I could get from my own local NVMe drive. That's cloud processing outperforming local hardware, not by throwing money at it, but by tuning the pipeline to the workload.
The Real Win Isn't Speed, It's Scale and Cost
I could have kept going: more threads, more CPU, more memory, building some fire-breathing super-verifier that finishes in seconds and costs a fortune to run. That misses the point entirely.
The real insight is this: while each verification is serial, the number of verifications you can run simultaneously is not. If you've got 50 images to verify, you don't need 10 machines or even one machine. You need a web browser.
Then there's prioritisation. Verification is important, though it doesn't have to happen right now. It can slot in at any point in your workflow, which means you can run it when the cloud has spare capacity. A Fargate Spot instance during off-peak hours can be up to 70% cheaper on compute, delivering the same result at a fraction of the cost with no compromise on quality.
Going Fully Parallel: File Extraction
Now take something that is genuinely embarrassingly parallel: file extraction from a disk image. If you can identify where files sit in the image, you just need start and end offsets to copy them out.
Say you have 100,000 files to extract. The identification step is serial: you read through the image and log the offsets as you find matches, potentially running one thread per file type you're targeting. Every time you hit a match, you push that offset out to a pool of workers who extract directly. Your serial reader isn't slowed down by carving, and your workers fan out to perhaps 100 instances, each handling around 1,000 files.
I built this using Elastic File System, Fargate, and Lambdas, and the architecture works well. It's worth being transparent about the economics, though: EFS pricing is a killer for forensic-scale operations. Storage runs at $0.30/GB and reads cost $0.03/GB, which means just reading a 1TB image end-to-end costs $30. It was a valuable experiment and we've since found a better path, though it serves as a good reminder that cloud architecture is as much about economics as it is about engineering.
What This Means for Large Organisations
The takeaway isn't about any single benchmark. Sprite lets you treat forensic processing as a pipeline rather than a workstation task. Work gets distributed, infrastructure scales to the job and then disappears, and cost is driven by what you actually use rather than what you've provisioned. Whether you're a single lab or a national operation, the architecture remains the same, and that consistency is the point.
Next time, I'll look at how Sprite's plugin model and open-format approach give organisations control without lock-in.