https://www.pistack.xyz/tags/data-pipelines/ Recent content in Data-Pipelines on Pi Stack Hugo en-us Fri, 18 Sep 2026 00:00:00 +0000 https://www.pistack.xyz/posts/2026-09-18-elixir-background-processing-oban-broadway-quantum/ Fri, 18 Sep 2026 00:00:00 +0000 https://www.pistack.xyz/posts/2026-09-18-elixir-background-processing-oban-broadway-quantum/ <p>Elixir makes concurrency so cheap that teams quietly accumulate <code>Task.start/1</code> calls all over their codebase — until the first deploy kills 4,000 in-flight jobs mid-execution and nobody can say which emails went out. The BEAM gives you lightweight processes for free, but <strong>durability, retries, and back-pressure are not free</strong>, and those are exactly the three things that separate Elixir&rsquo;s background processing libraries.</p>