All Articles

Presentation: Building a High Volume Data Processing Pipeline in JavaScript

I gave a presentation at The Live Series JavaScript meetup on the 28th January 2021. My talk was about the process of designing and building a high volume data pipeline in JavaScript and the lessons I learned from the process.

Some of the topics I covered:

  • What we do at Vortexa
  • The challenge we had
  • Building stateless processes that can scale
  • Replicating complex pipelines locally for development and testing
  • Overview of the infrastructure
  • Telemetry and error catching
  • Lessons learned
  • The outcome of the project

You can watch the entire presentation on YouTube here: