I gave a presentation at The Live Series JavaScript meetup on the 28th January 2021. My talk was about the process of designing and building a high volume data pipeline in JavaScript and the lessons I learned from the process.
Some of the topics I covered:
- The challenge we had
- Building stateless processes that can scale
- Replicating complex pipelines locally for development and testing
- Overview of the infrastructure
- Telemetry and error catching
- Lessons learned
- The outcome of the project
You can watch the entire presentation on YouTube here: