consultancy speaking content contact
LED panels showing logs

100x'ing bulk import speed in Rails with gnarly SQL

Processing masses of data is always tricky, and it's tempting to reach for specialised tools the moment we hit problems like this, but it's my firm belief that pushing forward with the tools you're already using has compounding benefits.

In the above presentation, we deep-dive on some Rails and PostgreSQL APIs and features which allowed me to optimise bulk imports in a large-scale Rails application down to less than 1% of it's original processing time.

If you'd like to follow along in code, the exercise is available on Github – you can fork it to your own account and push changes to see the performance impact in Github Actions. It could make a good training exercise for your team, or even to pair on in an interview.

The final optimised version is also available on Github. You'll see this takes import time from 25 seconds down to a few hundred milliseconds, memory down from 10 megabytes to less than half a megabyte and allocations from over 140,000 to less than 7,000 – this gives enormous scope for scalability.

Hero Image Credit: Leif Christoph Gottwald

Get in touch

Sending message…

Error sending message

Message could not be delivered — try the full contact page for alternatives

Message received

Thank you for your message — I will get back to you within 24 hours.