Benthos Lab is a website where users of the Benthos stream processor can write, format, execute and share their pipeline configurations. This app is now a critical part of our Benthos workflow, but it would never have existed were it not for Meltwater Hackathons.
This post explains how we built Benthos Lab and how the Meltwater Hackathons have enabled us to do so.
Why Benthos Needed a Lab
As heavy users (and maintainers) of Benthos our team has often floated the idea of a shareable config builder app. However, each time the topic was raised we struggled to commit to a static list of minimum requirements for the backend.
The main controversy was supporting the execution of configs, which is a fairly critical part of the workflow. Adding it would require a solution for sandboxing the execution environment and has large implications on the development time needed.
Our team is generally not particularly front-end savvy which, compounded with our uncertainty over feature requirements, left us in a situation where we were unlikely to ever prioritise this work over our other commitments.
Hackathons to the Rescue!
Meltwater runs company-internal hackathons a couple of times per year. We use these events to foster collaboration between different teams and departments, as a creative outlet for crazy ideas, and for learning and self-improvement of everybody working in Product & Engineering.
Hackathons have given us ideal opportunities to explore the value and technical feasibility of Benthos Lab. This is partly because there is no expectation for us to deliver, but also because taking those days out of our sprint relieves us of the pressure of our other commitments (provided we aren’t on meltdown).
For the first Hackathon we made a proof of concept config builder using a static template config. It was promising but ultimately not suitable for real use. It did, however, help refine the features necessary for the next iteration. In short, it turns out that execution is pretty vital.
Third Time’s the Charm
For the latest hackathon we decided to start again from scratch. We had vague ideas of what the front end would be like, taking inspiration from apps such as JSFiddle and the Go Playground. The execution mechanism was still a big unknown.
As a shot in the dark we decided to try to compile the entirety of Benthos into Web Assembly. This would potentially allow us to execute in the sandbox of the users browser rather than the riskier option of building our own sandbox on the server side.
Our initial assumption was that this was dead effort. That even if by some magic all of the dependencies brought in by Benthos would all compile to Web Assembly, we would still have an unfeasibly large artifact at the end of it. However, crucially, we were taking part in a hackathon, and were therefore not accountable if our project explodes (are you jealous, rocket scientists?)
So on we went into the darkness, prepared to have our hearts broken, bodies bent, spirits dissolved. All unnecessary, as it was actually fine and within an hour or two we had a build working, compressed to a manageable 10MB, and had it loaded in the browser. It turns out Web Assembly with Go is pretty easy, who would have known? There’s a blog outlining the technical steps we required for compiling Benthos to Web Assembly.
Once we had Benthos running in the browser everything else quickly fell into place. We now have it deployed at lab.benthos.dev and after just a couple of weeks it has paid dividends in boosts to productivity and collaboration between teams already. You may also want to check out this video of Benthos Lab in action.
In retrospect we could have easily fit this work into a sprint and it would certainly have been worthwhile. However, due to the circumstances we were in, we never actually would have. That’s where the magic of Hackathons comes in.