I’ve been advising a startup on the data transformation space. As part of this, we re-wrote the core engine in Clojure. The new version is, at the worst case, 16 times as fast in the same hardware, and in some cases over 200 times faster. And it does it with a fraction of the lines of code.
We did this in under 3 months of part-time work. We couldn’t focus our entire attention on it, as we had other concerns as well - I was involved with general team and management tasks, and the second developer was helping on other internal projects as well. To further raise the bar: we had to keep it functionality-compatible with the current version, so I had to get acquainted with the existing feature set, and it was the other developer’s first Clojure project.
Clojure made our lives so much easier. But this is not a post about why Clojure is cool.
Note: I wrote this in 2018, while I was helping a young startup and convincing them to open source their core application. If I were to write it nowadays, in 2024, I’d elaborate a bit more, including James Governor’s evergreen remark about how “You can make money with open source, but it’s extremely hard to make money from open source.” (source). Leaving it as-is, however.*
I’ve been advising a startup on the data transformation space. As part of this, we re-wrote the core engine in Clojure. The new version is, at the worst case, 16 times as fast in the same hardware, and in some cases over 200 times faster. And it does it in a fraction of the lines of code.
We did this in under 3 months of part-time work. We couldn’t focus our entire attention on it, as we had other concerns as well - I was involved with general team and management tasks, and the second developer was helping on other internal projects as well. To further raise the bar: we had to keep it functionality-compatible with the current version, so I had to get acquainted with the existing feature set, and it was the other developer’s first Clojure project.
Clojure made our lives so much easier. But this is not a post about why Clojure is cool.
I’ve been arguing about why they should open source anything that is not enterprise-specific, including this layer. There’s many advantages, which I won’t go over right now, but there’s also a looming threat.
Layers are getting commoditized faster and faster. More and more, there is demand for people who are good at wiring things together (beyond the gem install hairball approach), or tools that help with that wiring.
I suspect that’s a big part of what’s driving how many companies like Seldon are going open source-first, or how Unreal opened their code as a way to compete with Unity.
“what’s driving how many… are going” is a doozy
Remember: This rewrite took about 3 man-months, with our attention pulled in multiple directions, while we strove to remain feature-compatible with the old engine. You have to assume anyone else who has the technical chops but doesn’t have that baggage can do it as well. Do you want to be disrupted by some motivated, random person who thought what you were doing was cool, but neither needed your entire feature set nor wanted to be shackled to your cloud version?
Better to commoditize yourself before someone else does it to you. You get to have a say in how it happens, use it as fuel to propel you somewhere new. And you get to tell your customers “if you think this thing we give away is cool, you should see the part we charge for”.
Have you gone over the fundamentals on Reading Clojure? Did you shake off that preconception that there’s some magic syntax to declaring and evaluating things beyond the list?
Great! Let’s now go over other things you might encounter when looking at a random source file. I’m first going to give you an overview of types and related things. After it, we’ll then we get into the good stuff like going over a project, and the weird stuff like ->, ->>, #, the quote and other squiggles.
Back in early December I held a Clojure workshop for the Bucharest Functional Programming meet up. Having to explain the language to developers who were completely new at it was an interesting experience. I decided to start from the ground up: how to even read the blasted thing.
Once again I’m doing a personal project that requires me to do some HTML parsing. If you have had to look at it, the landscape of Clojure HTML parsing libraries seems to be littered with dead projects.
Let’s look at the options.
Now that we have our Clojure application tested every time we push it to Gitlab, let’s configure another stage to deploy it to Heroku.
This assumes that:
master is ready to deploy, There’s two ways we can do this.
Gitlab allows you not only to have free private repositories, but also to test them using free runners. These can run automatically, on push, for any branch or tag.
I keep a few private repositories with them, for personal projects and small experiments. I decided to give Gitlab CI a shot for a PostgreSQL-backed Clojure project.
There’s a basic example on the Gitlab CI repository. It gets and installs lein, which isn’t necessary. Instead, we’ll build use the clojure:lein Docker image.
J. Pablo Fernández has recently posted a piece with the incendiary title of “Emacs is Hurting Clojure“.
I disagree with the idea behind the title, but then again, he seems to do so himself. He promptly clarifies:
When I started working with Clojure, I wanted everything to be Clojure.
HTML? Vectors through Hiccup.
Configuration? Clojure maps.
SQL queries? Clojure lists.
After all, I might one day need to take advantage of Clojure’s functional abstractions to… um… well… I might need it, right?
There have been several questions floating lately to the effect of “How do I get started fast?” or “What’s the bare minimum I need to know to get hired?”.
In case you’re wondering how to do that… you’re not going to like this answer.