Shiveen Pandita

90% of software engineering is knowing what not to write

2026-02-15T03:59:31Z

It's clear that software engineering is changing. Gone are the days of code output being the core bottleneck for most projects. In any medium to large company, the plan-code-test cycle is the most efficient way to write software going forward. If you're still in denial, you're falling behind.

However, that is not what defines a software engineer or engineering in general. The following is the first line defining engineering in wikipedia:

Engineering is the practice of using natural science, mathematics, and the engineering design process to solve problems within technology, increase efficiency and productivity, and improve systems.

Notice how it consists of two broad active verbs, practice and solve. But there's a third word hiding in plain sight: design. Design is inherently about tradeoffs — what to include and what to exclude.

We're seeing an inflection point in the solution part. Just like punch cards were made obsolete by handcrafting assembly, followed by ever higher level languages ever since; I expect the same thing will happen on a larger scale with LLMs. We're no longer constrained by code output anymore. And as these technologies improve, they will only allow for better integration and output patterns.

So, what is left for an engineer to do? Ironically, a lot more. Pre-AI/Agentic coding, planning and coding constrained the delivery funnel for most "funded" engineering projects. With the coding part getting ever cheaper, you might think the funnel is now constrained by just planning.

I believe you're wrong to assume that!

See, the funnel didn't automatically get thinner in the middle just because code was inexpensive. In fact, the planning part of the funnel got wider and with higher stakes.

Inexpensive code means harder decisions #

When code is nearly free to produce, more things can be built. And when more things can be built, there are more decisions about what should be built. The bottleneck shifts from "can we build this?" to "should we build this?", which almost always is a much harder decision.

The real value of an experienced engineer is increasingly in knowing what not to write.

Knowing which features to say no to, even when they're easy to build. It's harder to reduce code than it is to add.
Knowing when a third-party service or library is better than building in-house, including knowing when including it will cause debt which can only be known if you have the general tacit world knowledge only humans can possess
Recognizing when a "simple" feature has hidden complexity — edge cases, operational burden, security surface area. LLMs are constrained by their context window, their model, and the narrow window of an execution decision tree they're operating in.

An AI-implemented solution leans heavily towards one path or the other in most circumstances. It will indiscriminately build a custom rate limiting solution and then suggest using redis for caching in another solution, not knowing that the problem domains are compatible across requests and sessions.

Creation is inexpensive, ownership is not #

Ownership is the crux of the difference between vibe/slop-coding apps and a real SaaS provider. Vibe coding begets vibe coding. As you don't develop the global context for your own project, a 2am page will be a nightmare for any complex app. Without understanding the internals of the system you're building, you will never realise that your picture-perfect custom auth solution is breaking because provider X did not implement the OAuth2 refresh tokenizing in the exact same way as all the other providers.

Vibe coding begets vibe coding

This is where "knowing what not to write" becomes critical. The engineer who generates reams of code per day cannot truly compete with an engineer who glanced at the problem and said, yes, we should not roll our own auth.

Most real customers, willing to pay you money, expect uptime, responsiveness, and a clean roadmap of features. This is not possible if your system grows like a jungle as you smash out another prompt. The only way this is possible is by knowing the real world constraints you're operating in and deeply understanding your codebase before opening your next claude code session or cursor tab.

Judgment wins #

What separates senior engineers has always been judgment. But for a long time, that judgment was masked by the coding bottleneck. When writing code was slow and expensive, raw output was a reasonable proxy for contribution. Now that bottleneck is gone, and judgment is laid bare.

Engineers who were primarily valued for code output are exposed. Those valued for architectural thinking and the ability to say "no" are more important than ever. The skill isn't typing faster or prompting better — it's pattern recognition built over years of seeing what works, what breaks, and what quietly becomes a liability.

Practice makes the engineer #

This brings us back to the engineering definition. The word practice implies accumulated judgment, pattern recognition, and a deep understanding of what works and what doesn't. That part doesn't get automated away. It only amplifies with age and experience. An engineer with strong judgment and AI tools is dramatically more effective than either alone.

This has been said in other online circles before, but the AI multiplication factor isn't linear. It sort of goes like:

An engineer with poor pattern recognition and low world knowledge: Maybe 2x faster with coding agents and tools
An engineer with years of honed pattern recognition and broad world knowledge: At least 10x faster with coding agents and tools

The second lot is the engineer that can guide the company and the product to a sustainable cadence of delivering customer value, not marred by outages and tech debt.

The future of software engineering isn't about writing more code. It's about writing less of it, more deliberately, with a clearer understanding of why each piece exists and what it costs to keep it alive.

My Short-Lived Experiment with Readwise and Reader

2026-02-07T00:00:00Z

Sometime around the end of 2025, I took stock of my reading habits. Over the years, I had started spending more time on social media (bad) and less time on curated feeds (bad, again). Where I usually would end my days reading or consuming feeds or saved articles, I now found myself endlessly doomscrolling.

Considering one of my core goals of 2026 to restore balance back into my personal life, I decided to cut down on my social media consumption and fix how I consumed information about the world around me. My mainstays for the last few years had been Feedbin and Instapaper. Two fast and to the point apps, that simply do one job and do it well. Unfortunately, my queue in both of those had grown and covered the place like weeds.

I also had been recently reading a lot more non-fiction and started toying with the idea of perhaps highlighting and revisiting passages more. Up until now, I was manually pulling the highlights from my kindle using calibre and putting them in Obsidian. It wasn't the end of the world, but it was definitely a chore. I looked into Readwise and learnt that they had branched out of their niche into a fully integrated read it later app that worked for RSS feeds, saved articles and newsletters.

I wasn't a stranger to Readwise, having used it a few years back when all they offered was a Kindle sync to Roam or Evernote. I toyed with it but eventually gave up as I didn't quite find as much value then. Ironically, this time I gave up for the opposite reason, but we will get to that in a minute.

I exported out my feeds from Feedbin and saved articles from Instapaper and fed them to Readwise. I also connected my Kindle and Obsidian vault and had them linked and ready to go.

Right off the bat, I was impressed with the feature set. Readwise has a ton of features (overwhelmingly so). The way the Reader app works is that it allows you to save articles and pull in your feeds. And then everyday it will pick a few and send them to your phone or email to review. It also has a neat "later" feature which allows you to defer reading things (but might still want to) for later. Think of it as your personal adaptive daily newspaper for the things you care about. Their mobile and web apps are elite and far better than any competition I had tried, and make the experience of using it enjoyable.

On top of that, it's filled to the brim with features (these are the ones I interacted with, there might be a lot more):

Clean sorting out of new, quick and long reads - auto categorised.
A Ghostreader feature that uses AI to fully read an article and summarise (also has audio transcription i think but I never tried)
Ability to save pdfs and epubs and highlight them as well
Great highlights and daily review feature, which is super useful addition from their other product and is now integrated in this product as well
One underrated feature, at least for me, was the addition of a nice set of keyboard shortcuts. As a software engineer, I truly appreciate when someone spends time to add refined keyboard shortcuts to their app

However, the novelty faded real quickly. After the first few weeks of opening the app with fervor every morning and reading the daily digest emails – I slowly started feeling disillusioned with the promise. The problem wasn't that the app did not provide me with the right information, rather it was that it provided too much of it.

The problem wasn't that the app provided me with the right information, rather that it provided too much of it.

See, as much as I love reading other interesting peoples thoughts or random post-its I have collected from all over the internet, I realised I like reading them at my own pace. Readwise felt like it was a firehose aimed right at my eyeballs. Even worse, the constant vectors of notifications, push and email, with their various varieties was just stealing my attention. The reason I wanted to consolidate was to move away from social media and its constant need for attention. Instead, I ended up recreating it in Readwise.

I know I could have turned off all the notifications, and that would be that - however, even after doing that, I felt the whole interface was also designed with the aforementioned goals in mind i.e., little UI elements to draw attention on the home page. That coupled with a few papercuts in regard to the reading experience (tiny images, bad HTML parsing etc.) I decided to end the experiment after a month.

I am back to using Feedbin and Instapaper again. I missed the feel of both and how they stay out of my way and let me choose. I like that they don't clamour for attention and I also like in a weird way that they are not filled to the brim with features. I like nice, simple and boring software.

Will I miss basic Readwise kindle highlights? unequivocally, yes. That said, I have also come to realise I care less about the highlights in general and anything important I want to note, I usually note it in the moment anyway. Sometimes restoring balance means choosing less over more.

Default Apps 2026

2026-01-29T11:25:29Z

📨 Mail Client: Gmail (web and iOS) with ImprovMX for domain routing
📮 Mail Server: Gmail
📝 Notes: Obsidian
✅ To-Do: TickTick
📷 Phone Photo Shooting: Default iOS Camera App
🟦 Photo Management: Lightroom or Google Photos
📆 Calendar: Fantastical with Google Calendar backend
📁 Cloud File Storage: Google Drive
📖 RSS: Feedbin (web + iOS)
🙍🏻‍♂️ Contacts: Cardhop (only because it comes bundled with my fantastical sub), rarely opened
🌐 Browser: Google Chrome
💬 Chat: iMessage + Whatsapp
🔖 Bookmarks: Raindrop.io
📑 Read It Later: Instapaper
📜 Word Processing: Google Docs
📈 Spreadsheets: Google Sheets
📊 Presentations: Google Slides
🛒 Shopping Lists: A frankenstein system made up of custom shortcuts + Apple Reminders + TickTick
🍴 Meal Planning: PlanToEat
💰 Budgeting and Personal Finance: I don't budget ever (used to use Banktivity in a previous life)
📰 News: Google News, but I rarely read news.
🎵 Music: Spotify
🎤 Podcasts: I barely listen, if I do then it's Spotify
🔐 Password Management: 1Password

Also see all the gear I use in Uses page.

Lightroom CC to Google Photos

2026-01-29T05:45:31Z

I use Lightroom for all my edits. I also use Google Photos for all my photo sharing and backups. Until a few months ago, I was using Lightroom Classic. My workflow then involved editing on my desktop, exporting the JPEGs manually, and pushing them to Google Photos.

This was not arduous really, but it required me to keep my computer on for larger edits for the sync to fully go through.

However, after moving to Lightroom CC, I discovered this is a lot easier and genuinely a great quality of life improvement over my previous workflow.

Now, I can simply edit the photos and have them uploaded to the cloud. Then open Lightroom Web, and directly batch transfer to Google Photos. There are actually two share buttons, so you have to choose the right one. The classic share button (shown below) does not have the option.

The correct way is to Cmd+A in your album or directory and then select share from the floating action bar that shows (highlighted below).

This makes the process so much smoother than before. My photos are now batch uploaded in the background and available when done without me needing to keep tabs on the operation. This is another solid reason to move to the cloud version of Lightroom.

Four Computers

2025-10-25T23:00:00Z

I currently have 4 different tiers of computing devices, listed in no particular preference order:

An iPhone 16 Pro
An iPad Pro 11"
An iPad Mini
A Macbook Pro 14" M4 Pro

On the outset, it seems like too many devices. And in all fairness, it probably is. But I get enough use out of all of them that I find them worth it.

Here's how I think about each one.

iPhone #

This is my main device and is almost always on or near me. I use it for the basics: calling and texting people (duh), quick media consumption, some social media (I'm not big on socials anymore), and taking photos.

I don't play games on it, and I don't do any intensive reading either. I realized a few years back that too much time on the small screen just makes me grumpy. Maybe it's the text size, or maybe it's how I hunch like a gremlin when I use it. Either way, if something takes more than 10 minutes to consume, this isn't the device for it. That honour belongs to my beloved iPad Mini.

iPad Mini #

I can't overstate how much I love this device. I was hesitant to get it at first—wasn't sure I needed another i(something) Apple thingamajig—but boy was I wrong. This device has surpassed all my expectations.

This is my main "put the phone down" device, other than my e-reader. I use it for basically everything. Reading news? Tick. Booking flights? Tick. Consuming newsletters? Tick. Watching YouTube? Tick. Watching TV shows while the family uses the bigger screen? Tick. I could go on.

It's the perfect device for casual consumption and surprisingly great for research too. I come home, put my phone away, and use the Mini for everything. I even got the cellular version, so sometimes I'll leave my phone at home if I just want to read and chill somewhere without it burning a hole in my pocket. This is also my go-to travel device.

I just really wish it had a better screen with a higher refresh rate, but that's a minor gripe.

iPad Pro #

This is the oldest device in the lineup. I bought it in 2021—it still has the M1 chip and works flawlessly for my needs. It stutters a bit more than it used to, but nothing that's enticed me to upgrade. Speaks volumes about how game-changing the M series hardware from Apple is.

I use this sometimes as a consumption device, but truly this is my "get my life in order and sometimes write" device. I'm writing this post right now, with a cup of espresso on the side, lounging on my deck with this iPad. I use it with the standard Apple keyboard, and it's a joy to write in this form factor. I love that the small screen keeps me focused.

I also do any technical or long-form reading on this device—research papers, lengthy articles, that sort of thing. Calendar scheduling and todo management happen here too. The multitasking shines with split screen working perfectly for these workflows.

MacBook Pro #

The workhorse of the family. I have the (almost) latest M4 Pro version, and it does everything I need. If there's one device here I would keep, it would be this. It handles basically any task thrown at it: photo editing in Lightroom, building apps, sending tax documents—all on this beast of a machine.

I remember the Intel MacBooks getting so hot on my lap when doing anything intensive. The current MacBooks, though? Truly meant for lap use. I can do hours of intensive work without it ever feeling uncomfortable.

For me, the MacBook is for anything the rest can't do—usually programming or graphics-intensive tasks. I also find some websites work better on the MacBook than the other devices, where the screen real estate gives me a better sense of what I need to do.

So yes, four computers might seem excessive. But each one has carved out its own niche in my daily life, and together they make up a system that just works for how I live and work. Could I pare it down? Probably. Will I? Probably not.

How I use AI

2025-10-13T00:49:31Z

Let's get a few things straight first.

I dislike AI.

There are many rational reasons I can put forward on why, such as its a stochastic word box with an ability to override our judgment with its confident tone. It's a poker machine of banal ideas mixed in from a corpus of open source books and the internet, skimming the words but not understanding the reason. It has an eccentric bubble of overenthusiastic snake oil sellers that seem to associate with it. Last but not least, this part inherently more dangerous than anything else, is that it is indeed extremely useful in a subset of scenarios and initial conditions, which are hard to predict.

Like any big technological or intellectual leap in human understanding or interfacing of the world, AI feels like a big jump from how we used to interact with the simple oracle like text box. We graduated from the simple HTML text <input/> of Google search days to a sparkling text box that does much the same but with less degree of freedom on discernment.

Alas, this post is not about my dislike.

I did not want to be the only luddite left behind, perfectly encapsulating the grumpy old man wishing the halcyon old days of crunching reading 17 stackoverflow threads to get an answer to a simple question should come back. Rather, I wanted to embrace this new interfacing with the world knowledge.

Committed now, I decided to buy a Claude pro subscription as my more AI-inclined tech friends reckoned it was the more elite highbrow option (I do not understand why, yet). After being a few hundreds of dollars poorer, I decided to put it through its paces so to speak and after a few months of using it, here are the things I use AI for.

As a less cumbersome way to google search #

This for situations where I know I will follow the hivemind and don't care about confirmation bias.

These are things like "what's a good butter brand in my local supermarkets?" or "What's the expected split for ETF investment for someone my age and country?" I find Claude's research mode extremely useful as it does offer vetted sources, places where I would have checked myself and more or less come to the same conclusion.

I also use it as a first pass builder for things like travel itineraries or recipes. I like cooking but hate parsing through cookbooks for simple stuff. So sometimes I just ask Claude to give me basic steps for things like churning butter in our stand mixer.

Admittedly, these are all tasks I could do myself, perhaps even better. But, as I said, my error margins for these asks or so high that even if I get put in the hallucinating answer part of the error graph, I will be okay.

As a junior dev with photographic memory #

I have used Claude code. I like Claude code. I love the way the text moves in my terminal and makes pretty little graphics while doing so and says silly little whimsical things like "Photosynthesising" or "Discombobulating". But, here's the thing, it's too confident, and it does not think beyond a layer and half of the problem.

Perhaps it's a prompting issue or in general imprecise instructions, but regardless, I am not baby sitting an "AI" to give me a perfect answer. I could build software before, and I can build software now—the only problem is that now, I also have to read while I write, which is not sustainable for me.

So for me, I have settled on the following pattern of working with it:

Make it write tests after I have set up the patterns in the codebase. It still sometimes spits our absolute garbage, but saves me maybe 10 mins of typing worst case, which I still consider a win as I can fire this off async in between doing other things.
Make it explain a class of bugs. There's a class of errors that are simple misconfigurations or too much or too little boilerplate. For this class, I ask the AI to try and fix it and I find that it generally works really well. For example, add the right config to connect to Postgres for this library in this YAML file.
Finally, start any new tech. Recently, I have been experimenting with React native for some non-important work, and I get Claude to setup the project with best practices for me. These are usually beyond one-shot style tasks, so, I ask claude to setup a task plan and then get me to review it, before following the plan to do the thing.

Most people telling you that AI can do more than this, are perhaps a degree or two separation away from what real software really is or are prompt gods or have an ulterior motive, possibly all three combined.

As a technical assistant #

One of the things I have recently started getting into is homelab'ing. I have a setup at home with a BeeLink CPU and a QNAP NAS. I run portainer to manage containers which range from media management and handling to productivity software.

One of the more cumbersome things of setting up a homelab is that you have to read the documentation of a dozen of different apps and docker containers, figuring out the right folder, networking, permission and metadata structure. This process can get extremely tedious.

I use claude to do the dirty work for me. I heavily use the Claude projects feature here as it lets me compartmentalize the work, and I constantly feed it text and reference documents to keep track of.

This works really well for me, most times. Sometimes it forgets the instructions I just gave it, such as using a particular group id for future docker compose files, or it hallucinates a docker image that just does not exist (and never did).

But more or less it works really well, and I can smash out running services quickly whenever I feel like it without spending a whole evening reading getting started pages or blogs on how to do things.

In the end, I think I get good use from the AI. I'm still not convinced the pro sub was worth it, as there seem to be a few VC-funded AI apps out there including the unfortunately ubiquitous ChatGPT. However, it has given me enough value to not make me completely discount it.

I think AI is a great technological leap if we separate the hype from the truth and use it for what it makes sense -- accelerating low-stakes mechanical pieces of work and using it as a way to lower the inertia to start more critical pieces of work.

My Backbone Apps

2025-02-09T07:04:28Z

It’s been a while since I have posted here, so today I wanted to share a bit about one of my favourite past time: productivity apps. Over the years, I have tried in some way or the almost all the popular (and sometimes not so popular apps). Through those years, I have gained an insight into, not only into the kind of apps that work with my particular bend of grey matter, but also the ones that are most likely to stick with me even if they don’t offer me the latest and greatest in features and UX.

Does this mean they will stick with me forever? Probably not! I love redefining what personal and professional productivity means to me relatively often (but not as often as I used to) and that means there’s a chance that this list will be out of date in a year. However, as I’m getting older, I am less inclined to shake up my anchor digital systems, so who knows — this might be the end game for a while.

Let’s dive into the system.

Things #

If there’s one app in my arsenal that has stood the test of time, jobs, personal milestones and innumerable other changes, that would be Things. This app is an indispensable tool and has been a constant companion for me almost 7 years. It’s the app that runs my life and is also the app, that I am most sensitive to missing features from when I switch to one of the other competitors. No matter how much the more shiny features in those other apps entice me, I always come back to Things, just because my brain works the fastest within the apps buttery smooth interface and strict bounds of structuring work.

Over the years I have perfected my own system in Things, that works exactly like I want it to. It’s a system that I am hard pressed to replicate anywhere else other than the guilded cage of Things elegant GTD system and UX.

If I had to pick just one app from this list as my forever productivity, it will unequivocally be Things.

My current Things 3 Setup

Obsidian #

Obsidian is the app where I come to think. Frankly, when I first tried Obsidian almost 2 years ago, I found the app ugly. However, almost a year ago, I was managing a lot of projects both personal and professional. Up until then I had been satisfied with Bear (especially the second version), which had been an mainstay app for managing my knowledge for a while - but, I seemed to keep hitting the limits of the app frequently. I realised I needed more degrees of freedom with my notes to allow lateral thinking and I decide to give Obsidian a go. To my surprise, not only had the app improved by leaps and bound, the sync, plugins and overall look and feel that could be customised to look quite nice now, made me really get back into it.

I haven't looked back since. I don't follow any PARAs or a million other Obsidian guru's selling short courses on how to organise your "PKM". I simply dump my notes, tag when I feel like, file them in folders when I feel like it. The paid sync service has been flawless for me and I do all my thinking, archiving and most importantly writing in it now (including this post).

I still like Bear and I think the app has a lot of potential; but at this point of my personal and professional life, obsidian fits me better.

A screenshot of my obsidian

Fantastical #

The flip side to Things, Fantastical is the place where I commit to things. I have been a long time calendar user, getting super into it during my uni days. I have continued using calendars as immutable event ledger of my life since then. Every important commitment makes its way back into my calendar. I used and enjoyed Apple calendar for quite some time, until I finally pulled the plug and got Fantastical around 3 years ago and in that time it has become a core part of my daily workflow.

I enjoy using the fluid NLP to add events, switch between calendar sets using keyboard shortcuts, interesting calendars (i subscribe to the F1 calendar and Aussie holidays). I also love that it syncs with my work google workspace account, fastmail calendar and our shared family iCloud calendar in an instant and just always works. I can't count the number of times Apple calendar has missed an appointment request via google workspace, completely lost an event or just simply decided to not alert me for an event change.

I don't subscribe to the school of tasks and events in the same app. I prefer to keep the separate. My tasks tell me what to do, but they're are more or less fungible. But my events are not. It's very rare that I put something in my calendar that is fungible.

A screenshot of Fantastical (note I have the personal calendar set for obvious privacy and non disclosure reasons)

Devonthink #

This has been a blackhorse entry into my workflow over the last few months. Initially, I hesitated mentioning it in this post as I do think it sits in a separate island to the apps above. However, while the past apps are my golden trifecta; this is such an indispensable tool for me, that it's worth a mention.

Devonthink is many things (far more than I probably will ever use it for), but I essentially use it as my paperless office + research vault. I have a few databases for personal and financial documents. This is where anything that is saved to my desktop or scanner folder in iCloud gets auto moved to (thanks to Hazel and Devonthinks ctrl+c).

I also use the companion Devonthink to Go app for iPhone and iPad; and it has already saved me from going home and fetching documents a few times. There are two main features of devonthink that are core to my workflow now:

Auto moving files to the right database and folder, based on it learning my behaviours for previous files
Being a really good pdf reader and highlighter/note taker on my iPad - which I use to read research papers and pdf ebooks

It's an expensive one time purchase app, but I find it's good value for me. Might not be for everyone.

Current Devonthink setup with databases

Overall, super happy with this setup for now, as I find it help me stay productive; but also allows me the freedom to flex hidden or more advanced capabilities of these apps when the need calls for.

I plan to do an individual in depth workflow posts for each of these apps in the near future, detailing my own workflow.

The problem with VC model

2024-11-09T06:59:14Z

The core problem with VC funding model is that they want you to pay for the failures of their 10 other ventures.

I have been thinking about this for a while and am writing down my thoughts from my lens. Most VC-funded or run businesses are not primarily running to make the consumer's lives better in a sustainable way. It does happen, but it is a side effect rather than the core goal. Regular lending and or bootstrapping does not have this issue. This is the biggest problem for the majority of startups and companies these days. These sorts of companies have to raise a lot of money and entice people into their ecosystem with free or almost free offerings, hoping to lure in solo/team users. They also sweeten the pot with a pleasing user experience, excellent onboarding, and other little delighter features that make people stay with them.

To be honest, I see nothing wrong with this approach in principle. It's a viable strategy to building a business. However, back in the 'olden days ', which I would roughly define as the pre-VC era, most companies started with an ambition to keep it running sustainably as long as possible and weren't tainted with the VC money printing machine ideology. Or that's how I anecdotally feel the difference is.

These days, most tech businesses start with the expectation to monetize as quickly as possible. This often leads them to fall into the trap of maximizing all the usual signalling metrics for sale, such as user count, retention, and media hype. Once a critical threshold of market and hype mindshare is reached, they pivot to some exorbitant usage fee, especially concerning their added value. Once the business reaches this point, two things happen:

Either the product is kept alive to generate more revenue till the VC overlords can then sell to another company or even different VC overlords and make the founders (and sometimes the founding team) a fuckload of money
Or, the product plateaus and the team tries to find a way to extract as much money as possible till it changes hands to a private equity firm.

Personally, I find both of these approaches distasteful. The software and businesses I value the most are those that are value-driven, with a strong product-market fit and a desire to be in it for the long haul. They're not pressured to fail fast and fail hard, burning stacks of cash in the process to quickly build market share by adding tons of features. In my opinion, slow and steady wins the trust race and stands the test of time.

Balancing strategic vs tactical work in software engineering

2024-05-19T00:42:31Z

I have been thinking about a sapling of an idea for a while now. It all started when I was helping my team at work with several important projects over the past few months. The basic idea I’ve been playing with is that in software engineering, there is a tension between things that are important in the short term and things that are important in the long term. Switching between these two levels of mental context can be (or was for me) challenging. For the sake of simplicity, I call the work to be done for the longer-term “strategic” work and the work to be done for the short-term “tactical” work.

ICs that have progressed far enough in their career are always making a tradeoff between these two abstract levels of work. This results in a single day from dealing with strategic concerns such as product direction and architectural fitness to tactical concerns such as our deployment pipeline can’t handle integration tests with a new database. Consequently, one might be in a whirlwind of context shifts daily. There is a similar enough parallel here to the now famous PG Essay on Maker vs Manager Schedule ^[1].

However, whilst it’s easy to know the difference between manager vs maker schedule and work around it, this might not always be the case between strategic and tactical work. For example, on a given heavy meetings day I might have a calendar that looks like this:

Busy meeting day

and I instantly knew today would be running on a manager’s schedule. Usually, I can anticipate well in advance (I say usually because sometimes things happen, like an incident at work, and the fallout can throw a spanner in everything) and plan around it. I can bunch up admin tasks in between the meetings or small coding tasks like updating a yml file to use a more efficient AWS instance in a region on prod — things that require minimal context loading in my brain and are on auto-pilot for me.

Similarly, during my weekly review, I can identify my core “maker” schedule days, and depending on my top tasks for the day/week, I can schedule things to maximise context and focus.

Now the problem arises within the context of these days and it doesn’t matter whether it’s on a manager or a maker day. I have to be switched on and go from “stratosphere to mantle” on different topics.

Easy to differentiate and control manager and maker days, hard to do so with strategic and tactical context of thinking

There might be a meeting about our next quarter’s engineering planning where I need to zoom out and look at projects to deliver a coherent vision as Lego blocks. But then there might be a meeting right after that might pull me into the weeds of why the calls on a specific REST endpoint run slower when the same call manages a decent response latency when made via graphql.

In the same way, on my maker days, I might have to think between coming up with answers for a system-wide integration of a new greenfield service and figuring out why the headers sent by service X are not being parsed by the service(s) we run and manage.

Unfortunately, there are no concrete rules to follow (unlike maker vs manager schedule) to allow an easy transition between these two states. And I am trying to figure this out myself.

That said, a few techniques have helped me here in the past. These include:

Context #

Have “context docs” running for everything. This is my top tip. You might be better than me and be able to hold a lot of context, but my brain is a bootleg NVRAM, and not much survives soft resets (a soft reset happens anytime when I have looked away from the problem for > 20 minutes) here. So, I keep context logs of everything so I can quickly return to them when necessary. It can be as simple as a piece of paper or as complicated as an obsidian database, whatever floats your boat.

Morning vs evening blocks #

Even within strategic and tactical work, it can be divided between doing discovery work vs known work. Discovery work involves figuring stuff out before you can put the approach or implementation on paper. Whereas known stuff is what you can derive from your existing knowledge. When possible, I keep any discovery-based strategic or tactical work in the morning vs keeping known work in the evening. I find mornings work better because I am open to more ideas and can spin my gears better. This might be different for you, so do what works. The core idea is to keep discovery-based S/T work at the time when you find most of your creative juices flowing. Planning is critical here.

Batch, batch, batch #

This is an obvious one, but if possible, batch Strategic or Tactical work. This is not quickly done, but based on your specific role or job profile, batching work, similar to a maker vs. manager schedule, can reap huge benefits.

Acceptance #

Finally, but perhaps most importantly, be okay with being interrupted. I think the biggest flaw in over-optimising this flow (which I have done to myself a few times as well) is to over-index too hard on when you need to be doing strategy vs when you need to be deep in the code trenches. Sometimes, shit just happens and you gotta roll with it. Keep that perspective.

https://paulgraham.com/makersschedule.html ↩︎

Circuit Breakers For Your Life

2024-05-06T10:09:31Z

In today’s fast-paced world, we often find ourselves juggling multiple tasks and responsibilities, leaving us feeling overwhelmed and exhausted. Just like electrical circuits have circuit breakers to prevent overloading, we too need a similar mechanism to prevent ourselves from being overwhelmed.

Circuit breakers can be small or large, depending on the degree of overwhelmingness we are experiencing. For different people, these circuit breakers can take different forms. It could be something that tells us that we need to take a break, slow down, and prioritize our well-being.

For me personally, it’s a combination of feedback from my wife and my own mental state that serves as a circuit breaker. If I find myself dreading going to work, or if my wife tells me that I’ve been looking down and stressed, I know it’s time to take a break and reassess my priorities. It could mean taking a few days off, planning a holiday, or simply slowing down.

Just like electrical circuits, our lives can benefit from circuit breakers. They help us prevent burnout, keep us operating at peak capacity, and allow us to take stock of our priorities. However, it’s essential to use them wisely. Too many circuit breakers in quick succession may indicate that we need to reevaluate our current priorities and regain our focus.

To end this, having circuit breakers in our lives is vital to prevent ourselves from being overwhelmed. It’s crucial to take breaks, slow down, and prioritize our well-being to operate at our best capacity. So, the next time you find yourself feeling overwhelmed, take a step back, and find your circuit breaker to recharge and regain your focus.

Fix Git Branch Prefix Uppercasing

2024-01-29T02:59:31Z

Context #

Sometimes git gets confused and uppercases the branch names. This happened to me today while I was working on a bug and on git checkout issue/* prefix kept getting renamed to ISSUE/* and this usually breaks the auto build mechanism at work for me, since we have it configured to look for a particular regex. This is not a particularly complex or even a unique problem, but I encounter it so infrequently (second time now), that I thought its worth just writing about even if for my own reference.

How to fix #

First step is to fix the refs. This is taken from this thread:

cd .git/refs/heads
mv ISSUE issue

the command above should fix the issue ref (folder) name so new issues don’t have the uppercased prefix.

Now, we tackle existing branches. If a branch has been created but not pushed do:

git branch -m <old_name> <new_name>

My git complained that there was already a branch with an existing name when all i did was change the branch name from ISSUE/existing-branch-name to issue/existing-branch-name so I renamed it as issue/existing-branch-name-2 (same name but with a prefix).

however, if a branch has been pushed, after the previous step do the following:

git push origin -d <old_name>
git push origin <new_name>

this will fix up any broken branch names already pushed, whilst preserving the changes.

That should fix it up nicely ✅

Make Next DNS work with Arc Browser

2024-01-29T02:59:31Z

I recently jumped on the bandwagon (again) for Arc, as I'm really not feeling Chrome's changes to shovel more ads down users throats.

One of the little pieces of set and forget software that I really like is NextDNS. I have been using them for coming up to 3 years now. The way I use it on my Mac is via profiles, which means it should be available to any app that uses system dns. Chrome doesn't follow system dns but the setting is easily avaiable in preference. However, I couldn't find anything that allowed me to change this setting readily in Arc, since their settings panel is kinda sparse. which resulted in:

However, you can override the macos system dns settings to make it work. Just run:

defaults write company.thebrowser.Browser BuiltInDnsClientEnabled -boolean false

and just like that, my boi is back again

Contributor vs Maintainer Mentality

2023-07-16T03:41:31Z

One of the main things I have been chewing on over the last few weeks is the quality of technical decisions made by different people working on a shared codebase. The thoughts then led me to think about the contributor vs maintainer mentality.

Although the terminology has been used extensively in the context of open source software (OSS) ^[1] - the writing here is more focused in terms of proprietary software written in SaaS-based companies (both startup and enterprise). This is because, unlike OSS, where there are one or few people that own and guide the repo, in software-based companies, that's different. In these companies, you'll have a team or a set of teams owning a service domain, which could be powered by one more service. Hence, the distinction becomes more nuanced and less obvious than in OSS, where code repositories directly promote the code owner/maintainer vs contributor designations. This in turn, means that the distinction is more of a mentality than designations.

One set of people has the contributor mentality. They see their job as coming in and making changes for a well-defined end goal, i.e., adding more features, fixing bugs, and patching security vulnerabilities. This set of people, however, will only care about moving the needle forwards within their local concern. They might optimise the software just in time to a local maxima that serves their current need but rarely go the extra mile. However, this is not to say that this group of people have the wrong engineering motivations or can't be trusted to help evolve the code. More often than not, people with a contributor mindset are also the "get shit done" types and view software as a means to an end.

The other set of individuals fall in the camp of maintainer mentality. These are the individuals who not only look at the codebase as a means to an end but instead as an evolving machine that needs to be curated and tended to routinely. For many companies, these are individuals in a tech lead or architect role (not always true). This particular set of people is always striving to improve the software's quality which benefits them and anyone else working in the system.

It's hard to say which one is better. The chances of your software lasting over 3-5 years ^[2] without a huge rewrite are slim. Hence, is it worth it for maintainers to expend time and energy in curating the software instead of focusing on a collaborator mentality?

For the most part, and most software (especially enterprise) it pays to be a contributor. Being a maintainer is usually low visibility and low payoff. Your PM does not care that you added architecture unit tests or cleaned up the packages in the repo to improve future code changes. Or they may care, but overall the memory of the effort of these changes is low. However, they notice when you can get that next whizz-bang feature out.

On top of that, as a maintainer, you will most likely have the same amount of work and duties as anyone else, which means you not only have to optimise locally to deliver but also have to constantly use up focus, energy and time to think of how you could evolve the software better. The problem gets exacerbated by the fact more often than not, the kind of changes that help everyone or move the needle on maintaining the software to be more robust is usually medium-big changes that can't be done in a day or two.

However, there are distinct advantages to adopting a maintainer mentality. The chief benefit, in my opinion, is that it's a great way to break through the monotony of the current state of software engineering, which is just yet another flavour of a CRUD app. Unless you're in a true deep engineering discipline (think quantum computing, building machine learning models from scratch etc.). Not only that, being in maintainer mode creates a great intuition for reviewing PRs and system designs, allowing you to spot potential future problems in both without needing to wait for things to play out in production. This will give you the confidence to make technical decisions that don't sacrifice the health of the software or the service to reach a local maxima since it's not a zero-sum game.

To sum up, both choices are good. I think in most instances people can mould their behaviour to either mentality based on the current needs of their productπ and team.

Code Review Principles

2023-04-25T03:55:31Z

I have been thinking a lot about code reviews lately and the impact it can have on delivery for teams. Whilst not a topic that has never been touched before, I found the existing material either was too verbose or too much geared towards how to raise a PR rather than how to review them. In light of that, I have compiled a small set of principles to guide them for myself and I am sharing them here.

Tag them #

In principle, all comments should have a tag. A popular strategy is:

nit ⚠️ - optional, feel free to ignore (e.g. [nit]: there is an extra space between these lines)
followup ♻️ - not optional, let's ticket it, ship and come back (e.g. [followup]: Let's clean this file once xyz team finishes project abc)
blocker 🛑 - fix it right now (e.g. [blocker]: this will make the code here run in O(n^2) instead of O(n) - let's discuss on zoom)

It's important to strike the right balance between code correctness and delivery, otherwise it can lead to unfortunate build - comment - fix - build - comment -build again purgatory. Following this approach not only makes the intention of the comments clearer to the PR owner, it makes you as a reviewer think aware of your comment. In several cases, you may find that the thing you were about to write off as a blocker is more of a follow up and that follow up is more of a nit.

Avoid being vague #

In general avoid making vague and maybe style comments (this is especially true for comments tagged as blocker). If you have an alternative to authors approach mention that, but hand wavy comments help no one. Whilst at it, I have noticed greater engagement when reviewers qualify their comment and think about the pros and cons of their suggestion - it implies that you're not simply ivory towering your way through the review.

Confirm, then suggest #

For most non-trivial suggestions (which should be all of them since we don't like nits so much) try to validate the proposed solution locally before suggesting. This is not a blanket rule and depends a lot on individual comfort with language, framework, tools etc. But, it's usually better to conserve the collective mental brainwidth of the team and wrong solution can end up wasting the reviewer and code owners time. Having a seperate copy of the project checked out and use that for code reviews. This approach simplifies separating dev environment from review environment.=

Nit as an exception #

Nits end up crowding the review. If it's a nit, it should appear if it adds value above and beyond adding it to a linter or the company/team code guidelines page would add. Yet, in saying that, there are grey areas. Not all languages lend themselves to a good automated linter and team guidelines can sometimes go stale. In these situations, having a social contract of sorts within the team where people acknowledge that nits are optional . Instead of adding 20 nits on a PR, grab the person who owns the change and walk them through the team code guidelines and add new ones where they don't exist.

Prior Art #

Feedback Ladders: The Code Review System We Follow at Netlify

The (written) unwritten guide to pull requests - Work Life by Atlassian

How to do a code review

On Note Taking Tools

2022-06-18T15:41:31Z

Update 2023

I have settled back on Bear as I kept coming back to it and the new Bear 2.0 looks amazing 🤩

I have always been a bit of a bike shedder regarding productivity apps, and note-taking apps are probably the most bike-shedded. Personally and professionally, notes have been integral to my learning journey since I was in University when I started to use them thoughtfully.

In this post, I will compare various note-taking tools I have used. I will not list the pros and cons per se, but rather my take based on my use cases. The views are anecdotal, and I wish to evaluate them all subjectively.

Apple Notes #

The humble Apple notes was my first proper foray into a note-taking app that wasn't a bunch of text files synced through my dropbox.

I loved using it. Being a native app by Apple, it was snappy and free. I viewed the lack of too many features and stripped-down formatting as a plus. The table feature was a game-changer for me since I tend to organise quite a few things in a matrix-like format. The collaboration and the recently introduced tags were also excellent, although I had already moved on from Apple notes by the time tags were introduced.

However, as much as I liked using it, I soon ran into limitations with the interface. The search seemed lacking; a simple text search wasn't cutting it for me. As a software engineer, I also store a lot of code snippets in my notes, which Notes does not support. I also am pretty used to the quick formatting offered by markdown-based tools, which in Notes takes several clicks or keyboard shortcuts to remember. The lack of wiki or backlinking is also a big downside for me as it's a vital tool to add structure to my notes sprawl.

Evernote #

I confess that I jumped on the Evernote bandwagon quite late. I tried Evernotes briefly in 2016 and stuck with it for six months. I tried it again in 2021, but this time gave up in less than a day. Both times it failed me on my single most important criterion of note-taking apps: speed.

Writing in Evernotes felt laggy and devoid of any simpleness to it. I also felt that Evernote was trying to be a kitchen sink for everything productivity-related (tasks, calendars, project management etc.) but with a janky UX. I admit this works for many people, and I know colleagues that breeze through their documents in Evernote like it's nobody's business. But, for me, it was a chore.

Bear #

I started using Bear about four years ago, which was love at first sight. It felt like a breath of fresh air compared to anything else I had tried. It hit all my standards, and I was sold on the tag-based organisation, so much so that I can't imagine using anything else to organise my notes anymore.

Another plus for me was the seamless markdown editing experience. It feels well created and gets out of your way when you need it to but is readily available when needed as well. It's easy to tell the amount of love and care that has gone into the editing experience for Bear. Bear also has powerful search macros like @todo to find all notes with open checkboxes and @today to find all notes created today. I tend to use them quite frequently. Although not as powerful as something like Roam Research or Obsidian, the wiki linking was good enough for my use case, which involves structuring notes that have accumulated a lot of cruft - to tell an unbroken story as much as possible.

However, it hasn't always been smooth sailing with Bear. I miss tables that are almost table stakes ;) for most note-taking apps these days. The wiki link, as good as it may be, I wish it had backlinks and, perhaps even better, a graph view similar to Obsidian or LogSeq to understand and consolidate bits of knowledge that get scattered across multiple notes. The app also hasn't seen any significant changes in the time I have been using it. Bear is good if you want a Markdown-based alternative to Apple Notes, but it was not suited to my workflow for anything more complicated than that.

Notion #

I tried using it for the first time three years ago, in 2019. I drank the kool-aid before it became hip at the start of the remote work wave. At that time, I felt like I was doing too much, or maybe Notion was not sure.

Whenever I tried Notion, I kept wasting time trying to work out the compelling features under the hood. I would start with a basic template and then get carried away trying to waste too many hours thinking if what I wanted to put in the tool was a database or not. The constant needing to curate rather than create was a drain. The UX also felt a bit slower than the plain text note-taking, and the no offline access was a pain when I put some camping recipes in it, and I didn't have access to it when I reached the actual campgrounds.

However, having given it a go over the last few months, I can conclusively say Notion is on track to be my second brain for the foreseeable future. The thing where I went wrong with Notion was focussing on form over function. These days I write whatever, wherever in Notion and every two weeks, I review my newly created pages and structure them. I heavily use the database function to track my projects and things such as subscriptions, finances and an upcoming Europe trip. This workflow allows me to get stuff in Notion without worrying about where it needs to go but also provides a framework for patterns to emerge organically.

Obsidian #

I must have been living under a rock since I only learnt about it from a friend who uses it religiously. Notes get stored in plain markdown, which is a plus for longevity, and even though it's an electron app, I never felt it was slow for day-to-day use. The most robust feature for me was the graph view which automatically finds links between notes based on words and phrases written in other notes. This type of Omnidirectional note linking is advantageous if you are practising the Zettlekasen method of note-taking.

I stuck with Obsidian for a while but found it unsuitable for me in a few but essential ways. Firstly, the search was lacklustre and confusing even. Compared to searching in various note-taking apps, the search in Obsidian tends not to find what I'm looking for most of the time. Another drawback for me was viewing inline images. I like having images in my notes as I am a visual learner. However, there's no way to view pictures while editing in Obsidian notes. Last but not least, the mobile app, while well designed to handle the complexity of the problem Obsidian is tackling, seemed cluttered for my day-to-day use and didn't have the seamless feel I had with other markdown-based apps (like Bear).

Craft #

I started using Craft after I got frustrated using Notion. Craft pitches itself as a fast and native alternative to Notion. First impressions were good, it was fast, and the writing felt fluid. It also had tables (hell yeah) and the ability to track Daily notes as an inbuilt feature. I liked that I could also link my calendar and take meeting notes within the same app.

However, it didn't work out for me long term. First, Craft may be fast, but it doesn't match the ease of editing and wrangling text of a simple markdown text editor. The choice of block system (same as the one Notion uses) caused a lot of friction when editing and moving around text which happens a lot when I'm in the first few fluid stages of a new note, especially on mobile. I also found that a big emphasis for the tool was to make text look pretty, which is not a wrong goal, but it seemed to be at odds with how I take notes constantly. I also came spoiled by the tag-based approach to note-taking that started with Bear and not having that shackled my workflow. With all these limitations and a hefty price tag (and probably well-deserved price tag, good native/cloud hybrid apps aren't cheap) - I decided it was not worth it for me as a long-term tool.

Drafts #

I learnt about Drafts through one of those Apple app store stories. After trying it for a few weeks and reading this detailed review, I knew this was it. Drafts now serve as my hub for daily notes, random snippets of text, unformed thought, and anything that's not fully formed yet in my knowledge base. It also has tags that work best for me as an organisation system, and I can quickly jump to different categories of notes. I love the powerful integrations and the uncluttered interface to get some text on the screen as soon as possible.

However, in saying that, it might not be the app for everyone. Drafts also doesn't have a polished look that some others like Craft and Notion have. It can be a bit disconcerting to land on an empty note every time.

Conclusion #

Over the last few months, I have decided that a single app won't work for me for my use cases. So, I have decided to adopt a workflow that involves two different apps. All my thoughts, random todos, small and big ideas and initial inspirations of text land in Draft. It has become my most used app, and for a good reason. It's so simple to defrag my mind and drop everything there. Periodically I look at the notes accumulated in Draft and use the extensions to move data across to Notion or Things (my task management app).

Notion has become the central hub for my PKM system. All my long-term notes and projects live there, and I actively use them to collaborate with my family.

The choice of using two apps helps me optimise and use the benefits of both of them together. Drafts with its quickness and rich sharing support for daily things and Notion with its structure and built-in backlinks, as well as database support for long-term storage.

Daily Routine

2022-01-16T15:41:31Z

Ever since the pandemic threw the world in chaos, I have been consciously thinking about my work and life and how interconnected they have become, partly due to WFH, partly because I am growing old. Pre pandemic, I never really cared much about daily routines, other than basic things, like I need my coffee by 9 am to function, or, I need to go gym atleast twice a week to not feel shit through the week. However, I have come to appreciate the profound power of routine and structure on my overall mental health and thus in turn on my relationships and the work I produce. Here is my current daily routine,

Early Morning (7 - 10 am) #

Wake up
Exercise
Stretch + Meditate
Breakfast + Coffee
Define my intention for the day in the daily log and work on the most important tasks first
Focussed work

I avoid looking at any news or internet in the morning and try to conserve my mental energy for the most important tasks of the day. This usually starts by me quickly taking stock of all my main tasks of the day and then setting an intention to complete the top priority task first thing in the morning on my daily notes. I tend to do atleast 1-2 30 minute focus sessions during this time. I follow pomodoro technique for my focus sessions and usually have slack off till 10am.

Late Morning (10 am-12 pm) #

Focussed work
Read/respond on Slack
Walk
Lunch

This is where I pick up steam on the context gained from morning focus sessions. These may not be the most creative hours for me but I tend to be most output geared during this stretch and it involves either getting clear answers to the tasks I resolved to do in the morning or copius amounts of note taking to make sure I don't miss things. This is also the time when I respond over slack and plan my lunch for the day. If weather permits, I always tend to go for 30 min walk in this stretch.

Afternoon (1p - 5 pm) #

Admin tasks
Meetings
Knock out small dev tasks
Read/response emails
Read/respond slack

This is the time when I can feel the slump slowly creeping in. This is also the block of time when I have the most amount of meetings scheduled. I try to knock out atleast 1 or 2 focus sessions again but it's a rarity. During this time, if I haven't made huge breakthrough on my main task in the morning, depending on the urgency, I will shift gears and start focussing on low hanging fruit style things, for example, fix the documentation here, rework the build file there kind of stuff. I also use this time to go over my emails of the day and read or respond to any pending slack messages.

Evening (5pm - 10 pm) #

Dinner
Spend time with my wife
Personal Project or Reading
Reading news (mostly twitter)

In the evening, I like to spend time doing non work stuff. This time block ins unstructured and I usually let my mood decided what I feel like doing. Some days it's cooking, other days I spend my time watching something on Netflix or listening to a Podcast. I try to priortise majority of this block to do atleast one thing with my wife, whether that be playing a board game or watching a TV show. Cooking together has also been a fun activity to do together. As far as news is concerned, I tend to avoid doomscrolling and have no social media apps installed on my phone. If I need to browse twitter I open the web app on my iPad.

And that's all. Hopefully this gives you some motivation to structure your day as well!

Software Is Not A Race

2021-08-12T08:00:29Z

Too often I see people building software and doing changes in an existing piece of code as if they're texting one...excruciating...line...at...a...time. As if it's important to get one disjointed piece of functionality out and making up what's next on the go.

This is bad. Software like any other piece of well thought out functionality should be deliberate. Don't rush into adding a change because the current circumstances confirm it. Check if the change needs to be done based on the requirements of the system. Confirm it with your teammates. More often than not, you will realise that you have missed a key detail of the system. And that the change can be done in another part of the system or not needed at all.

I have long held the belief that a key and critical milestone in the path of young precocious programmer progressing from making computers go bleep blop to building programming megastructures, is the ability to pause and question, deeply, pragmatically, WHY. It's easy to forget in the pace of building a new systems and delivering rapid value, that the best software is the one that's not needed at all. Each extra line of code in the system brings with itself an extra line to maintain and code reading overhead.

Since most of the code ever written is write once and maintain/read many times - it is imperative to question the need. Each line of code needs to earn its place there and should have a rock solid reason to be there.

There are some good strategies that can be applied to enable well thought out software in the org:

No matter what the change, always communicate the need. Writing a detailed why on pull requests is good enough for trivial changes. For non-trivial changes, write a small RFC or some form of Approach Validation Document ™ and get a consensus sign off from your team or a subset of the team. This adds a small overhead to delivering your work but in the end your team will thank you for it.
Treat your codebase not as a junkyard to throw shit in instead a garden to be cultivated. Encourage this mentality within your team as well.
Hone and cultivate a strong bullshit meter for changes that are done to cover and compensate the deficiencies of the system or the process. Quick hacks to get stuff shipped are okay, living with those hacks without even discussing fixing the problems elsewhere seldom is.

I'd like to end this post with a caveat that there are cases where it's necessary to add quick & dirty hacks to fix production or hustle changes to prod to meet a deadline. Such things are a part and parcel of any engineering org to a certain degree and no one should expect you to write a detailed write up for every single commit you make in these cases. But, recognising the time and place for such changes and balancing that with careful followup is the key to not let the system fall under ever increasing tech debt.

Equality Checks in Kotlin With Unstructured Data

2021-06-17T11:03:44Z

This is a small blog post on equality checks in Kotlin when one of the objects may contain arbitrary data.

Let's say you have an object that looks like:

data class ArbitraryData(val id: UUID, val someData: Map<String, Any>)

where someData is any json map of data that could contain any untyped data at runtime.

Now, if you have two instances of this object with the same data and try to compare the two using the regular equals method, you would get a false result.

arbData1 == arbData2 results in false.

This is due to the default Map equality implementation for Kotlin, basing the check on same references for the objects. Hence, even if you have the someData maps in both objects populated with the same value they will fail.

To get around this little dilemma, you need to override the default equals implementation for ArbData class and manually compare the values.

data class ArbitraryData(val id: UUID, val someData: Map<String, Any>) {
    override fun equals(other: Any?): Boolean {
        if (this === other) return true
        if (other !is ArbitraryData) return false
        if (id != other.id) return false
        return someData.entries.all { it.value.toString() == other.someData[it.key] }
    }
}

Dynamodb Gotchas

2021-05-19T09:56:29Z

This is just a collection of things that required some google searches to resolve while working with DynamoDB at work recently. All the code examples here use Kotlin, though, I'm confident that they would still appear in Java.

Persisting a DynamoDb Object that extends a class #

DynamoDB doesn't automatically work with Abstract classes. For example, let's say you have an abstract class which contains some common fields (such as the hashKey and the range fields),

abstract class AbstractBook @JvmOverloads constructor(
    @DynamoDBHashKey(attributeName = "isbn")
    var isbn: String? = null,

    @DynamoDBTyped(DynamoDBMapperFieldModel.DynamoDBAttributeType.S)
    @DynamoDBRangeKey(attributeName = "author")
    var bookType: String? = null
)

And then you define the actual class, that extends this base class,

class Novella @JvmOverloads constructor(
		isbn: String? = null,
		bookType: String? = null, 
        var content: String? = null
 ) : AbstractBook (isbn, bookType)

When you try to use a DynamoDBMapper to save the object it will throw an exception.

To fix the exception, you'll need to add the @DynamoDBDocument to the AbstractClass to let the mapper know that it's the abstract version of the actual persisted entity. This is what the JavaDoc for the annotation says:

An annotation that marks a class which can be serialised to a DynamoDB document or sub-document. Behaves exactly the same as DynamoDBTable, but without requiring you to specify a tableName.

Which means that this annotation is necessary to serialise objects types that are not directly part of the actual stored object type.

Persisting a List of Objects in DynamoDB #

Another little snowflake behaviour I encountered was when persisting a list of objects. Let's say we go ahead and add some more data to the Novella object, like a List of Publishers.

data class Publishers(
    val name: String,
    val address: String?
)

class Novella @JvmOverloads constructor(
		isbn: String? = null,
		bookType: String? = null,
        var content: String? = null, 
        @DynamoDBTypeConvertedJson 
        val publishers: List<Publishers>? = null
 ) : AbstractBook (isbn, bookType)

@DynamoDBTypeConvertedJson is the annotation DynamoDB recommends for storing objects when using DynamoDBMapper. It has a strange behaviour where it can auto serialise a List<T> but it loses type information on deserialisation and deserialises the object as an amorphous map. Which means you get exceptions like: java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to Publisher.

Based on this thread this has to do with type erasure, wherein T : Object which results in bad behaviour at deserialisation time (the default Jackson marshaller is smart enough during serialisation).

The best way to solve this is to define a custom serialiser and deserialiser for your object.

So, in this case it's a matter of defining a class like,

class PublishersMapListConverter : DynamoDBTypeConverter<String, List<Publishers>> {
    private val objectMapper = jacksonObjectMapper()

    override fun convert(publishers: List<Publishers>): String {
        return objectMapper.writeValueAsString(publishers)
    }

    override fun unconvert(publishers: String): List<Publishers> {
        val type = object : TypeReference<List<Publishers>>() {}
        return objectMapper.readValue(publishers, type)
    }
}

And then adding the @DynamoDBTypeConverted(converter = PublishersMapListConverter::class),

class Novella @JvmOverloads constructor(
        isbn: String? = null,
        bookType: String? = null,
        var content: String? = null,
        @DynamoDBTypeConverted(converter = PublishersMapListConverter::class)
        val publishers: List<Publishers>? = null
) : AbstractBook(isbn, bookType)

Hopefully this post helps someone else save some time as well.

Building a Reactive Oauth Client App with SpringBoot and Kotlin Coroutines

2020-06-09T10:40:49Z

Background #

In this post, I’ll walk through two new exciting things that have happened in the spring ecosystem recently. First is spring webflux support for kotlin couroutines and second is the overhaul of spring security, and the addition of the out of the box oauth2 client support for social logins.

For me one of the most impactful new features is the integration with spring coroutines throw the Flow primitive. The integration makes writing reactive code a lot more straightforward - no more subscribeOn and observeOn operations. Instead of thinking in terms of Mono and Flux based primitives borrowed from reactor, it allows a straightforward way to generate cold streams instead. Kotlin flow allows us to write purely asynchronous code in an sequential/imperative manner - which in turn means an existing codebase using blocking code can be converted to use the non-reactive paradigm a lot easier. Another advantage of using coroutines based non-blocking code is that coroutines have been implemented as lightweight threads whereas schedulers with reactor can incur a lot of performance overhead as they have to context switch between threads. As we progress forward in this post, people who are already familiar with composing non-reactive code in webflux will see what I mean.

Let's get started! #

To demonstrate the way this works, we will try to develop an asynchronous api that connects to a GitHub account and gets all the starred repos for a user. The api will use spring webflux with Kotlin Flow and we will integrate that with the spring oauth2 client to ensure the user is logged in.

To start off, let’s create a new gradle project in whatever IDE you prefer (I use Intellij IDEA) and add the following set of dependencies:

implementation("org.springframework.boot:spring-boot-starter-webflux")
implementation("org.springframework.boot:spring-boot-starter-security")
implementation("org.springframework.security:spring-security-oauth2-client")
implementation("org.springframework.security:spring-security-oauth2-jose")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
implementation("io.projectreactor.kotlin:reactor-kotlin-extensions")
implementation("org.jetbrains.kotlin:kotlin-reflect")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-reactor")

Once setup, lets start building our controller first, creating a new file Router.kt

@RestController
class Router {

    @Bean
    fun routes(handler: Handler) = coRouter {
        "/".nest {
            GET("", handler::getStarredRepos)
        }
    }
}

As you can see we have simply used the normal way of defining a new spring based REST api controller but instead of using the standard router annotations we took the approach of using the coRouter. It’s more of a personal preference since it allows me to see my whole api surface area in a more concise format. However, you could easily replace it with the more common Spring MVC style annotation based approach.

Next we will build a handler that let’s us handle our service response.

@Component
class Handler(val service: Service) {

    suspend fun getStarredRepos(req: ServerRequest): ServerResponse {

        return ok().bodyAndAwait(service.getUserStarredRepos())
    }
}

Nothing special about the handler except two main distinctions when compared to equivalent reactor based approach. First is that we have made the function getStarredRepos() a suspending function - which means it tells the compiler that this code will be run inside a couroutines context. The second interesting thing to note is theat we used a bodyAndAwait() instead of body() method for a server response. This extension allows us to correctly get the body from suspending couroutine context while the service is generating a response without blocking the calling thread.

Now let’s jump into the service code - we will use the new spring Oauth2 client with comes with first class webflux support to make our app support the GitHub Oauth login.

To begin with, register a new app in Github. Once done, add a new application.yml file and add your oauth details like:

spring:
  security:
    oauth2:
      client:
        registration:
          github:
            clientId: <add-github-client-id-here>
            clientSecret: <add-github-client-secret-here>

This tells the Oauth client what credentials to use when redirecting a non authenticated user to Github authentication page. The default authorization callback url setup by the spring is: http://localhost:8080/login/oauth2/code/github .In and of itself all oauth client will do is grab the github authorization grant and store that data in a JSESSION cookie. If that's all you want to do then that's fine - however, I would also like to show the user their starred github repositories. To achieve that we need to be able to somehow get the access token and make an authenticated call on behalf of the user. Spring provides a nice and secure interface to achieve that as well - enter the authenticated webclient.

Webclient is the reactive counterpart of the old and trusty RestTemplate from the Spring MVC days introduced in Spring webflux. It allows us to make calls to APIs in a non-blocking api and comes with nice composition and testing support. In our case, we will build a spring config that will populate the current context with an authenticated webclient for the current logged in user.

@Configuration
class GithubWebClientConfig {

    @Bean
    fun webClient(clientRegistrations: ReactiveClientRegistrationRepository?,
                  authorizedClients: ServerOAuth2AuthorizedClientRepository?): WebClient? {
        val oauth = ServerOAuth2AuthorizedClientExchangeFilterFunction(clientRegistrations, authorizedClients)
        oauth.setDefaultOAuth2AuthorizedClient(true)
        return WebClient.builder()
            .filter(oauth)
            .build()
    }
}

The key to setting the right authentication credentials is the line oauth.setDefaultOAuth2AuthorizedClient(true) which allows the github client bean to be automatically authenticated with the current users github tokens, simple and delightful.

Now we'll go ahead and wire up the last piece of this puzzle which is the service layer that allows us to tie the authenticated webclient with an API call. So let's go ahead and do that:

@Service
class Service(val client: WebClient) {

    suspend fun getUserStarredRepos(): Flow<String> {
        return client
            .get()
            .uri("https://api.github.com/user/starred?page=1")
            .retrieve()
            .bodyToFlow()
    }
}

To explain a bit of what's going in this code snippet, we have created another suspendable function getUserStaarredRepos() that retrieves the starred repos from github and uses our pre authenticated oauth webclient to do so. One callout here would be the return type which is of the type Flow<String> . This possible due to Kotlins extension based programming approach where the coroutines team have created an extension for the webclient which is functionally equal to reactor's bodyToFlux() method.

That concludes our build for this app. To test, let's fire it up using the ever helpful gradle bootRun command (you should run this in your terminal with the current directory set to this sample project root):

./gradlew bootRun

Once done, this will start up our Springboot app and you should be able to navigate to localhost:8080 to see it in action.

Comparing with non-reactive flow #

Thanks to clojj the example project for this post has been updated. It now supports demonstrating how the HTTP responses work for Flow type vs sending a normal response. To demonstrate, we now have two endpoints that return the same data as the endpoint described originally in this post but have been segregated into a non-blocking vs blocking function. Our Router.kt file now looks like:

@RestController
class Router {

    @Bean
    fun routes(handler: Handler) = coRouter {
        "/".nest {
            GET("flow", handler::getStarredReposAsFlow)
            GET("list", handler::getStarredReposAsList)
        }
    }
}

where we now have a getStarredReposAsList endpoint to demonstrate sending the data as List. Consequently, the handler and service now look like,

Handler.kt

@Component
class Handler(val service: Service) {

    suspend fun getStarredReposAsFlow(req: ServerRequest): ServerResponse {
        val starredRepos = service.getUserStarredReposAsFlow()
        return ok().bodyAndAwait(starredRepos)
    }

    suspend fun getStarredReposAsList(req: ServerRequest): ServerResponse {
        val starredRepos = service.getUserStarredReposAsList()
        return ok().bodyValueAndAwait(starredRepos)
    }
}

Service.kt

@Service
class Service(val client: WebClient) {

    suspend fun getUserStarredReposAsFlow(): Flow<Repository> {
        return client
            .get()
            .uri("https://api.github.com/user/starred?page=1")
            .retrieve()
            .bodyToFlow()
    }

    suspend fun getUserStarredReposAsList(): List<Repository> {
        return client
            .get()
            .uri("https://api.github.com/user/starred?page=1")
            .retrieve()
            .awaitBody()
    }
}

As expected, the Flow based non-blocking endpoint does not know what the full size of the response will be so it adds the transfer-encoding: chunked header. On the other hand, the blocking type endpoint which returns a list directly knows the response size beforehand, so it adds the Content-Length header letting the client know the size of the payload being received straightaway.

Depending on the specs of your application and how you want the browser to behave it is probably worthwhile understanding which of these two http responses suit your usecase.

Conclusion #

As someone who moved from springboot with spring mvc building blocking code to using spring webflux and spending hours to understand the non-blocking paradigm - the Flow support in webflux is a welcome evolution. The potential of migrating existing codebases to non-blocking stream based paradigm has been greatly amplified, and I can already see the impact it is having in making my own side projects a lot easier to build and reason about!

If you'd like to experiment yourself the source code for this post can be found on github.

References: #

https://medium.com/better-programming/asynchronous-data-loading-with-new-kotlin-flow-233f85ae1d8b
https://spring.io/blog/2019/04/12/going-reactive-with-spring-coroutines-and-kotlin-flow
https://medium.com/@elizarov/execution-context-of-kotlin-flows-b8c151c9309b
https://todd.ginsberg.com/post/springboot-reactive-kotlin-coroutines/
https://spring.io/guides/tutorials/spring-boot-oauth2/
https://spring.io/guides/tutorials/spring-boot-oauth2/#github-register-application
https://www.httpwatch.com/httpgallery/chunked/

Right Way To Shadow Jar When Using Jetty With Http4k

2020-04-18T03:10:59Z

If you’ve been spring boot for a while, you’re probably familiar with Spring’s bootJar functionality that lets you create a new executable “fat jar” with all its dependencies pre defined. However, while I was building a kotlin app using http4k, which doesn’t come with any such built in tooling I had to resort to using the shadowJar plugin to build a fat jar.

However post deploy, the app stopped starting up and I noticed the following error in the logs:

Exception in thread “main” java.lang.ExceptionInInitializerError
	at org.eclipse.jetty.http.MimeTypes$Type.<init>(MimeTypes.java:98)
	at org.eclipse.jetty.http.MimeTypes$Type.<clinit>(MimeTypes.java:56)
	at org.eclipse.jetty.http.MimeTypes.<clinit>(MimeTypes.java:175)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:806)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
	at org.eclipse.jetty.websocket.server.WebSocketHandler.doStart(WebSocketHandler.java:84)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:407)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
	at org.eclipse.jetty.server.Server.doStart(Server.java:371)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	at org.http4k.server.Jetty$toServer$3.start(jetty.kt:33)
	at io.taggit.AppKt.main(App.kt:175)
	at io.taggit.AppKt.main(App.kt)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
	at org.eclipse.jetty.http.PreEncodedHttpField.<clinit>(PreEncodedHttpField.java:68)
	… 20 more

Perplexed, since when I used the same plugin with another one of my projects I had never noticed this error.

After much googling, turns out Jetty bundles HttpField encoders by referencing META-INF/services/org.eclipse.jetty.http.HttpFieldPreEncoder so when shadow jar created the fat jar it didn’t have the data that Jetty was looking for which in turn meant Jetty threw an initialisation exception.

The fix?

Adding the mergeServiceFiles() attribute to the shadowJar task, as per the shadow documentation here.

shadowJar {
    baseName = ‘taggit-api’
    zip64 = true
    mergeServiceFiles()
}

And voila, the service was back up and running again. This is becauase multiple libraries potentially use the same service descriptor files (usually META-INF) and in case of creating fat jars its generally desired to merge the service descriptors to make sure all libraries have their service descriptors loaded at runtime.

P.S. - It might be worth mentioning here as well that the reason I didn’t encounter the same error for one of my other services is due to that project using Netty as the underlying webserver.

Brows3r - Pure Kotlin S3 Browser

2019-07-29T08:52:50Z

Background #

Recently I found myself with some extra development time on hand. Now usually, most of my projects start off as big grand ideas and as soon as I start working on them, I lose steam or life comes in the way and things just sit their eating dust as a private github repo.

However, I stumbled upon Kweb which is a server side rendered web app building library, but written entirely in Kotlin. Kweb provides a nice dsl like interface to build web apps by programmatically defining the html elements for the app instead of writing the html and CSS by hand. For those who don't know, Kotlin provides a really nice way to build type safe declarative builders.

Since it's all created declarative in the kotlin code, we get access coroutines, extensions and all the nice things that make working with Kotlin such a joy. For those of you who have worked with pure code based server side rendered frameworks before, this might remind you of Vaadin which is an industry leader in its space, but there are subtle and not so subtle differences which you can be found on the Kweb FAQS.

Going through their codebase, I though it would be a really good opportunity to try and a build a quick and easy app by just using pure kotlin. Why you ask? Cause it was quick and I could smash it out in a few hours.

Simple Kotlin S3 Client #

To start off, I wrote a basic S3 browsing class, starting off small - I copied the code from AWS examples on creating a new S3 client and then added the ability to search for all public keys in a given bucket and get some metadata and download links:

class S3Client(private val endpoint: String, private val bucketName: String) {


    private val client = AmazonS3ClientBuilder.standard()
        .withPathStyleAccessEnabled(true)
        .withEndpointConfiguration(AwsClientBuilder.EndpointConfiguration(endpoint, "ap-southeast-2"))
        .build()

    fun listAllKeys(): List<S3Data> {
        val req = ListObjectsV2Request().withBucketName(bucketName).withMaxKeys(10)
        val keyList = mutableListOf<S3Data>()
        client.listObjectsV2(req).objectSummaries.forEach {
            keyList.add(S3Data(it.key, "$endpoint/$bucketName/${it.key}", it.size.toString().toDouble() / 1000.0, it.lastModified.toString()))
        }
        return keyList
    }
}

the listAllKeys() functions returns a list of all keys in that given bucket, which I can then map to a custom S3Data class:

data class S3Data(
    val key: String,
    val downloadUrl: String,
    val size: Double,
    val lastModifiedAt: String
)

Building the UI with Kweb #

Once done, I got cracking on the UI interface. I wanted something quick and simple, crude even, just to demonstrate that it all works as proposed. So I setup a container with some form fields, a search button and a table to input the S3 region link and the name of the bucket.

I also needed a table to display all the keys (sans pagination, who builds pagination in PoCs anyway? 🤠). To enable holding the data, I used something called KVAR which is simply a state store used by Kweb to support propagating state changes to the web app via Observer Pattern.

The following code fragment gets the initialises the S3 data Kvar (setup as an empty list initially) - which eventually propagates it to the table:

div(fomantic.ui.main.container).new {
    div(fomantic.ui.vertical.segment).new {
        div(fomantic.ui.header).text("Welcome to S3 Browser 💻")
    }

    val keyData = KVar(emptyList<S3Data>())

    val loader = div(mapOf("class" to "ui active text loader")).addText("Retrieving keys...")
    loader.setAttribute("class", "ui disabled text loader")
    createInputSegment(loader, keyData)
    createKeysTable(keyData)
}

Now, here is where Kwebs deep integration with kotlin really comes in handy, since it allows us to use kotlin coroutines to handle tasks with considerable i/o (such as retrieving data from an AWS bucket). I have recently started using coroutines frequently in production code and I can without doubt say they're the best way to write asynchronous tasks without worrying about threads. The low touch syntax setup and the results are so easy it almost feels like cheating.

The following code fragment uses the S3Client introduced earlier to launch a coroutine and when the user hits the search button, and displays a loading icon until all the data is retrieved or an error is thrown:

private fun ElementCreator<DivElement>.createInputSegment(
    loader: Element,
    keyData: KVar<List<S3Data>>
) {
    div(fomantic.ui.vertical.segment).new {
        div(fomantic.ui.input).new {
            val endpointInput = input(type = InputType.text, placeholder = "Enter S3 Endpoint Url")
            val bucketInput = input(type = InputType.text, placeholder = "Enter S3 Bucket Name")
            button(mapOf("class" to "ui primary button")).text("Search").on.click {
                GlobalScope.launch {
                    loader.setAttribute("class", "ui active text loader")
                    val s3Client =
                        S3Client(endpointInput.getValue().await(), bucketInput.getValue().await())
                    try {
                        keyData.value = s3Client.listAllKeys()
                    } catch (ex: Exception) {
                        p().execute(ERROR_TOAST)
                        loader.setAttribute("class", "ui disabled text loader")
                    }
                    if (keyData.value.isNotEmpty()) {
                        p().execute(SUCCESS_TOAST)
                        loader.setAttribute("class", "ui disabled text loader")
                    }
                }
            }
        }
    }
}

So far so good, now that we have successfully pulled in the data in out Kvar container, we can start rendering a table. Now I also wanted to show a nice little icon to show that the retrieved object was a file - and also allow the ability to click name of the key as a link so the user can download.

Now Kweb as far as I could tell didn't have the ability to specify that via a DSL object, however, it does provide the ability to specify nested HTML inside a table element to add my own custom behaviour.

private fun ElementCreator<DivElement>.createKeysTable(
    keyData: KVar<List<S3Data>>
) {
    table(mapOf("class" to "ui celled striped table")).new {
        thead().new {
            tr().new {
                th().text("Key")
                th().text("File Size (in KB)")
                th().text("Last Modified At")
            }
        }
        tbody().new {
            keyData.map {
                it.forEach {
                    tr().new {
                        td(mapOf("data-lable" to "Key")).innerHTML("<i class=\"file outline icon\"></i> <a target=\"_blank\" href=${it.downloadUrl} download=${it.key}>${it.key}</a>")
                        td(mapOf("data-lable" to "File Size")).text("${it.size} KB")
                        td(mapOf("data-lable" to "Last Modified At")).text(it.lastModifiedAt)

                    }
                }
            }
        }
    }
}

The code above creates a new table and generates a new row in the table for each public key present in the provided bucket. If no data is present, nothing gets rendered.

This probably also the time to give a shoutout to the Kweb creators for an integration with Fomantic UI which comes pre-configured with nice UI elements. Although, the integration doesn't end there and there is a nice APi for anyone to write a new plugin with their favourite UI elements library.

Final Notes #

And that is all that's needed to write a simple S3 browsing web app using Kweb. Here is the app deployed on heroku and the full working code is on github.

Gif Demo:

Youtube Demo:

Test Multipart/Form Data with WebTestClient

2019-04-21T05:52:55Z

Background #

For the past year or so, I have been working extensively with spring, especially spring webflux; building scalable reactive micro services for our customers.

Coming from spring MVC, learning webflux and getting used to reactive programming in general has been a great and worthy learning experience and I highly suggest going through the references section if you haven’t heard of reactive programming and/or have been thinking about giving it a go and don’t know where to start. But essentially reactive programming involves a model of creating, requesting and manipulating data in a controllable (from a consumers perspective) and non-blocking manner.

WebTestClient is a reactive testing high level http client with fluent assertions, packaged in spring web flux. Recently, while integration testing an application that accepted data as multipart/form-data I had to figure out how to test the data effectively using the webtestclient and personally found the lack of comprehensive resources on the internet lacking, so I wrote this blogpost to share my own learnings.

Web Form Testing with Webflux #

Let’s suppose that we’re trying to send the request to fill a form api that accepts a document (image, text, plain binary etc.) and some textual data.

To aid with our example, lets imagine the form is a profile setup for an document share service and takes the following input:

Profile Image (api label: profileImage)
Username (api label: username)
Email (api label: email)
PDF document to share (api label: userDocument)

For us to begin sending the data, we’ll use the spring library called MultipartBodyBuilder which provides a nice api for setting up the body for multipart requests.

To send the first part, the profile image we can set it up as:

val bodyBuilder = MultipartBodyBuilder()

bodyBuilder.part("profileImage", ClassPathResource("test-image.jpg").file.readBytes()).header("Content-Disposition", "form-data; name=profileImage; filename=profile-image.jpg")

To explain a bit about what’s going on there, we’re simply telling the body builder to upload an image found in src/test/resources folder with the name test-image.jpg as the profile image part of this body. The real kicker here is setting up the Header part as that is what’s used by the webtestclient internals (specifically the Synchronoss-nio library which webflux uses internally) to determine the type of form data being sent and how to process it.

Also, note that the real file name that will get uploaded in the web server receiving the request is the profile-image.jpg filename that gets sent as part of the headers,.

Similar to the profile image, we can also send the document part of the whole request payload:

bodyBuilder.part("userDocument", ClassPathResource("user-document.pdf").file.readBytes()).header("Content-Disposition", "form-data; name=userDocument; filename=my-thesis.pdf")

Similar to the previous payload we test the body builder 💪 to read a file in the test resources folder called user-document.pdf as bytes and send the document with the name my-thesis.pdf to the form web api.

As you can already see, compared to some other ways of doing it, such as in this excellent blog , using the MultipartBodyBuilder is rather conveneient.

Now for the last two remaining pieces of the form api, which are usually only plain text, we can set them up as:

bodyBuilder.part("username", "shiveenpandita", MediaType.TEXT_PLAIN).header("Content-Disposition", "form-data; name=username").header("Content-type", "text/plain")

bodyBuilder.part("email", "shiveenpandita@gmail.com", MediaType.TEXT_PLAIN).header("Content-Disposition", "form-data; name=email").header("Content-type", "text/plain")

Woohoo! 🎉 We’ve got all our form fields wired now.

Now to see it all in action and bring it all together, we can simply setup a spring integration test and use our freshly setup body builder as:

@RunWith(SpringRunner::class.java)
@SpringBootTest
@AutoConfigureWebTestClient
class WebClientTest {

    private lateinit var webclient: WebTestClient

    @Test
    fun `test webform api`() {
        val bodyBuilder = MultipartBodyBuilder()

        bodyBuilder.part("profileImage", ClassPathResource("test-image.jpg").file.readBytes()).header("Content-Disposition", "form-data; name=profileImage; filename=profile-image.jpg")

        bodyBuilder.part("userDocument", ClassPathResource("test-document.pdf").file.readBytes()).header("Content-Disposition", "form-data; name=userDocument; filename=my-thesis.pdf")

        bodyBuilder.part("username", "shiveenpandita", MediaType.TEXT_PLAIN).header("Content-Disposition", "form-data; name=username").header("Content-type", "text/plain")

        bodyBuilder.part("email", "shiveenpandita@gmail.com", MediaType.TEXT_PLAIN).header("Content-Disposition", "form-data; name=email").header("Content-type", "text/plain")

        webClient.post()
            .uri("/v1/test-api")
            .contentType(MediaType.MULTIPART_FORM_DATA)
            .body(BodyInserters.fromMultipartData(bodyBuilder.build()))
            .exchange()
            .expectStatus().isOk
    }
}

The above code snippet will successfully send the required data to our test api and the webtestclient asserts that the response is 200 OK.

References #

Simple Oembed Service Using Http4k

2019-03-18T00:27:06Z

Recently on my usual #githunt prowl I came across a new promising http library called http4k. The library is based on the philosophy of Application as Function based on the twitter paper Your Server as a Function and promises a lightweight a server toolkit alongwith a very modular approach to adding functionality on top of the core set of capabilities built in. But the best part of all, http4k is written in pure Kotlin and follows a consistent functional approach in handling http services.

Now, as much as a like the kitchen sink approach of Spring framework in getting an almost enterprise ready service running with a couple of tutorials and minimal effort , I really wanted to try my hands at something lighter and different.

Setting up a micro server with http4k #

Http4k provides a simple yet highly configurable way to setup a microserver using nothing but just the core client and an underlying webserver of choosing.

To start off, create a new base gradle project in Intellij or any IDE of your choosing and add the following dependencies:

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version"
    compile group: "org.http4k", name: "http4k-core", version: "3.113.0"
    compile group: "org.http4k", name: "http4k-server-netty", version: "3.113.0"
}

This will add the base http4k library and the netty server library, which these days is my underlying server of choice when building JVM microservices. However, http4k supports various other server implementation as well such as Jetty, Apache Tomcat etc. and some other prominent jvm web servers.

First thing I tried to do was to experiment setting up a simple http server, this is where http4k is a joy to work with as a library.

Setting up a basic server is fairly simple and requires almost no effort.

fun main() {

    val app = routes(
        "/alive" bind GET to { Response(OK).body("The crew is more kahless now than vogon. biological and tightly dead.") },
        "/api" bind routes(
            "/embedLink" bind GET to { request ->  Response(OK).body(getOembedData(request.query("link")!!))}
        )
    )

    val nettyServer = app.asServer(Netty(9000)).start()
}

The snippet above starts up a basic Netty server on port 9000. The startup time is very small, I calculated less than <2 seconds in my observations but the tests were not rigorous so take it with a grain of salt. Once started, the app will start serving the two routes /alive and /api on the local machine. the alive endpoint is simple to make sure we can ping the app so I;m not going to talk about it anymore. The /api link however has an embedded route to /embedLink which allows us to make GET calls to the server with the query param link which contains the link to the resource we are wanting to get the oembed link for.

So, let's talk a little bit more about how we do that...

Extracting Oembed data #

Now oembed as a protocol is fairly old and many services directly support getting oembed data for the content on their sites using public APIs. One of such sites is Instagram which has a nice api that properly adheres to oembed standards. Without going to deep into the oembed specification (which if you're keen you can read more about here), we'll see how we use the powerful lensing capability of http4k modules to extract oembed data.

The getOembedData() function from the routes code snipper in the previous section can be implemented as:

fun getOembedData(link: String): String {
    val request = Request(Method.GET, "https://api.instagram.com/oembed/?url=$link")

    val client: HttpHandler = JavaHttpClient()

    val igLens = Body.auto<IgOembedResponse>().toLens()

    return igLens.extract(client(request)).html
}

Here we provide the function an actual link to the resource (in this case a link to a post instagram) and use the hardcoded instagram oembed url to grab the oembed html.

To actually parse the returned response from instagram, we use a very nifty feature in http4k called Lenses which allows us to use typesafety while working with the request/response from the http client calls. Lenses are a very powerful feature and allow not only immutable parsing of request/response objects but also use ADTs to parse the responses in an Maybe using functional extension libraries for kotlin such as Arrow.

To actually extract the data, we have to define the type of the returned response first and nothing better to do that, than the ever so simple kotlin data classes:

@JsonIgnoreProperties(ignoreUnknown = true)
data class IgOembedResponse(
    val version: String?,
    val title: String?,
    @JsonAlias("author_name")
    val authorName: String?,
    @JsonAlias("author_url")
    val authorUrl: String?,
    @JsonAlias("author_id")
    val authorId: Long?,
    @JsonAlias("media_id")
    val mediaId: String?,
    @JsonAlias("provider_name")
    val providerName: String?,
    @JsonAlias("provider_url")
    val providerUrl: String?,
    val type: String?,
    val width: Int?,
    val height: Int?,
    val html: String,
    @JsonAlias("thumbnail_url")
    val thumbnailUrl: String?,
    @JsonAlias("thumbnail_width")
    val thumbnailWidth: String?,
    @JsonAlias("thumbnail_height")
    val thumbnailHeight: String?
)

In this example, I've used the @JsonIgnoreProperties and @JsonAlias annotations from jackson which is a well known serialization/deserialization library for java. However, http4k does not limit to just one such library and provides various options such as Gson, moshi etc. as plugins. Just remember to add the library of your choice in your build.gradle.

Finally, going back to our code snippet for getOembedData introduced earlier, we first setup a new client:

val client: HttpHandler = JavaHttpClient()

which gives us a new Java based http client, which is a simple implementation of a simple Request -> Response client as a function introduced in the [previous section](## Setting up a micro server with http4k). All it does is take a request and parses the response as a bytestream.

To parse the bytestream in a typesafe way, we attach it to our lens which contains the data class we just defined:

igLens.extract(client(request))

if the request is successful, we will get an object parsed into the IgOembedResponse type. For our purposes, what we really need is the html, which gives us the full oembed html which we can use in an iframe.

Final Notes #

In the end, I ended up deploying the app on aws lambda by by setting up an API gateway and a lambda function that called the final app. The final cleaned source code for the app is located here and it also contains the instructions on how to call the aws lambda function to get the oembed responses.

All in all, I was pleasantly surprised at how productive I was working with http4k and how easy it is to set up a FAAS type application using kotlin and AWS.

Post Boot Actions Using Application Events

2019-01-10T07:23:18Z

Background #

Recently I faced an interesting problem at work. A production app written in Kotlin with Spring Webflux as the webframework had suddenly started taking more than 189 seconds on average to start (more than 3 minutes 😱).

Now as much as I love spring for the easy to use and battle tested APIs, everyone knows its not the nimblest of frameworks around, but having a boot time of 3 minutes was simply unacceptable. The problem was made even worse by the fact that our production kubernetes cluster, on which the app was deployed, usually checks if the app is alive (a simple alive endpoint on the REST api) and triggers container restart if the app doesn't respond withing a given time threshold. Due the delay in the app boot, the kubernetes kept on assuming the app has not started yet and it triggered a flurry of container restarts - resulting in our very talented and very hard working support team being pinged at ungodly hours. Something needed to be done...

After spending some time analyzing the application, we found that most of the time was being spent in the startup for Elastic Search Transport client, which loads up several plugins, one of which is the Transport Client which relies on the negotiating underlying netty thread which also happens to be the same underlying server that spring webflux uses.

To test the theory that this was the reason the build was slowing down, we commented out the code that builds up the client and noticed that the app was starting in less than 10 seconds now! WOAH! that's a big gain...

Setting up post bootup logic #

Spring always has had events inside the system since the very beginning as a loose way to exchange application context information inside a running spring app and there are variety uses, most frequently being the ability to detect any local or global event change and then associating actions or events that take place post the event taking place.

To solve out problem, we created a class called event listener where all the startup logic was shifted (the code is in Kotlin):

class EventListener {

    private fun expensiveThings() {
//        do expensive intialization steps here
    }

    @EventListener(ApplicationReadyEvent::class)
    fun bootStrap() {
        expensiveThings()
    }
}

The code above is very simple, it sets up a class called event listener which will contain all our spring application event listener logic. We have a function named expensiveThings() , which does something that involves heavy initialization (think network negotiation, I/O bound work etc.) steps and something we want to do pot application startup. Ideal candidates for this scenario might be plugins or utilities that let you interact with third party services like elastic search but that are not absolutely necessary to have when your app is first starting to serve it's controller endpoints. Inside expensiveThings() we can add something with heavy init. Function bootStrap() is where all the magic takes place. We use the spring EventListener which gives us the ability to link any void returning function to an application event, in other words, at runtime spring will proxy the call for that event to be triggered when the registered event is received inside the springboot application context. In our case that is the springs ApplicationReadyEvent which gets sent out the first time a spring boot application fully starts up.

But our job is not done yet, we would like a way to access the initialized variable throughout our application, and we would like to do it in as idiomatic spring way as possible.

To achieve that we will modify our previous code as follows:

@Configuration
class EventListener {

    lateinit var poststartupVar: String

    private fun expensiveThings() {
        poststartupVar = "this is just an example"
    }

    @EventListener(ApplicationReadyEvent::class)
    fun bootStrap() {
        expensiveThings()
    }
}

here we've modified the class to include a lateinit (if you don't know, a lateinit variable in kotlin is simply a variable that doesn't need to be initialized at declaration, this makes the code cleaner but you run the risk of an ugly exception if the variable is still initialized at the time of accessing); which then gets instantiated in expensive things. We've also marked the class as a configuration object, which makes it a candidate for injection autowring througghout your spring managed beans like Components, Services etc.

This should get you up an running with a variable that gets initialized post the ApplicationReadyEvent which, if your application has REST controllers, happens post the app starting to listen on those endpoints.

However, what if this component is a core component and we want to hard exit as soon as possible.

No problem, as demonstrated below:

@Configuration
class EventListener: ApplicationContextAware {

    lateinit var poststartupVar: String

    private var context: ApplicationContext? = null

    private fun expensiveThings() {
        poststartupVar = "this is just an example"
    }

    @EventListener(ApplicationReadyEvent::class)
    fun bootStrap() {
        try {
            expensiveThings()
        } catch (ex: Exception) {
            panic()
        }
    }

    private fun panic() {
        val containerContext = this.context as ConfigurableApplicationContext
        SpringApplication.exit(containerContext)
    }
}

Here we use the help of ApplicationContext to inject the current running springboot application context in the even listener configuration (more details here). In case an exception is encountered in the bootStrap(), the panic() function will be invoked and the spring application will be gracefully exited (the graceful bit depends on your configuration but that's for another blogpost).

Final notes #

A code very similar to that with some business speicifc sauce is what I used to solve the problem we encountered in the previous section and it brought down the application startup time to less than 15 seconds, which was a major win for a critical production application. Utilizing the APIs and utilities provided by Spring out of the box in my opinion provides the easiest and most convenient way to solve most production problems and is a testament to maturity of Spring and its status as one of the top jvm web framework out there.