Grisha Kruglov

When Gradle composite builds don't work, but you really want them

Sat, 19 Nov 2022 11:03:13 +0000

(I wrote this post a while back while still at Mozilla, but I didn’t publish it then. looking at the codebase today, even though some stuff got monorepo’d, this flow is still used as of Nov 2022 for dependencies that are still silo’d in their own repositories)

I work on Firefox for Android (aka Fenix). It has a highly modular architecture - composed out of hundreds of libraries (aka components) that mostly plug-and-play, like a Lego set; albeit, with some distinctively non-Lego-like glue here and there. The Fenix codebase could be thought of that “glue”, to some extent. These different pieces of the system live across several repositories. They have somewhat aligned lifecycles, and the final app is composed out of a set of published, tested-to-work-together versions of these modules.

It all mostly works out, most of the time. This post won’t discuss the trade-offs between a mono repo vs splitting code across multiple repositories. That’s a much larger topic that should touch on organizational dynamics almost more than the technical aspects involved. That hypothetical post will need to start with a definition of Conway’s Law.

Instead, I’ll simply describe how we make the local development experience not-terrible when working across these multiple large gradle projects.

Here is how you’d build Fenix and most of its dependencies split across multiple repositories:

checkout the main repo and the “library” repos you care about
in a Fenix local.properties config file, specify which overarching sets of libraries you’d like to build locally
open Fenix in Android Studio, press ‘play’
done!

This will build everything locally - some libs in Rust, some in Java, some in Kotlin - and tie it all together. If you didn’t specify local versions of some set of libraries, they’ll be pinned to whatever is listed in the regular build config.

Demo

A while back I recorded a quick little demo of this flow, to help out external contributors and encourage internal folks to work across our entire stack. Names I use in the video are Fenix for Firefox for Android, application services are various Rust libs that do auth, storage, sync, etc, and android components are all the Kotlin libraries you’d need to create your own browser.

Gradle Composite Builds?

You’d assume we’re using Gradle Composite Builds for this - and I did try! And perhaps, I should try again. These Composite Builds are simply builds that include other builds - sounds exactly like what we want here. But at the time, something about our projects got Gradle really freaked out - to the point of poisoning the local Android Studio, forcing its victims (aka users) to reinstall the world and bang their heads against nearby walls.

So, if Gralde Composite builds don’t work for you for whatever reason, but you really need to, uhm, compose multiple gradle projects together into a build, what do you do?

Local publication workflow

What you could do manually is fairly simple, but tedious:

run builds of all configured dependencies
publish them locally into a maven repository; don’t forget to bump their versions!
configure your “parent” project - add local maven repository, make sure gradle uses locally published dependencies when available
finally, build the parent project and hope you got all these steps just right.

Nobody’s excited to do this, and when they’re forced to they often mess up something and lose time figuring out why their code isn’t working (when perhaps it’s not even in the build being tested!).

Autopublication workflow

Let’s automate this! Roughly in three steps:

when parent project builds, as a prerequisite Gradle needs to trigger builds/publication of its dependent projects. Simplest way to do that is by “abusing” Gradle’s initialization phase, by adding some code to settings.gradle which will runCmd your dependencies’ publish-if-modified script that lives within a dependent project. Here’s a real example.
create a publish-if-modified flow for dependencies. Simple Gradle task can work here, but they’ll be slow - in case of no modifications, Gradle needs to do a bunch of stuff before it can figure out there’s nothing to do. Instead, let’s pick something we already have and that’s hyper-optimized for checking if a project has been modified - Git! So, use git commands to produce a hash of everything you care about, keep track of it locally, and check if that hash changed between builds. If it did, then build & publish locally. Here’s a python script we wrote for this.
allow dependent projects to change how parent consumes their modules. E.g., we need to make sure that what’s picked up is the latest published versions in a local maven repository. You can do this by prepending mavenLocal() to list of repositories, and switching versions of every relevant module to 0.0.1-+. That last ‘+’ bit tells Gradle to resolve relevant modules to the latest available version, which will happen to have been just published to maven local. Here’s an example gradle script that does this, which you’d then apply during a Configuration Phase of the parent project.

Once you put it all together, the dev experience is actually quite nice. You make a change in some Rust or Kotlin file in some separate repository, click ‘Play’ in Android Studio, and it “just works”. Magic! And if you make no change there’s no perf penalty for your local builds, since git is really good at hashing stuff.

In closing…

I realize that this is basically a giant hack, and if I look at it long enough my head starts to hurt, mostly from disappointment with our build systems. But, at risk of sounding apologetic - this flow works! It took less than a week to figure out/implement, and requires no maintanence. It involves no manual steps. It adds almost no additional overhead to the builds. And most importantly: it saved real engineering teams a lot of time, and enabled team members to work better across the stack, helping us deal with the Conway’s law.

An evening sketch

Thu, 15 Nov 2018 03:31:13 +0000

Burnt out, just depressed, or is this actually just very hard? Maybe that’s stress, there’s been a lot of that lately. I’ve read somewhere that stress is good in moderation.

A good number of evenings during these past years follow the same pattern. It’s probably more of a sketch than a pattern. Imagine a couch, a bed, wondering around the apartment, a numbing headache, a stream of semi-complete phrases popping out of consciousness, physical exhaustion even though there’s been nothing but sitting all day. A day spent solving some problem, and then another, going to meetings about this or that. Trying to figure out how a system someone spent years building works. “Keep at it” seems like the only action for which there’s energy, but brain is too spent anyway. Problems and puzzles never seem to go away, and they’re all consuming. Feeling of mediocrity doesn’t quite leave, either. When faced with infinitity - in complexity, in scope, in requirements, in ability to repeat the same mistakes - who has a chance to avoid that feeling?

A problem is fixed, another comes up. Two years later you give up and start over, only to go through the same thing all over again. It’s comfortable, being back with your old friends; they now speak a different language and promise a brighter future. The game is solving enough of these problems. Winning looks like getting dealt harder problems to play with in front of a larger audience.

Mornings are more varied, from productive to kicking yourself for inability to focus on the simplest of things. Actually, the kicking usually comes in around lunch.

Often I ask myself if the trade offs are worth it, but then it’s easier to just keep going back. My twenties were spent in front of the screen, for the most part. And so were my teens. Weekends turn into a race to get out, escape, climb, bike, hike, ski, anything to break up the routine. There’s even a verb for this: “to recharge”. So this life really is draining you out.

Impacts of a phone addiction

Sat, 31 Mar 2018 07:17:13 +0000

(this was originally written in March of 2018, but I didn’t finish it at the time, and didn’t publish. now in 2022 i stumbled into this post on an old laptop, with no memories of writing it; but the topic is as relevant to me today as it was then. publishing as-found)

As many others, for some time now I’ve struggled with a “phone addiction”. I am becoming increasingly aware of the negative impact daily phone usage has had on me, and just starting to think about longer term, social implications. While this topic has been beaten to death (i.e. this Spiegel article from 2016), my personal experience suggests that the situation is getting worse.

TLDR:

information is highly addictive (and often soothing), especially in bite-sized chunks
frequent interruptions over the long term are harmful to our cognitive abilities
always-connected phones are the ultimate enablers
phone addiction feels like a form of mental illness
human intertia and incentive structures suggest that tech experiences will not change for the better without our active participation
pessimistically extrapolating current trends creates a scary vision of the future

There is an excellent Russian phrase which applies quite well: “the rescue of the drowning is the task of the drowning themselves”. So many of us are drowning in this neverending sea of tweets, instagram posts, comments, headlines, facebook updates. We can’t wait for the tech companies to come to their senses and remove “engagement metrics” from their quarterly goals, change product monetization strategies, etc. The various incentive structures are too powerful, the problems aren’t well understood, and freedom of our minds is on the wrong side of the barricades.

If I’m being optimistic, I can see how our current use of phones will seem just as harmful and unhealthy in the future as smoking cigarettes is starting to seem today. The pessimist in me sees an Orwellian future enabled by Brave-new-World-ish patterns and “manufactured consent.” It’s a slippery slope: an addicted population with diminished cognitive abilities is a prime material for manipulation. Add to the mix rapid iteration, excellent metrics, tools such as A/B testing This kind of manipulation is already happening, both by private companies and state actors; I’m sure what we’re reading about in the news is just a tip of the iceberg.

The culture

Here are some of the more obvious changes I’ve noticed in myself:

it’s much harder to focus
consuming larger chunks of information is more and more difficult
keeping focus for longer periods of time is more difficult
reading literature and long-form content is difficult
I’m often disconnected from what’s going on, from people around me
I’m not “mindful” of “the moment”
information that i am consuming isn’t “sticking” as well, it’s harder to form “webs of knowledge”
it’s getting harder to think through problems that aren’t “tweet-sized”
etc, etc, etc.

While there are a lot of other environmental factors, it certainly feels that phone usage - and in particular, obsessively checking Instagram, Twitter, HackerNews, WhatsApp groups, select subreddits, etc - is one of the main culprits. Oftentimes, spikes in phone usage happen when I’m more stressed or particularly anxious: safe, consistent information intake provides a short-lived relief, an escape, and comes with a brief calming effect.

But there is a dark side here: over time, frequently “tapping” into the phone for a relief creates a different kind of anxiety, with largely misunderstood longer-term implications for our cognitive abilities. I can only observe some of the more obvious changes by comparing how I feel now versus how I rememeber feeling at some point in the past. I’m sure there are more subtle changes.

Comment threads are particularly interesting. They’re highly addictive, as I’m sure a lot of reddit and HN users will attest. Often, I’m choosing what article to read mainly based on how many comments it has (e.g. on HN). I don’t even read the original article, just the comments. This is painfully obvious to anyone who frequents such websites, but it’s helpful to actually spell out. While reading a large comment thread I get a feeling similar to what it feels walking into a loud room with lots of overlapping conversations going on all at once. It’s similarly deafening, except that it’s the outside world that is tuned out. As I’m reading through a thread, people’s voices are reading out these comments - just like a poor, low-resolution version of imagining book characters. It’s not a pleasant experience, it’s been getting worse over time, and yet it’s oddly addicting.

While websites like HN and reddit aren’t just accessible via phones, the always-connected, always-there nature of phones turns them into ultimate enablers. Our tiny little pocket computers allow for unscheduled, frequent, mindless interruptions, during which the mind is washed in a flow of information and thoughts of others, images and projected feelings - all for a brief moment until the phone is tucked away. Rinse and repeat, every 15-20 minutes, every day, for years.

Imagine trying to focus on a task, and then suddenly stepping into a loud room, listening to a few monologues and conversations for a few minutes - all entirely unrelated to what you’re trying to achieve - and then going back to whatever you were focusing on. Your attention span is gone, and context switching will be hard.

While this pattern isn’t new, the latest iteration of information overflow is certainly proving to be the hardest to fight against. Several years ago, I’ve deleted Facebook, which helped a lot at the time. Several years before that, I’ve deleted most of my chat applications (MSN, ICQ), and found that I gained hours of free time in each day. Both of these actions had very concrete social implications: I’ve largely stopped interacting with a number of people. This was a good trade-off, as long as my social network doesn’t disappear entirely.

I do enjoy a lot of benefits of having a smartphone: decent enough camera, instant access to information, communication with my friends and family, keeping a log of what “interesting” things I’ve been up to (via Instagram), mapping applications, podcast app, music, various utilities, backup emergency communication, etc. Quitting cold-turkey feels like a wrong way to approach this problem.

Instead, I’m making subtle changes to what my phone feels like. The goal is to turn it into a tool as opposed to a slot machine”

Luna Mono 2.0 Sandals - four month review

Tue, 05 Sep 2017 03:31:13 +0000

I bought these sandals on May 5th, 2017. This review is written on September 4th, 2017.

Usage

I’ve been wearing these sandals every day. On an average week that includes cycling or walking (~4km) to work, walking around the city, walking in the office to get coffee, and hiking on the weekends. I run in Lunas as well, but I don’t run often - perhaps a couple of times per month, for about 5-10km. I did some multi-day backpacking trips in these sandals as well. The single worst thing I’ve done in them is probably foot breaking on a longboard - but only once, since you can just feel the tread disappearing.

I guess most of the “wear and tear” happens during my hikes. A typical hike would include a day pack and 5-8 hours of walking up and down coastal mountains near Vancouver.

Impressions

Overall, I love these sandals. They are some of the most comfortable footwear I’ve had the pleasure of wearing. That goes for all of the uses I’ve put them through, but they’re especially great while hiking. The tread is quite sticky, and I’ve never felt that they weren’t “enough of a shoe” on any of my hikes. With their current tread (see below) are a bit slippery on really loose and steep surfaces. Soles “molded” after a few weeks, hugging my feet a little bit. Lack of a toe box really helps on long slogs down fire roads, or just on any prolonged downhill sections. In my “regular” hiking shoes, or in VFFs, something would inevitably start to rub, or my toes would hit fronts of the shoe over and over again, to great discomfort. Not so in these sandals - or most other similar ones, I imagine - which is a huge plus. While backpacking, after a few long days of walking through mountains with an overnight pack, I would overuse my ankles to the point of needing a good few days of rest to fully recover. Currently I’m chalking this up to “my feet aren’t strong enough”. Normally I would go on multi-day trips in my regular Merrel Moab hiking boots, and so this is new territory for me.

Lunas came with ankle straps, they call them “tech straps” - you attach them around your ankles for added support. I personally don’t find them very useful, and almost never wear them. Sandals work just as well without these straps on my feet.

These sandals get a lot of attention on the trails. I’m used to people looking at my feet while hiking, since often I’d be either wearing my Fivefingers or going barefoot, and Lunas are no different.

They were not very comfortable at the start - specifically the front strap did irritate me for a couple of weeks. Also, for my taste they’re a tad too thick. However, now that they’ve worn down quite a bit, I find them nicer to walk in.

I often get these sandals wet on the trails - cooling down my feet in a creek (so nice!), crossing muddy sections, etc. They’re not great while wet - comfortable enough to keep going, but my feet do slide around a bit, and I look forward to the moment they dry out (which happens in about 10 minutes on a hot day). Add in a steep trail with a heavy pack on my back, and that sliding starts to become a consideration. I can feel the extra stress on the straps as my feet move while walking up or down, which makes me worried that something might tear or break. There’s no sign of that, but I’ve read stories about Lunas breaking mid-hike. On longer hikes and multi-day trips I always take my VFFs as a backup option.

Wear and tear

Frankly, I’m disappointed. I’ve only had these Lunas for four months, and looking at their tread I don’t think they will last another summer. It’s a tricky design constraint - soles need to be soft enough to be comfortable, to mold to your feet, and to offer a “minimalist” feel, but they also must be durable. It’s hard to say if Luna did a good job combining these two requirements - or rather, if these Vibram soles were the right choice. Other than “tread”, there’s not much to these sandals. Footbed started to peel off a bit (see photos below). The straps and the buckles are as good as new and feel pretty bulletproof - they’re also basic nylon straps, similar to my climbing slings; I don’t anticipate much happening to them.

Daily wear gotchas

Sandals are great for wearing around the city. However, one thing didn’t occur to me four months ago - there’s a lot of dirt in the cities! Vancouver looks and feels clean, but my feet are still dirty at the end of the day. That’s not something I had to be very conscious of before. Sandals need to be washed thoroughly about once a week, otherwise, bad smell starts to become a problem. They’re certainly a bit more fussy than my other shoes in this regard.

Summary

Overall these Lunas are very comfortable sandals, I love the idea of them, but given their current condition I’m projecting that they’re going to be poor value at ~$120CAD. I anticipate retiring them sometime in spring or early summer of 2018, which is about a year after purchasing (I’m not planning to wear them on the cold and rainy winter days). In the meantime, I’m going to look into building similar sandals myself.

For comparison, my 7 or 8 year old hiking fivefingers are still going strong, and I bought them for about the same price. My 5 year old Merrel Moabs goretex boots, which I use in all four seasons, are going strong as well. Both of these shoes are much more complicated, as well. Of course, there’s a matter of “economies of scale”, and various other things at play here - a popular book that we’ve all read, some good marketing…

Regardless, paying so much money for a piece of rubber and some nylon straps feels a bit ridiculous if the whole thing won’t actually last more than a year. The whole affair strikes me as irresponsible. Companies should strive to make products that last as long as possible, and to reduce their impact. Sandals that you can wear for one-and-a-half hiking season are no better than disposable, non-repairable garbage sneakers that are sold in your local mall. Perhaps their minimalism is their saving grace - less material will eventually end up in the landfill.

Here is what my Monos look like after four months of daily wear:

As you can see, the tread is not doing well. Extra wear on the left sandal is from a single longboard ride, stopping a couple of times via a “foot brake”. Little holes are from aggressive, spiky pedals on my mountain bike.

For reference, here is what Luna Mono 2.0 sandals look like when they are brand new (random internet image):

Hollyburn and Strachan lift-assisted ski tour

Fri, 06 Jan 2017 04:54:13 +0000

After a fun little January 1st outing, Stefan and I decided to go exploring Hollyburn/Strachan area a bit more. I’ve bought new downhill boots (Technica Mach1 LV 130), so wanted to see if I can tour in them at all (answer: yes! 11am-3:30pm trip was acceptably comfortable, with excellent support on the way down).

We’ve parked our cars at the overnight Cypress Mountain parking lot, and started to skin up SW face of Hollyburn around 11:30am. We took a bit of a break as we got to Hollyburn shoulder, and then continued onto the peak. We skied down NW, and on the way down slightly veered skier’s left to avoid going down a gully. This got us to the downhill runs. With help of two lifts got to top of Strachan. We skied down SW on the out-of-bounds slope of Strachan which we scoped out earlier from top of Hollyburn, which was great fun. We saw signs of a small slab slide with a defined crown starting on a convex slope, and some snow debry here and there. As I was skiing down, once there was a small sluff slide under me. The slope leads down to a meadow adjacent to Collins downhill run. This was followed by two more out-of-bounds runs, NW from top of Sky Chair through the glades down to Howe Sound Crest Trail, and then following it back to the downhill area at the bottom of Top Gun.

All of the skiing was great: steep, fun, and lots of untouched snow (especially on Hollyburn, lots of out-of-bounds skiers/riders on Strachan). Snow was significantly heavier going NW from Strachan towards HSCT. I had an unpleasant crash there in the woods: my left ski binding detached as I was making a turn on it, and I tumbled down tweaking my shoulder slightly. Perhaps the DIN settings are too permissive? This isn’t the first time it happened, so I might dial it up a notch.

I was back at home in Vancouver’s West End only by 4:30pm due to heavy Lions Gate bridge traffic.

Hot tea break on the shoulder. Look NE towards the peak. It's fun to remember that we skied down these trees just a couple of days ago.

Looking SW, back the way we came.

Our line down Hollyburn is clearly visible on this 82 year old photograph! Source.

Our faint line down Strachan, if I'm reading this (very) old photograph correctly. Source.

Amazing views of the Howe Sound, next to Sky chair, on top of the out-of-bounds forest leading down to HSCT; getting ready to ski down.

Scoping out our future ski adventures? View of the HSCT, St Marks summit, Mt Unnecessary and the Lions.

Hollyburn Mountain ski tour

Mon, 02 Jan 2017 04:54:13 +0000

New year began with a quick trip up and down SW face of Hollyburn Mountain. This was my first self-propelled skiing of the 16/17 season!

We parked at Cypress Mountain’s overnight parking, and started to skin up SW face at 2:30pm. At 3:45pm we were just past Hollyburn shoulder, and a bit later we were at the peak. We got back to the parking lot just as the dusk started to thicken and terrain became hard to make out as you’re skiing in the glades. This was too late for my taste. It was the three of us on the trip, and the most rookie person struggled a bit on the way down through steeper sections and on the top while transitioning their equipment. That’s really no fault of theirs as this was a first trip of the season for everyone, but we should have left way earlier.

Skiing itself was super fun! Lots of soft, deep snow. I wore my old boots (which are very soft and probably a few sizes too large), so I felt a little clumsy but really enjoyed it. We veered too much left as we were going around the hill SW of the Hollyburn shoulder, and had to keep cutting skier’s right afterwards. Next time I’d probably go around the right side and cut left a bit instead.

Skinning up. We'll be skiing down here in a less than two hours.

Looking SW from just above Hollyburn shoulder, back the way we came. You can see how windy it is! English Bay in the distance.

Looking towards the ski slopes.

On peak of Hollyburn.

Sun was setting just before we started to ski down.

Abstract adventures

Sun, 09 Oct 2016 10:09:13 +0000

What is it like to explore and will into existence creations abstract in nature? Explore their shapes, understand the intricate fabric, the coarse skin, the surface tension of the many layers. What is it like to see their colours, know how they feel to the touch, observe their flow through time, know them in the past, present, imagine them in the future?

From musical landscapes to manifestations of software, these worlds exist all around us. They present themselves in familiar shapes, but beyond the familiar are vast adventures, sometimes taken solo, sometimes in small bands. These worlds are there for you to visit and explore. They take shape of vast cities, small villages, countries and continents, creaking cabins, of sand castles. At times they are orderly, easy to grasp and navigate. Some are friendly. Others - chaotic, changing dangerously as they are traversed.

We are both adventurers in these worlds and their creators, often unknowingly stepping in and navigating the mazes, the forests, the multi-dimensional and the bizarre. We wander through the worlds of others, trying to grasp their shapes, find some meaning. We live in these worlds, for days, weeks, months, for longer. We revel in the abstract, finding joy in understanding, in exploration, in the unlimited ability to create. We chart maps to share, to not forget. Sometimes others are invited to come along, to build in this space together, to share the struggle, to help each other. Sometimes you are alone, and that is how it must be.

These spaces we will into being or discover might fill us with pride, or joy, or satisfaction, or fear and confusion, but ultimately they just are, not here for us or to serve a purpose. Their projections might stir reality into action, might cause a light to blink, a plane to fly or a concert hall to grasp. But the abstract exists in the abstract, in the shared consciousness of those who venture out to explore, and inevitably - to get lost.

Joys of work

Sat, 25 Jun 2016 01:40:13 +0000

Why do we work? The first thought that seems to inevitably come to mind is that most people can’t avoid it. Not out of any noble sense, but simply because they can’t afford to not work. Personally, I feel that this view of work is unfortunately perverse. If I was well-off enough to be able to sit around and do whatever I please, surely in time I would start to perform regular activities, make some progress on personal projects, perhaps work on a garden, learn pottery, etc. What would I call my activities, once they start to take shape of something regular and give me a sense of purpose? I might start calling them “my work”, but in a loftier sense, with more air to it - and perhaps foolishly, with more weight and meaning?

When I’m thinking of the word “work” in its day-to-day sense, and how people often perceive it, another word comes to mind - “burden”. Having to work is often viewed as a personal burden. Nothing is absolute, of course - and what one finds to be a burden could still very well provide pleasure. Personally, I’ve observed that pleasure to be two-fold. One side of it is a sense and realization that I’m helping to create a comfortable material existence for whoever is in any way dependent on me. It is very satisfying to see that I’m able to provide for my family, and help out my friends when and if they need help. That’s one aspect, and surely this in itself should be enough to sustain a career. Another side of this is that any type of work could be in many ways interesting - and not just in a sense of solving puzzles or “hard problems”, as computer programming jobs are often pitched to potential employees and students, but in a much broader sense of personal interest.

I used to walk paper routes; the position itself was called “adult paper carrier”, and I imagine it wasn’t that dissimilar from a job of a postal worker. I carried a rather heavy bag or cart of papers - especially heavy on Fridays, with all of the extra advertisements - for around five hours. It was great to be by myself, to think, to listen to music, to walk, and just observe the world around me. I got to do all of these things, and I was getting paid for it! I find that I rarely get to do these things now, or have as much time for them.

Another job I remember fondly was as a house painter. It was physically demanding work, but very satisfying to see old, fainting houses get a new life, a better look. Sometimes the whole neighbourhood would start to look a bit more pleasant. When I pass by one particular house which I painted, I’d often proudly point it out. The work days were long, and I would be very tired in the evenings - but my mind would be fresh. After all, I’ve spent the whole day outdoors, working with my hands, climbing ladders, sitting on roofs! And in the evenings I had all the energy I needed to read, think and pursue more intellectually demanding activities. For example, I would often come home and do programming for fun - I very rarely do that now. In that job, I never had a sense that I needed to compete with someone, or to prove my abilities - I felt free to be myself, and thinking about it now I realize that it was very liberating.

“Flipping burgers” in a fast-food chain was another job I remember in a positive light. It was my very first job in Canada, and the first paying job in my life. I barely spoke English, and at the time I very much wanted to pursue mountain biking. To that end, I borrowed money from my parents to buy a mountain bike, and in the meantime found a job to pay them back. I would go in a few days after school, and work until 11pm or so, when the restaurant would close. I was involved in food preparation, cleaning the kitchen, taking out garbage, etc. It was interesting in a few ways: it forced me improve my English; it gave me a hint of what it’s like to feel financially independent; but I think the most important bit of learning was that once I finished paying back my parents, I largely lost any interest in going to work - it was time to move on.

In my middle school, children would take turns to perform various tasks around the school. If I remember correctly, once a month we would take a few days from our classes and take up positions in the kitchen (cleaning tables, serving food to our classmates), with the janitorial staff (cleaning the school grounds, wiping hallways, etc), doing secretary work, running errands, and so on. I enjoyed it at the time because it gave you a break from sitting in the classes, and now it seems like an immensely valuable educational experience. We all rotated in our duties, so everyone got involved in every type of work. There was a similar setup in the kindergarden, as well - it was possible to bail out of the immensely boring “nap time”, and instead clean the floors. As a result, I loved cleaning floors as a kid. I remember doing that at home, but unfortunately without the impeding doom of boredom of having to nap it wasn’t as thrilling.

Thinking back to these experiences, I see that there was often a sense of personal purpose - I needed to pay back my parents, I needed cash to buy an airplane ticket to see my girlfriend, I needed to pay for college, etc. Another common thread is a joy of working because it saves me from doing other activities, and helps break up the routine.

My first job after university made me absolutely miserable. It paid well, it gave me a chance to be near my girlfriend - in fact, we were in the same city for the first time in years! The job itself was technically interesting, I got to do some decent programming, and contributed to buildind a product - but one I couldn’t care for at all. Having worked throughout university (I lived at home, started programming for money at some point, and then took internships), I didn’t have any real debt at the time, so money wasn’t a great motivator. I also wasn’t supporting anyone other than myself. This job taught me importance of spending my days with at least some sense of purpose. I wasn’t mature enough to realize it at the time, but the alternative was that in time I stopped caring - and that is a dangerous road to take. In that particular case I also started to fall apart emotionally since I couldn’t quite figure out what was going on. In the end I was fired, and that was both a great slap in the face, and a push in the right direction. I now realize it was the best possible outcome; otherwise, learning potential wouldn’t have been the same. Overall this was an interesting experience, and I’m glad I had it early enough.

Two different quotes come to mind [0] when thinking about work in a sense of “burden”. First is Spinoza’s: Emotion, which is suffering, ceases to be suffering as soon as we form a clear and precise picture of it. Another is Nietzsche’s: He who has a ‘why’ can bear any ‘how’. If our work has meaning, as small and personal or as grandiose as it might be, it has full potential to not only be bearable, but also very enjoyable. My own society seems to strive to provide its members with a sense of “default meaning” - I work, I pay taxes, and as a result I contribute to a common goal of building a more comfortable, stable and predictable existence. That’s the promise at least, and it’s easy to see how dissatisfied people get when that promise appears to be broken.

Coming back to the distinction between “work as a burden”, and “work as a lofty pursuit”, perhaps it really isn’t as black and white. After all, most things aren’t. We get to choose our own attitudes, and our own outlook on life and its activities. This is one ultimate freedom we all have, and mindfully applying it in our daily lives is bound to help. Perhaps “burden with a purpose” has a good ring to it?

[0] I’m guilty of lifting these from Viktor Frankl’s most striking book Man’s Search for Meaning. Do yourself a favour and read it!

Easily change SQLite user_version pragma with a hex editor

Fri, 03 Jun 2016 07:17:13 +0000

Do you find yourself trying to change sqlite database file’s user_version back and forth (perhaps, you’re making databases for migration testing… or it’s a boring Wednesday evening, or…), but don’t want to mess around with libraries for such a simple task?

Get out your favourite HEX editor, and open up the SQLite file in question, and look at offset 60. That’s part of database file’s header, and it’s a 4-byte, big-endian number which represents the user_version pragma. Change it (keeping in mind that number you’re seeing is in base 16), save the file, and you’re done.

In the image below I have it set to version 32.

My interview with Mozilla

Mon, 28 Mar 2016 08:13:13 +0000

On January 21st, just past midnight, I filled out a form on careers.mozilla.org describing why I believe I’ll be a good fit for the Firefox for Android team, clicked submit, and went to bed. I woke up to an email from a recruiter at mozilla - they wanted to chat. I’ll summarize the process here if anyone’s curious:

phone screen with a recruiter. a 15-20 minute chat about my experience, and why I’m interested in mozilla. General information about how the process works.
phone interview with team’s manager lasted just over an hour, we’ve talked about my past professional experience, various projects I’ve been involved in. Chatted about mozilla, the team, the work, and why I’m interested in mozilla in particular
homework assignment was the next step. Essentially take a working android app and add some functionality to it. I had about a week to complete it, and was asked not to spend much time on it. It took me a few hours (around 3 iirc), involved doing a few things which I haven’t had to do with Android prior (so lots of googling around) - nothing complicated, it’s just that my android experience at that point was somewhat particular.
engineering interview to discuss the assignment lasted about an hour, and was conducted by one of the team members. We’ve chatted about my design decisions, how I would expand on the solution, how I would approach certain tasks, etc.
on-site interview was conducted in Vancouver. Five one hour interviews, lunch, and a chat afterwards. Everyone was very friendly and supportive, and although I was pretty tired at the end the whole process was pretty fun. Discussions were about my past experience, especially at CommandWear, general software engineering and android questions, tooling, etc. One of the interviews was a live coding session, which at a point included use of a somewhat obscure javascript API that I had to quickly learn - that was good fun, and I’ve only managed to crash the browser once :)
job offer! I received a formal offer on February 24th (so just over a month after applying), and it was signed a few days later.

Generally it really felt that the process, in a way, “sets you up for success” - spoken with a clear selection bias! Chatting at length with my future co-workers was great, certainly a good way to gauge what I’m getting myself into. As I went through the steps above and interacted with more people at the company, it all started to become progressivly more interesting - people are smart, open, problems are interesting and plenty. Mozilla’s openness and commitment to its open source community is certainly unique in the industry, and an intriquing thing to experience. People talked openly about problems they’ve experienced, and various peculiarities of work - no organization is perfect, after all.

In the end though, it’s a bit like a flower that keeps opening up, and then a month later you find yourself coming in for the first day of work :-)

Fully distributed web applications?

Fri, 20 Nov 2015 08:51:00 +0000

Full disclosure: I have almost no idea what I’m talking about :)

But, wouldn’t it be at the very least interesting if web applications that we use day-to-day were entirely distributed, with no centralized, single point of failure?

I’ve been thinking about this for a while - in a very pedestrian sort of way, not even touching pen and pencil to actually draw out some ideas. But, it seems to me that a lot of web applications would benefit a whole lot from a p2p architecture. There are several obvious things to point about the current model that the web seems to be following.

Most web apps that people use day-to-day are ran by private companies
These private companies act as a single point of failure
Interests of these companies (i.e. their strategic direction and how they attempt to monetize their user base) change over time
It’s rare that openness of data is a priority for these companies

As a result, we get walled gardens. We get companies that try their very best to silo data and social graphs, and keep users from leaving. As consumers, even if we’re paying for the product, we’re essentially relying on good will of others to keep our data safe or even keep it from perishing. The old (internet-time-old, that is) saying “if you’re not paying, you’re the product” certainly applies - but I’d argue it goes even further. Even if you’re paying, your data might still be the product. The “you paying” bit is just an icing on the cake for someone. Dear reader, I’m sure you have personal examples of “corporate betrayal”, or really just companies doing whatever they please, with data you produced and that is important to you - because they can, because they had to, whatever it was. Products disappearing, lawsuits lost, personal data going down with them. Sometimes it’s an annoyance, and sometimes you loose something of an intangible, sentimental value in our ever-more-oh-so-digital lives.

The alternative, as I see it, is to recognize the money-driven nature of the vast majority of products that we use daily. It’s to realize the inherent freedom we have - at lease for now. If we, users, truly care about our data and about continuity of service - we must take matter into our own hands. This also seems vitally important for continued freedom and openness of the Internet as a whole - lack of centralized control, distributed nature, ideas that persist beyond an individual - and certainly far beyond reach of any one corporate entity.

I actually need to sit down and start drawing this out. P2P-based web application architectures powered by distributed data structures - what will that look like in practice?

This certainly feels like a worthy cause.

Running Fedora 23 on Macbook Air 5,2 (mid-2012 model)

Mon, 09 Nov 2015 06:31:00 +0000

Update (june 2nd) : so, I couldn’t fix usb issues; it’s probably my hardware though, as the ports could act pretty wonky in OSX as well (although, at least they work). And over time, Fedora has been a tad unstable on the macbook. Overall so far I’ve had much better luck with running Fedora on my work-provided Thinkpad Carbon X1 - after some tweaking, it turned into a great little dev machine.

I finally got around to installing a desktop linux distro on my daily laptop! Last time I’ve used linux full-time was somewhere around 2007-2008, and since then it was all Macbooks and OSX. I briefly mentioned in my post about dealing with “Developer’s Block” that over the years I have felt that not using a Linux desktop environment must have had a certain negative effect on my desire to tinker with stuff and build new things. Fedora 23 was just released, so I figured why not give it a spin!

Installation

Surprisingly very easy! I’ve downloaded the workstation iso, made myself a bootable SD Card using dd command (which took its sweet time, average copying speed was around 300kb/s). I’ve ensured that I had enough free space on the hard drive, and using Disk Utilities shrunk my main OSX partition to free up around 100gb for Fedora. I even installed rEFInd boot manager, but it turned out not to be necessary - Fedora installs its own boot manager, which is “good enough” (although I can’t boot into OSX from it - have to use the Option key right after the “chime”). The rest of the installation was simple - boot into the live environment, and go through the installation wizard. I’ve pointed it to the 100gb “free space” to auto-partition for its needs.

Tweaking

Out of the box, at the first glance, everything seemed to work. WiFi works, screen looks fine, backlight adjusts, keyboard backlight works. Suspend seems to do the correct thing. So far so good!

Once I started to use it more, I’ve realized that there are a few annoyances. The biggest one for me was the behaviour of the Command and Control Keys. After years of OSX, I’m used to using Command key as the Control key - so they needed to be swapped. Thankfully, gnome-tweak-tool made it super easy.

Another thing I really needed to install was Guake. It’s a slide-out, always accessible terminal. I spend a good portion of my time in OSX using TotalTerminal, so this is a direct replacement. So far I’m pretty happy with it!

Problems

gnome Out of the box some of the UI stuff was… wonky. Especially the “overview” mode, or “activities” mode, or whatever that thing is called. But, running dnf update and rebooting seems to have fixed all of the issues I noticed.

usb The biggest hurdle so far is that I can’t get fedora to recognize USB devices. Specifically, I needed to upload some books to Kobo ereader, and transfer some files to-from an external USB3 hard-drive. For both tasks I had to resort to re-booting into OSX. I did had some hardware issues with this laptop (spilled jam onto the keyboard…), but over time things just started to work again in OSX. But not here. I’ll update this post if I’ll get this stuff to work correctly.

Another problem is the touchpad. It generally works, and switching to tap-to-click was a huge improvement. But it doesn’t feel as good as in OSX. Generally issues seem to be around sensitivity and accidental clicks. I haven’t had time to investigate this further, but there are multiple guides online to adjusting macbook touchpads in linux.

battery? While battery on my macbook seems to be dying after three years of use (OSX asks me to replace it ASAP, and sometimes can’t seem to figure out how much time is left), Fedora seems to be struggling with it as well. Or is it? I’ve only spent couple of days with it and have been mostly plugged in, but generally it seemed that charge levels never climbed about 30-40% (while OSX would show 70-80%). It’s hard to say which one is more accurate, since I haven’t really timed how much life I get out of a full charge just yet; but there’s certainly a discrepancy in reporting. Time will tell which OS does a better job.

Feels good though!

After spending some time with Fedora, getting some of the biggest annoyances out of the way, I’m starting to really enjoy it. It feels very good to be running an open source OS again - and not just on yet-another VM, but as my main desktop :) The biggest things still to overcome is tweaks to how touchpad behaves and the USB issues.

Developer's block - and how I'm trying to deal with it

Wed, 04 Nov 2015 06:31:00 +0000

For the past several years now I’ve been going through what I’m going to call here a “developer’s block”. It’s very much like a writer’s block, in that as time passes I’m finding it increasingly harder to create anything outside of my 9-5 work. This post is a way to document some of the back story, analyze how I got to this point, and an attempt to build a path forward.

Some back-story

I’ve been creating all sorts of software for the web since around grade 8. I won’t list particular projects - they don’t matter. What matters is that I’ve been actively writing code and putting it out there for people to use for about 12 years now. And then at some point I’ve stopped. It seems that, tracing back my progress - or rather, regress - I’ve slowed down work on personal projects and ideas sometime around finishing university and starting my first post-graduation job (I did three of internships and bunch of freelance dev work throughout school). I didn’t stop completely and suddenly, but rather everything just sort of tapered off. It became hard to start things, and even harder to finish them. I used to have an active blog (in Russian, never really got used to keeping one in English), which has been entirely abandoned. I can’t even really remember for sure the last thing that I’ve made and published, other than helping setup custom map hosting for my partner. That’s not to say that I haven’t coded anything at all outside of work in those years - there were a few little projects here and there - some to learn new tech, and some to fill a personal need - but even “internal use” tools just for myself have stopped coming off my keyboard, last one being a personal diet tracking tool.

Why is this a problem? There is a rich, beautiful life outside of programming!

Partly because it certainly feels like one. I have this constant, nagging feeling that I should be creating something. I’m used to creating stuff. Whenever I think back to what I’ve used a computer for, there was almost always a project, some users, some feedback, some writing. I love that stuff… at least I definitely used to. So now it just feels like a strange void in my life. I have generally filled it with what people might refer to as “life”, but the nagging feeling and lack of satisfaction is there.

Additionally, one of my ambitions is financial independence and making the world if only a little better, and for some reason I have this silly thought in my head that I should be able to accomplish by creating truly useful, thoughtful things. It seems that inability to maintain and push forward “side-projects” is quite detrimental to this goal.

Work

When you write code for a living and push out products into production, spending 8-10 hours daily at the office - well, it would be foolish to ignore that as a factor. However, I’m not convinced that work itself is the problem. Perhaps it was what stopped me from working on personal projects when I’ve spent almost a year trying to get a company off the ground. When you’re in that “founder” mindset and on a short timeline, it’s hard to justify working on something other than your product. But generally speaking, work has made me a better developer, exposed me to new tech, new ways of working, and gave me certain personal freedoms (in exchange for time). Over the years I got a little better at mentally compartmentalizing work - it’s hard, but certainly possible and seems to be absolutely crucial.

So work by itself isn’t necessarily the. problem. But it could be a trigger for certain issues, if not thought of properly. In no particular order:

Burnout & mental health

I’ve experienced what seemed like a burnout, but in retrospect was more of a depression. It seems like a good idea to dedicate another blog post to the topic in order to do it justice. I’ll just mention that after experiencing a complete breakdown of my professional life at the time, I’m acutely aware of dangers of spending most of your time working on the wrong things, for the wrong reasons, and not seeing and dealing with mental problems in an honest, sustainable way. Thankfully after a lot of retrospection I have more or less figured out what caused my problems at the time, and how not to put myself into similar situations.

However, as I was dealing with those issues, I’m afraid that I’ve created mental blocks around certain activities and patterns, and created negative associations with “work outside of work”. I’m realizing now that they are not entirely necessary, and everything could be just fine in moderation.

Time

Recently I’ve spent a brainstorming session trying to, yet again, figure out - what is it that I actually want to do? I’m closing in on my third decade, and I still haven’t really figured it out. And I don’t mean finding some entirely absorbing passion, or a grand life plan, or anything along those line. I’d be happy with a general, coherent direction and perhaps some goals.

It seems that I’m currently following something along these lines: “work, get paid, try to lead a good life, be healthy, build meaningful relationships, and explore my inner self and the world around me”. So, that’s something, if just a little vague.

During my brainstorming session I kept coming back to the fact that I need time, more time - to read, to think, to construct and develop ideas, to talk with people. The most precious resource I seem to miss is time. This way of thinking makes deciding what to work on even harder - if I’m telling myself that I dearly need more time, I can’t just squander it on learning some new JavaScript library and building a little project with it. I could be instead reading up on some philosophical ideas that have been bothering me lately, or perhaps trying to brainstorm my way out of yet another mental dead end. That is, of course, unless I can justify learning and building tech stuff and work it into the bigger picture (whatever that might be). It’s all about my perception lens, after all.

I did take a very concrete first step, however. I’ve talk to good people at my work, and we have agreed that I will be taking every other Friday off. So, every other weekend is a three day weekend, and sometimes they even turn into four day weekend, thanks to statutory holidays. This has been fairly recent so too early to report on long-term effects, but I feel it’s a great step in the right direction. This directly gives me more free time in continuous blocks, and has a lot of other positive effects that are harder to quantify - more rest, ability to focus on things for longer periods of time, etc.

Pattern of consumption

Reddit, HackerNews. HN, reddit. Rinse and repeat. The fact that content is often interesting and sometimes even thought-provoking only seems to postpone realization that its constant consumption is not actually any good for me. I feel that over the years I have shifted my behavior online from active participation in select few communities, to passive consumption of many communities. Additionally, most of the content leaves very little lasting impact, but takes up a large amount of time.

What this did, over time, is created a paradigm shift. It changed the baseline from “participate, contribute and get feedback” to “read, hopefully learn (but most likely just forget), rarely participate”. I feel that this must trickle down into other parts of my life, it has to - it just seems like a general behavioral pattern.

Cutting down the amount of time spent mindlessly consuming content is important. Increasing signal-vs-noise ratio, spending more time reading literature I’ve wanted to for a while - these steps should help with motivation and perhaps will even inspire. Interesting side-note about reading good literature - for me it became a very important language preservation tool. Living in a largely English-speaking environment is not doing my Russian any favors.

Tinkering

I used to tinker with my computer all the time. When I was a kid, I treated its physical body as a malleable LEGO set. I was lucky enough to have been raised by parents who provided me with enough parts to assemble, re-assemble, and sometimes modify them to make things fit. Later on I started running linux full time, and tinkered endlessly with it - oftentimes just to get some basics working. I’ve learned a lot, did a bunch of cool stuff with it. Then I switched to OSX. At first, it was a bit of a relief - finally, things “just worked”. But it feels that over the years I’ve been using it (around 6-7 years now, I’m onto my second personal macbook, and I’m using one at work as well), I’ve lost that passion for tinkering with my own computer. There wasn’t much need. Yes, things generally work. Perhaps it made me more productive? That’s debatable and very hard to quantify. But I keep asking myself - at what cost? What did I loose - or rather, not gain - in those years?

An obvious thing to do here is to switch away from OSX. About a year ago I’ve installed Arch on my MBAir, but never really followed through with it. I remember there was some annoyance, and some things just seemed needlessly complicated. In the end, I didn’t have patience for it - and went hiking instead. It often seems that spending time screwing around with linux distros isn’t the best use of my time - but then again, perhaps it’s part of a some longer-term plan that actually makes sense.

Either way, since starting writing this post I have been inspired enough to install the new Fedora 23 release, and I’ve now mostly have it configured! And it’s been really fun. Cool beans.

Building things “offline”

Recently I’ve signed up for a membership at a local maker place - MakerLabs. They give you access to a whole bunch of tools - a woodworking shop, laser cutters, CNC routers, 3D Printers, etc. My goal is to learn some new skills, perhaps meet people that I wouldn’t normally get to interact with, and build some tangible things. I’ve started by making a maple cutting board, which was simple and surprisingly very enjoyable. Being exposed a little to the local maker community also made me look at my own home with different eyes, and I’ve started working on 3D modeling some simple custom furniture ideas. More projects to come!

Leaving off on a positive note

Writing this down had a mild therapeutic effect. It’s much better to have a problem, understand it a little better, and have some ideas on how to move forward - instead of just wishing for some state from the past. As part of this process and in a way of creating a social contract, I will keep this blog updated with what I’m up to, projects I’m working on, and generally how it’s all going.

Onwards and upwards!

Hosting MBTiles maps, fast and cheap

Sat, 05 Jul 2014 21:37:00 +0000

Intro

Ekaterina was working on a data visualization project, creating a map of Vancouver’s buildings colourized by their age of construction. She has created one several months ago in ArcGIS to aid in our apartment hunt in Vancouver, but now we decided to make it interactive and publish it online. I’ve pointed her towards Mapbox’s great map design tool TileMill, and in a couple of days she came up with a great looking, at times trippy map.

Check out the interactive version (click-click!):

Now, how do we get people to see it? TileMill, after some persistence, produced an MBTiles export, which is a SQLite database consisting of a bunch of rendered map tiles and some meta information (including map legend, UTFGrid). Mapbox.com offers a map hosting service, but with Ekaterina’s map we were looking at $50/month at least - which would only go up if the map saw significant use.

DIY to the rescue, and the birth of Love Your Map

I’ve decided to host everything myself, and in the process create a service others can use. I called this service Love Your Map, and currently getting it ready for a beta launch sometime later this week.

Quick rundown of technical details

After exploring different tile libraries, I went with TileStache. It’s a Python-based map server with WSGI support, which can easily serve pre-rendered tiles from MBTiles and render Mapnik maps (something I was trying out at that time, as TileMill has a Mapnik XML export option). After getting all of the dependencies installed and tweaking TileStache configs inside of a Vagrant VM, I’ve packaged everything using Docker for deployment. This project was a good excuse to try out Docker, and after getting a hang of the basics it’s been a great experience.

I’ve been meaning to try out DigitalOcean’s VPS service (handy referral URL) for sometime now, having read good things about them on HackerNews. The experience so far was stellar. Droplets (their term for a VPS instance) launched quickly (although, one took a few minutes to get going), behaved well, and they have a ton of tutorials on a variety of things you can do with your servers. I haven’t had to use their support, but I’ve read good things about it.

I’ve used their Docker-ready droplet to deploy the map server, and another droplet to run Nginx as a load balancer.

Initially when we launched the map, I’ve exposed the Gunicorn instance to the public (laziness, and a little bit of “let’s see what happens”). This worked fine until ~20 concurrent users started to actively browse the map, at which point the poor map server started to timeout and generally die. Nginx went in, Route53 records were updated, and it’s been holding up really well since. Some quick load testing with loader.io confirmed that this setup should serve me well for now.

Note on DigitalOcean coupons: they’re out there! After a 30 second search on google, everything is hosted entirely free of charge (for a month, and perhaps loveyourmap.com will start paying for itself afterwards).

In the six days that it’s been online, this map received around 20k unique pageviews, and got some amount of attention from the local community, blogs and media.

Driving across Canada in February

Mon, 04 Feb 2013 20:54:13 +0000

A few years ago, I lived in Ontario. In February of 2013 I went for a rather long drive with my dad - from Mississauga, ON, to Victoria, BC. A winter roadtrip is one hell of a way to move to British Columbia.

Shot with a GoPro mounted on the dashboard and taking a photo every 10 seconds. I’ve combined about 44,000+ photos into video chunks using ffmpeg, and stitched it all together with iMovie.

Music:

K.C. Accidental - Residential Love Song
Yppah - Never mess with Sunday
Caribou - Sun
Smile - Jean Michel

Getting started with Django on AWS Elastic Beanstalk

Sun, 16 Sep 2012 05:31:00 +0000

Note #1: this writeup made it into the Python Weekly Newsletter #54!

Note #2 (~2014): I wrote this post two years ago. I haven’t used Beanstalk much in the past year, and have mostly used Heroku and gunicorn/nginx running off of VPS for python projects. I’ve ported this post from my old wordpress blog since it’s still getting a lot of traffic from Google, but keep in mind its age.

Let’s go!

I needed to pick a home for an ongoing Django project. In the past I’ve deployed Django to my own Linodes, to WebFaction (both of which have been really good to me) and to AppEngine. This time, I needed something robust and more or less hassle-free, meaning that time dealing with the infrastructure is, ideally, minimized. I also didn’t want to develop the project within the AppEngine’s ecosystem. On the other hand I wanted free tiers, ability to go “under the hood” if necessary, and ability to sanely migrate away from the vendor.

Just over a month ago, Amazon announced Python support for their Elastic Beanstalk. After doing some research, I decided to go with this option. It doesn’t cost anything extra beyond whatever Amazon resources Beanstalk provisions for you, and fits really well with all my other requirements. It was also really easy to set up and get my new Django project on the way. I found the documentation just a little thin, so perhaps this quick writeup will be helpful to others in the same boat. Keep in mind that Python’s support is only a few months old in Beanstalk, and that I myself never set up AWS services before. This is very much a learning process.

Basics

Amazon’s own Deploy a Django Application tutorial is a good starting point. I won’t repeat the steps listed there, so do follow them and come back here. Their steps won’t take you more than a few minutes.

Your infrastructure looks like this now, which is a great starting point, especially provided that it took you minutes to set up:

Here is an excellent presentation on Scaling Django Apps with AWS, and we’re not terribly far away from what they’re talking about there. As a nice bonus, all of this should fit into the Amazon’s free usage tier.

Here’s is Amazon’s guide on how to hook up your domain name to this setup: Using AWS Elastic Beanstalk with Amazon Route 53 to Map Your Root Domain to Your Load Balancer.

Here’s how Amazon illustrates their Beanstalk architecture:

Notes on Amazon’s tutorial

Provided createadmin.py script didn’t work for me (getting PASSWORD_HASHERS errors when trying to log in), so I’d recommend using something like this:

from django.contrib.auth.models import User
if User.objects.count() == 0:
    admin = User.objects.create_user('username', 'email@example.com', 'password')
    admin.is_superuser = True
    admin.is_staff = True
    admin.save()

If you follow all of the steps, you’ll end up with python.config file looking like this:

container_commands:
 01_syncdb:
  command: "django-admin.py syncdb --noinput"
  leader_only: true
 02_createadmin:
  command: "scripts/createadmin.py"
  leader_only: true
 03_collectstatic:
  command: "django-admin.py collectstatic --noinput"
option_settings:
 "aws:elasticbeanstalk:container:python:environment":
  DJANGO_SETTINGS_MODULE: "myproject.settings"
 "aws:elasticbeanstalk:container:python":
  WSGIPath: "myproject/wsgi.py"
 "aws:elasticbeanstalk:container:python:staticfiles":
  "/static/": "static/"

leader_only: true is an important flag, it tells Beanstalk to run the commend once per deployment only on the “leader” instance, and not once per EC2 instance.

You’re also now able to make some changes to your project, commit, and deploy a new version like this:

git add whatever/file.py
git commit -m "my changes"
git aws.push

Simple local development environment with local_settings

In your settings.py file, make sure the database section looks something like this:

if 'RDS_DB_NAME' in os.environ:
 DATABASES = {
  'default': {
  'ENGINE': 'django.db.backends.mysql',
  'NAME': os.environ['RDS_DB_NAME'],
  'USER': os.environ['RDS_USERNAME'],
  'PASSWORD': os.environ['RDS_PASSWORD'],
  'HOST': os.environ['RDS_HOSTNAME'],
  'PORT': os.environ['RDS_PORT'],
  }
 }

We need that if statement because in all likelihood, your local environment won’t have these environmental variables set up and you’ll end up with KeyErrors.

Now, add the following to the bottom of your settings.py file:

try:
 from local_settings import *
except ImportError, e:
 pass

And put something along these lines into your local_settings.py (don’t add it to source control though):

DATABASES = {
 'default': {
  'ENGINE': 'django.db.backends.sqlite3',
  'NAME': 'db.db',
  'USER': '',
  'PASSWORD': '',
  'HOST': '',
  'PORT': '',
 }
}

MEDIA_ROOT = ''
MEDIA_URL = ''
STATIC_ROOT = ''
STATIC_URL = '/static/'
STATICFILES_DIRS = ()
TEMPLATE_DIRS = ()

S3 Storage with django-storages and boto

We want to use an S3 bucket that Beanstalk kindly created for us in order to server static files, and if you’re going to handle uploading of files in your project, these also should go straight to the S3.

Let’s use django-storages and boto to handle AWS communication.

pip install django-storages boto
pip freeze > requirements.txt

Add ‘storages’ to your INSTALLED_APPS. Set the following variables in your settings.py:

STATIC_ROOT = os.path.join(
 os.path.dirname(
  os.path.dirname(
   os.path.abspath(__file__))), 'static')

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'your_key'
AWS_SECRET_ACCESS_KEY = 'access_key'
AWS_STORAGE_BUCKET_NAME = 'bucket_name'

You can get the name of the S3 bucket that Beanstalk created from the AWS S3 console. It will look something like this: elasticbeanstalk-us-west-2-1234567890.

The above achieves two things:

collectstatic script will now move everything to your S3 bucket, and all of your static files will be served from there
FileFields in your models will now also use the S3 bucket for storage

Check out more docs on setting up django-storages with S3.

Schema migrations with South

I’m using Amazon RDS (which is an instance of MySQL). We’d like to manage our schema migrations in a sane way, and South is as good of a way to achieve that as we have available.

pip install south
pip freeze > requirements.txt

Add ‘south’ it to INSTALLED_APPS. Make manage.py executable, and create your first app:

chmod +x manage.py
./manage.py startapp myapp

Create your app’s models.py file, and run the initial migration:

./manage.py schemamigration myapp --initial
./manage.py migrate myapp

You want your migrations to run on the remote RDS as well, so add a new command to your python.config file which will execute migrations during deployment. Below I’ve added 04_migrate_myapp, and the python.config now looks something like this:

commands:
 01_syncdb:
  command: "django-admin.py syncdb --noinput"
  leader_only: true
 02_createadmin:
  command: "scripts/createadmin.py"
  leader_only: true
 03_collectstatic:
  command: "django-admin.py collectstatic --noinput"
 04_migrate_myapp:
  command: "./manage.py migrate myapp --noinput"
  leader_only: true
option_settings:
 "aws:elasticbeanstalk:container:python:environment":
  DJANGO_SETTINGS_MODULE: "myproject.settings"
 "aws:elasticbeanstalk:container:python":
  WSGIPath: "myproject/wsgi.py"
 "aws:elasticbeanstalk:container:python:staticfiles":
  "/static/": "static/"

Wrap up

Amazon’s own Django deployment tutorial plus the steps I’ve outlined above should, fairly quickly and without too much hassle get you up and running with a semi-decent development environment, and on a very nicely infrastructure consisting of various AWS components, with auto-scaling, load balancing, and a great web console to manage it all.

I still need to figure out the best way to handle test/staging/production environments, and ideally throw in something like Jenkins CI into the mix, but this is a really good starting point.

Any comments, additions, and corrections are very welcomed.