But it works on my machine!

Don’t let “it works on my machine” be a thing!

It’s convenient to just git pull a project, install some dependencies and get on with development. But that doesn’t work well enough, even with runtime version managers like ruby’s RVM and python’s virtualenv , to be reliable. You may be able to control the runtime environment but what about all the binary dependencies?

And so technologies like vagrant with virtualbox and now docker , have transformed one-off snowflakey development environments into virtualised sandboxes that can be brought up and down as we hop from project to project without dependency versioning tripping anyone up.

But then you have to consider the differences between development and production environments.

Docker allows container images to be (almost) identical between development and production, which is the ideal. But a vagrant+virtualbox workflow, which we use at my workplace due to limitations with Amazon’s ECS is dangerous without a means of keeping development in sync with production. The solution is to base vagrant and EC2 images on the same version of linux and then to provision both types of environment using the same definition, for example using chef cookbooks.

Of course a development environment will probably contain ‘extra’ stuff, for example database servers that are externalised on the production platform (e.g. by using RDS), in which case keeping the versions consistent becomes an easily-solved issue within the cookbooks. Again, docker provides a more elegant solution with its support (especially via docker-compose) for multi-container deployments.

How can you share your chef configuration between vagrant and AWS EC2? It used to be difficult/a hack, but the vagrant-aws plugin makes life easier. It’s not perfect since the EC2 instances created by one developer end up being private to him and not able to be managed by other developers, but bringing up new instances instead of updating existing ones is a best practice anyway so that’s not a real blocker.

But the Hashicorp folks, who own Vagrant, are rolling out a reallt interesting and rather complete suite of tools that appear to wrap up all these concerns and much more. The part of the jigsaw that solves the problem of sharing development and production platform builds is the concern of their Packer product. While the end-to-end solution includes Terraform to build AWS infrastructures, as well as more tools to deal with secrets management, service orchestration and scheduling that I’m very interested to look at for the future.

Finally, to reiterate… don’t let ‘required’ tools or dependencies creep onto developers’ laptops outside of your virtualised and controlled development environments. Because then you’re back to “it works on my machine”. If there are tools required that are shared between projects or that are difficult to install inside project environments then make a sandbox (container or VM) specifically for the tool.

Don’t be satisfied with “OK let’s make sure everyone does brew install foo” because you’ll end up in a world of “it doesn’t work anywhere except my machine”.

The AWS “new features” email continues to grow longer and longer with the passing weeks! Here are some recent announcements that caught my eye.


Amazon noticed the success of Slack and the interest in Skype: online collaboration is the next battleground for integration technologies as these products are pressed into service as a human-oriented “Enterprise Service Bus”. Not to mention that the clunky old-school VOIP solutions are hopelessly outclassed by these ad-hoc solutions that expose APIs for app developers to hook into.

Chime is clearly receiving some attention within AWS as it matures nicely, see Amazon Chime Now Allows You to Log Console Actions Using AWS CloudTrail and Amazon Chime Now Supports Quick Actions on iPhone, iPad, and Apple Watch .

Cloudfront TLS Policies

Recent security breaches and the continuing exploitation of SSL weaknesses puts the spotlight on the aging protocols that terminate our “secure” web infrastructures. AWS does a great job of keeping up to date with the latest and greatest in SSL/TLS protocol suites at the Load Balancer layer. With Amazon CloudFront now lets you select a security policy with minimum TLS v1.1, v1.2, and security ciphers for viewer connections! we have some welcome parity at the Cloudfront layer.

Network Load Balancer on Elastic Beanstalk

And then there were three varieties of Elastic Load Balancer on the AWS platform! The original (and now pretty much deprecated) Classic Load Balancer has been joined by the Layer 7 focussed Application Load Balancer for flexible routing, and the Layer 4 focussed Network Load Balancer for high-performance network balancing.

Now all three can be deployed using Elastic Beanstalk as described in the recent announcement.

Care Needed When Navigating the Chef Ecosystem

Chef is a config management tool that allows you to put the build of a server under version control as a text (ruby) description of the steps needed to bring it to the correct state. It is one of several tools that allow you to automate server builds which is essential for reliable deployment of solutions. Similar tools include Puppet and ansible, but my most recent experience is specific to Chef on MacOS and Linux, the basis for this discussion.

As is the norm these days, a Chef stack is built on a towering pile of open source dependencies that decay and expire with the changing seasons, leaving behind a trail of failed builds and much swearing.

Chef itself, with its dependency manager Berkshelf, is installed as a bundled ‘Chef DK’ that even includes the underlying ruby runtime (such is the fragility of these tech stacks). So installing the tool itself is now fairly fool-proof. But there’s trouble brewing of you plan to use the tool to actually do some work!

You drive Chef with ‘cookbooks’ that contain ‘recipes’ holding the instructions to deploy the components of a platform. Cookbooks are written in ruby and can have dependencies between them that you resolve using the Berkshelf tool, similar to ruby’s native ‘bundler’ or node’s ‘npm’ (that’ll be ‘yarn’ these days).

This tends to be one source of pain when a cookbook depends on a third party cookbook… which gets updated in such a way as to become incompatible. I’m not sure why this is a major problem, given that semantic versioning should be standard, but it is. Versioning is just a hard problem.

The other important source of incompatibility is between cookbooks and Chef itself. Chef has been growing and changing for years now, and it seems quite common to come across cookbooks that only work with v11 of chef and not v12, or vice versa.

I suspect part of the problem is that deployments tend to fail only when you actively update cookbooks, which is less important to do compared with keeping libraries updated for security reasons; cookbooks themselves rarely need updating on the basis of security. And so a deployment that worked last year and hasn’t changed since then, can suddenly break when a cookbook needs to be updated for some reason (perhaps to install an updated version of some package when the version is hard-coded into a recipe… more common than you might hope).

The take-away message is that a Chef deployment can easily decay unnoticed over time, so taking regular time out for maintenance is essential to avoid panic when an emergency deployment of a fix turns into a traumatic chef-cookbook-dependency-tracking-down-why-doesn’t-it-work-athon.

Not that that ever happens in real life.

Docker is an Immature Technology on AWS ECS

I’ve been working on converting some backend API development environments from a vagrant VM-based platform onto docker containers, provisioned using a Chef configuration that also builds the EC2 instances that production services are deployed onto.

Originally, I intended to move containerised production deployments onto Amazon’s Elastic Container Service (ECS) but backed out when I realised how immature the technology is. I’m sure it won’t stay that way with the velocity that AWS is moving at, but right now ECS is under-developed.

I’d expected ECS to be similar to the Lambda platform, which allows functions to be pushed onto AWS for deployment in a hidden pool of compute resource that can scale on demand. I imagined we’d be pushing docker containers into a similar pool of essentially infinite resource for which we’d pay a simple usage fee.

Unfortunately that’s not how it is. On ECS you define and manage a pool of EC2 resource that containers are deployed onto.

For some use-cases this is probably not an impediment, but when you’re dealing with a large number of projects that rarely require more than a handful of servers, the requirement to manage both the container layer and the underlying compute resource layer is an overhead that surprised me.

I can’t wait for deployment of containers on the AWS platform to remove the need to manage compute resource beyond the simple specification of operating characteristics to service an expected workload.

Making Enterprise Architecture Relevant in the Land of Devops

In 5 Tips to Automate Your Cloud Infrastructure I mentioned automated compliance testing. This is the idea that once your entire solution (both infrastructure and software) is defined using artefacts under version control, then the way is open to checking those artefacts for compliance against an enterprise-wide set of rules… in an automated workflow… not in meetings with an “enterprise architecture team” who wield the fiery blame hammer to punish teams ignorant of the latest Powerpoint edict from on high.

That’s a bit of a harsh characterisation. Harsh, but unfortunately true for some of the large businesses I’ve worked with over the years, where it seems that the traditional enterprise architecture role has managed to avoid the magnetic pull towards automation that drives efficient solution delivery. Things are sometimes so toxic that EA is seen as an optional extra, or worse, an impediment for delivery teams.

And so there may be an unhealthy tension between delivery teams who see themselves as “actually doing stuff” and the EA team, which is often far removed from the concerns of project teams working with specific business units to automate their processes.

But there is a useful role for a central authority that understands a wide landscape and its history. This is the context in which the totality of IT solutions supports the business as efficiently as possible right now, with a view to the need to support a likely path of flight for the future.

The EA role can be a helpful force for project teams when it becomes integrated into the automated workflows that modern delivery uses. When it sits outside the automation of infrastructure build and deployment then it becomes a cumbersome obstacle, but what if compliance against the enterprise architecture was simply a part of the automated testing? A build that failed the EA tests would instantly make visible an issue that could be addressed immediately. The alternative of non-compliance going unnoticed until the next ‘architecture review board’ months in the future looks barbaric and not very helpful at all.

Of course this approach depends entirely on automation of the entire project delivery from infrastructure through build and deployment, which only a “cloud” approach (i.e. API-driven infrastructure) can bring.

How can compliance be automated? By building and maintaining common “recipes” (to use a Chef term) that projects use, which are regularly audited and tweaked by the EA function. By attaching metadata to project repositories that bespoke tools can scrape and analyse. By running tools such as Amazon Inspector and any one of the myriad third-party tools. By using AWS Config to ensure that infrastructure changes are compliant. And on it goes. We’ll return to this topic in the future.

This is a brave new world in which EA is a helpful enabler not a grumpy librarian. A world in which EA is intimitely bound to the project teams and their processes. An aid for projects, able to bring relief to stretched teams by identifying potential for reuse of existing solutions (infrastructure recipes, micro-services, APIs, components, COTS). And a source of compliance to ensure that teams are building value within the growing IT estate.