Stephen Newey


Would Like To Build: Kotaka the Deployer

16 February 2015

Following up on Django app deployer I want to get down some of my ideas for the deployment tool I’d like to build.

It’s provisionally named Kotaka after a Japanese torpedo boat from the late 19th century, considered to be a forerunner to Destroyers according to Wikipedia. It turned out I wasn’t the first person to think of the pun “X the Destroyer” / “X the Deployer” and so all the comic book examples I found were already taken.

Don’t worry, I’ll still be here after you’ve exited the wikiloop I just sent you on.

The goal

The end goal is I’d like a tool that allows me to create a minimal amount of configuration to describe where to find a Python web project and how to run it.

  • The tool should be able to automatically provision servers for the application, including any databases. Abstractions should allow multiple providers for this function, including Docker, Amazon EC2, Azure Virtual Machines, Linode and DigitalOcean.

  • The servers should allow for failover by have database replication and near-realtime syncronisation of media uploaded by users. Abstractions should allow support for multiple databases and database-like services, including PostgreSQL, MySQL, MongoDB, Redis and Memcache.

  • The configuration should include a primary domain used by the application and any other domains that should redirect to it.

  • The configuration should include definitions of cron jobs and custom daemon processes, like Celery workers.

  • The tool should be able to manage multiple environments for a project, including staging and production and allow for execution of tests.

  • The tool should be able to update DNS entries for projects. Abstractions should allow support for multiple DNS providers/servers including PowerDNS (where it is database backed), Amazon Route 53 and Linode.

  • The tool should use JSON configuration files to define projects. The structure should be written in such a way as to avoid repetition/duplication of configuration (Don’t Repeat Yourself).

  • The tool should provide an easy to use set of command line tools to run the deployment process and to enact and restore from failovers.

  • The tool should include a web-based user interface to create and modify the JSON configuration files easily.

  • The resultant environments created after the tool has been used should be easy to operate in. Upon cd’ing into a particular project the shell environment should be populated with all the information required to operate on the project (Django settings path, database connection details, etc).

Here’s an example of what the configuration might look like:

{
    "project": "Example Project",
    "slug": "example",
    "builds": {
        "default": {
            "servers": [
                "primary.exampleinstance",
                "failover.exampleinstance"
            ],
            "repo": "git@bitbucket.org:stevenewey/example.git",
            "branch": "master",
            "project_type": "django<1.7_south",
            "settings": "example.settings.staging",
            "wsgi_app": "example.wsgi",
            "databases": {
                "default": {
                    "type": "postgresql",
                    "host": "localhost",
                    "username": "example",
                    "name": "example_staging",
                    "password": "xxxxxx"
                }
            },
            "primary_domain": {
                "name": "staging.example.com",
                "with_www": false
            },
            "cron_jobs": [
                {
                    "job": "{manage} update_index",
                    "hour": 4,
                    "minute": 15
                }
            ]
        },
        "production": {
            "branch": "production",
            "settings": "example.settings.production",
            "databases": {
                "default": {
                    "name": "example_production"
                }
            },
            "primary_domain": {
                "name": "example.com",
                "with_www": true,
                "redirect_non_www": true,
                "ssl_enabled": true,
                "ssl_key": "...",
                "ssl_cert": "..."
            },
            "domains": [
                {
                    "name": "another.com",
                    "with_www": true,
                    "redirect_non_www": true,
                    "nginx_additional_config": "rewrite ^legacy_url$ /new_url"
                }
            ]
        }
    }
}

The definition is not yet complete, but it covers a lot of my requirements. Use of the “default” object serves as a base set of configuration options which can be extended or overridden by specific instances (as in the builds block).

This is in all likelihood a big project. I’m going to get started building the minimum product I need to get up and running, supporting only Docker, PowerDNS and PostgreSQL to begin with.

Feedback welcome. I’ll post again if there’s ever something to look at on Github.

Tags: development, deployment, python, hosting, kotaka