How Do I GDS?


Learnings from early-adoption of open-source code

ODI Tech Team ยท @ukoditech

WTF is the Open Data Institute?

  • Launched October 2012
  • Non-profit
  • Funded by
    • UK Government
    • Matched by others
  • Dedicated to Open Data
    • Increasing supply
    • Increasing demand
    • Education and training

Open culture

Who are the tech team?

  • Mixture of skills
  • Building tools for open data
    • CSVlint
  • Showing best practice
  • Open culture in code

How do we work?

  • agility
  • TDD
  • Lots of automation

What were we trying to fix?

  • We inherited a Drupal install
  • None of us knows PHP or Drupal
    • Nor do we want to
  • Drupal is hard to
    • Scale
    • Automate
    • Test

What did we really want?

  • A web-publishing platform which we could bring into our workflow
    • Testable
    • Automatable

The web is not just for humans

  • API first - not bolted-on (c.f. Drupal)
  • Content-as-data

Do web-publishing better

  • Decoupled - many parts, loosely joined
  • Multiple views as artefacts of the core platform
  • This stuff matters to the ODI because Open is the most important consideration for us - the platform needs to reflect who we are

OpenTech 2013

  • Saw Tom Loosemore and Jordan Hatch speaking about their platform
  • They had the parts we needed
    • It was built in the right way
    • Using tools we know
    • Let's not re-invent any wheels - building on the work of others is one of our core principles

The Spike

  • Took a 1-week spike, got it kind-of working
  • Realised that while this stuff is open-source, nobody had ever considered that it might be re-used by somebody else
  • But it looked like it would fit our needs

Pushing Upstream

  • Made a setup script
  • Wrote some documentation
  • Made things a little more generic

Time to validate our assumptions

  • Went to speak to some people at GDS
  • There are humans at the other end of this

Resolved some mysteries

  • Discovered that some things that confused us were like that because GDS had been in a hurry
  • There was positive intent, but we're all human

We needed to make it more generic

But this will help others, so that's OK

How Do I Fork?

  • When to fork?
  • When to break your fork?
  • How to keep up to date?
forkbomb.io
mazz-ops.com

How and where to customise?

  • Extend
    • e.g. odi_content_models
  • Monkeypatch
    • e.g. odidown
  • Fork and customise
    • Publisher
    • Panopticon
  • Can apps be customised on the fly with Engines?

Make your improvements opt-in

Don't change default behaviour, so you don't need upstream to care

Refactoring

  • How aggressively do we feel we can refactor?
  • Requires a lot of confidence

Tests are essential

  • If GDS didn't have tests, we would never have been able to do this
  • If we push back to them without tests they'll never accept our PRs

What should we contribute back?

  • Do GDS want this shit?
  • Do they have time to deal with it?

Could others pick up our version?

  • Probably not
  • Yet
  • Would be great to get to a fully-generic, fully-customisable version

Ongoing process

  • We're still working on it all
  • Building new front-ends for different projects

What makes this hard?

  • Piecing it all together from blogposts and scraps of documentation
    • This is not apt-get install gds
  • Not having visibility on GDS's roadmap
  • Not being their primary customer
  • Differences in tooling choices
    • e.g. test::unit
  • Mysterious missing parts
    • this is overwritten on deploy all over the place

WHAT IS THIS THING EVEN CALLED?

What did we fuck up?

  • Sending mail to winston@alphagov.co.uk because GDS had left this hard-coded in an initialiser
  • Occasionally accidentally opening pull-requests onto alphagov's master branch
    • and we still owe them cake
  • Emailing half of GDS on a failed Travis build
  • And only discovering these things when we got emails from GDS saying 'please stop doing this'

Summary of our learnings

  • Know what you're getting into
  • Push back, but consider the impact
  • Upstream, there are humans
  • Use, fork, or split?
    • Easier on a decoupled project
  • Rule 5: there are no rules
Stuart Harrison Sam Pikesley James Smith Jeni Tennison

Open Data Institute Tech Team
@ukoditech
info@theodi.org
irc.freenode.net #theodi

ODI

http://theodi.github.io/presentations

ODI Creative Commons