Negative Code

Subscribe via RSS

Recent Entries

• It’s Better to Work Together Than Alone 14 Dec 2012
• Joining GitHub 13 Jul 2012
• Fibers & Resumable Recursive Descent JSON Parsing 26 Mar 2012
• Why I Love Programming in Ruby 16 Feb 2012
• Vines XMPP Server, Now With Clustering! 30 Jan 2012
• Metaprogramming ActiveRecord Connection Pools 16 Dec 2011
• Introducing Vines Shell 03 Nov 2011
• Vines XMPP Server 0.2.0 Released 11 Jul 2011
• Why XMPP? 01 Jul 2011
• Hello World! 15 Jun 2011

I'm David Graham, a software designer in Denver, Colorado. I work at GitHub, making github.com a little more wonderful every day. Follow me on Twitter!

• It’s Better to Work Together Than Alone

  Last week we shipped image attachments on GitHub issues, a feature that I’ve personally wanted for a long time. We worked on it for just over six weeks and shipped over 25 pull requests before turning it loose for everyone to use.

  Living with a feature for a few weeks is the best way to fine-tune it before pushing it live. And by the time we did release it to the world, the pull request was just a single line of code. We knew it worked great, we just had to turn it on.

  But the real story is how we ship things fast without any formal coordination. Attachments started when Kyle asked, “Who wants to help me make this a reality today?” in the chat room, with a link to Jason’s pull request sketching out the drag-and-drop user interaction design. It didn’t take long for a flurry of new pull requests to open and ship into production.

  Looking back through our GitHub issues, more than a dozen people hacked on this one small feature before its release.

  We don’t have managers at GitHub and we don’t tell each other what to work on. Just asking for help is enough. Invite someone to hack on something with you. You’ll be surprised by the response you get.

  And please use the animated gif support for good, not evil.

  - David Graham
  14 Dec 2012

  It’s Better to Work Together Than Alone

• Joining GitHub

  It’s been a busy couple months since I joined the crew at GitHub. Designing, testing, and shipping code to millions of people has been a blast, but it’s taken a few weeks to understand why I’m so happy to be a GitHubber.

  Great Software Starts With Great Design

  I’ve loved GitHub’s product since the day I signed up. In all the time I’ve been building software, I’ve used very few tools as well-crafted as github.com.

  Now that I’ve been on the team for a while, it’s obvious why I love it so much. The design team has an obsessive attention to detail. Every pixel and interaction is carefully considered, discussed, and refined before it ships.

  It’s not perfect (yet), but we’re getting closer every day.

  Great Engineering

  All of this design work is for nothing if it’s not backed up by some serious engineering. GitHub’s internal systems are impressive. We’re shipping code, live to the site, dozens of times a day.

  When I was building enterprise software we’d ship once every six months. At GitHub we even update the Enterprise version every two weeks. That’s unheard of in enterprise software. Always be shipping.

  Optimize for Happiness

  Maybe most importantly, Tom’s talked a lot about building a company that’s optimized for happiness. And he’s not joking. I’ve never been surrounded by as many people who care about each other and their customers as much as this team.

  So when they approached me about joining them on their mission for world domination, I thought a long time about it, but in the end the decision was easy. I’m so proud and honored to work at GitHub.

  - David Graham
  13 Jul 2012

  Joining GitHub

• Fibers & Resumable Recursive Descent JSON Parsing

  The first thing I tried while developing the json-stream gem was writing a recursive descent parser. They’re great because the parsing code looks like the syntax description of the language being parsed. Unfortunately, they can’t pause while waiting for more text to complete the parse.

  We need to provide a recursive descent parser a complete JSON document, then wait for the entire parse to finish before continuing on. This is fine for a programming language compiler, but doesn’t work when we’re receiving JSON data in small pieces over the network.

  Now that Ruby 1.9 is widely used, I thought it would be fun to revisit this problem with Fibers.

  Resumable Parsing

  Here’s a sample program that demonstrates a worst-case scenario: JSON text arriving to our parser one character at a time.

  parser = Parser.new
  json = '[1, 2, 3, [4, 5, 6]]'
  json.each_char do |ch|
    if result = parser << ch
      puts result.inspect
      # parse finished, process result object
    end
    # do other work
  end

  We pass a JSON chunk into the parser and process the result object that’s returned when parsing is complete. But if this piece of data didn’t complete the parse, we need the parser to remember its current state, so we can feed it more data when it’s available.

  Pausing Recursion With Fibers

  Each Ruby Fiber gets its own call stack separate from our main program. They’re similar to threads in that way; however, fibers always run on the thread that calls them.

  So if we wrap a Fiber around our recursive descent parser, we can feed it data, parse up to the end of the available chunk, and suspend the call stack until we have more data with which to complete the parse. The level of recursion inside the parser is maintained so it can pick up where it left off.

  Let’s look at the two classes responsible for parsing a stream of JSON data: Parser and Scanner. These classes use a Fiber to coordinate the parsing.

  Scanner

  The Scanner is responsible for maintaining the buffer of JSON data that hasn’t been parsed yet. It’s basically just the data source for the Parser. In a traditional JSON parser the data source would typically be a String or StringIO object.

  class Scanner
    def <<(data)
      # more data has arrived
      @buf += data.split('')
    end
  
    def peek
      # pause when we're out of data
      Fiber.yield if @buf.empty?
      @buf.shift
    end
  end

  The trick to this data source is that it yields/pauses the Fiber it’s running in when it’s out of data. It will feed the Parser characters, one at a time, until there’s nothing left to parse and pause the Fiber’s stack. Now we need something that will resume the Fiber when there’s more data in the buffer.

  Parser

  The Parser is responsible for reading characters from the Scanner and building Ruby objects from the JSON tokens it recognizes. If it encounters a character it didn’t expect, it raises an error for the improperly formatted JSON it was given.

  Most importantly, in this case, it must resume the Fiber when it’s given more data to parse.

  class Parser
    def initialize
      @scanner = Scanner.new
      # run the parse method in a fiber
      @fiber = Fiber.new do
        Fiber.yield parse
      end
    end
  
    def <<(data)
      # resume parsing when more data arrives
      @scanner << data
      @fiber.resume
    end
  end

  The Scanner yields the Fiber and the Parser resumes it. This back and forth happens until a full JSON object is parsed from the stream. Fibers allowed us to write an easy-to-read recursive descent parser that also handles partial JSON data streams.

  I simplified these code samples to highlight just the Fiber interaction. A working, fibered JSON parser is available at this gist, though.

  So, is this technique just for crazy people who need specialized JSON parsers or are there other places this is useful?

  - David Graham
  26 Mar 2012

  Fibers & Resumable Recursive Descent JSON Parsing

• Why I Love Programming in Ruby

  A friend asked me recently why I enjoy programming in Ruby so much and I thought I’d share my thoughts here as well. We’ll focus on one piece of sample code, letting Ruby do most of the talking, that highlights some of my favorite features.

  This example comes from some real code I was working on. We’ll refactor it step by step until we get something we’re proud of. Each time through, we’ll improve the code by using more and more of Ruby’s features.

  Split an Array Into Keys and Values

  Our goal is to write a split method that accepts an array and partitions it into two different arrays of keys and values. The minitest spec for split would look something like this.

  it 'splits an array into keys and values' do
    params = ['foo', 12, 'bar', 42]
    keys, values = split(params)
    keys.must_equal ['foo', 'bar']
    values.must_equal [12, 42]
  end

  Array#each

  The first attempt creates a couple arrays, loops through params, and adds the item to keys if its loop index is even or to values if it’s odd.

  def split(params)
    keys = []
    values = []
    index = 0
    params.each do |param|
      if index % 2 == 0
        keys << param
      else
        values << param
      end
      index += 1
    end
    [keys, values]
  end

  The neat thing about this is the Array#each iteration method that accepts a block of code (between the do/end keywords) and runs it once for each item in the array. Internal iterators are one of my favorite Ruby features.

  Array#each_with_index

  We can tighten this up a bit by using multiple assignment to define the keys and values arrays on one line. And rather than define and increment the index variable ourselves, we’ll let the Array#each_with_index iterator do it for us. Now the outer scope isn’t cluttered with index variables that are only needed for the iterator block.

  def split(params)
    keys, values = [], []
    params.each_with_index do |param, index|
      if index.even?
        keys << param
      else
        values << param
      end
    end
    [keys, values]
  end

  As a bonus, we used the Fixnum#even? method on the index, which reads a little easier than the modulo operator.

  Array#partition

  Array’s each and each_with_index methods are nice, but not as powerful as Array#partition. This iterator splits one array into two, based on the block you’ve given it. In our case, if the block returns true, the item goes into the keys array, false and it goes into values.

  def split(params)
    index = 0
    keys, values = params.partition do |param|
      even = index.even?
      index += 1
      even
    end
    [keys, values]
  end

  So Array#partition is convenient, but now we’re back to maintaining the index variable again. This still feels like too much bookkeeping, just to split an array.

  Enumerators

  It turns out that if you don’t pass a block of code to Array#partition, it returns an Enumerator object instead. Enumerators have a with_index method that will maintain the current loop index for us.

  def split(params)
    params.partition.with_index {|param, index| index.even? }
  end

  By combining two Enumerators together, we can partition an array based on item indexes in a single, readable, beautiful line of code.

  What Do You Think?

  Is there an easier way to split this array that I missed? What’s your favorite language feature, in Ruby or otherwise? Let me know in the comments!

  - David Graham
  16 Feb 2012

  Why I Love Programming in Ruby

• Vines XMPP Server, Now With Clustering!

  We’re proud to announce the fourth preview release of the Vines XMPP server! In addition to bug fixes and performance improvements, there are three important new features in this release.

  Two Servers Are Better Than One

  Thanks to EventMachine, a single Vines chat server can handle plenty of traffic. But eventually, we’ll need a second server (or more) to handle higher loads and provide high-availability should one server fail. In this release, several Vines servers can cluster together, using Redis to route messages among themselves.

  I’ll write more in-depth about how the clustering feature works in the future. For now, it’s easy to get started by adding this snippet to your config file.

  cluster do
    host 'redis.wonderland.lit'
    port 6379
  end

  Point two or more Vines servers to the same Redis database and you’re done!

  Publish/Subscribe

  The XMPP standard includes an extension to support the Publish/Subscribe (a.k.a pubsub) design pattern. This pattern allows connected users to publish messages to a channel without necessarily knowing who’s subscribed to that channel.

  This feature is big because Vines can now support more than just instant-messaging applications. Publish/Subscribe is the basis of a new service we’re about to launch into beta. Stay tuned for more on that in the near future!

  MongoDB

  And finally, user accounts can now be stored in MongoDB databases. Vines can connect to a single MongoDB instance or to a replica set for high-availability.

  So Vines can store information in pretty much any database out of the box: simple YAML files, SQL databases, Redis, CouchDB, and MongoDB. And it still has an easy to customize storage API in case one of these options doesn’t quite fit your environment.

  Try It Out!

  If this is the first time you’re installing Vines, make sure to follow our Ruby install instructions and then run gem install vines. If you already have Vines installed, just update the Ruby gem with gem update vines to install the new version.

  - David Graham
  30 Jan 2012

  Vines XMPP Server, Now With Clustering!

• Metaprogramming ActiveRecord Connection Pools

  Using ActiveRecord database connections in a multi-threaded app, outside of Rails, is a bit of a mystery. Since Rails typically takes care of connection cleanup, there’s not much information on how to handle connections when using ActiveRecord in a stand-alone app.

  But after piecing together a few helpful blog posts and finding the right class to use inside ActiveRecord, it’s actually pretty simple.

  def save_user(user)
    ActiveRecord::Base.connection_pool.with_connection do
      found = User.find_by_jid(user.jid)
      found.name = user.name
      found.password = user.password
      found.save
    end
  end

  ActiveRecord’s with_connection method yields a database connection from its pool to the block. When the block finishes, the connection is automatically checked back into the pool, avoiding connection leaks.

  We don’t use or even assign the yielded connection to a block variable. Any ActiveRecord call inside the block will use the connection automatically.

  Don’t Repeat Ourselves

  There are seven methods, today, in the Vines SQL storage class that need to use with_connection. And while that’s not a terrible burden, it would be nice if we didn’t need to repeat this block of code in each place.

  The storage layer in Vines is meant to be extended for custom database layouts, so it should give users an easy way to handle database connections out of the box.

  Metaprogramming to the Rescue

  Using instance_method together with define_method is a simple technique for decorating a method with extra behavior. This takes an existing method and redefines it with some extra behavior wrapped around a call to the original implementation.

  def self.with_connection(method)
    old = instance_method(method)
    define_method method do |*args|
      ActiveRecord::Base.connection_pool.with_connection do
        old.bind(self).call(*args)
      end
    end
    defer(method)
  end

  In this case, the new method calls the original inside an ActiveRecord with_connection block. This makes sure that connections are released back to the pool every time we use an ActiveRecord model in our methods.

  Finally, the decorator defers the method call onto the EventMachine thread pool. ActiveRecord uses blocking IO, which would block the EventMachine reactor thread. Pushing these calls onto a thread pool allows the reactor thread to continue processing IO while we’re waiting for the database query to return.

  We won’t get into the defer implementation here, but if you’re interested, check out the code.

  Putting It to Use

  Now that we have the with_connection decorator written, we can use it to wrap all other methods that talk to ActiveRecord. Here’s the save_user implementation using the decorator rather than dealing with ActiveRecord pools directly.

  def save_user(user)
    found = User.find_by_jid(user.jid)
    found.name = user.name
    found.password = user.password
    found.save
  end
  with_connection :save_user

  And that’s just one of our query methods. We can wrap the others just as easily and consolidate connection pool handling in one place.

  Metaprogramming is a pretty big hammer to use to solve a simple connection pool problem. But define_method is a nice tool here, because it provides a simple way to write custom storage code without accidentally losing connections or blocking the EventMachine reactor thread.

  - David Graham
  16 Dec 2011

  Metaprogramming ActiveRecord Connection Pools

• Introducing Vines Shell

  We’re proud to announce the third preview release of the Vines XMPP server! Along with many bug fixes and improvements, we’re introducing Vines shell, a new way to chat with servers.

  We ❤ the Command Line

  Vines shell connects us with a bash session on a remote server through the same instant-messenger we’re using to chat with friends. We can send the unix commands we already know and love to the server and it replies with the commands’ output.

  

  Chatting with a server is novel, but it’s not much better than simply using ssh. That’s where services and permissions come in.

  Services

  One limitation of the command line is that we can only control one machine at a time. So, Vines lets us group machines together into “services,” based on attributes they have in common. We might have a Rails Cluster service for our web application or even separate groups for our Fedora and Ubuntu servers.

  We can chat with a service just like a single machine, except every server in the group runs the command simultaneously. For example, rather than patching a few dozen servers independently, we can send yum update to the service and patch all of them at once.

  Trust No One (Or Just Some People)

  Granting access to servers is a pain. We need to create a user account, add it to the right groups, and update the sudoers file. Then repeat those steps on each server to which we need access. On top of that, we need to manage passwords.

  A Vines service defines which unix accounts we’re allowed to use when chatting with those servers. For example, we can limit our Development Web Servers users to the apache and postgresql accounts. Each developer doesn’t need an account on these servers because they can get access to the tools they need through Vines.

  We can give root access, to services, to the system administrators we already trust with that responsibility.

  Try It Out!

  With services and centralized permissions, Vines shell adds some unique features to our trusty command line. Interacting with remote shell sessions is just the beginning of what we can do with servers connected to a real-time message system. There’s much more to come.

  Head on over to the Vines website to get started!

  - David Graham
  03 Nov 2011

  Introducing Vines Shell

• Vines XMPP Server 0.2.0 Released

  We’re proud to announce the second preview release of the Vines XMPP server! Along with many bug fixes and improvements, there are three important new features in this release.

  File Transfers

  The chat server now supports in-band and out-of-band file transfers. You can drag and drop a file onto any XMPP chat client and transfer it to anyone on your buddy list.

  HTTP Server

  We’ve enhanced our web support with a simple HTTP server capable of serving static files. You’ll typically run this, in production, behind a caching proxy server like nginx, Apache mod_proxy, or HAproxy to gain TLS and clustering support.

  Web Chat

  This release includes a new web chat client built entirely with CoffeeScript. There’s no application server to configure (e.g. Rails) as the app connects directly to the XMPP server using HTTP. The web application is available immediately after starting the chat server at http://localhost:5280/chat/.

  

  The user interface is designed for desktop browsers as well as tablets. We’re still tweaking some details on iPad, but give it a try and let us know what you think!

  Try It Out!

  If this is the first time you’re installing Vines, make sure to follow our Ruby install instructions and then run gem install vines. If you already have Vines installed, just update the Ruby gem with gem update vines to install the new version.

  - David Graham
  11 Jul 2011

  Vines XMPP Server 0.2.0 Released

• Why XMPP?

  We’re using XMPP (eXtensible Messaging and Presence Protocol) because it’s a well-defined, open standard, real-time protocol. We could use HTTP for messaging, but it requires cumbersome workarounds for its request/response design to get real-time behavior. Because XMPP was designed as an event-driven messaging system, it also features presence, discovery, and strong security.

  Presence

  Status events, or presence in XMPP, are broadcast to interested parties as they happen, so there’s no need to frequently poll components for their current state. When you sign off of your instant-messenger, a presence event is broadcast to your friends, notifying them of your change in status.

  But presence is much more than just changing your buddy icon from green to red. For example, servers can broadcast changes in their capacity and load to other components, which can then route work to idle systems.

  Discovery

  Each component on an XMPP network can discover the features and capabilities of every other component. Your chat client can ask your buddy’s program if it supports file transfers or video chat. A web application can dynamically discover the location of back-end message queues to send tasks to. Being able to learn about other components on the network allows systems to scale and more easily adapt to failure.

  Security

  XMPP has flexible SASL authentication baked right in. It also mandates the use of TLS encryption on connections to the server. In fact, our Vines server doesn’t support a plain-text, unencrypted mode. All client connections must be encrypted.

  An XMPP domain is a closed system, in the sense that messages can only be sent between two authenticated users in the same domain. This prevents spam and unwanted messages passing through the server. Although XMPP servers can be federated to allow multiple domains to talk to each other, it’s up to you to decide which domains to allow to connect.

  Get Started!

  XMPP is more than just instant-messaging. It’s a mature protocol that can be used to connect components in any scalable web application. Take a look at our open-source Vines XMPP server. We’d love to hear what you think!

  - David Graham
  01 Jul 2011

  Why XMPP?

• Hello World!

  We started Negative Code because we love building software and helping customers. In the past, we’ve built and deployed enterprise systems at some of the world’s biggest companies. But building complicated systems isn’t the only way to go. Elegant, well defined products acting together are easier to understand and maintain. We believe less code is better code.

  Design

  Everything at Negative Code starts with design. Before we write a line of code we’re designing the product. This isn’t just the user interface design. We craft the full user experience, from installing the product, to programmer APIs, all the way through to support calls. It definitely shows when a product has been painstakingly designed. That’s the only kind of software we make.

  Support

  How many times have we been satisfied with a support call or product documentation? Not many. So, we’re always happy to pick up the phone and you’ll always speak with a highly-qualified engineer to help with problems right away. The only reason we build software is to make customers happy. We don’t have a bronze tier for support. We only do platinum.

  Craft

  We love the craft of designing and building software. We’re passionate about testing, programming languages, user interface design, and scalable server systems. We’ll be sharing techniques we’ve learned along the way on this blog and at local user groups and conferences.

  Open Source

  Writing open source software is some of the most fun we have. The community around most projects is welcoming and helpful. As much as possible, we release our software under open source licenses and contribute to other projects upon which we depend. We’re extremely proud of our code and love to share it.

  Hopefully you will enjoy using our software as much as we enjoy building it. Stay tuned!

  - David Graham
  15 Jun 2011

  Hello World!