Safety in Numbers

Posted by Andy on Wednesday, September 01, 2010.

Finding a GOOD home for Wowcrowd

Quick version: Wowcrowd, our web application for crowdsourced funding allocation, has been acquired by our friends at GOOD.

Now here’s the background.

We started Project Fund as our main corporate philanthropy initiative a year ago to fill a need we saw in the climate community: neighborhood-scale climate projects needed quick access to small grants, and they were willing to work hard to get them.

To make a long story short, Project Fund was a runaway success, one we’re very proud of. With 199 projects submitted and over 100,000 votes cast, we were able to give $50,000 in grants to 10 spectacular causes.

But Project Fund was also successful in another unexpected way: the “crowdsourced funding allocation” model itself was a hit, winning us a Social Innovation award from Financial Times. When the Pepsi Refresh Project launched in January, it took the concept to a totally different scale. That’s when “How could I start my own Project Fund?” became one of the most common questions we at Brighter Planet heard.

So this past winter we spun up a new Ruby on Rails app codenamed wlpf1 (“white-label Project Fund”) and started filling it with Project Fund code extracted from our main web application. A few months later Wowcrowd was born.

Recently we realized that somebody else could probably do a better job of running with Wowcrowd. After all, we’re a climate company. The very first group that came to mind was GOOD, not only because they’re friends of ours, but also because they are an active partner in the Pepsi Refresh Project and have been closely involved since its launch earlier this year. They’re also probably the world’s best folks at connecting great people with great causes.

And a handful of weeks later, the ink is dry. Here’s the press release. GOOD, she’s in your hands. We can’t wait to see where you take this.

Discussion at the permalink
Posted by Andy on Friday, August 13, 2010.

Announcing Carbon Middleware

Brighter Planet has been building technology for climate change since 2007, including a fully automated carbon offset provision, retirement, and assignment system (to interface with Bank of America and power our credit and debit cards) and what we feel is the world’s best carbon profiler for individuals.

We’re opening up our technology as a platform, and we can’t wait to see what can be done with these systems. Have a look at the new site and tell us what you think.

Discussion at the permalink

Posted by Seamus on Wednesday, August 11, 2010.

Use Amazon SQS to get emission estimates

You can get emission estimates by queueing up messages on Amazon SQS:

$ curl -v https://queue.amazonaws.com/121562143717/cm1_production_incoming -X POST --data "Action=SendMessage&Version=2009-02-01&MessageBody=emitter%3Dautomobile%26make%3DNissan%26guid%3DMyFavoriteCar%26key%3D86f7e437faa5a7fce15d1ddcb9eaeaea377667b8"
* About to connect() to queue.amazonaws.com port 443 (#0)
*   Trying 72.21.211.87... connected
* Connected to queue.amazonaws.com (72.21.211.87) port 443 (#0)
  # eliding some standard server messages for brevity . . .
> POST /121562143717/cm1_production_incoming HTTP/1.1
> User-Agent: curl/7.20.0 (i386-apple-darwin9.8.0) libcurl/7.20.0 OpenSSL/0.9.8m zlib/1.2.3 libidn/1.16
> Host: queue.amazonaws.com
> Accept: */*
> Content-Length: 158
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 200 OK
< Content-Type: text/xml
< Transfer-Encoding: chunked
< Date: Wed, 11 Aug 2010 08:20:56 GMT
< Server: AWS Simple Queue Service
< 
<?xml version="1.0"?>
* Connection #0 to host queue.amazonaws.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
<SendMessageResponse xmlns="http://queue.amazonaws.com/doc/2009-02-01/"><SendMessageResult><MD5OfMessageBody>fb29551a9e1c36dcf1b8ba624695210a</MD5OfMessageBody><MessageId>5d48a63b-5416-4d35-8bc5-65ace74f4d42</MessageId></SendMessageResult><ResponseMetadata><RequestId>a3e0d71a-9e47-493b-a92c-29b37393e899</RequestId></ResponseMetadata></SendMessageResponse>

You need to use this SQS queue: (but you don’t need an SQS account, it’s just a standard HTTP POST)

https://queue.amazonaws.com/121562143717/cm1_production_incoming

As you can see, the MessageBody is the url-encoded form of a querystring:

emitter=automobile&make=Nissan&guid=MyFavoriteCar&key=86f7e437faa5a7fce15d1ddcb9eaeaea377667b8

Wait a minute, isn’t that an empty response body?

Correct. This is an asynchronous way of doing things. The result in JSON format will appear as soon as it is calculated (usually in a few seconds).

If you need an answer in realtime, then you should skip SQS and hit mostly the same querystring.

Why use SQS?

Depending on your environment, you may already have an excellent SQS client library. You’ll have the high availability of Amazon web services combined with competitive Brighter Planet pricing for asynchronous (rather than realtime) emission estimates.

Depending on your application, set-it-and-forget-it may be a natural fit. We’ll store the result for you in JSON format at an effectively randomized URL, which is perfect for AJAX calls that display results in a browser. In general, if you can queue up the emission estimate now and count on a JSON-enabled client to pull the results later, this is a good way to go.

How did you calculate the result URL?

Just SHA1 the string key plus guid. No salt or separators or anything, just 86f7e437faa5a7fce15d1ddcb9eaeaea377667b8MyFavoriteCar in this case.

ruby-1.8.7-head > require 'digest/sha1'
 => true 
ruby-1.8.7-head > Digest::SHA1.hexdigest('86f7e437faa5a7fce15d1ddcb9eaeaea377667b8MyFavoriteCar')
 => "745c4bcda8234186178e8430ae55f38913a5f042"

Other notes

We host the storage on Amazon S3, so you’ll benefit from high-availability there too.
If you’re using Ruby, you can use our carbon gem with the :guid option.
If you would rather have us POST the result back to your waiting servers, you can pass &callback=http%3A%2F%2Fmyserver.example.com%2Fcallback-receiver.php.

Discussion at the permalink

Posted by Andy on Monday, August 09, 2010.

Sharing Rails views with Jekyll

In my last post I discussed how we share a single layout between Rails apps. This has been a lifesaver for us as we manage a half-dozen production apps. But a couple of our sites—our developer hub and this here blog—don’t use Rails. They’re both Jekyll sites running on GitHub Pages.

Obviously we can’t rely on the Rails Engine features in our shared layout gem to load the layout into the right places. What are we to do?

One step at a time

Luckily Jekyll already includes the basic building blocks of our solution: layouts and includes. Layouts, described here are Liquid templates, and Jekyll ships with a custom Liquid extension that enables includes.

All we need to do then is transform our Rails layout into a Jekyll layout and use includes instead of partials. Ready, set, go.

When in Rome

Jekyll is a static site generator. Following this lead, our transformations will be manually executed and staticly stored within your Jekyll site. The easiest way to get started is to set up a task in your Rakefile:
```
require 'net/http'
require 'uri'
require 'erb'
require 'lib/stubs'
namespace :layout do
  task :build do
    File.open File.join(File.dirname(__FILE__), '_layouts', 'default.html'), 'w' do |f|
      f.puts ERB.new(Net::HTTP.get(URI.parse('http://github.com/brighterplanet/brighter_planet_layout/raw/master/app/views/layouts/brighter_planet.html.erb'))).result(Layout.new.get_binding  { |*pages| '{ { content } }' if pages.empty? })
    end
  end
end
```
What’s going on here? Rake is fetching the raw ERB of your layout from the gem’s repository, sending it to ERB for processing, and then storing the result as your Jekyll site’s default layout.

I should call your attention to a couple of tricky bits here.

Bindings

First, this business about bindings. ERB needs a “binding” to work–that is, a context within which it can access instance methods, variables, etc. Rails takes care of this for you, but since we’re invoking ERB here directly, we have to tell it where to bind. Why is this important? Your layout probably uses methods like stylesheet_link_tag or render to get its job done. If we don’t provide those methods in ERB’s context, we’ll get NoMethodError all over the place. The easiest way to fool ERB is with a fake context, which we’ll put in lib/stubs.rb:
```
class Layout
  def stylesheet_link_tag(*sheets)
    sheets.collect do |sheet|
      "<link rel=\"stylesheet\" type=\"text/css\" href=\"/stylesheets/#{sheet}.css\" />"
    end
  end
  
  def javascript_include_tag(*args); end
  
  def render(options = {})
    "{٪ include #{options[:partial][/[a-z_]*$/]}.html ٪}"
  end
  
  def get_binding
    binding
  end
end
```
You can see how we re-interpret these method calls in a way that’s meaningful to Jekyll. (Note that since binding is a private method we have to publicize it with the get_binding wrapper.)

Yields

The second tricky bit is dealing with your standard Rails layout’s multiple yield calls, the consequence of using content_for blocks in your views. We have to anticipate this and set up ERB to act accordingly. Where do we even capture arguments to yield? Turns out the correct place to do this on get_binding, our wrapper to the private binding method. Now the yield we’re interested in—the one where we want Jekyll content to go—is the one called without any arguments. So we set the block to output the content Liquid tag when it sees yield called with an empty argument set. Other yield calls—to dump content_for material, which could never be prepared by Jekyll anyways—are simply ignored.

And we’re done

Your complete layout package will probably include several partials—each with its own fake context class in stubs.rb—as well as asset files. To build your layout from its source in the cloud, just
```
$ rake layout:build
```
Check out our developer hub’s Rakefile and stubs.rb for all the details.

Discussion at the permalink
Posted by Derek on Wednesday, July 28, 2010.

Bundler to the Max

I have been spending the past few weeks creating and refactoring our carbon model gems, with the goal of making them easy to enhance, fix, and test by climate scientists and Ruby developers. I wanted to make contributing a simple process and bundler fit the bill quite well.

A not-so-widely-known feature of the Rubygems API is the ability to declare a gem’s development dependencies, along with its runtime dependencies. If one planned on making changes to one of the emitter gems and testing it, she could run gem install <emitter_gem> --development and have any needed testing gems installed for the emitter gem.

This is all fine and good, but I chose to use bundler to manage our dependencies, as it adds a few extras that have been a tremendous help to us. To contribute to any of our gems, a developer can follow a simple process:
```
$ git clone git://github.com/brighterplanet/<gem>.git
$ cd <gem>
$ gem install bundler --pre  # this is needed until bundler 1.0 is released
$ bundle install
$ rake
```
And Bob’s your uncle!

Bundler + Gemspecs

The first goodie that bundler provides is the ability to use the gem’s own gemspec to define the dependencies needed for development. For instance, our flight gem has a gemspec with dependencies:
```
Gem::Specification.new do |s|
  # ...
  s.add_development_dependency(%q<activerecord>, ["= 3.0.0.beta4"])
  s.add_development_dependency(%q<bundler>, [">= 1.0.0.beta.2"])
  s.add_development_dependency(%q<cucumber>, ["= 0.8.3"])
  s.add_development_dependency(%q<jeweler>, ["= 1.4.0"])
  s.add_development_dependency(%q<rake>, [">= 0"])
  s.add_development_dependency(%q<rdoc>, [">= 0"])
  s.add_development_dependency(%q<rspec>, ["= 2.0.0.beta.17"])
  s.add_development_dependency(%q<sniff>, ["= 0.0.10"])
  s.add_runtime_dependency(%q<characterizable>, ["= 0.0.12"])
  s.add_runtime_dependency(%q<data_miner>, ["= 0.5.2"])
  s.add_runtime_dependency(%q<earth>, ["= 0.0.7"])
  s.add_runtime_dependency(%q<falls_back_on>, ["= 0.0.2"])
  s.add_runtime_dependency(%q<fast_timestamp>, ["= 0.0.4"])
  s.add_runtime_dependency(%q<leap>, ["= 0.4.1"])
  s.add_runtime_dependency(%q<summary_judgement>, ["= 1.3.8"])
  s.add_runtime_dependency(%q<timeframe>, ["= 0.0.8"])
  s.add_runtime_dependency(%q<weighted_average>, ["= 0.0.4"])
  # ...
end
```
Instead of defining these dependencies in both flight.gemspec and in Gemfile, we can instead give the following directive in our Gemfile:
```
gemspec :path => '.'
```
Bundler + Paths

We have a chain of gem dependencies, where an emitter gem depends on the sniff gem for development, which in turn depends on the earth gem for data models. In the olden days (like, 4 months ago) if I made a change to sniff, I would have to rebuild the gem and reinstall it. With bundler, I can simply tell my emitter gem to use a path to my local sniff repo as the gem source:
```
gem 'sniff', :path => '../sniff'
```
Now, any changes I make to sniff instantly appear in the emitter gem!

I had to add some special logic (a hack, if you will) to my gemspec definition for this to work, because the above gem statement in my Gemfile would conflict with the dependency listed in my gemspec (remember, I’m using my gemspec to tell bundler what gems I need). To get around this, I added an if clause to my gemspec definition that checks for an environment variable. If this variable exists, the gemspec will not request the gem and bundler will instead use my custom gem requirement that uses a local path:
```
# Rakefile  (we use jeweler to generate our gemspecs)
Jeweler::Tasks.new do |gem|
  # ...
  gem.add_development_dependency 'sniff', '=0.0.10' unless ENV['LOCAL_SNIFF']
  # ...
end
```
```
# Gemfile
gem 'sniff', :path => ENV['LOCAL_SNIFF'] if ENV['LOCAL_SNIFF']
```
So now, if I want to make some changes to the sniff gem and test them out in my emitter, I do:
```
$ cd sniff
  # work work work
$ cd ../[emitter]
$ export LOCAL_SNIFF=~/sniff
$ rake gemspec
$ bundle update
  # ...
sniff (0.0.13) using path /Users/dkastner/sniff
  # ...
```
And then Bob is my uncle.

Bundler + Rakefile

This next idea has some drawbacks in terms of code cleanliness, but I think it offers a good way to point contributers in the right direction. One thing that frustrated me about Jeweler was that if I wanted to contribute to a gem, my typical work flow went like:
```
$ cd [project]
  # work work work
$ rake test
LoadError: No such file: 'jeweler'
$ gem install jeweler
$ rake test
LoadError: No such file: 'shoulda'
  # etc etc
```
I attempted to simplify this process, so a new developer who doesn’t read the README should be able to just do:
```
$ cd [emitter]
  # work work work
$ rake test
You need to `gem install bundler` and then run `bundle install` to run rake tasks
$ gem install bundler
$ bundle install
$ rake test
All tests pass!
```
I achieved this by adding the following code to the top of the Rakefile:
```
require 'rubygems'
begin
  require 'bundler'
  Bundler.setup
rescue LoadError
  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
end
```
This was convenient, but it created a chicken and egg problem: in order to generate a gemspec for the first time, bundler needed to know which dependencies it needed, which meant that it needed the gemspec, which is generated by the Rakefile, which requires bundler, which requires the gemspec, etc. etc. I overcame this problem by allowing an override:
```
require 'rubygems'
unless ENV['NOBUNDLE']
  begin
    require 'bundler'
    Bundler.setup
  rescue LoadError
    puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
  end
end
```
So, if you’re really desparate, you can run rake test NOBUNDLE=true

More on Local Gems

Now that I had a way to easily tell bundler to use an actual gem or a local repo holding the gem, I wanted a way to quickly “flip the switch.” I wrote up a quick function in my ~/.bash_profile:
```
function mod_devgem() {
  var="LOCAL_`echo $2 | tr 'a-z' 'A-Z'`"

  if [ "$1" == "disable" ]
  then
    echo "unset $var"
    unset $var
  else
    dir=${3:-"~/$2"}
    echo "export $var=$dir"
    export $var=$dir
  fi
}

function devgems () {
  # Usage: devgems [enable|disable] [gemname]
  cmd=${1:-"enable"}

  if [ "$1" == "list" ]
  then
    env | grep LOCAL
    return
  fi

  if [ -z $2 ]
  then
    mod_devgem $cmd characterizable
    mod_devgem $cmd cohort_scope
    mod_devgem $cmd falls_back_on
    mod_devgem $cmd leap
    mod_devgem $cmd loose_tight_dictionary
    mod_devgem $cmd sniff
    mod_devgem $cmd data_miner
    mod_devgem $cmd earth
  else
    mod_devgem $cmd $2
  fi
}
```
This gives me a few commands:
```
$ devgems enable sniff
  # sets LOCAL_SNIFF=~/sniff
$ devgems disable sniff
  # clears LOCAL_SNIFF
$ devgems list
  # lists each LOCAL_ environment variable
```
I now have a well-oiled gem development machine!

Overall, after a few frustrations with bundler, I’m now quite happy with it, especially the power and convenience it gives me in developing gems.

I’m really interested to hear any of your thoughts on this. Drop me a line at @dkastner.

Discussion at the permalink

Who's behind this?

We're Brighter Planet, and we deliver meaningful, transparent carbon information to businesses and developers.

Who's blogging here?

Seamus Abshere Engineering director
Robbie Adler Biz dev director
Ian Hough Staff scientist
Derek Kastner Rails developer
Matt Kling Science analyst
Patti Prairie CEO
Andy Rossmeissl Co-founder

What are we talking about?

Feed

Site generated by jekyll at Wed Sep 01 15:23:19 -0700 2010

Brighter Planet

Safety in Numbers

Brighter Planet's blog about climate science, Ruby, Rails, data, transparency, and, well, us.

Finding a GOOD home for Wowcrowd

Announcing Carbon Middleware

Use Amazon SQS to get emission estimates

Wait a minute, isn’t that an empty response body?

Why use SQS?

How did you calculate the result URL?

Other notes

Sharing Rails views with Jekyll

One step at a time

When in Rome

Bindings

Yields

And we’re done

Bundler to the Max

Bundler + Gemspecs

Bundler + Paths

Bundler + Rakefile

More on Local Gems

Who's behind this?

Who's blogging here?

What are we talking about?

Shortcuts

Brighter Planet

Carbon Middleware

Live applications