-
A new offset project
We’ve got a new offset project, and once again it’s the first of it’s kind for Brighter Planet. The Rentech Fertilizer Plant project supports a new N2O destruction system at midwestern fertilizer plant, and is certified under the Climate Action Reserve.
-
Split XML files with `sgrep`, a classic UNIX utility from 1995
sgrepis better thansplitorcsplitfor breaking up XML files by element – you can even use it to create a constant-memory streaming “parser.”$ sgrep -o "XXXSTART%rSTOPXXX" '"<TourismEntity" .. "</TourismEntity"' transmission_file.xml XXXSTART<TourismEntity> <State>New York</State> <Saying>I♥NY</Saying> </TourismEntitySTOPXXXXXXSTART<TourismEntity> <State>Virginia</State> <Saying>Is For Lovers</Saying> </TourismEntitySTOPXXXXXXSTART<TourismEntity> <State>Wisconsin</State> <Saying>America's Dairyland</Saying> </TourismEntitySTOPXXX(see below for why that output is useful)
-
Upsert for MySQL, PostgreSQL, and SQLite3 (and Ruby)
Our
upsertlibrary for Ruby gives you NoSQL-likeupsertfunctionality in traditional RDBMS databases. How?- MySQL’s native
INSERT ... ON DUPLICATE KEY UPDATE - PostgreSQL’s canonical
CREATE FUNCTION merge_db - SQLite3’s
INSERT OR IGNOREplus a trailingUPDATEstatement
50%–80% faster than ActiveRecord
New in 0.4.0: When used in PostgreSQL mode, database functions are re-used, so you don’t have to be in batch mode to get the speed advantage.
You don’t need ActiveRecord to use it, but it’s benchmarked against ActiveRecord and found to be up to 50% to 80% faster than traditional techniques for emulating upsert:
# postgresql (pg library) Upsert was 78% faster than find + new/set/save Upsert was 78% faster than find_or_create + update_attributes Upsert was 88% faster than create + rescue/find/update # mysql (mysql2 library) Upsert was 46% faster than find + new/set/save Upsert was 63% faster than find_or_create + update_attributes Upsert was 74% faster than create + rescue/find/update Upsert was 28% faster than faking upserts with activerecord-import (which uses ON DUPLICATE KEY UPDATE) # sqlite3 Upsert was 72% faster than find + new/set/save Upsert was 74% faster than find_or_create + update_attributes Upsert was 83% faster than create + rescue/find/update(run the tests on your own machine to get these benchmarks)
- MySQL’s native
-
Graphite and statsd – beyond the basics
The Graphite and statsd systems have been popular choices lately for recording system statistics, but there isn’t much written beyond how to get the basic system set up. Here are a few tips that will make your life easier.
-
Analyze CREATE TABLE SQL with pure Ruby
You can use the new create_table library to analyze and inspect CREATE TABLE statements (what is the primary key? what are the column data types? what are the defaults?) You can also generate SQL that works with different databases.
>> require 'create_table' => true >> c = CreateTable.new(%{ CREATE TABLE employees (employeeid INTEGER NOT NULL, lastname VARCHAR(25) NOT NULL, firstname VARCHAR(25) NOT NULL, reportsto INTEGER NULL); }) => #<CreateTable> >> c.columns.map(&:name) => ["employeeid", "lastname", "firstname", "reportsto"] >> c.columns.map(&:data_type) => ["INTEGER", "CHARACTER VARYING(25)", "CHARACTER VARYING(25)", "INTEGER"] >> c.columns.map(&:allow_null) => [false, false, false, true]
Read archived posts in our Company, Science, and Technology sections.
Who's behind this?
We're Brighter Planet, the world's leading computational sustainability platform.
Who's blogging here?
What are we talking about?
Site generated by jekyll at Mon Oct 01 16:35:27 -0700 2012