I’m hoping this patch is reverted and instead rolled out for 3.0.

Thanks to capistrano, the deployment tool, this turned out to be fairly easy. Capistrano already collects the deployed version and places it into a file named REVISION in the deployment directory. The contents hold whatever is reported by your version control software. For git, this is a SHA1 hash.

For the deployment timestamp, thankfully capistrano uses a timestamp for the deployment directory name. So we can just parse the path, looking for the timestamp.

Below is the helper code, please let me know if there’s a better way.

  # pulls from the capistrano directory structure
  def deploy_timestamp
    if Rails.env.production?
      RAILS_ROOT.split('/').last
    else
      "Undeployed"
    end
  end

  def deploy_version
    if Rails.env.production?
      File.read("#{RAILS_ROOT}/REVISION")
    else
      "DEV - " + `git rev-parse HEAD`
    end
  end

I know you just got bought by Sun, and are now swimming in money and possibilities. Please continue this momentum by formalizing your connection to the Web by including a RESTful end point in your next version. The Web is where you grew up and matured, and it’s time to give back.

The architecture wars are nearly over, and REST has proven time and again to be the dominant architectural style for Web applications. You, MySQL, need to expose your data via REST to promote interoperability across clients and further cement your place as one of the top databases for the Web.

This is low hanging fruit.

Your URL space is well defined, as you have databases, tables and views, rows, and columns. The HTTP methods (GET, PUT, POST, DELETE) map very well to your SQL DML methods of SELECT, INSERT, UPDATE, and DELETE.

I’d imagine your URL space to be something like:

• /databases
• /databases/NAME/tables
• /databases/NAME/tables/NAME/rows
• /databases/NAME/tables/NAME/rows/PK

Use Content Negotiation to return either XML or JSON. Use hypermedia to encapsulate the connections between rows. Use HTTP Basic Auth and plug in to your existing user management (use SSL for security.) Take advantage of your fondness for Memcached.

With CouchDB promoting a REST interface, any language that can speak JSON and HTTP can build a trivial client.

MySQL, the Web is where you flurished. Time to join the Web.

Sincerely,

Seth

{ "_id": "_design/counts",
  "language": "javascript",
  "views": {
    "all": {
      "map": "function(doc) { emit(null, 1); }",
      "reduce": "function(keys, values, combine) { return sum(values); }"
    }
  }
}

I started with my crazy hack to use bit strings to calculate combinations. I didn’t want to resort to creating bit strings, as I wanted to minimize the amount of work I needed to do. Therefore, I thought about simply using bitwise operators.

Below is my Java algorithm for creating calculations using iteration and bitwise operators to create all combinations from an array.

Thoughts?

public static void generate(String[] list) {
	int max = (int) Math.pow(list.length, 2)-1;
	System.out.println(max + " combinations");
	for (long i = 0; i < max; i++) {
		String[] combo = new String[Long.bitCount(i)];
		int comboPos = 0;
		for (int j = 0; j < list.length; j++) {
			if ((i & (1L< 0) {
				combo[comboPos++] = list[j];
			}
		}
		//System.out.println(Arrays.toString(combo));
	}
}

Update: handle up to 64 slots. Will need to move to something like BigInteger to handle greater than 64 bits I suppose. Wonder how that will affect performance?

Assuming you have FreeTDS installed (easily through macports) and iODBC (comes out of the box with Mac OS X) then this is all you need in your database.yml file:

development:
  adapter: sqlserver
  mode: odbc
  dsn: DRIVER=/opt/local/lib/libtdsodbc.so;TDS_Version=8.0;SERVER=10.0.6.20;DATABASE=awesome_development;Port=1433;uid=sa;pwd=password;

No configuration of a DSN required, and no mucking about with freetds.conf!

Now, if I could just use database.yml to get rid of SQL Server all together, I’d be a happy developer.

Several of us engineers at Twitter, Inc. are learning Scala as the language in which to develop new components for our system.

From Graceless Failures: Hello World

Make sure you install the Rails 2.1 branch of Active Scaffold:

git clone git://github.com/activescaffold/active_scaffold
cd active_scaffold
git branch -r (just lists the branches)
git checkout origin/rails-2.1

(Check out the full thread for Rails 2.1 and Active Scaffold compatibility)

Second, I had to patch vendor/plugins/active_scaffold/lib/extensions/generic_view_path.rb. On line 53, make it look like this:


if !@template.controller.is_a?(ActionMailer::Base) && @template.controller.class.uses_active_scaffold?


I had to add !@template.controller.is_a?(ActionMailer::Base) &&

All my tests are now passing!

I still love Active Scaffold, even though it’s a bit behind the times.

The first question you need to ask is, Do you need exact counts or will approximate counts be good enough? I bet in many situations, an approximate count will be perfectly reasonable. Think about the use case of tracking web hits. When you’re talking about millions of hits, what is the difference between 1,000,000 and 1,000,001? Of course, only your business expert will know if approximate or exact answers are required. The decision, though, is crucial because it’s the difference between an easy implementation and a hard (costly) implementation.

Let’s say, for the purposes of this article, that you’ll need very close to accurate counts, plus you need to scale a lot. The first step is to pre-calculate the count, and cache the results. When a new web hit occurs, grab the current count, add one, and put it back. This approach will scale for a while, but the chance of missing a count goes up as load goes up. Because we’re not explicitly locking on the row (which can be expensive), the last person to write the record back to the database wins.

The next option is to wrap the “grab record, increment, put record back” inside a locking transaction. This will ensure that only one writer can access the counter at a time. This ensures an accurate count, but will greatly slow down the site as contention around the single counter increases.

The third option, and the best option, is to split the counter up into smaller counters. When it’s time to get the full, single count, simply grab all the counter partitions and add them up. For very high loads, increase the number of partitions. The theory is it’s quick to add up 100 partitions, while you’re providing 100 different counters to lock around.

How do you pick which partition to increment? One easy way is to create a hash of the timestamp (or some other part of the request that changes frequently) of the request, and mod it on the number of partitions in the system. The theory here is you’ll be spreading the load across the partitions as the number of concurrent requests increases.

In any scalable web system, reads should be by key and writes are expensive. Do whatever you can to read a single object by a key, and minimize your writes. Minimize the contention around objects in the data store, too. Realize that ad hoc queries can almost always be implemented by pre-calculating the answers, so that an ad hoc query is simply retrieving a record by a key (instead of scanning through all rows, computing the answer as you go.)

For more on this technique, I recommend the excellent video Builing Scalable Web Applications with Google App Engine.