The Dark Balloon

§

Running Mercurial (and hg record) on top of Subversion is a delicious proposal.

OK, assume you have a Subversion repository. Everybody has one. It’s the cool thing to have these days. They’re the pet rocks of the internet. You read about Mercurial. You think it’s cool. You decide to hoist it over a Subversion repository. So you install it and you come here. Lucky you.

Some preliminaries: You’ve installed Mercurial. You have a dashing sense of danger. You’ve created your hgrc file and added “.svn” to your hgignore file. You’ve inserted hgignore into your hgrc file. Everything’s copacetic.

You’ll also need to add .hg to your svn:ignore property using propset on your SVN repository root.

If you can’t tell, this is all done on Windows. That’s just how old-school I am. Note also that we’re not using any fancy extensions or converters or converter-extensions at all.

> hg init

> hg add .

> hg commit -m "Initiation is go."

Now we’ve committed everything into the Mercurial store. This is normal. Everything’s wavy gravy.

> hg parents
changeset:   0:988dad87d289
tag:         tip
user:        shadytrees <@>
date:        Tue Mar 04 18:46:39 2008 -0500
summary:     Initiation is go.


> hg st

> svn st

See? Everything’s wavy gravy. Now you can just goof around in Emacs.

> rem "Goof around in Emacs."

Oh, if Emacs were only that easy.

> hg diff
diff -r 988dad87d289 befriend_weiss.m
– a/befriend_weiss.m  Tue Mar 04 18:46:39 2008 -0500
+++ b/befriend_weiss.m  Tue Mar 04 19:04:32 2008 -0500
@@ -1,3 +1,6 @@ function befriend_weiss(folder, limit)
+% Look for the limit on modifying Travel.uua until we see a difference
+% between 4p2003 (which should not frameshift properly but does anyway
+% right now) and prfB (which is our golden boy).
 function befriend_weiss(folder, limit)
      multipliers = [9.7:0.01:9.9];
      classify(folder, 'befriend_weiss', @helper);
diff -r 988dad87d289 get_travel.m
– a/get_travel.m      Tue Mar 04 18:46:39 2008 -0500
+++ b/get_travel.m      Tue Mar 04 19:04:32 2008 -0500
@@ -5,9 +5,15 @@ function travel = get_travel(codons, TAV

     max_hund = max(TAV);
     min_hund = min(TAV(find(TAV)));
+
+    % nloopcalc vectorized: convert TAV values to abundance ratios.
+    a = find(TAV == 0);
+    b = find(TAV ~= 0);
+    TAV(a) = 1000;
+    TAV(b) = max_hund/min_hund - floor(TAV(b)/min_hund);
+
     for i = 1:limit
-        k = nloopcalc(i, TAV, max_hund, min_hund);
-        str = [str sprintf('''%s'', %g,', codons{i}, k)];
+        str = [str sprintf('''%s'', %g,', codons{i}, k)];
     end

     % Delete trailing comma.

I’m flaunting my biotechnology project here, but this is the general gist: I’ve (uh-oh) edited two files for two separate reasons. This is, by modern standards, bad since each commit into a repository should represent one modular change to the code. And there’s no easy way to fix this in Subversion. We don’t want a dirty commit; we also don’t want to pull our hair out splitting this commit. (Well, this commit isn’t so bad. But imagine 10+ separate little changes.) Fortunately, Mercurial will set us free.

For the record, the two changes I made were (1) I documented a function I recently wrote named befriend_weiss.m and (2) I rewrote a section of the function get_travel.m. Yes they have silly names. That’s the mark of genius, where by “genius” we mean “annoying programmer.”

> hg record -m "Documents befriend_weiss"
diff —git a/befriend_weiss.m b/befriend_weiss.m
1 hunks, 3 lines changed
examine changes to 'befriend_weiss.m'? [Ynsfdaq?] f
examine changes to 'get_travel.m'? [Ynsfdaq?]  d

> hg log
changeset:   1:0d864c2ec867
tag:         tip
user:        shadytrees <@>
date:        Tue Mar 04 19:05:56 2008 -0500
summary:     Documents befriend_weiss

changeset:   0:988dad87d289
user:        shadytrees <@>
date:        Tue Mar 04 18:46:39 2008 -0500
summary:     Initiation is go.


> svn st
M      befriend_weiss.m

OK, we’ve used the record extension to extricate the befriend_weiss change. Already SVN is complaining with its britches all in knots, wanting to commit that change. Resist the britches!

To review briefly, the record extension goes through each change you made in a revision and interactively asks you to keep it or throw it away. What it does, why you need it, and how it works is beyond this tutorial but check out the link above. I typed “f” to mean “include this file”; I typed “d” to mean “I’m done, ignore everything else which currently is only get_travel.” There’s quite a lot of meaning in those letters. Never underestimate the Roman alphabet.

> hg record -m "Vectorize get_travel (no testing done, warning warning)"
diff —git a/get_travel.m b/get_travel.m
3 hunks, 11 lines changed
examine changes to 'get_travel.m'? [Ynsfdaq?] f

> hg st
M get_travel.m

> hg log
changeset:   2:154c45bdefda
tag:         tip
user:        shadytrees <@>
date:        Tue Mar 04 19:10:39 2008 -0500
summary:     Vectorize get_travel (no testing done, warning warning)

changeset:   1:0d864c2ec867
user:        shadytrees <@>
date:        Tue Mar 04 19:05:56 2008 -0500
summary:     Documents befriend_weiss

changeset:   0:988dad87d289
user:        shadytrees <@>
date:        Tue Mar 04 18:46:39 2008 -0500
summary:     Initiation is go.

OK, we’ve now extricated the other change. If we were to type svn st right now, can you guess what it would display? It would display that both files are modified. There’d be no way to separate those changes out. Fortunately, the Mercurial history knows about our changes separately. We just need to translate that knowledge into SVN-fu.

> hg update -r1
1 files updated, 0 files merged, 0 files removed, 0 files unresolved

I call this the time machine command, which has yet to catch on but it will. It goes back to the first revision, when we first changed befriend_weiss only. Those were simpler times.

> svn st
M      befriend_weiss.m

Hooray! Time machines rock hard.

> svn ci -m "Documenting befriend_weiss"
Sending        befriend_weiss.m
Transmitting file data .
Committed revision 582.

> hg update -r2
1 files updated, 0 files merged, 0 files removed, 0 files unresolved

> svn st
M      get_travel.m

> svn ci -m "Vectorize get_travel (no testing, warning, warning)"
Sending        get_travel.m
Transmitting file data .
Committed revision 583.

Here you can see we’ve done the same for revision two. Now Subversion has two separate, new commits.

There are some disadvantages to doing this, which I hope do not outweigh the advantages. First, there’s no way to access prior Subversion history. I’m sure you could finagle it by copying over the .hg folder from an imported Subversion repository, but I haven’t tried. (In general, Mercurial is pretty well-contained in that nondescript folder so try it and see.) Second, there’s a lot of redundant typing here and this process is just dying to be made into a Mercurial extension if you could get past the poorly documented Mercurial Python modules that are ui and repo.

Advantages: It works. You can use everything available to you in Mercurial to get your ducks (read: commits) in order, including Mercurial Queues and of course branching-merging. And much much more. It works without any complicated intermediate parties, no less. It was made be me, and my motto is “I’m awesome, therefore everything I produce is awesome.” And mottoes never lie.

And now, your moment of zen.

> svn st

> hg st

§

Hum-drum.

This started as really ugly Perl code. But, like a flower or a cocoon or a flower named Cocoon, it blossomed into something beautiful after an hour of reduction, that wonderful sweet process where everything clicks and hums. Java folks call it “refactor,” Haskell folks call it “second-level static typed meromorphism arrow structure on the category Dog”: the fact of the matter is that this is a universal, pan-language feeling. Anyway, what this does is take a Genbank full listing for an organism’s genes, for example Salmonella’s genes, and convert that into a tab-delimited file with–for each gene–the name, start and stop of the gene, and the description. If you have the Salmonella genome, you can then pull all the genes from Salmonella using slurp_ecogene.pl or whatever file we’ll have 1000 revisions from now. A word of caution: The Genbank links go to extremely large web pages in the order of megabytes, megabytes, like all four of them.

use strict; use warnings;

open(my $h, shift);
my @lines = <$h>;
close($h);

sub infer {
    my ($i, $start, $stop) = @_;

    # Lucky for us, the next line is always the /gene. Then we'll
    # grope around in the dark and try to find /product.
    my ($gene) = $lines[++$i] =~ /gene="(.*?)"/;
    $i++ until $lines[$i] =~ /product=/;

    # Some products span more than one line. We'll need to remove
    # initial whitespace and newlines before extracting $desc.
    my $desc;
    for (join '::', @lines[$i .. $i+3]) {
        s/::\s+//g; s/(\r|\n)/ /g;
        ($desc) = /product="(.*?)"/;
    }
    return "$gene\t$start\t$stop\t$desc";
}

# Doing a foreach loop over <$h> (linear) and parsing (decidedly
# nonlinear) is hard, so we'll sacrifice elegance for sanity.
foreach my $i (0 .. $#lines) {
    local $_ = $lines[$i];
    next unless /^\s+CDS/;
    my ($start, $stop) = /(\d+)..(\d+)/;

    # We're matching for complement(#..#). It's good to know, so
    # we'll mark it with a negative $start.
    $start *= -1 if /complement\(/;
    print infer($i, $start, $stop), "\n";
}