Skip to: Navigation | Content | Sidebar | Footer


Weblog Entry

Sneakernet SVN

August 18, 2008

Over the past three months I’ve gotten myself used to checking all my projects into a pair of Subversion repositories, one for personal projects and another for all my client work. In both cases I’m the only person using them, so neither are your typical SVN use case scenarios, but I’m still finding my system enormously handy for a few reasons:

  • I work off two computers during any given week, and though I’ve been good about keeping them synchronized, I have had the occasional few hours of work wiped out by a wrong-direction file transfer. Plus the mental overhead of manually keeping track of which files have changed meant I didn’t do it as often as I should.
  • I have at least three copies of my work on hand, which means I’m well situated if any one or two copies crap out on me, or get stolen, or whatever.
  • I can revert to a previous version of my work if I ever realize down the road I needed something from a file I deleted three months ago. This one is mostly theoretical for now, as I haven’t had to do it yet, but it’s nice to know the option is there.

But here’s the twist: my repositories exist on USB keys. I decided against storing them on a server, so I needed some other way to create relatively portable repositories. USB drives fit that bill. Now there are two immediate flaws with this setup:

Security
A USB key is a physical object that could wind up in someone else’s possession, through theft or carelessness or any other reason. I don’t want anyone else having access to work in progress or my personal files.
Data Loss
Also through losing the key, or through physical damage, my work could disappear.

The second point is relatively easy to shrug off: I have the master files in three locations, it’s just one of those that I’d be losing. Okay, so I’d also lose the change history, but in my mind the most important piece is the current master copy of the files, which I’d have safely backed up in two other places. So, while being forced to recreate the repository wouldn’t be ideal, it also wouldn’t be a huge hardship.

However it’s that first point that concerned me more. I had to get a little more proactive about securing the USB key, but luckily OS X makes that fairly simple to accomplish. Using the built-in Disk Utility (Applications > Utilities > Disk Utility) it’s easy to create an encrypted disk image that protects its contents from prying eyes:

  1. Open Disk Utility, hit the “New Image” button.
  2. Give the image a name; I’d highly recommend naming it differently from your USB key to avoid confusion.
  3. Select your USB key from the left hand device listing.
  4. Select a size that will fit on your key. For an 8GB USB key I went with a 7.5GB disk image to account for byte count differences, though I probably could have gone a bit higher.
  5. Set encryption to AES-128.
  6. Leave format as “read/write”, then hit Create. Enter your password when prompted, and verify it. Do not lose or forget this password.
  7. Wait; it will take quite a while to write a large disk image over USB.

What you’ve created is a secondary virtual disk on the USB key, but this one’s encrypted. 128 bit AES isn’t uncrackable, but if it’s secure enough for classified government documents, it’s good enough to render the disk image useless to virtually anyone who doesn’t have your password.

Now to use this image you first need to manually mount it by inserting the key, then double-clicking on the image that sits on the key. You’ll know it’s mounted when you see two drive icons in your Mac’s device listing, one for the physical USB key and one for the virtual disk image.

Once the encrypted image is mounted you can read and write files inside of it like you would any other disk, so this is where the repository goes. I took the easy way out and just grabbed the latest beta of Versions and used it to set up my initial repository by pointing it at a master copy of my project folder on one computer, and then checking out the repository on the other computer.

And from then on it’s SVN as usual; my projects are kept up to date on both systems, and I can make changes on either end with the confidence that I’m not going to lose critical data. And this have been an extremely useful system for me, as evidenced by the fact I’m still using it religiously three months later. But there are a few minor quibbles I should point out.

I do have to keep swapping the USB key back and forth between machines (hence the term Sneakernet in the title of this post). I also need to hit eject twice every time I want to remove the USB key: once for the disk image and once for the USB key itself. I wouldn’t have to do this if I hosted the repository on a server somewhere.

I also need to remember to keep the key with me when I’m likely to require it; having it on my keychain was working fine for that… up until the metal clasp broke. But I’m also finding the process of checking in my work is making me more aware of when one computer or the other is out of date, and usually when I grab the MacBook to go work in a cafe or elsewhere I’ll remember that I need the USB key with me. So far so good, but I can at least see the potential for this being a problem now and then.

And finally, a USB key is limited in space; I’ve been checking in all my work, from HTML and CSS to PSD and AI files. Some of these are many megabytes in size, and the space requirement for multiple revisions is not insignificant.

While I can foresee the need to maintain the repository down the road by trimming the oldest revisions to free up space, I haven’t had to cross that bridge yet. And I also half suspect that simply moving the repository to a larger USB key will be the path of least resistance, as larger capacities are introduced and prices come down.


Daryl says:
August 19, 16h

Hi Dave, that’s an interesting system. I’m a little unclear on why you don’t use a central server. For me, a major benefit of the central server is the fact that if I need to work on the code on a completely different computer, it’s accessible (I just check it out from the server).

If I were you, I’d be deathly worried I’d lose the key! That’s all your backups.

August 19, 16h

What made you shy away from keeping the repository on a server?

Andrew says:
August 19, 16h

I’ve never used subversion (used alienbrain briefly, I assume it’s similar). and If i read this right you’re keeping all your files on the USB key?

how big is that USB key? my working clients folder is huge!

John B says:
August 19, 16h

I’ve recently started checking stuff into an SVN repository too! I haven’t been at it as long as you have, but I’m really enjoying it. I ended up putting my repos on the PC that I don’t use very much since I bought a mac - it was much easier than I thought, (click my name above and scroll down for a full description).

One note - I recently had to go back to an older revision, and it is a bit of a pain in the behind. Also, I have been having problems with Versions if I have to do anything out of the ordinary, (like go back to an older revision).

In the end, I am enjoying my SVN server. I am now collaborating with someone on a project using it and it works extremely well.

5
Kenzie Campbell says:
August 19, 16h

I wonder if anyone offers colocation for USB keys?

Dave S. says:
August 19, 16h

@Daryl - I thought it was clear in the post, but I have a backup of the current work in three different places including the key, so I’m not at all worried about losing it.

@Josh - Initially the main reason for not going with a server is because I still intend to get the hell off Dreamhost one day and I only want to set up a remote repository once. Then when I thought about the times I don’t have wifi USB started making sense, and now that I have it all in AES-128 I’m thinking I’m more secure this way anyway.

@Andrew - One’s 8GB, one’s 4GB. I archived a bunch of non-active work before the first check-in, so at the moment it all fits comfortably.

cam c. says:
August 19, 17h

Cool idea… I’ve been looking into setting up an SVN repo for my work as well but the logistics have been holding me back; your way would definitely solve some of the problems I’ve been thinking about. I’m planning on setting up a collocation server though, which might make keeping a repo on a live server less of a chore… still an interesting alternative though.

I’ve also been hearing a lot about Git these days and how it treats even your local copy as a repository, with version history etc. That would be another potential solution, but as far as I can tell, there’s not much Windows or Mac support for that version control system.

August 19, 17h

Dave,

You might want to look into Git. I’ve converted from SVN and it’s pretty straight-forward, most of the Rails community has now moved onto it.

It’s a distributed VCS meaning you can commit changes locally and then ‘sync’ with a centrally stored master copy. It also seems to be *much* faster at pushing changes over the wire.

I use http://github.com for all my source code needs… it really is rather splendid.

9
Thomas B says:
August 19, 18h

Very cool. I’ve been meaning to start using SVN (currently I just use my IDE [Aptana Studio] and my server to upload/download projects). I’m in the same situation (I’d be the only one to check anything in or out of the repository), but it would still be useful for the same reasons that you mentioned.

Using a server instead of USB drive would be easier of course, but I can understand your desire to set up a remote repository only once

10
Jason M says:
August 19, 18h

@cam c.
Distributed version control (like Git) is really nice. Both Mercurial http://www.selenic.com/mercurial/wiki/ and Bazaar http://bazaar-vcs.org/ are good systems with good windows support. I personally use Bazaar.

August 19, 21h

Seconding the DVCS recommendation. It sounds like the lack of a centralized server would really benefit you in this case where there’s a portable setup and there might be a case where multiple repos exist and need to sync from one another without necessarily having the centralized one available. Git can be built on Macs, and there’s a msysgit port available on Windows that’s fairly usable.

For a Subversion user, though, I’d recommend Mercurial. It’s roughly equivalent to Git in terms of features (notable exception: ability to rewrite history as easily), but the command set is much closer to SVN and works better on Windows/Macs because it’s written in Python. It’s also, like many (all?) DVCSes, much faster than SVN.

August 20, 03h

Uh oh, you’ve mentioned VCS in public Dave, these days that always results in a chorus of “use git! no, use bzr! no, use mercurial!” :)

But none of these DVCS have nice GUI apps like Versions, which sounds like it’s important to you.

That said, anyone using SVN and willing to use a command line should take a look at SVK (1). Like the DVCS mentioned it allows offline commits and has better merge than SVN itself. SVK uses the SVN ‘filesystem’ for it’s storage format, so you can still use GUI tools to look at your repository, just don’t use them to work on the repository.

With SVK you’d keep the main SVN repository on one of the machines and use SVK to keep a copy of that repository on your usb key. You’d commit changes to the usb key then at a later point push changes back to the main SVN repo.

This way you’d have a single master repo in a safe place, that can be backed up with all your other stuff (repositories do get corrupted sometimes!), but are still able to commit changes whilst away from the ‘main’ machine or a network connection.

Because you keep your original SVN repo you’re also in a position to just revert to SVN alone if you decide you don’t like SVK.

(I know git does a lot of this stuff too, but it’s commands are quite different from svn, harder to learn I think.)

(1) http://svk.elixus.org/view/HomePage

August 20, 06h

Another advantage to having a physical USB key nearby is reduced latency times; storing files remotely often incurs a serious time overhead when browsing and checking out of the repository. My employer’s SVN server takes about 15 minutes to pull down a new export of at least one of its projects (that’s “only” 150M). I imagine such a task would take 30 seconds with a direct connection.

That said, our SVN server is an old P4 somewhere in Moscow.

Bramus! says:
August 20, 07h

Using an USB stick as SVN repo too here. Not using an encrypted disk image on it though (that, and the fact that my main machine still is my XP box … recently switched over).

Never would’ve thought about it; Just ingenious!

Darren says:
August 20, 08h

I do have a suggestion for you that will eliminate the double-eject requirement.

1. Back up the data on your encrypted volume.
2. Install TrueCrypt (http://www.truecrypt.org)
3. Wipe the USB key
4. Use TrueCrypt to make the whole USB key one encrypted volume.
5. Put your data back on this volume

Now, you still need to do an extra step when you mount the USB key (mounting it via TrueCrypt), but you don’t have to double-eject.

This also has the following advantages:

* You can select the crypto algorithm (though I agree that AES-128 is plenty)
* TrueCrypt is available for Windows too, so your image is portable to OS X, Win2000, XP, and Vista.

August 20, 09h

Interesting alternative. I can’t imagine dragging the drive back and forth.

I actually have the “server” portion set up on my Mac and then I can check things out locally (the usual) or over the network to another machine. I found the hardest part was getting svnserve setup in Leopard. Maybe someday I’ll add in a sync/backup of the repositories to a remote location, but for now my local backup is sufficient.

Frank says:
August 21, 20h

I’m with Darren with using TrueCrypt but I’m not 100% sure it works on Mac’s. If you want the “ultimate” in encrypted UBS keys check out IronKey (https://www.ironkey.com/).

18
Jeff says:
August 22, 13h

I’ve been doing this myself for a while, for all of the same reasons. For my setup I added a linux box to house the repos and then hooked everything together with a Hamachi VPN. I suppose you could do the same by leaving the USB key in one of the machines and making it the server.

August 28, 08h

Like you, I just adopted Subversion for all my client work, even though in most cases I’m the only one accessing it. I’m using Dreamhost as the Subversion server. It’s a bit slow for big commits, but has worked well for me. I also like Cornerstone very much.

The one piece of the puzzle that bothers me is that Subversion doesn’t support the resource forks of Mac files. Not an issue mostly except for Applescripts which come back into the working copy as 0 byte files. My workaround is to always include a ZIP version of any Applescript that has to be refreshed as the script is updated. The 0 byte file can be deleted and replaced by the version protected inside the ZIP file.

20
Berta Berlin says:
September 11, 07h

Two more issues: When your files grow your stick will not only be to small, it will also become to slow. Data transfer is usual about a fifth of i.e. USB drives.
And a pretty cool solution to encryption: There are drives with build in finger print readers! Encryption is done by the hardware inside of this drives, independent from your OS.

Anyway, I prefer to use a https-svn repository on the net (for reasons already mentioned here).

21
Lorenzo Bolognini says:
September 12, 05h

Hi Dave,

just use Assembla.com it’s free and they provide SVN, even with Trac integration. But I prefer keeping all my docs under Google Documents.

L.

22
Matt says:
September 17, 22h

Like other commentators, I would strongly recommend a server if you’re using subversion. The centralized repository method just really lends itself to a server. I tried the SVN + USB route and gave up after I lost one drive, then accidentally broke another, then kept forgetting to bring my USB drive with me. You may not be worried about losing your revision history, but it’s still a pain to create a new repository from the working copy.

If you’re using a USB drive I’d encourage you to consider a distributed system like git or Mercurial. Then, your entire repository (history and all) will be in three places. And git makes it really, really, really easy to create new branches to try out ideas.

Dave says:
November 06, 14h

I tried the two computer thing for a while, and it began to become a real pain in the butt trying to keep everything synchronized. I eventually realized it was just easier to use my
notebook solely.

January 07, 17h

Why don’t you use DropBox instead? SVN can be useful for projects with multiple contributors, and for some other more robust features, but DropBox has a lot of the bases covered (including version control).

I work on 3 different computers, all linked up via my 50gb DropBox account. I save active projects directly to my DropBox, in their own respective folders. The files are saved locally to my HDD, so they’re quick to work with. When saved, they are uploaded to the Amazon S3 cloud in the background, and then pushed to the other 2 computers.

Everything is seamless. I don’t have to worry about uploading, or checking files in or out, or taking a USB key with me. Files are simply automatically synced up. It has been a lifesaver, because otherwise working on the same projects on 3 different computers (both Windows & OS X) would be a bitch!

deizel says:
February 02, 09h

Three reasons for using Git came to mind when reading your post:

a) History. This is stored in each repository on each machine, so you don’t lose this if your USB key goes missing.

b) File size. Git can diff binaries, not just text, so when you save multiple versions of a PSD file, it will only save the actual changes.

c) Offline. Since each copy is itself a repository and your ‘server’ isn’t on a USB key, you could leave your USB key behind when you go to the cafe, commit your changes locally, and push/pull them when you return to your desk.

There is a bit of a learning curve, and you would have to replace Versions with GitX (http://gitx.frim.nl/), but it does solve a few of the issues you mentioned above.

(P.S. There is a 5px notch out of the right column of your template in Chrome)

26
Ed says:
March 07, 17h

I’m setting up my USB key now. Excellent idea, perfect for my needs. One question: Since you wrote about this, have you run out of space? If so, I’m curious to know how you dealt with that.