Having fun with UNIX commands

Like all good programmers, I usually spend more time figuring out ways to save me time than it would actually take to do the task. It must be a programming thing; building tools to build tools and loathing menial tasks. Maybe it's some kind of programmer's syndrome? Anyway, I digress.

One of the things I love about UNIX (and UNIX based systems, so, Mac OS X) is that it's full of ways to speed up your processes. Take a simple example. I wanted to move a load of directories in Mercurial into a sub-directory. Now, I could do this manually:

$ hg mv README.txt third_party/blog/README.txt
$ hg mv LICENSE.txt third_party/blog/LICENSE.txt
$ hg mv libraries/ third_party/blog/libraries/
...

But that would be very boring, wouldn't it? Plus, how plebian it would be! I'm a programmer, there's no WAY I'm doing it manually. So, what's the quickest way to get that done? Let's think about what we actually want to achieve, and try to map this to commands. 

1. Get a list of the files in the directory
2. Remove all the files we do not wish to move - as well as the file we're copying everything into, so, third_party/blog - from that list.
3. Stop tracking them in the current directory
4. Copy them to the new directory
5. Add them again

We can actually pretty easily map each of these desires into UNIX shell commands:

1. ls
2. grep
3. hg forget
4. cp
5. hg add

If any of these commands are unfamiliar, they essentially do what they say on the tin, with the exception of grep, which I'll explain in a second. We'll have to pipe these into each other in order to make it a swell, one-step process. Here's my first attempt:

$ ls | grep !^third_ | hg forget | cp . third_party/blog | hg add .

Which, of course, was very naïve indeed. For starters, bash reads the exclamation point as an event designator, so it looks for a command to execute that has been previously executed in the shell's history beginning with the designated string. So grep !^third_ is invalid.

Additionally, I'm piping the output here from one to another. ls to grep is fine, but when we get to hg forget, we can see additional stuff that the command pushes into the command. cp will trip out when it sees this as it will be looking for files that don't exist. Additionally, the command syntax of cp is hard to bypass. We'd need to recursively loop through the output from hg forget, remove the cruft and pipe it into cp correctly. That's just a pain.

Okay, so, let's look at alternative solutions. There's a hg mv command that can solve a lot of our problems - it essentially combines steps 3-5 into one command. We like that. But before we fix that, let's take a look at the grep syntax and see what we can do about that.

grep takes an input and a regular expression, and allows you to search through a string using the regex. We want a regex that can be used in the command line. Since the universal "NOT" operator is the exclamation point, and we can't use that for aforementioned reasons, we'll have to figure something else out. Luckily for us, grep supports a particularly useful flag, -v, which returns everything that DOESN'T match the pattern. It basically whacks a bloody great exclamation point on the regex for us. Sweet.

Let's take that knowledge, and pipe it into hg mv:

$ ls | grep -v ^third_party | hg mv . third_party/blog

We're getting there. This is starting to look possible. The issue here is that we still need to loop through the output of grep to pass it into hg mv. Discovering grep's -v flag got me intrigued, and made me ask a question; can we pass a regex into hg mv? That would be perfect. I took a look at the help page for hg mv.

$ hg mv --help
hg rename [OPTION]... SOURCE... DEST

aliases: mv

rename files; equivalent of copy + remove

    Mark dest as copies of sources; mark sources for deletion. If dest is a directory, copies are put in that
    directory. If dest is a file, there can only be one source.

    By default, this command copies the contents of files as they exist in the working directory. If invoked with
    -A/--after, the operation is recorded, but no copying is performed.

    This command takes effect at the next commit. To undo a rename before that, see hg revert.

options:

 -A --after    record a rename that has already occurred
 -f --force    forcibly copy over an existing managed file
 -I --include  include names matching the given patterns
 -X --exclude  exclude names matching the given patterns
 -n --dry-run  do not perform actions, just print output

use "hg -v help rename" to show global options

Wow. It's our lucky day! The eagle-eyed among you will have noticed the -X flag lingering on this command: 'exclude names matching the given patterns'. Fantastic. We can use the splat operator, or the asterisk, to match our files and move them, on a regex, all in one line. All that boils down to this: 

$ hg mv ./* third_party/blog/ -X ^third

I encourage you to play about with UNIX commands and see what sort of tricks you can discover. Actually, if you're not confident with the terminal, give the Bash documentation a read and learn the shortcuts and the operators you get with Bash. It'll make your life a LOT easier, trust me. And plus, it's kinda cool.

Comments

[Martin, CycleStreets]

Erm, you missed this fantastic command...! (Don't worry, it is safe!)


ls -R | grep ":" | sed -e 's/://' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'