simonh.uk - software

Backing Up Retro Game Library using Rsync

2026-02-10T00:00:00+00:00

In this post, I thought I’d share my simple method for keeping my (mainly) retro games backed up on a USB hard drive. To start with, I was doing it all manually, but that soon grew tiresome so I decided to use the tried and trusted rsync to save time. Before I ran any commands on my proper game directory, I created a test folder to make sure everything worked as I wanted. Rsync is a program with lots of options, but here I’m only using it for a simple backup from one tree to another. Let’s get started!

If you’re not familiar with rsync, it generally takes a source folder and a destination folder, preceded by whatever options you want to use. Below, I’m using -navi which means:

-n – run in test mode (dry run)
-a – run in archive mode (which includes lots of options to save typing)
-v – run in verbose mode to give us info
-i – itemize what files are going to be added, deleted etc.

As you can see from the output below, rsync when used with -navi gives us a summary of everything that it’s going to do.

[simon@computer 14:38:24] ~/tmp
$ rsync -navi Games/ Games-backup
sending incremental file list
created directory Games-backup
cd+++++++++ ./
cd+++++++++ gb/
>f+++++++++ gb/A-mazing Tater (USA).gb
>f+++++++++ gb/Adventures of Lolo (Europe) (SGB Enhanced).gb
>f+++++++++ gb/Adventures of Star Saver, The (USA, Europe).gb
>f+++++++++ gb/Kirby's Block Ball (USA, Europe) (SGB Enhanced).gb
>f+++++++++ gb/Kirby's Dream Land (USA, Europe).gb
>f+++++++++ gb/Kirby's Dream Land 2 (USA, Europe) (SGB Enhanced).gb
>f+++++++++ gb/Kirby's Pinball Land (USA, Europe).gb
>f+++++++++ gb/Kirby's Star Stacker (USA, Europe) (SGB Enhanced).gb
>f+++++++++ gb/Wario Blast Featuring Bomberman! (USA, Europe) (SGB Enhanced).gb
>f+++++++++ gb/Wario Land - Super Mario Land 3 (World).gb
>f+++++++++ gb/Wario Land 3 (World) (En,Ja).gbc
>f+++++++++ gb/Wario Land II (USA, Europe) (SGB Enhanced) (GB Compatible).gbc
>f+++++++++ gb/Yoshi (USA).gb
>f+++++++++ gb/Yoshi's Cookie (USA, Europe).gb
cd+++++++++ gba/
>f+++++++++ gba/Celeste 2 (v1.0) (Aftermarket).gba
>f+++++++++ gba/Celeste Classic (World) (v1.2) (Aftermarket) (Unl).gba
cd+++++++++ gg/
>f+++++++++ gg/Castle of Illusion Starring Mickey Mouse (USA, Europe, Brazil) (En).sms
>f+++++++++ gg/Chase H.Q. (USA).sms
>f+++++++++ gg/Choplifter III (USA).gg
>f+++++++++ gg/Excellent Dizzy Collection, The (Europe).sms
>f+++++++++ gg/GG Aleste (Japan) (En).gg
>f+++++++++ gg/GG Aleste 3 (Japan) (En) (Aleste Collection).gg
>f+++++++++ gg/GG Aleste II (Japan) (En) (Aleste Collection).gg
>f+++++++++ gg/Galaga 2 (Europe).gg
>f+++++++++ gg/Sonic & Tails (Japan) (En).gg
>f+++++++++ gg/Sonic & Tails 2 (Japan) (En).gg
>f+++++++++ gg/Sonic The Hedgehog (World) (Rev A).gg
>f+++++++++ gg/Sonic The Hedgehog 2 (World).gg
>f+++++++++ gg/Streets of Rage (World).gg
>f+++++++++ gg/Streets of Rage 2 (World).gg
>f+++++++++ gg/Super Space Invaders (USA, Europe).gg
>f+++++++++ gg/Tails Adventure (World) (En,Ja).gg
>f+++++++++ gg/Terminator 2 - Judgment Day (World).gg
>f+++++++++ gg/Terminator, The (USA, Europe).gg
cd+++++++++ ngc/
>f+++++++++ ngc/Bust-A-Move Pocket (USA).ngc
>f+++++++++ ngc/Crush Roller (World) (En,Ja).ngc
>f+++++++++ ngc/Fantastic Night Dreams - Cotton (Europe).ngc
>f+++++++++ ngc/Fatal Fury F-Contact - Pocket Fighting Series (World) (En,Ja).ngc
>f+++++++++ ngc/King of Fighters R-2 - Pocket Fighting Series (World) (En,Ja).ngc
>f+++++++++ ngc/Last Blade, The - Beyond the Destiny (USA, Europe).ngc
>f+++++++++ ngc/Magical Drop Pocket (USA, Europe).ngc
>f+++++++++ ngc/Rockman - Battle & Fighters (Japan).ngc
>f+++++++++ ngc/SNK Gals' Fighters (USA, Europe).ngc
>f+++++++++ ngc/SNK vs. Capcom - Card Fighters' Clash - SNK Version (USA, Europe).ngc
>f+++++++++ ngc/SNK vs. Capcom - The Match of the Millennium (World) (En,Ja).ngc
>f+++++++++ ngc/Sonic The Hedgehog - Pocket Adventure (World) (En,Ja).ngc
cd+++++++++ pce/
>f+++++++++ pce/Bomberman '93 (USA).pce
>f+++++++++ pce/Bomberman (USA).pce
>f+++++++++ pce/Chase H.Q. (USA).pce
>f+++++++++ pce/Darius Alpha (Japan) (En) (SG Enhanced).pce
>f+++++++++ pce/Darius Plus (Japan) (En) (SG Enhanced).pce
>f+++++++++ pce/Final Soldier (Japan) (En).pce
>f+++++++++ pce/Galaga '88 (Japan) (En).pce
>f+++++++++ pce/Gradius (Japan) (En).pce
>f+++++++++ pce/ImageFight (Japan) (En).pce
>f+++++++++ pce/Pac-Land (USA).pce
>f+++++++++ pce/R-Type I (Japan) (En).pce
>f+++++++++ pce/R-Type II (Japan) (En) (V1.1).pce
>f+++++++++ pce/Raiden (USA).pce
>f+++++++++ pce/Salamander (Japan) (En).pce
>f+++++++++ pce/Soldier Blade (USA).pce
>f+++++++++ pce/Space Invaders - Fukkatsu no Hi (Japan).pce
>f+++++++++ pce/Super Star Soldier (USA).pce
>f+++++++++ pce/Turrican (USA).pce

sent 3,171 bytes  received 266 bytes  6,874.00 bytes/sec
total size is 54,620,304  speedup is 15,891.85 (DRY RUN)

Once we’re happy that our command is going to do what we intend it to do, we can remove the -n switch to make the initial backup:

[simon@computer 14:38:40] ~/tmp
$ rsync -avi Games/ Games-backup
sending incremental file list
created directory Games-backup
cd+++++++++ ./
cd+++++++++ gb/
>f+++++++++ gb/A-mazing Tater (USA).gb
>f+++++++++ gb/Adventures of Lolo (Europe) (SGB Enhanced).gb
>f+++++++++ gb/Adventures of Star Saver, The (USA, Europe).gb
>f+++++++++ gb/Kirby's Block Ball (USA, Europe) (SGB Enhanced).gb
>f+++++++++ gb/Kirby's Dream Land (USA, Europe).gb
>f+++++++++ gb/Kirby's Dream Land 2 (USA, Europe) (SGB Enhanced).gb
>f+++++++++ gb/Kirby's Pinball Land (USA, Europe).gb
>f+++++++++ gb/Kirby's Star Stacker (USA, Europe) (SGB Enhanced).gb
>f+++++++++ gb/Wario Blast Featuring Bomberman! (USA, Europe) (SGB Enhanced).gb
>f+++++++++ gb/Wario Land - Super Mario Land 3 (World).gb
>f+++++++++ gb/Wario Land 3 (World) (En,Ja).gbc
>f+++++++++ gb/Wario Land II (USA, Europe) (SGB Enhanced) (GB Compatible).gbc
>f+++++++++ gb/Yoshi (USA).gb
>f+++++++++ gb/Yoshi's Cookie (USA, Europe).gb
cd+++++++++ gba/
>f+++++++++ gba/Celeste 2 (v1.0) (Aftermarket).gba
>f+++++++++ gba/Celeste Classic (World) (v1.2) (Aftermarket) (Unl).gba
cd+++++++++ gg/
>f+++++++++ gg/Castle of Illusion Starring Mickey Mouse (USA, Europe, Brazil) (En).sms
>f+++++++++ gg/Chase H.Q. (USA).sms
>f+++++++++ gg/Choplifter III (USA).gg
>f+++++++++ gg/Excellent Dizzy Collection, The (Europe).sms
>f+++++++++ gg/GG Aleste (Japan) (En).gg
>f+++++++++ gg/GG Aleste 3 (Japan) (En) (Aleste Collection).gg
>f+++++++++ gg/GG Aleste II (Japan) (En) (Aleste Collection).gg
>f+++++++++ gg/Galaga 2 (Europe).gg
>f+++++++++ gg/Sonic & Tails (Japan) (En).gg
>f+++++++++ gg/Sonic & Tails 2 (Japan) (En).gg
>f+++++++++ gg/Sonic The Hedgehog (World) (Rev A).gg
>f+++++++++ gg/Sonic The Hedgehog 2 (World).gg
>f+++++++++ gg/Streets of Rage (World).gg
>f+++++++++ gg/Streets of Rage 2 (World).gg
>f+++++++++ gg/Super Space Invaders (USA, Europe).gg
>f+++++++++ gg/Tails Adventure (World) (En,Ja).gg
>f+++++++++ gg/Terminator 2 - Judgment Day (World).gg
>f+++++++++ gg/Terminator, The (USA, Europe).gg
cd+++++++++ ngc/
>f+++++++++ ngc/Bust-A-Move Pocket (USA).ngc
>f+++++++++ ngc/Crush Roller (World) (En,Ja).ngc
>f+++++++++ ngc/Fantastic Night Dreams - Cotton (Europe).ngc
>f+++++++++ ngc/Fatal Fury F-Contact - Pocket Fighting Series (World) (En,Ja).ngc
>f+++++++++ ngc/King of Fighters R-2 - Pocket Fighting Series (World) (En,Ja).ngc
>f+++++++++ ngc/Last Blade, The - Beyond the Destiny (USA, Europe).ngc
>f+++++++++ ngc/Magical Drop Pocket (USA, Europe).ngc
>f+++++++++ ngc/Rockman - Battle & Fighters (Japan).ngc
>f+++++++++ ngc/SNK Gals' Fighters (USA, Europe).ngc
>f+++++++++ ngc/SNK vs. Capcom - Card Fighters' Clash - SNK Version (USA, Europe).ngc
>f+++++++++ ngc/SNK vs. Capcom - The Match of the Millennium (World) (En,Ja).ngc
>f+++++++++ ngc/Sonic The Hedgehog - Pocket Adventure (World) (En,Ja).ngc
cd+++++++++ pce/
>f+++++++++ pce/Bomberman '93 (USA).pce
>f+++++++++ pce/Bomberman (USA).pce
>f+++++++++ pce/Chase H.Q. (USA).pce
>f+++++++++ pce/Darius Alpha (Japan) (En) (SG Enhanced).pce
>f+++++++++ pce/Darius Plus (Japan) (En) (SG Enhanced).pce
>f+++++++++ pce/Final Soldier (Japan) (En).pce
>f+++++++++ pce/Galaga '88 (Japan) (En).pce
>f+++++++++ pce/Gradius (Japan) (En).pce
>f+++++++++ pce/ImageFight (Japan) (En).pce
>f+++++++++ pce/Pac-Land (USA).pce
>f+++++++++ pce/R-Type I (Japan) (En).pce
>f+++++++++ pce/R-Type II (Japan) (En) (V1.1).pce
>f+++++++++ pce/Raiden (USA).pce
>f+++++++++ pce/Salamander (Japan) (En).pce
>f+++++++++ pce/Soldier Blade (USA).pce
>f+++++++++ pce/Space Invaders - Fukkatsu no Hi (Japan).pce
>f+++++++++ pce/Super Star Soldier (USA).pce
>f+++++++++ pce/Turrican (USA).pce

sent 54,639,115 bytes  received 1,310 bytes  109,280,850.00 bytes/sec
total size is 54,620,304  speedup is 1.00

Make sure to include the trailing slash after the source directory otherwise rsync will copy the Games directory as well (which is not what we want):

[simon@computer 14:46:29] ~/tmp
$ rsync -navi Games Games-backup
sending incremental file list
cd+++++++++ Games/
cd+++++++++ Games/gb/
>f+++++++++ Games/gb/A-mazing Tater (USA).gb
>f+++++++++ Games/gb/Adventures of Lolo (Europe) (SGB Enhanced).gb
>f+++++++++ Games/gb/Adventures of Star Saver, The (USA, Europe).gb
>f+++++++++ Games/gb/Kirby's Block Ball (USA, Europe) (SGB Enhanced).gb
>f+++++++++ Games/gb/Kirby's Dream Land (USA, Europe).gb
>f+++++++++ Games/gb/Kirby's Dream Land 2 (USA, Europe) (SGB Enhanced).gb
>f+++++++++ Games/gb/Kirby's Pinball Land (USA, Europe).gb
>f+++++++++ Games/gb/Kirby's Star Stacker (USA, Europe) (SGB Enhanced).gb
>f+++++++++ Games/gb/Wario Blast Featuring Bomberman! (USA, Europe) (SGB Enhanced).gb
>f+++++++++ Games/gb/Wario Land - Super Mario Land 3 (World).gb
>f+++++++++ Games/gb/Wario Land 3 (World) (En,Ja).gbc
>f+++++++++ Games/gb/Wario Land II (USA, Europe) (SGB Enhanced) (GB Compatible).gbc
>f+++++++++ Games/gb/Yoshi (USA).gb
>f+++++++++ Games/gb/Yoshi's Cookie (USA, Europe).gb
cd+++++++++ Games/gba/
>f+++++++++ Games/gba/Celeste 2 (v1.0) (Aftermarket).gba
>f+++++++++ Games/gba/Celeste Classic (World) (v1.2) (Aftermarket) (Unl).gba
cd+++++++++ Games/gg/
>f+++++++++ Games/gg/Castle of Illusion Starring Mickey Mouse (USA, Europe, Brazil) (En).sms
>f+++++++++ Games/gg/Chase H.Q. (USA).sms
>f+++++++++ Games/gg/Choplifter III (USA).gg
>f+++++++++ Games/gg/Excellent Dizzy Collection, The (Europe).sms
>f+++++++++ Games/gg/GG Aleste (Japan) (En).gg
>f+++++++++ Games/gg/GG Aleste 3 (Japan) (En) (Aleste Collection).gg
>f+++++++++ Games/gg/GG Aleste II (Japan) (En) (Aleste Collection).gg
>f+++++++++ Games/gg/Galaga 2 (Europe).gg
>f+++++++++ Games/gg/Sonic & Tails (Japan) (En).gg
>f+++++++++ Games/gg/Sonic & Tails 2 (Japan) (En).gg
>f+++++++++ Games/gg/Sonic The Hedgehog (World) (Rev A).gg
>f+++++++++ Games/gg/Sonic The Hedgehog 2 (World).gg
>f+++++++++ Games/gg/Streets of Rage (World).gg
>f+++++++++ Games/gg/Streets of Rage 2 (World).gg
>f+++++++++ Games/gg/Super Space Invaders (USA, Europe).gg
>f+++++++++ Games/gg/Tails Adventure (World) (En,Ja).gg
>f+++++++++ Games/gg/Terminator 2 - Judgment Day (World).gg
>f+++++++++ Games/gg/Terminator, The (USA, Europe).gg
cd+++++++++ Games/ngc/
>f+++++++++ Games/ngc/Bust-A-Move Pocket (USA).ngc
>f+++++++++ Games/ngc/Crush Roller (World) (En,Ja).ngc
>f+++++++++ Games/ngc/Fantastic Night Dreams - Cotton (Europe).ngc
>f+++++++++ Games/ngc/Fatal Fury F-Contact - Pocket Fighting Series (World) (En,Ja).ngc
>f+++++++++ Games/ngc/King of Fighters R-2 - Pocket Fighting Series (World) (En,Ja).ngc
>f+++++++++ Games/ngc/Last Blade, The - Beyond the Destiny (USA, Europe).ngc
>f+++++++++ Games/ngc/Magical Drop Pocket (USA, Europe).ngc
>f+++++++++ Games/ngc/Rockman - Battle & Fighters (Japan).ngc
>f+++++++++ Games/ngc/SNK Gals' Fighters (USA, Europe).ngc
>f+++++++++ Games/ngc/SNK vs. Capcom - Card Fighters' Clash - SNK Version (USA, Europe).ngc
>f+++++++++ Games/ngc/SNK vs. Capcom - The Match of the Millennium (World) (En,Ja).ngc
>f+++++++++ Games/ngc/Sonic The Hedgehog - Pocket Adventure (World) (En,Ja).ngc
cd+++++++++ Games/pce/
>f+++++++++ Games/pce/Bomberman '93 (USA).pce
>f+++++++++ Games/pce/Bomberman (USA).pce
>f+++++++++ Games/pce/Chase H.Q. (USA).pce
>f+++++++++ Games/pce/Darius Alpha (Japan) (En) (SG Enhanced).pce
>f+++++++++ Games/pce/Darius Plus (Japan) (En) (SG Enhanced).pce
>f+++++++++ Games/pce/Final Soldier (Japan) (En).pce
>f+++++++++ Games/pce/Galaga '88 (Japan) (En).pce
>f+++++++++ Games/pce/Gradius (Japan) (En).pce
>f+++++++++ Games/pce/ImageFight (Japan) (En).pce
>f+++++++++ Games/pce/Pac-Land (USA).pce
>f+++++++++ Games/pce/R-Type I (Japan) (En).pce
>f+++++++++ Games/pce/R-Type II (Japan) (En) (V1.1).pce
>f+++++++++ Games/pce/Raiden (USA).pce
>f+++++++++ Games/pce/Salamander (Japan) (En).pce
>f+++++++++ Games/pce/Soldier Blade (USA).pce
>f+++++++++ Games/pce/Space Invaders - Fukkatsu no Hi (Japan).pce
>f+++++++++ Games/pce/Super Star Soldier (USA).pce
>f+++++++++ Games/pce/Turrican (USA).pce

sent 3,187 bytes  received 232 bytes  6,838.00 bytes/sec
total size is 54,620,304  speedup is 15,975.52 (DRY RUN)

Now that we’ve run rsync and copied all our games to our backup folder, running the same command again should not produce any changes. Let’s check:

[simon@computer 14:55:15] ~/tmp
$ rsync -avi Games/ Games-backup
sending incremental file list

sent 2,957 bytes  received 17 bytes  5,948.00 bytes/sec
total size is 54,620,304  speedup is 18,365.94

rsync hasn’t done anything by the looks of it, which is as we expect. Now, lets add a couple of new games to our source folder. We’re in the mood for a bit of hockey so we’ll add two games, one Gameboy game, one Game Gear one. Let’s see if rsync spots them:

[simon@computer 14:55:20] ~/tmp
$ rsync -navi Games/ Games-backup
sending incremental file list
.d..t...... gb/
>f+++++++++ gb/NHL Blades of Steel (USA).gbc
.d..t...... gg/
>f+++++++++ gg/NHL All-Star Hockey (USA).gg

sent 3,040 bytes  received 29 bytes  6,138.00 bytes/sec
total size is 56,193,168  speedup is 18,309.93 (DRY RUN)

It sure did! Let’s have rsync add them to our backup directory:

[simon@computer 15:00:19] ~/tmp
$ rsync -avi Games/ Games-backup
sending incremental file list
.d..t...... gb/
>f+++++++++ gb/NHL Blades of Steel (USA).gbc
.d..t...... gg/
>f+++++++++ gg/NHL All-Star Hockey (USA).gg

sent 1,576,360 bytes  received 69 bytes  3,152,858.00 bytes/sec
total size is 56,193,168  speedup is 35.65

I really like the -i option, as we can see exactly what rsync has done.

What about files I’ve deleted?

If you’re trying to manage a backup directory manually, it can be a lot of work, keeping track of files and folders added, modified and deleted. rsync has a handy --delete option which will delete files from the destination folder that aren’t in the source folder. For our example use case in this post about managing our retro game library backups, we may have a rom that doesn’t work. So we’d like it gone from both folders.

We decide to get rid of a Wario game and a Turrican game. These have already been backed up, but the --delete option tells rsync to delete anything that’s in the destination folder, but not in the source folder. Let’s run rsync in test mode:

[simon@computer 15:16:04] ~/tmp
$ rsync -navi --delete Games/ Games-backup
sending incremental file list
*deleting   gb/Wario Blast Featuring Bomberman! (USA, Europe) (SGB Enhanced).gb
*deleting   pce/Turrican (USA).pce
.d..t...... gb/
.d..t...... pce/

sent 2,942 bytes  received 120 bytes  6,124.00 bytes/sec
total size is 55,668,880  speedup is 18,180.56 (DRY RUN)

What about file / directory renames?

Let’s say we decide to stop being so lazy and rename gb to Gameboy. Will rsync be smart enough to notice that and simply rename it for us at the backup? Or, will it assume this a brand new directory and copy all the same files again? Let’s find out:

[simon@computer 11:23:14] master ~/tmp
$ rsync -navi --delete Games/ Games-backup/
sending incremental file list
*deleting   gb/Yoshi's Cookie (USA, Europe).gb
*deleting   gb/Yoshi (USA).gb
*deleting   gb/Wario Land II (USA, Europe) (SGB Enhanced) (GB Compatible).gbc
*deleting   gb/Wario Land 3 (World) (En,Ja).gbc
*deleting   gb/Wario Land - Super Mario Land 3 (World).gb
*deleting   gb/NHL Blades of Steel (USA).gbc
*deleting   gb/Kirby's Star Stacker (USA, Europe) (SGB Enhanced).gb
*deleting   gb/Kirby's Pinball Land (USA, Europe).gb
*deleting   gb/Kirby's Dream Land 2 (USA, Europe) (SGB Enhanced).gb
*deleting   gb/Kirby's Dream Land (USA, Europe).gb
*deleting   gb/Kirby's Block Ball (USA, Europe) (SGB Enhanced).gb
*deleting   gb/Adventures of Star Saver, The (USA, Europe).gb
*deleting   gb/Adventures of Lolo (Europe) (SGB Enhanced).gb
*deleting   gb/A-mazing Tater (USA).gb
*deleting   gb/
.d..t...... ./
cd+++++++++ Gameboy/
>f+++++++++ Gameboy/A-mazing Tater (USA).gb
>f+++++++++ Gameboy/Adventures of Lolo (Europe) (SGB Enhanced).gb
>f+++++++++ Gameboy/Adventures of Star Saver, The (USA, Europe).gb
>f+++++++++ Gameboy/Kirby's Block Ball (USA, Europe) (SGB Enhanced).gb
>f+++++++++ Gameboy/Kirby's Dream Land (USA, Europe).gb
>f+++++++++ Gameboy/Kirby's Dream Land 2 (USA, Europe) (SGB Enhanced).gb
>f+++++++++ Gameboy/Kirby's Pinball Land (USA, Europe).gb
>f+++++++++ Gameboy/Kirby's Star Stacker (USA, Europe) (SGB Enhanced).gb
>f+++++++++ Gameboy/NHL Blades of Steel (USA).gbc
>f+++++++++ Gameboy/Wario Land - Super Mario Land 3 (World).gb
>f+++++++++ Gameboy/Wario Land 3 (World) (En,Ja).gbc
>f+++++++++ Gameboy/Wario Land II (USA, Europe) (SGB Enhanced) (GB Compatible).gbc
>f+++++++++ Gameboy/Yoshi (USA).gb
>f+++++++++ Gameboy/Yoshi's Cookie (USA, Europe).gb

sent 2,999 bytes  received 724 bytes  7,446.00 bytes/sec
total size is 55,668,880  speedup is 14,952.69 (DRY RUN)

As you see, rsync went with the second option. At the time of running this, I really wasn’t sure what would happen. After doing a bit of research, it’s because rsync doesn’t keep any state between runs. Each time you run it, it simply compares the source and the destination and because we passed --delete rsync simply sees a folder (and all it’s contents) need deleting, and a new folder needs copying to the backup.

It’s the same result if you rename a file or multiple files:

[simon@computer 11:30:08] master ~/tmp
$ rsync -navi --delete Games/ Games-backup/
sending incremental file list
*deleting   Gameboy/Blades of Steel (USA).gbc
.d..t...... Gameboy/
>f+++++++++ Gameboy/NHL Blades of Steel (USA).gbc

sent 2,953 bytes  received 60 bytes  6,026.00 bytes/sec
total size is 55,668,880  speedup is 18,476.23 (DRY RUN)

With files only a few kilobytes in size, it’s not really an issue. However, if you’re backing up big files taking up gigabytes, this is going to be a problem. There are patches and workarounds, but I think you should take this into account up front and try to avoid renaming directories in particular!

Conclusion

So now, I’m happy that the command:

$ rsync -avi --delete Games/ Games-backup/

is going to backup anything newly added, and delete any games that I deliberately deleted. By the way, rsync also keeps track of modified files. So if you replace a game with another one shaing exactly the same name (but different contents), rsync will copy that one to your backup folder too.

I think rsync is the perfect program for managing a backup of your games / roms. You only have one folder that you need to keep tidy and rsync will keep a perfect duplicate of that folder for you.

A lot of people (myself included) may have had some fear of using rsync for this sort of task but once you’ve tried it on a test directory and dialled in your options, I don’t tnink anything can beat it.

Using Emacs as a Word Processor

2025-09-27T00:00:00+01:00

Ever since Tavis Ormandy got Wordperfect 8 working on modern Linux it’s been bugging me that we don’t have many options for word processing in the terminal.

Thanks to his amazing work, we do now have an enterprise grade word processor running in our terminals. However, there are problems.

Problems with Wordperfect 8

The most glaring problems are that it’s closed source, hasn’t been updated in 27 years, and it’s not exactly, user friendly. The PDF manual is ~700 pages! There’s also the file format problem. It’s not a publicly documented format. You can certainly open a .wp file in plenty of software:

Libre Office
Abiword
Google Docs

will all open an old wordperfect document, thanks most likely to libwpd but you’ll only be able to edit it in Wordperfect. Which raises the question,

“will I be able to open / edit these documents in twenty years time?”

So, while wpunix is an interesting piece of software, it’s probably not the best idea to do all your writing with it.

So, as I’m inclined to do, I set about seeing what other options I have for editing documents on the command line.

JOE Editor (Jstar)

I was already familiar with JOE and vaguely remembered that it had a Wordstar emulation mode. I’ve never used Wordstar, but know it’s still considered irreplaceable by some notable users. So, I had a play with it but it’s not quite what I’m after. The only formatting that one could consider somewhat out of the ordinary with Jstar is being able to centre text.

What Should a Word Processor Do?

To my mind, at a minimum, a word processor should be able to:

bold, underline and italicize text
Centre, left and right align text
Justify text
Have various types of headings

I was amazed that no such command line program seemed to exist. Yeah sure, we’ve got loads of Markdown editors, but I don’t consider Markdown to be anything related to word processing. I even thought about attempting to make my own format (how hard could it be?!). But luckily, I came across:

RFC 1896 The text/enriched MIME Content-type

For some reason, I always forget than any computer problem was almost certainly solved years ago. And so it was for this problem. RFC 1896 was suggested in 1996 to allow formatting of email messages. Unfortunately, it never caught on. Still, it exists and is a standard and Emacs has an enriched mode! See here for online documentation. Here’s the “source” code for the screenshot at the top of this post:

Content-Type: text/enriched
Text-Width: 70

<flushright>This is right aligned 

</flushright><center>

Here is some <bold>bold </bold>text and here is some that is <italic>italic</italic>.

</center><flushboth>

Enriched mode is a minor mode for editing formatted text files in a
WYSIWYG (What You See Is What You Get) fashion. When Enriched mode is
enabled, you can apply various formatting properties to the text in
the buffer, such as fonts and colors; upon saving the buffer, those
properties are saved together with the text, using the MIME
‘text/enriched’ file format.


Enriched mode is typically used with Text mode (see Text Mode). It is
not compatible with Font Lock mode, which is used by many major modes,
including most programming language modes, for syntax highlighting
(see Font Lock mode).  Unlike Enriched mode, Font Lock mode assigns
text properties automatically, based on the current buffer contents;
those properties are not saved to disk.

</flushboth><center>

<underline>Underlined Heading</underline>

</center>

The only thing that seems to be missing from enriched-mode is increasing the font size. There’s a discussion on the mailing list that goes into detail:

Making selected text larger in enriched mode

Apart from that, enriched-mode seems pretty much perfect for writing.

Using Emacs Enriched Mode

I won’t go into great detail as the documentation link above explains everything, but here are my brief notes:

Type M-x enriched-mode on a new file to activate the mode.¹ Then, you only really need to remember two shortcuts:

Alt-o to bring up the Options(?) menu and Alt-j to bring up the Justify menu.

Then, you can just press a single letter:

Alt-o b to bold for example

Alt-j b to justify

Plain Text Benefits

As enriched text is simply plain text with a bit of markup, it’s easily versionable (RCS, Mercurial / Git et al.). You can write a script to convert your enriched text to another format etc.

Conclusion

As I wrote in Why I use GNU Nano post, I don’t use Emacs as my text editor. But, I’ll probably start using it as my word processor instead…

Footnotes

¹ I’m trying to find an easy way to start Emacs in enriched mode. Will update post if I find it!

Microsoft Edit vs GNU Nano

2025-09-23T00:00:00+01:00

I’ve been seeing a few posts / videos claiming that Microsoft Edit is better than GNU Nano. Sorry, it isn’t. Furthermore, it hasn’t been designed to be better than anything!

Microsoft explained why they created Edit on their Devblog:

What motivated us to build Edit was the need for a default CLI text editor in 64-bit versions of Windows. 32-bit versions of Windows ship with the MS-DOS editor , but 64-bit versions do not have a CLI editor installed inbox. From there, we narrowed down our options…

Many of you are probably familiar with the “How do I exit vim?” meme. While it is relatively simple to learn the magic exit incantation, it’s certainly not a coincidence that this often turns up as a stumbling block for new and old programmers.

Because we wanted to avoid this for a built-in default editor, we decided that we wanted a modeless editor for Windows (versus a modal editor where new users would have to remember different modes of operation and how to switch between them).

This unfortunately limited our choices to a list of editors that either had no first-party support for Windows or were too big to bundle them with every version of the OS. As a result, Edit was born.

It should be noted that this new, Rust written Edit, is a modern version of an old DOS program of the same name. Here’s what that looked like:

Edit is designed for very simple editing. Below is the entire man page:

Nice and clear, and the program is learnable in approximately 5 minutes. In contrast, you’ll probably need a couple of hours to fully learn and become productive with GNU Nano. They have different histories and purposes. Here is the first paragraph from the Nano README:

Nano began life as a clone of Pico¹, then added features that users wished Pico had, whilst retaining its simple, easy to use nature. Edit is a re-implementation of DOS edit.com. The stated goal of Edit is

…to provide an accessible editor that even users largely unfamiliar with terminals can easily use.

You can type characters, delete them, search and replace, and that’s about it. Which is fine. Remember, this will ship with Windows. Now I haven’t used Windows at home since, maybe, 2006. MS Edit is not targeted at me, or anyone other than their customers. It’s there because they should have a terminal text editor. It’s not fancy or powerful but it will allow their customers to edit text files from the command line in a stress free way.

Personally, I like what they’ve done. There’s no configuration and no options to muck about with. You fire it up and do what you need to do.

Conclusion

I think Edit is a great bit of software. I like that Microsoft have released it as open source, and that they release Linux builds. I’ve had a play with it and haven’t felt the need to delete it. To my mind, it has already achieved it’s stated goal. There’s not really anything else that it needs at this point. I’m sure Microsoft are fully aware that if they start adding new features, it becomes harder to learn, harder to use. GNU Nano doesn’t have that contraint. It wants to be simple, but also useful.

Microsoft Edit is good but GNU Nano is better and always will be.

Footnotes

¹ Pico (pi)ne (co)mposer was the editor for the PINE email program. Very popular in the 90’s and kept alive in the fork Alpine which I use on my servers.

Why I use GNU Nano as my Default Editor

2025-09-13T00:00:00+01:00

The Backstory

For many years, my default text editor has been xed as it comes with Linux Mint, my distro of choice for some years. I’ve used xed for most of the bottle.py programs I’ve written. I never really thought about it, just used it as it got the job done. I generally use Nano for editing files over SSH and have also used that for many years.

A few weeks ago, I was thinking about my workflow and wondered why I wasn’t using a terminal based editor? Here’s how I use bottle.py

1. Fire up tmux
2. Run “python3 name-of-program.py“
3. Open another tmux window for Mercurial checkins

Here’s a screenshot of my terminal whilst writing this post:

It started bothering me that I was faffing about having to use my mouse to navigate between the terminal and my text editor. So I decided to find a terminal (non GUI) editor that would suit me.

My Editor Requirements

I was well aware of the hundreds of editors available for Linux, having had a play with most of them at one time or another:

Emacs, Vim, Jed, Vile, Jove, Joe, Nano, mg, etc….

So I had a think about what was vital and what would be nice-to-have features.

Vital

Multiple buffers — I usually have at least two source code files open at a time when I’m coding.

Good support for keyboard shortcuts — Of course, this is subjective. Emacs and Vim let you do anything, but I don’t normally need that much power! However, most things that I may need to do on a daily basis should have a shortcut.

Remapping of keys — I’ve used Colemak since about 2008, so what might be great for QWERTY, may not be great for me.

Quickly delete lines — One of the features I really liked with Vim is dd to delete a line, followed by . to delete subsequent lines.

Nice to Have’s

Sensible Defaults — This is one of the things I’ve always liked about Nano. You can just start using it. Shortcuts are visible at the bottom of the window if you need them.

Show a line at 80 columns — I don’t like long lines of code.

Quickly comment / uncomment code — I got very used to xed’s Ctrl-/ for this purpose!

Auto ident — I use Python more than any other language, so auto indenting is hard to live without.

Syntax Highlighting — Not 100% critical, but is often useful for seeing what’s what.

Bookmarks — Very useful to be able to mark lines in files and jump between them.

Scroll without moving cursor — I liked this feature I first saw in Vim. You move the screen but the cursor stays where it is

Insert a newline above / below current line — Some editors have a shortcut that will insert a newline without having to go to the end of the line, press enter

Where I Looked Closer at Nano

I won’t lie, Nano came first to mind. I was already familiar with it and it seems to be installed everywhere I need it. But did it do what I needed (and would like it to do)?

Yes, it would (I found out after an hour or so)!

I discovered pretty quickly that version 8, released in 2024 had introduced “modern bindings”:

Command-line option -- modernbindings (-/) makes ^Q quit, ^X cut, ^C copy, ^V paste, ^Z undo, ^Y redo, ^O open a file, ^W write a file, ^R replace, ^G find again, ^D find again backwards, ^A set the mark, ^T jump to a line, ^P show the position, and ^E execute.

I look at that as a sensible default. The original shortcuts aren’t terrible, but I like the CUA ones more. So I downloaded the latest tarball (8.6) and compiled it. I added alias nano="nano -/" to my .bashrc (and on all my servers).

What I Needed / Wanted

✓ multiple buffers: set multibuffer
✓ comment / uncomment code: Alt-3
✓ show end of line: set guidestripe 80
✓ syntax highlighting: include /usr/local/share/nano/*.nanorc
✓ quickly delete lines:
Alt-delete to delete entire line
Alt-backspace to delete words
✓ bookmarks: “anchors” in Nano speak.
Alt-Insert to set/unset.
Alt-PgUp / Alt-PgDn to cycle through them
✓ move viewport up and down: Alt-- / Alt-+
✓ insert a newline above / below current line
bind ^N "{home}{enter}{up}" main
bind ^k "{end}{enter}" main

My Tweaks

Many, many years ago,¹ I came across the idea of having the arrow keys accessible on the home row using either the CTRL or ALT keys. Similar to this keyboard:

As I was hoping to somewhat optimize my text editing, I figured I’d try and get some often used keys where I wanted them.² Using QWERTY notation, I now have:

Alt-j — Left
Alt-k — Down
Alt-i — Up
Alt-l — Right
Alt-u — Home
Alt-o — End
Alt-h — Delete
Ctrl-k — Page Down
Ctrl-i — Page Up

Very nice! I’m already used to them after only a few days. If you’re interested, below is my .nanorc

My .nanorc

# $Id: .nanorc,v 1.6 2025/09/15 13:29:21 simon Exp simon $

set multibuffer
set tabsize 4
set tabstospaces
set autoindent
set wordbounds
set quickblank
set trimblanks
set linenumbers
set smarthome
set mouse
set indicator
set guidestripe 80
set positionlog
set quickblank
set stateflags

set brackets ""')>]}"
set matchbrackets "(<[{)>]}"

set keycolor cyan
set functioncolor cyan
set numbercolor yellow

bind M-5 cycle main

#=============================================
# Experimental: bring useful keys to home row
#=============================================

bind M-n left main
bind M-i right main
bind M-u up main
bind M-e down main
bind M-l home main
bind M-y end main
bind M-h delete main

bind ^u pageup main
bind ^e pagedown main

# newline above current line
bind ^n "{home}{enter}{up}" main

# newline below current line
bind ^k "{end}{enter}" main

include /usr/local/share/nano/*.nanorc
include "~/.nano/tpl.nanorc"

Why Not Just Use…

Of course, anyone who comes here may have their own, possibly strong, opinions about which text editor is the one true editor. Fair enough, but I can’t be bothered to spend weeks learning how to use a text editor.

I set out to find a TUI editor to do my work with and have found it with GNU Nano.

I’ll quickly mention why some other, obvious editors didn’t beat Nano for me.

1. Emacs

Sorry, but having to type Ctrl-X Ctrl-C to exit isn’t for me.

2. Vim

Philosophically, I like the idea of a modal editor. However, in practice, I’ve always found Vim’s learning curve not worth the time. If I wrote software for a living, instead of as a hobby, I might invest more time in learning it. Maybe…

3. Micro

Micro does seem interesting. However, Nano with --modern-bindings possibly makes it less interesting.

Conclusion

Like a lot of people, I’ve been unaware of exactly what Nano is capable of. I’ve been pressing pageup repeatedly to get to the top of the screen when I could have just pressed Ctrl-home or Ctrl-end to go to the beginning or end of the file.

Likewise, I never knew that Alt-home and Alt-end would take me to the first / last lines in the viewport.

Embarrassingly, I didn’t know that Nano supported multiple buffers. So like a fool, I’d edit one file on a server, close it, open another one, close that, open the first one again…

Today I’ve just found that Nano supports soft wrapping using Alt-s which has proven useful writing this post in Textile where unwanted newlines are inserted.

I think some people in the Linux world see Nano as our Windows Notepad. Not something to be used by serious people with serious things to do. I never thought that, but after reading the excellent documentation on the Nano website (and man / info pages) I’m now aware of exactly what it can do.

I hope that you too, will have another look at this awesome little editor.

Footnotes

¹ I believe it was Ergoemacs that introduced me to this idea.

² If you want to do this yourself, here are a couple of tips. You press control by moving your palm outwards and not by stretching your pinky! Alt is pressed by moving your thumb outwards. It takes a bit of getting used to but is worth it, I think.

Trying Out OpenBSD Game of Trees (Got)

2025-06-08T00:00:00+01:00

Background & Motivations

I’ve been a Mercurial user since about 2017. It’s always worked well for me, but I’ve found myself wondering over the last couple of years if another VCS might be a better fit for me. One reason is that I’m not an advanced user, using only a fraction of what Mercurial has to offer. I don’t collaborate with anyone else so I only need to record changes, push my code out to backup servers, revert (occasionally), email patches to my production servers etc.

So today, I decided to use Got to track changes on a real project and this post documents how it went.

Installation

We’ll need to download and compile Got from here. At the time of writing version 0.113 is the latest. Then, it’s the usual:

./configure
make
sudo make install

Which will install Got to /usr/local

To check everything worked, run:

simon@computer:~$ got -h
usage: got [-hV] command [arg ...]
commands: init import clone fetch checkout update log diff blame tree status ref branch tag add remove patch revert commit send cherrypick backout rebase histedit integrate merge stage unstage cat info

The Got Way

One of the things I found most appealing about Got is that repositories and work trees have to live in different locations. When you think about it, that makes complete sense. Most of us are used to having our repo’s inside our working directory. Yes, it’s convenient but I’ve now come to the conclusion that it’s not optimal. Your repo is one thing, your working directory is another thing.

Creating the Repo and importing code

As this project is a static web site, I dectided to create a separate directory for my repo:

got init /home/simon/tmp/repo/first_project.got

There’s no requirement to append .got to the project, but it reminds me what I’m using.

simon@computer:~/tmp$ tree repo/first_project.got/

Let’s have a look to see what Got has created for us:

tree repo/first_project.got
repo/first_project.got/
├── config
├── description
├── HEAD
├── objects
│   └── pack
└── refs

4 directories, 3 files

Looks good. The next step may seem a bit clunky, but is what we need to do:

Import code into our repository
Check out the repository to a worktree

Right now, our repo is empty. Let’s do our first checkin:

simon@computer:~/tmp$ got import -r repo/first_project.got/ first_project/
A  /home/simon/tmp/first_project/css/pico.min.css
A  /home/simon/tmp/first_project/css/tacit.min.css
A  /home/simon/tmp/first_project/fonts/Inter-Italic-VariableFont_opsz,wght.woff2
A  /home/simon/tmp/first_project/fonts/Inter-VariableFont_opsz,wght.woff2
A  /home/simon/tmp/first_project/software/visitors/index.html
A  /home/simon/tmp/first_project/banner.png
A  /home/simon/tmp/first_project/index.html
Created branch refs/heads/main with commit 95f800a6d501db76d7fe56978e6c396d0c3736bb

Now that our working directory’s contents have been added to our repo, we can delete the working directory and check out a worktree:

simon@computer:~/tmp$ rm -r first_project

simon@computer:~/tmp$ got co repo/first_project.got/ first_project
A  first_project/asus.png
A  first_project/css/pico.min.css
A  first_project/css/tacit.min.css
A  first_project/fonts/Inter-Italic-VariableFont_opsz,wght.woff2
A  first_project/fonts/Inter-VariableFont_opsz,wght.woff2
A  first_project/index.html
A  first_project/software/visitors/index.html
Checked out refs/heads/main: 95f800a6d501db76d7fe56978e6c396d0c3736bb
Now shut up and hack

Now, inside our worktree

simon@computer:~/tmp/first_project$ got log
-----------------------------------------------
commit 95f800a6d501db76d7fe56978e6c396d0c3736bb (main)
from: Simon Harrison <mail@simonh.uk>
date: Sun Jun  8 20:16:54 2025 UTC

 First checkin

Got uses a hidden directory to keep track of things — .got. Let’s see what’s in there:

simon@computer:~/tmp/first_project$ ll
total 116K
drwxr-xr-x  6 simon simon 4.0K 2025-06-08 21:21 .
drwxrwxrwx 11 simon simon 4.0K 2025-06-08 21:21 ..
-rw-r--r--  1 simon simon  84K 2025-06-08 21:21 asus.png
drwxr-xr-x  2 simon simon 4.0K 2025-06-08 21:21 css
drwxr-xr-x  2 simon simon 4.0K 2025-06-08 21:21 fonts
drwxr-xr-x  2 simon simon 4.0K 2025-06-08 21:21 .got
-rw-r--r--  1 simon simon 4.6K 2025-06-08 21:21 index.html
drwxr-xr-x  3 simon simon 4.0K 2025-06-08 21:21 software
simon@computer:~/tmp/first_project$ tree .got
.got
├── base-commit
├── file-index
├── format
├── head-ref
├── lock
├── path-prefix
├── repository
└── uuid

1 directory, 8 files

Obviously, we’ll leave all that alone as Got will update them as we check in changes.

So at this point, we’re all setup to track changes to our code. The last thing I want to do right now is set up a backup that I can push to one of my servers.

We’ll do that in our repo:

simon@computer:~/tmp/first_project.got$ nano got.conf

And add the following, adjusting to fit your server

remote "simonh" {
  server myserver.com
  protocol ssh
  repository "/home/simon/repo/got/first_project.got"
}

For this to work, we need to create a Got repo on our server that matches the repository value above.

With this done, we can got send our changes. By the way, most commands have aliases:

ci — commit
co — checkout
se — send
fe — fetch
pa — patch

Consult the excellent man pages for the full list. Let’s send our changes to keep them safe:

simon@computer:~/tmp/first_project$ got se simonh
Connecting to "simonh" ssh://myserver.com/home/simon/repo/got/first_project.got
2 commits colored; 3 objects found; 6 trees scanned
packing 1 reference; 3 objects; deltify: 100%; uploading pack:  2.7K 100%
Server has accepted refs/heads/main

Got also comes with Tog, a CLI repository browser:

Creating & Listing Tags

Tags are equivalent to Mercurial bookmarks — a label for a commit.

simon@computer:~/tmp/first_project$ got tag -l
-----------------------------------------------
tag 0.2 807e609a829c43a9f3f9c8d4c1214b4371fe4485
from: Simon Harrison <mail@simonh.uk>
date: Sun Jun  8 16:12:03 2025 UTC
object: commit 3aa38f1608aee14a1a90a7eae3928f83e4c14baa

 Second test of using tags

-----------------------------------------------
tag 0.1 c291af4175f48db379169437ac1a09e381a631f8
from: Simon Harrison <mail@simonh.uk>
date: Sun Jun  8 13:34:21 2025 UTC
object: commit 8e431b7a506992d02d5e39b283652bb5d3ae9d03

 Just testing out tags

Passing -s gives us a short one line summary:

simon@computer:~/tmp/first_project$ got tag -ls
2025-06-08 commit:3aa38f1608 0.2: Second test of using tags
2025-06-08 commit:8e431b7a50 0.1: Just testing out tags

We can create a tag easily:

simon@computer:~/tmp/first_project$ got tag '0.3'
Created tag d1d6792cb83866567525f66875b983faf7dcd4d7

simon@computer:~/tmp/first_project$ got tag -sl
2025-06-09 commit:f257381e43 0.3: Third version
2025-06-08 commit:3aa38f1608 0.2: Second test of using tags
2025-06-08 commit:8e431b7a50 0.1: Just testing out tags

Branching & Merging

To be honest, I don’t branch. If that changes I’ll update this section.

Conclusion

Initial setup for basic tracking of changes should take no more than fifteen / twenty minutes. Once you’ve done that, you’ll have a nice, well-documented VCS that has a tight focus. I’m going to continue using Got for this one project. I should point out that Got is not Git. Many features are not implemented (and may never be). However, it’s perfectly usable right now and if you’re looking for a no frills VCS, Got may be just what you need.

First Look at OpenBSD Game of Trees (Got)

2024-10-29T00:00:00+00:00

Git, but simpler (to use, maybe…)

I’m a long time Mercurial user. Even so, I keep my eye on other VCS’s to see what new ideas are developing. A few weeks ago, I came across Got, a reimplementation / reimagining of Git for use by the OpenBSD people, under a BSD compatible licence.

Got claims to take some of the best features from CVS, SVN, Git, Mercurial and Fossil. This I found very interesting. Here are the most noticeable differences I’ve noticed (for a Git or Mercurial user).

Repositories must checked out to a worktree.

Hg and Git users are used to having the repository inside the worktree (working directory). Got treats a repository and a worktree as two completely different things. Or rather, Got treats the repository as one thing and you can have as many worktrees as you need. The benefit of this model is that you can have different branches checked out at the same time. You can checkout only a subset of the repository. This is also the same model that Fossil uses. When I tried using Fossil years ago, I intensely disliked this “feature”, but I’m now starting to think it makes sense.

No built in help

As always, OpenBSD prefers to have excellent man pages to document their software and answer any question you might have.

simon@computer:~$ got help
got: unknown command 'help'
commands: init import clone fetch checkout update log diff blame tree status ref branch tag add remove patch revert commit send cherrypick backout rebase histedit integrate merge stage unstage cat info

Whether that’s planned to change, I’m not sure. Probably not.

Some important features are currently missing

Last time I checked, file renames weren’t automatically handled. I believe this is planned to change. Little, quality of life features like hg email will probably never be implemented.

There’s a steeper learning curve

You really do have to RTFM when it comes to Got. I got stuck on some basic stuff and asked on IRC for assistance. The lead developer, Stefan Sperling is actually a very helpful guy and got me going. Still, be prepared to read the entire man page!

Features I’d love to see

hg shelve / Git stash

Set aside changes and bring them back later. Very useful, (nearly) impossible to live with once you’ve used it.

Easy emailing of patches

I use hg email -r tip all the time when I’m working on a project.

Local revision numbers

Mercurial is (I think) the only current VCS that has local revision numbers. That feature is just so handy. Much easier for humans to deal with revision 17 than revision bf63491f5a5b

Conclusion

Got makes it clear on their website they aren’t seeking to replace anything or to steal you away from your VCS of choice. In the FAQ they state:

If you are wondering why Got even exists, you can just ignore it.

Quite right. There are far too many people on the internet moaning that their thing is the only thing that matters and if everyone would just use it everyone would be far happier as a result. Got isn’t one of those projects.

It’s written by, and for, the OpenBSD crew. There’s a portable port for us Linux types. I’ll be keeping an eye on this one in case it ends up as useful as OpenSMTPD

Using GNU RCS for Managing Configuration Files

2023-08-25T00:00:00+01:00

As regular readers will know, I’m a big fan of version control for software projects. You may also know that I’ve recently started to use RCS for managing single files (shell/Python scripts, CGI scripts etc.). Tonight, I’d like to present my simple workflow for version controlling configuration files, also using RCS.

The Problem

We Linux users frequently have to create and edit config files. Here is a list of some of mine:

.vimrc
.nanorc
/etc/smtpd.conf
/etc/aliases
/etc/hosts
/etc/apache2/*
/etc/apt/sources*
...

Up until recently, I’d usually rename a config file to <filename>.bak in case something went horribly wrong. Come to think of it, that’s pretty much always the advice you’ll be given online.

Of course, there’s a better, more robust way of keeping track of changes you’ve made, with the ability to roll back to any previous revision. But, it does take a little bit of discipline!

The Solution

Let’s say you need to edit your .nanorc

simon@computer:~$ mkdir RCS
mkdir: cannot create directory ‘RCS’: File exists

OK, good. I’ve already got a repository setup in my home directory. Let’s open it up and see what it contains:

nano .nanorc
 GNU nano 6.2                         .nanorc                                  



















                                [ Read 0 lines ]
^G Help      ^O Write Out ^W Where Is  ^K Cut       ^T Execute   ^C Location
^X Exit      ^R Read File ^\ Replace   ^U Paste     ^J Justify   ^/ Go To Line

An empty file which means the file either didn’t exist or was empty. Let’s get to work. I’ll add the settings I want, plus the RCS metadata so I can see the current version number and the date it was checked in:

  GNU nano 6.2                         .nanorc                           I   S  
#=====================================================================
# $Revision$
# $Date$
# $Author$
#======================================================================

set autoindent
set tabsize 4
set tabstospaces
set softwrap









                               [ Read 10 lines ]
^G Help      ^O Write Out ^W Where Is  ^K Cut       ^T Execute   ^C Location
^X Exit      ^R Read File ^\ Replace   ^U Paste     ^J Justify   ^/ Go To Line

Now let’s checkin the file

simon@computer:~$ ci -l .nanorc 
RCS/.nanorc,v  <--  .nanorc
enter description, terminated with single '.' or end of file:
NOTE: This is NOT the log message!
>> First checkin 
>> .
initial revision: 1.1
done
simon@computer:~$

Let’s have a look at the file

simon@computer:~$ cat .nanorc 
#=====================================================================
# $Revision: 1.1 $
# $Date: 2023/08/25 19:11:01 $
# $Author: simon $
#======================================================================

set autoindent
set tabsize 4
set tabstospaces
set softwrap

So, three things are immediately evident:

1. The file is version controlled
2. I can see how many revisions have been made to the file
3. I can see the datetime of the last revision. Nice.

I read somewhere that it’s a good idea to add the .nanorc files from the /usr/share/nano/* directory to enable syntax highlighting for lots of filetypes. Let’s add that to my .nanorc file:

  GNU nano 6.2                         .nanorc                           I   S  
#=====================================================================
# $Revision: 1.1 $
# $Date: 2023/08/25 19:11:01 $
# $Author: simon $
#======================================================================

set autoindent
set tabsize 4
set tabstospaces
set softwrap

include "/usr/share/nano/*.nanorc"







                               [ Read 13 lines ]
^G Help      ^O Write Out ^W Where Is  ^K Cut       ^T Execute   ^C Location
^X Exit      ^R Read File ^\ Replace   ^U Paste     ^J Justify   ^/ Go To Line

Now I’ll checkin the change

simon@computer:~$ ci -l ./.nanorc 
./RCS/.nanorc,v  <--  ./.nanorc
new revision: 1.2; previous revision: 1.1
enter log message, terminated with single '.' or end of file:
>> Add all .nanorc files from /usr/share/nano/*
>> .
done
simon@computer:~$

If I accidently try and checkin an unchanged file, RCS will say:

simon@computer:~$ ci -l ./.nanorc 
./RCS/.nanorc,v  <--  ./.nanorc
file is unchanged; reverting to previous revision 1.2
done

If you’re ever unsure you can always run rlog:

simon@computer:~$ rlog ./.nanorc 

RCS file: ./RCS/.nanorc,v
Working file: ./.nanorc
head: 1.2
branch:
locks: strict
        simon: 1.2
access list:
symbolic names:
keyword substitution: kv
total revisions: 2;     selected revisions: 2
description:
First checkin
----------------------------
revision 1.2    locked by: simon;
date: 2023/08/25 19:20:20;  author: simon;  state: Exp;  lines: +6 -3
Add all .nanorc files from /usr/share/nano/*
----------------------------
revision 1.1
date: 2023/08/25 19:11:01;  author: simon;  state: Exp;
Initial revision
=============================================================================

Yep, 1.2 is the latest revision. Another option is to run rcsdiff:

simon@computer:~$ rcsdiff ./.nanorc 
===================================================================
RCS file: ./RCS/.nanorc,v
retrieving revision 1.2
diff -r1.2 ./.nanorc

No changes are shown (as no changes have been made).

What if we have changes? I’ll add the setting to show line numbers (without checking the file in):

simon@computer:~$ rcsdiff -u ./.nanorc 
===================================================================
RCS file: ./RCS/.nanorc,v
retrieving revision 1.2
diff -u -r1.2 ./.nanorc
--- ./.nanorc   2023/08/25 19:20:20     1.2
+++ ./.nanorc   2023/08/25 19:29:15
@@ -8,6 +8,7 @@
 set tabsize 4
 set tabstospaces
 set softwrap
+set linenumbers

 include "/usr/share/nano/*.nanorc"

I’ve now run ci -l .nanorc but decide I don’t really need to see to see line numbers. I want to go back to the revision before I added that line:

simon@computer:~$ co -l -r1.2 ./.nanorc 
./RCS/.nanorc,v  -->  ./.nanorc
revision 1.2 (locked)
writable ./.nanorc exists; remove it? [ny](n): y
co: ./RCS/.nanorc,v: warning: You now have 2 locks.
done

Let’s see the revision of the file now:

simon@computer:~$ cat .nanorc 
#=====================================================================
# $Revision: 1.2 $
# $Date: 2023/08/25 19:20:20 $
# $Author: simon $
#======================================================================

set autoindent
set tabsize 4
set tabstospaces
set softwrap

include "/usr/share/nano/*.nanorc"

Ah good, we’ve rolled back to where we were. And that’s all there is to it. Here is the new workflow

The New Workflow

1. mkdir RCS in the working directory
2. Add RCS metadata to the file if it doesn’t exist
3. Make changes to the config file
4. Checkin the changes (ci -l myfile.conf)
5. If you make a mistake or dislike a setting, co -l -r1.?

Some nice features would be to automatically create the RCS directory if it doesn’t exist and to have all these RCS directories backed up. That’s for another post though…

Running bottle.py with systemd

2023-03-25T00:00:00+00:00

I have a number of bottle.py web services that I run on my servers. For the last few years, I’ve been using tmux to launch these services and then detaching them so they continue running after I logout of the SSH session. Recently, one of those servers had an issue causing the provider to require moving my VPS to another machine. So, when I went to use it, I got the 503 Service Unavailable message (that no one likes to see)!

Today, I finally got around to setting up systemd service files for these services. It only took about an hour (including troubleshooting things not working). Below is the shopping.service file (for a web based shopping list I wrote):

# Save this file in ~/.config/systemd/user/shopping.service
# then run:
# $ systemctl --user daemon-reload
# $ systemctl --user enable shopping.service
# Below will prevent the service stopping when we terminate the SSH session
# $ sudo loginctl enable-linger username

[Unit]
Description=Shopping List (bottle.py)
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/simon/web/shopping
ExecStart=/usr/bin/python3 /home/simon/web/shopping/shopping.py
Restart=always

[Install]
WantedBy=default.target

Note: make sure to follow the comments. Otherwise, you’ll also have problems.

Once you’ve created and symlinked the file to ~/.config/systemd/user, you’ll be able to check all is good as you can for any system process.

simon@server:~/web/shopping [ssh] $ systemctl --user status shopping_list.service 
● shopping_list.service - Shopping List (bottle.py)
     Loaded: loaded (/home/simon/.config/systemd/user/shopping_list.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2023-03-25 20:26:29 GMT; 9min ago
   Main PID: 32632 (python3)
      Tasks: 11 (limit: 1129)
     Memory: 100.3M
        CPU: 7.471s
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/shopping_list.service
             └─32632 /usr/bin/python3 /home/simon/web/shopping/run_sl2.py

I’ll monitor this one service for a while, then I plan to use systemd service files for all my web services.

Using GNU RCS in 2023

2023-02-11T00:00:00+00:00

Old School Version Control

Since I started using Mercurial a few years back, I’ve realised the value in having everything important under version control. The thing is, tools like Mercurial and Git don’t seem to be quite the right tool when you’re wanting to manage single files.

Enter RCS

RCS dates back to 1982, and was last updated in 2022 (still active). I believe that one of RCS’s perceived weaknesses can actually be a strength: it was designed to work with single files. Using it is very easy. Let’s get started!

The first thing I do is add some variables that RCS will populate for us. I’ll add a bit of metadata to our fake program my_program

# $Date$
# $Revision$
# $Author$

These will be replaced when we “checkout” our file from RCS. But first we need to “checkin” our initial revision.

# $Date$
# $Revision$
# $Author$

This is the first version of my file. I've done a small amount of work that I want to save.

Let’s check in our first version.

simon@computer:~/tmp/my_script$ mkdir RCS

simon@computer:~/tmp/my_script$ ll
total 16K
drwxrwxr-x  3 simon simon 4.0K 2023-02-11 11:16 .
drwxrwxr-x 18 simon simon 4.0K 2023-02-11 11:09 ..
-rw-rw-r--  1 simon simon  128 2023-02-11 11:15 my_program
drwxrwxr-x  2 simon simon 4.0K 2023-02-11 11:16 RCS

simon@computer:~/tmp/my_script$ ci my_program 
RCS/my_program,v  <--  my_program
enter description, terminated with single '.' or end of file:
NOTE: This is NOT the log message!
>> Version 1 of my_program
>> .
initial revision: 1.1
done

So we’ve performed our first commit. Let’s see the state of our working directory

simon@computer:~/tmp/my_script$ tree
.
└── RCS
    └── my_program,v

1 directory, 1 file

As you can see, our original file has been destroyed! We now have one file in the RCS directory my_program,v. That file is our “versioned” file. Let’s take a look at it

simon@computer:~/tmp/my_script$ cat RCS/my_program,v 
head    1.1;
access;
symbols;
locks; strict;
comment @# @;


1.1
date    2023.02.11.11.16.25;    author simon;   state Exp;
branches;
next    ;


desc
@Version 1 of my_program
@


1.1
log
@Initial revision
@
text
@# $Date$
# $Revision$
# $Author$

This is the first version of my file. I've done a small amount of work that I want to save.


@
simon@computer:~/tmp/my_script$

As you can see, there’s quite a bit of information in there, along with the text of my_program

To retrieve our file and do some work, we need to check it out, with a lock, to prevent other people working on it at the same time as us (to avoid conflicts).

simon@computer:~/tmp/my_script$ co -l ./my_program 
./RCS/my_program,v  -->  ./my_program
revision 1.1 (locked)
done

Let’s see our working directory state now

simon@computer:~/tmp/my_script$ tree
.
├── my_program
└── RCS
    └── my_program,v

1 directory, 2 files

Great, the file is back! Let’s take a look at the contents

simon@computer:~/tmp/my_script$ cat my_program 
# $Date: 2023/02/11 11:16:25 $
# $Revision: 1.1 $
# $Author: simon $

This is the first version of my file. I've done a small amount of work that I want to save.

As I said earlier, the dollar variables have been expanded by RCS. Let’s make another change and see what happens.

# $Date: 2023/02/11 11:16:25 $
# $Revision: 1.1 $
# $Author: simon $

This is the first version of my file. I've done a small amount of work that I want to save.

I've simply added a blank newline and this line now.

Running rcsdiff will show us unsaved changes. Use the -u option to get unified diff output.

simon@computer:~/tmp/my_script$ rcsdiff -u ./my_program 
===================================================================
RCS file: ./RCS/my_program,v
retrieving revision 1.1
diff -u -r1.1 ./my_program
--- ./my_program        2023/02/11 11:16:25     1.1
+++ ./my_program        2023/02/11 11:29:41
@@ -4,4 +4,6 @@

 This is the first version of my file. I've done a small amount of work that I want to save.

+I've simply added a blank newline and this line now.
+

simon@computer:~/tmp/my_script$

All looks good, so we’ll commit our changes.

simon@computer:~/tmp/my_script$ ci ./my_program 
./RCS/my_program,v  <--  ./my_program
new revision: 1.2; previous revision: 1.1
enter log message, terminated with single '.' or end of file:
>> Did a bit of work
>> .
done

Very easy and painless, eh? Once again, our file will have been destroyed so we need to check it out again to either work on it (with the -l option to have a locked version of the file) or without a lock for a read only version.

simon@computer:~/tmp/my_script$ co ./my_program 
./RCS/my_program,v  -->  ./my_program
revision 1.2
done
simon@computer:~/tmp/my_script$ cat my_program 
# $Date: 2023/02/11 11:31:42 $
# $Revision: 1.2 $
# $Author: simon $

This is the first version of my file. I've done a small amount of work that I want to save.

I've simply added a blank newline and this line now.

The date and revision have been updated and our most recent changes are there. Because we didn’t lock our file, it’s readonly.

  GNU nano 6.2                       my_program                                 
# $Date: 2023/02/11 11:31:42 $
# $Revision: 1.2 $
# $Author: simon $

This is the first version of my file. I've done a small amount of work that I w>

I've simply added a blank newline and this line now.












                      [ File 'my_program' is unwritable ]
^G Help      ^O Write Out ^W Where Is  ^K Cut       ^T Execute   ^C Location
^X Exit      ^R Read File ^\ Replace   ^U Paste     ^J Justify   ^/ Go To Line

GNU Nano tells us at the bottom that the file is unwritable. Fair enough.

We’ll checkout an editable version now with co -l

simon@computer:~/tmp/my_script$ co -l ./my_program 
./RCS/my_program,v  -->  ./my_program
revision 1.2 (locked)
done

If we want to view our changes rlog will happily tell us

simon@computer:~/tmp/my_script$ rlog ./my_program 

RCS file: ./RCS/my_program,v
Working file: ./my_program
head: 1.2
branch:
locks: strict
        simon: 1.2
access list:
symbolic names:
keyword substitution: kv
total revisions: 2;     selected revisions: 2
description:
Version 1 of my_program
----------------------------
revision 1.2    locked by: simon;
date: 2023/02/11 11:31:42;  author: simon;  state: Exp;  lines: +5 -3
Did a bit of work
----------------------------
revision 1.1
date: 2023/02/11 11:16:25;  author: simon;  state: Exp;
Initial revision
=============================================================================

And that’s really all there is to it (for my purposes anyway). Take a look at the excellent manual for everything else you can do with RCS.

I’m sure you can see the value of RCS for shell scripts, single Python files, etc.

Manage Invoices with GNU Recutils

2022-08-13T00:00:00+01:00

I’ve been running an NDT business since 2017. After a couple of years getting annoyed at writing my reports in Libreoffice Writer, I decided to write my own software to generate the reports for me. This has saved me an enormous amount of time. A couple of years ago, I started thinking about how I should handle my invoices. I don’t generate a lot of invoices, but sometimes I’d lose track of who’d paid, who hadn’t and realised I needed something to keep track of them.

One option was just to simply add it to my existing system. But, I couldn’t really be bothered. So I decided to have a simple spreadsheet. This worked ok, I simply added a new row to my “invoice_log_2022.xlsx” and then email myself the new version so I had some redundancy. But it bugged me a little as it seemed there should be an easier way to handle a few records. For example, I had to use some spreadsheet program just to add / view my invoices.

So, I thought I’d see what other options I could find. Eventually, I came across GNU Recutils. Here is the description from their site:

GNU Recutils is a set of tools and libraries to access human-editable, plain text databases called recfiles. The data is stored as a sequence of records, each record containing an arbitrary number of named fields. The picture below shows a sample database containing information about GNU packages, along with the main features provided by Recutils.

This seemed to be worth a look! I converted my spreadsheet to csv, then imported it into the recfile format.

Invoice_No: N123456
Due_Date: 2022-07-31
Amount: £180.00
Customer: Some Customer
Description: MPI of fillet welds on fabricated assemblies
Amount_Paid: £180.00
Date_Paid: 2022-08-05

Invoice_No: N123457
Due_Date: 2022-07-31
Amount: £150.00
Customer: Another Customer
Description: DPI of condenser pipework
Amount_Paid: 
Date_Paid:

The above shows two records. The first one has been paid, the second one has not. If I want to add a new invoice to the log, I simply copy the last record, edit the fields and save the file. It really doesn’t get much easier than that. No need for anything more than a text editor.

I’ve only been using recutils for a week and have just started scratching the surface of the tools that come with it, recins, recsel, recdel etc. One thing I want to do is have a cron job that will email me any invoices that are overdue.

Conclusion

I really like Sqlite. I use it for loads of things, but sometimes even Sqlite is overkill. The next time you think about creating a new spreadsheet to save a few records, perhaps you’ll try Recutils instead…

A Guide to Choosing, Starting and Finishing Software Projects

2021-12-18T00:00:00+00:00

Intro

I’ve been a productive programmer for about four years now and believe I’m in a position to offer some friendly advice to others who might be struggling.

I tried, and failed, to learn programming for many years. The turning point for me was a real world project for a company that I worked for.

Step One

Pick a real world project

My first project was a system to record torque settings in a workshop. The way they had done it for many years, was for each technician to record the settings in a bound book. This would include

their clock number
the date
the job number
the torque value applied
the torque wrench used
the torque meter used

And finally, they’d stamp the last column instead of a signature (company policy) so if there was an issue, they could ascertain who’d torqued a particular bolt.

The problems with this method, as I saw it, were:

someone had to regularly print out the new books (ME)!
finding a record was like finding a needle in a haystack
no backup – if book is lost, all records gone

Bear in mind, this company had about twenty of these books in daily use. Also, it was not uncommon to have to try to find a record (if an engine or transmission failed in service).

So, I offered to attempt to write a web based system, accessed using a couple of tablets the company already had. It took maybe a month or two of work in my spare time to have a prototype built. We tested it using one station, and when that proved to work better than the old way, we introduced the second tablet.

This was not a tech company and the supervisors were genuinely surprised and delighted that they could now search on any field:

all records for a particular job
all records for a particular technician
all records on a date or in a date range

etc.

Unfortunately, their lazy, unskilled IT department didn’t like me in their turf and started causing problems, so I decided to turn the program off. But, it was very successful and well liked by its users and management who were directly involved.

Step Two

Make it better than the old way

Personally, I love writing software. Starting with nothing and ending up with something in use, better than how it was done previously. Every software project I’ve written has been to replace something that could easily be improved, i.e. to replace a spreadsheet, a white board, or paper.

When you’re starting out people can be nervous about change. So I always say to them:

“If you don’t like it, you can always go back to how you did it before.”

Knowing fine well that once they’ve used my system, going back will be the last thing they want to do! But it’s important to tell them, to reassure them that it’s just a trial. That they are in control. People are weird like that.

Step Three

Launch as soon as you can

Once you’ve picked your real world project, get to work! As with all things, you’ll get better and quicker and more confident as you write more code. You’ll also get feedback from your users that you couldn’t predict. The sooner you launch, the sooner your project improves. That’s been my experience.

Step Four

Work as much as you can to improve it

So your project is live, real people are using it and benefiting from your skills. Over the next few weeks / months, try and make it better. Add new features that your users suggest (within reason). Make things simpler and easier for your users. I guarantee you that it will take months until you can officially say it’s “done.”

Summary and Conclusion

So, lets recap

1. Pick a real world project
2. Make it better than the old way
3. Launch as soon as you can
4. Work as much as you can to improve it
5. GOTO step 1

And that’s how I work. That’s my system for writing software. Of course there’s loads of other things that you should consider (version control, backups, security etc.) but for my first couple of projects, I didn’t bother with version control software. I’d struggle to go back to not using it now, but I muddled through. The other stuff you’ll pick up as you need to.

In case you’re interested, here is the before and after for the torque meter software project I mentioned above:

Before:

After:

Send SMS from Linux Computer via ADB

2021-11-14T00:00:00+00:00

Since I learnt how to send emails from my servers and desktop, I thought it would be useful to send SMS (Short Message Service) texts as well. No specific use cases yet, but I’m sure one day it’ll be needed…

After looking into smstools, Gammu, and loads of others, I was getting nowhere and ready to give up. But then, I remembered ADB. Like most people, I’ve got numerous Android phones knocking about, surely I could use an old Android phone + ADB to send an SMS.

I started off with my first Android phone, an HTC Desire. Even though this phone is now 11 years old, it still works perfectly and it’d be nice to give it something to do!

Android 2 (Gingerbread)

The first thing to notice is that we put the phone number in just as we’d type in to to the phone. No +44XXXXXXXXX needed here. So let’s start sending some SMS messages.

To send an sms with spaces, which is almost certainly required, we either escape each space character with a back slash, or far better, enclose the string in single quotes:

simon@computer:~$ adb shell service call isms 5 s16 "07123456789" i32 0 i32 0 s16 "'Here is some text with spaces but not escaped.'"
Result: Parcel(00000000    '....')
simon@computer:~$

I wondered if I could send the output of shell commands (yep):

simon@computer:~$ adb shell service call isms 5 s16 "07123456789" i32 0 i32 0 s16 "'$(uptime)'"

Here are some of the messages I received on my mobile. The last message was the uptime message above.

I also had a Nokia 5 in a drawer.

Android 9

simon@computer:~$ adb shell service call isms 7 i32 0 s16 "com.android.mms.service" s16 "+447123456789" s16 "null" s16 "Hello\ from\ your\ home\ computer." s16 "null" s16 "null"

As you can see, the syntax is different depending on your Android version. I’ve only just started scratching the surface with ADB, but there’s lots of potential for all sorts of helpful notifications.

I’ll probably write a new post once I’ve done some more experimenting.

On the Pros and Cons of Software Testing

2021-06-10T00:00:00+01:00

Over the years, I’ve noticed that there are two types of coders

Those who don’t really see testing as something worth their time
Those who test everything

I fall into the first camp. There’s probably a transtion from 1 to 2, but it seems to be an all or nothing type arrangement to me. You don’t come across many coders who do “some” automated testing.

Now, of course, we all test functionally. You’ve just added a new feature to your untested code, so you run it and test it out. Try and break it. Often, it will break quite happily. You missed something obvious and then you fix the code, retest, and all looks good.

I run quite a lot of code like this. As do many others. I think, secretly, many coders know that test suites are a bit of a con. You may run 5000 tests and everything passes.

So, therefore the code is good?! Bug free? Not necessarily…

I mean, think about it. You are going to test every permutation of permissable inputs? Or outputs?

Nope (unless you’re sqlite).

For a lot of code, it doesn’t really matter.

You run it
It works
Something breaks
You fix it
You run it

That seems reasonable to me. And it’s exactly the same process with an automated test suite except there’s a write tests step.

Small time coders can get away with dealing with bugs as they come up. I’ve had a few chewy moments, but if you coded intelligently / defensively in the first place, you’ve built workarounds in to handle minor bugs.

My Workflow

When I start a new project, I initialize a Mercurial repository, bring in all the files I know I’ll need right away, and then start coding. I work alone, by the way, so there’s no corner of the code that I haven’t touched.

I work on one feature at a time. Anything that feature might touch, I functionally test as I go.

Example

Last year, I wrote a web based shopping list for my partner and me to use instead of a paper list. We’ve been using it every week since, and generally it works perfectly.

But, eventually a very annoying bug showed up. This cropped up after it running for about six months. What went wrong was she created a new list, and started to add items. And then she got a 500 error (Internal Server Error).

Now, I’d set up my database so that each shopping list was indexed by date. So if she created a list today (2021-06-11) and then tried to create a new list on the same date, she’d be prompted whether to add to the existing list or whether she wanted to delete the existing list. The reason being; my partner will create a new shopping list every Wednesday, add all the items she knows we need, then on the Thursday, one of us ticks off the items on our phones to remove them as we’re putting items in to the trolley.

But on this day, she created a new list (a week after the old one), started to add items to the list and got the 500 error for nearly every item she tried to add! I was at home at the time, so SSH’d into the server to have a look what was going on. The traceback pointed to an sqlite integrity error.

It turned out that instead of creating a new list, she was adding to the old list. Not her fault as I’d coded it that creating a new list would close the old list, and make a new one. I checked all the relevant code, and everything looked good.

When I designed the software, one of the features I thought she’d like, was being able to look at previous lists, and just click each item to add it to the new list. She generally gets a lot of items each week that are the same as every other week.

Here is a screenshot of the /lists page:

So from here she can click on the first date (or any date) and every item added on that date can be clicked and that will add it to the current list:

If the item has already been added, it won’t be added again.

It was on the /lists page that I realised that a new list hadn’t been created. The solution was to delete the old list, and create a new one.

Conclusion

I never found the actual cause of the bug. I think it was possibly a caching issue on her phone. It has never showed up again. Even if I’d written tests for everything I could think of, I wouldn’t have considered the possibility that today’s date would be anything other than: today’s date!

Why I still Use Windows XP

2021-05-21T00:00:00+01:00

Windows XP was on the first PC I owned, built by my brother in about 2001/2002. The computer had a 20GB hard disk, a 750Mhz processor, and served me well for many years.

To be honest, I always liked XP. I upgraded to Vista at some point, but that ran so badly, I switched to using Linux permanently in about 2008 or 2009.

Ever since I’ve had computers powerful enough to run VirtualBox, I’ve always had a Windows XP virtual machine.

Why?

Lightweight and fast
Good enough
Runs most of the Windows software I might need
Stable

I’m not a Free Software “fanatic”. There’s nothing as good as Adobe Acrobat XI that runs natively on Linux. Some people, myself included, consider Acrobat 11 far better than Acrobat DC. Recent Adobe software is more about them, and less about their users. Photoshop CS 5.1 also runs on XP and does pretty much anything you might need to do to a photo (if you can’t figure out how do it in GIMP).

Windows XP RAM Usage (Idling)

Linux Mint 20.1 Cinnamon (Idling)

MX Linux 19.4 (Idling)

Big difference between the three, aren’t there?!

Tips

My XP instance has no internet access (why would it?). I only use it for occasional Acrobat, a bit of WordPerfect and some IronPython stuff.

If you want to do some scripting, IronPython 2.7.8 is the last version that runs on XP (to my knowledge). I tried 2.7.9 to 2.7.11 (latest) but no dice, probably due to the .net framework 4.0 being the last to run on XP.

Make sure you’re running SP3, which was the last service pack released.

Getting Started with GnuPG (gpg)

2021-05-20T00:00:00+01:00

I’ve been meaning to spend a bit of time looking at use cases for gpg

I wasn’t aware that you don’t even need to set up any accounts. You can start using for symmetric encryption right out of the box. Symmetric encryption relies on a shared password / passphrase that is shared with recipients.

I’m using a fairly fresh installation of mxlinux in Virtual Box. GnuPG is already installed as with pretty much any distro.

As you can see below, lots of options are available.

simon@mx:~
$ gpg -h
gpg (GnuPG) 2.2.12
libgcrypt 1.8.4
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/simon/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Syntax: gpg [options] [files]
Sign, check, encrypt or decrypt
Default operation depends on the input data

Commands:

 -s, --sign                  make a signature
     --clear-sign            make a clear text signature
 -b, --detach-sign           make a detached signature
 -e, --encrypt               encrypt data
 -c, --symmetric             encryption only with symmetric cipher
 -d, --decrypt               decrypt data (default)
     --verify                verify a signature
 -k, --list-keys             list keys
     --list-signatures       list keys and signatures
     --check-signatures      list and check key signatures
     --fingerprint           list keys and fingerprints
 -K, --list-secret-keys      list secret keys
     --generate-key          generate a new key pair
     --quick-generate-key    quickly generate a new key pair
     --quick-add-uid         quickly add a new user-id
     --quick-revoke-uid      quickly revoke a user-id
     --quick-set-expire      quickly set a new expiration date
     --full-generate-key     full featured key pair generation
     --generate-revocation   generate a revocation certificate
     --delete-keys           remove keys from the public keyring
     --delete-secret-keys    remove keys from the secret keyring
     --quick-sign-key        quickly sign a key
     --quick-lsign-key       quickly sign a key locally
     --sign-key              sign a key
     --lsign-key             sign a key locally
     --edit-key              sign or edit a key
     --change-passphrase     change a passphrase
     --export                export keys
     --send-keys             export keys to a keyserver
     --receive-keys          import keys from a keyserver
     --search-keys           search for keys on a keyserver
     --refresh-keys          update all keys from a keyserver
     --import                import/merge keys
     --card-status           print the card status
     --edit-card             change data on a card
     --change-pin            change a card's PIN
     --update-trustdb        update the trust database
     --print-md              print message digests
     --server                run in server mode
     --tofu-policy VALUE     set the TOFU policy for a key

Options:

 -a, --armor                 create ascii armored output
 -r, --recipient USER-ID     encrypt for USER-ID
 -u, --local-user USER-ID    use USER-ID to sign or decrypt
 -z N                        set compress level to N (0 disables)
     --textmode              use canonical text mode
 -o, --output FILE           write output to FILE
 -v, --verbose               verbose
 -n, --dry-run               do not make any changes
 -i, --interactive           prompt before overwriting
     --openpgp               use strict OpenPGP behavior

(See the man page for a complete listing of all commands and options)

Examples:

 -se -r Bob [file]          sign and encrypt for user Bob
 --clear-sign [file]        make a clear text signature
 --detach-sign [file]       make a detached signature
 --list-keys [names]        show keys
 --fingerprint [names]      show fingerprints

Please report bugs to <https://bugs.gnupg.org>.

Now, we’ll take a man page (uptime as it’s nice and short) and save it to a file ready to encrypt and email to test encryption.

simon@mx:~
$ man uptime > uptime.txt

simon@mx:~
$ gpg --armor --symmetric uptime.txt

We could save some keystrokes by just typing gpg -ac <filename>

You’ll be prompted for a password (which you have to enter twice), I chose simonh

As you can see, gpg also compresses the encrypted file (uptime.txt.asc).

simon@mx:~
$ ll
total 300K
-rw-r--r--  1 simon simon  113 Apr 17 18:18 dead.letter
drwxr-xr-x  2 simon simon 4.0K Apr 15 19:17 Desktop
drwxr-xr-x  2 simon simon 4.0K Apr 15 19:25 Documents
drwxr-xr-x  2 simon simon 4.0K Apr 15 19:25 Downloads
drwx------  2 simon simon 4.0K Apr 16 15:36 mail
drwx------  5 simon simon 4.0K Apr 16 17:26 Maildir
-rw-------  1 simon simon  467 Apr 19 18:26 mbox
drwxr-xr-x  2 simon simon 4.0K Apr 15 19:25 Music
drwxr-xr-x  2 simon simon 4.0K Apr 15 19:25 Pictures
drwxr-xr-x  2 simon simon 4.0K Apr 15 19:25 Public
drwxr-xr-x  3 simon simon 4.0K Apr 17 17:53 src
drwxr-xr-x  2 simon simon 4.0K Apr 15 19:25 Templates
-rw-r--r--  1 simon simon 1.9K May 20 16:52 uptime.txt
-rw-r--r--  1 simon simon 1.4K May 20 16:52 uptime.txt.asc
drwxr-xr-x 12 simon simon 4.0K May  9 20:00 var
drwxr-xr-x  2 simon simon 4.0K Apr 15 19:25 Videos

Here is the (encrypted) contents of the uptime man page:

simon@mx:~
$ cat uptime.txt.asc 
-----BEGIN PGP MESSAGE-----

jA0ECQMCfoUA3aASbz7/0ukBogSiu3yxpUM+FfoTaCkTBqmUxYagw7ifq8R6WT4o
Za/3rgH7X7UXWuis9q+GKssabXdnZelUlUewnz5P/iHGYBP0C0KhCTHZm8r5abDg
L8E0nsLEYkSu1bcdz07Rdky0S6Pc3uqvFOJLnGBz+TUgWdzHEXtijCxYya1oasyf
izZD+QRGFcsCjh5Ji9BBK92Dza9lRRgvLeZ3PE9IDQjgtHz2qLEDJq82CRMqqzC5
Gv9xfG+sryEgWxKXjfa0XfiC9+9P1hZC4buaKxX26GS697ks2UikZ4EzOi6lpuUC
uGCwi3CdpjnrKnkkPqCixAVKmq6t81MMD1m6fbHnOJSHyRBdk+V7qgPQAq+wDbHM
BnQW4FmX5rVMPBKEezaM/HMJYjBwRjOpeBoAUF4gx8nr9YGTVJygNYgdH8P19mqf
rdxcCFf7OADpZOrxVl1aaaBC5EmEGhUO4zJ47a11m+Tg+0QNddRQOnUZQ6T76glM
Q6TvkxR/wewL3yAIfFqAYACWWyIlMsKOWi4BT8d4MQMauvRC039lQtrNb4E/DfSw
ZD4o4N28f5IrS2Icri0qpqCyK/5bhbSReXpcjgbCF61ckG8H6S6kucAidir7DCJS
gfxbTBuD+FmhN2D+tzerUV0WiMeHMBnvF24ng0zRS0bCG20yj3Kb2n4tsz64FIfj
bcDfPxjwnTNiSZUcmE75ClL9V/4M/NBIu0d1j/okdaTkbLHtsITrOSPenunhvFZp
cMcnDrvfAo+EpT4NIBJ6xrCRq/wRSSW5/cMAp3BF9RpYpmNr1MLQsVx4ax28Yid6
A2r865gPDVVyFBnIvmaFUpmGf9sGg3J2AnjcBpSAXzWnvn7nXXs84At4ZqwgBkhp
FLGWYaPv8MajifGjbjewjMJqYN/Xq8bBw+9x6HCL/AJT4XYeNhYYzHwqUhLFVJeS
PWisaW0bKx2lUYyHEMBW/+KSeTna+UiSEsnbhD+ZBrr9OBnRgPAgNlZwOUZPOVAM
RPvSJdMVkzMVDeHK7tJEUEUMHYpg/3VtVD/EjkHCRrkWStLx4IhJp4UszhH4H3Da
B/yklhNpbejjYOPFT75O/LCBM+cvzdBhO2QucsisyA6WD9fPTzR5clPsndHJSdKW
Ai8lfH5wI0D4Ci4kl2NTCEtIE/StbV4DhZx/VXVu08jSu+xuSribBGimGsAPu/lx
473lFEgjEujaXG2OvUZNBEQwqcBMd8cvkHT+9VyF1M/f8g==
=WzQF
-----END PGP MESSAGE-----

We’ll email it to ourselves now, read the email and save the message in the body to uptime

simon@computer:~$ gpg -d uptime > uptime-manual

Of course, you’ll be prompted for the password simonh and you get the original file back.

Here’s the original, decrypted text

UPTIME(1)                                 User Commands                                 UPTIME(1)

NAME
       uptime - Tell how long the system has been running.

SYNOPSIS
       uptime [options]

DESCRIPTION
       uptime  gives a one line display of the following information.  The current time, how long
       the system has been running, how many users are currently logged on, and the  system  load
       averages for the past 1, 5, and 15 minutes.

       This is the same information contained in the header line displayed by w(1).

       System  load  averages is the average number of processes that are either in a runnable or
       uninterruptable state.  A process in a runnable state is either using the CPU  or  waiting
       to  use  the  CPU.   A process in uninterruptable state is waiting for some I/O access, eg
       waiting for disk.  The averages are taken over the three time  intervals.   Load  averages
       are not normalized for the number of CPUs in a system, so a load average of 1 means a sin‐
       gle CPU system is loaded all the time while on a 4 CPU system it means it was idle 75%  of
       the time.

OPTIONS
       -p, --pretty
              show uptime in pretty format

       -h, --help
              display this help text

       -s, --since
              system up since, in yyyy-mm-dd HH:MM:SS format

       -V, --version
              display version information and exit

FILES
       /var/run/utmp
              information about who is currently logged on

       /proc  process information

AUTHORS
       uptime was written by Larry Greenfield ⟨greenfie@gauss.rutgers.edu⟩ and Michael K. Johnson
       ⟨johnsonm@sunsite.unc.edu⟩

SEE ALSO
       ps(1), top(1), utmp(5), w(1)

REPORTING BUGS
       Please send bug reports to ⟨procps@freelists.org⟩

procps-ng                                 December 2012                                 UPTIME(1)

So, just using this simple method, you can take any file, encrypt it (you can change the cipher by the way), compress it and then send it to someone who knows the key. If you accidentally send the message to someone who wasn’t expecting it, they’ll have little clue what it’s all about.

Awesome!

Using WordPerfect 6.2 in DOSBox (Part 1)

2021-05-17T00:00:00+01:00

Why?

WordPerfect for DOS, in the opinion of many knowledgeable users, is still the greatest program ever written. Some of its features have not been matched even by the latest Windows software, and its interface remains unequaled for efficiency and elegance. Its support for multiple font formats has not been equaled by any other program or operating system, and its support for multiple alphabets, languages, and symbol sets has only been equaled by twenty-first-century versions of Windows, Linux, and the Mac OS.

www.columbia.edu/~em36/wpdos/
www.dosdays.co.uk/topics/Software/wordperfect.php
blog.cmpxchg8b.com/2020/09/finding-console-word-processor.html
www.studymore.org.uk/wp51.htm

My Reasons

I use GNU/Linux and the obvious choices are LibreOffice, Apache OpenOffice and Abiword. They’re all fine and the first two can go toe to toe with MS Office any day of the week.

But sometimes, I think it’s a bit overkill. And LibreOffice isn’t exactly fast to load up. This isn’t the first time I’ve installed WordPerfect, but as I’ve got a bit of time on my hands, I thought I’d dig a bit deeper.

And to my knowledge, Linux never really had a great word processor (correct me if I’m wrong).

Add Colemak Keyboard Layout to Dosbox

I’m awkward so I use an alternative keyboard layout called Colemak. I’ve been using it since 2008 so the first order of business was to get that set up correctly.

Copy this mapper

Save that as colemak-0.74.map in you dosbox settings directory.

simon@computer:~/.dosbox$ tree
.
├── colemak-0.74.map
├── dosbox-0.74-3.conf
└── dosbox-0.74.conf

Open dosbox-0.74-3.conf and find the line beginning with mapperfile

mapperfile=mapper-0.74-3.map

I usually just comment out the original value and add the new one below:

#mapperfile=mapper-0.74-3.map
mapperfile=colemak-0.74.map

Obtain the Software

I searched for “Corel WordPerfect Suite 6.2 for DOS” and the filesize of the .7z file should be about 49.1MB.

Open with wine (5.0) and follow the prompts. I opted to only install WordPerfect, not the full suite.

Here are a few screenshots of the install process. Notice the help tips

When it’s finished installing you’ll need to launch WP.EXE

simon@computer:~/.wine/drive_c/COREL/WP62$ ls
CVDASC.CVX   HRF6.FRS      TIBM.VRS     VMP.COM       WP_OS2.ICO
CVDTKT.CVX   INSTALL.EXE   T_LES1K.TUT  WP350M.ORS    WPSTAND.DR
CVDWP5X.CVX  LIBRARY.STY   T_LES1M.TUT  WP350.ORS     WPUS.ICR
CVDWP60.CVX  MACROS        T_LES2.TUT   WP480M.ORS    WP{WPC}.LCN
CVDWPG1.CVX  MCV.EXE       T_LES3.TUT   WP480.ORS     WP{WPC}.QKL
CVDWPG2.CVX  QFIGENUS.EXE  T_LES4.TUT   WP62.INS      WP{WPC}.SET
CV.EXE       README.ARS    T_LIB.TUT    WP62.PIF      WP_WP_US.BRS
EGA512.FRS   README.CV     T_LTRS.TUT   WP.COM        WP_WP_US.HLM
EGAITAL.FRS  README.FAX    VAPINUL.COM  WP_CV_US.DLG  WP_WP_US.HLP
EGASMC.FRS   README.MCV    VGA512.FRS   WP_CV_US.HLP  WP_WP_US.MRS
EGAUND.FRS   README.VRS    VGAITAL.FRS  WP.EXE        WP_WP_US.QRS
EGA.VRS      README.WP     VGASMC.FRS   WP.FIL        WP_WP_US.TRS
GRAPHICS     STANDARD.IRS  VGAUND.FRS   WPICONS.DLL   WP_WP_US.TUT
HRF12.FRS    TEMPLATE      VGA.VRS      WP.LRS

Which will give you the following beautiful screen

The first thing you’ll want to do is choose Help > Tutorial from the top menu.

And spend about twenty minutes going through them. It’s worth your time, I assure you.

By the way, your mouse will be trapped in the dosbox window. To free it hit CTRL-F10 (took me a while to discover that)!

WordPerfect Tips

www.microcounsel.com/WPTIPS.HTM
www.youtube.com/watch?v=ML_GoEUhs4A

And the first link at top of this page, has lots of great resources for getting help, manuals etc.

Interesting UNIX, BSD, Linux programs and Packages

2021-04-19T00:00:00+01:00

Mail

Mailutils

Mailutils is a swiss army knife of electronic mail handling. It offers a rich set of utilities and daemons for processing e-mail.

mblaze

The mblaze message system is a set of Unix utilities for processing and intseracting with mail messages which are stored in maildir folders.

fdm

fdm is a program designed to fetch mail from POP3 or IMAP servers, or receive local mail from stdin, and deliver it in various ways.

getmail6

Welcome to getmail 6, a mail retriever with support for POP3, POP3-over-SSL, IMAP4, IMAP4-over-SSL, and SDPS mail accounts. It supports normal single-user mail accounts and multidrop (domain) mailboxes. getmail is written in Python, and licensed under the GNU General Public License version 2.

getmail

getmail is a mail retriever designed to allow you to get your mail from one or more mail accounts on various mail servers to your local machine for reading with a minimum of fuss. getmail is designed to be secure, flexible, reliable, and easy-to-use. getmail is designed to replace other mail retrievers such as fetchmail.

public-inbox

public-inbox implements the sharing of an email inbox via git to
complement or replace traditional mailing lists. Readers may
read via NNTP, IMAP, Atom feeds or HTML archives.

dbmail

Fast and scalable sql based email services
Storage of mail messages in a relational database
Frontends: IMAP4, POP3, ManageSieve, LMTP
Backends: MySQL, PostgreSQL, Oracle, and SQLite

Mailfromd

Mailfromd is a general-purpose mail filtering daemon for Sendmail, Postfix and MeTA1. It is able to filter both incoming and outgoing messages using criteria of arbitrary complexity, supplied by the administrator in the form of a script file. The daemon interfaces with the MTA using Milter or PMilter protocols.

QSF

Quick Spam Filter (QSF) is an Open Source email classification filter, designed to be small, fast, and accurate, which works to classify incoming email as either spam or non-spam. To recognise spam, QSF strips the text out of the email (using MIME decoding and HTML stripping) and then splits it into tokens (words, word pairs, URLs, and so on). These tokens are then looked up in a database and analysed using the Bayesian technique to see whether the email should be classified as spam or not.

Mercurial for the Lone Coder

2020-09-08T00:00:00+01:00

Background

I’ve been a (productive) programmer for about three years now.
My workflow has been the following:

set up a folder for the project
start working
create an archive of the folder with the current date each day and save that in multiple locations for backup
rename the folder to todays date
Back to step 2

I was aware pretty much from day one that this wasn’t the best setup, but it worked and I didn’t have to worry about any tool clobbering my work and leaving me with lost files

I remember that happening at the time, as Zed Shaw went from a massive fan of fossil, to, well, not a massive fan.

Over the years, I’ve tried Fossil, Git and Mercurial (and probably others).

I always came to the conclusion that I’d need to put in too much time to learn their intricacies to bother with them.

Anyway, last month I decided to revisit SCM again. As I’d been using SQLite a lot, I thought Fossil would be ideal as it’s used as the SCM for SQLite itself.

My Thoughts about Fossil

Fossil uses an sqlite database as the repository. One of it’s best features is that it’s an SCM, a wiki, a bug tracker, a forum, a website, and probably other stuff too.

What annoyed me very quickly though is the workflow:

Create a directory ~/fossil where you store your repositories
From that directory run fossil new project.fossil /path/to/where/actual/code/is
Then run fossil open /path/to/where/actual/code/is
Get a warning from fossil that the directory is not empty and must use --force to override

I mean, come on! So, I ditched the idea of using fossil and thought I’d see what Mercurial (hg) was like nowadays.

Here is the hg way:

Run hg init in the directory you want to track
Run hg add . to add everything
Make some changes, run hg commit -m 'I changed some things'
Run hg serve and have a web interface

I recommend using hg c 'Changed something else'

by adding the following to your ~/.hgrc

[alias]
c = commit -m

TO BE CONTINUED…

My New Favourite Text Editor

2016-01-19T00:00:00+00:00

Note: This post is from an old Pelican blog I used to have at simonh.me, which I found on a backup drive on 2025-01-07. It's probably of no use, but reposted for posterity!

Over the years, I've tried lots of text editors. Vim, EMACS, Sublime Text, Jedit, Nano, Notepad++ (and more). As I use Linux at home and Windows at work, I've been looking for a while, for one that I can use on either OS.

As much as I love Linux (and I have no desire to switch back to Windows after about six years), there is some software written for Windows that I think is fantastic. Foobar2000, imgburn (when I used to burn discs), Adobe Photoshop and namexif spring to mind.

So, a few weeks ago I set about (again) finding a text editor that I could use just as well at work as I could at home. I had another go with Notepad++. It seemed to be trying to do a bit too much. The interface has too much going on. I needed something simpler. Plus there were some oddities running on Wine. I somehow stumbled upon Notepad2-mod https://xhmikosr.github.io/notepad2-mod/.

A text editor that wanted to be slightly better than Notepad??? This doesn't sound good, I thought. But when I tried it, it seemed to be just what I was looking for. Syntax highlighting for Python and indentation guides are all I really need at the moment. Plus, it works great on Wine.

It does very little for you, except what you need. I class that as a feature. I have the utmost respect for editors like Vim, but that it is pretty much unusable "out of the box", without hunting around on the net for tips on how to set up your .vimrc seems strange to me.

If you haven't tried it, and are wanting to do some coding, rather than learning the intricacies of a particular text editor, I urge you to give it a go.

simonh.uk - software

Backing Up Retro Game Library using Rsync

What about files I’ve deleted?

What about file / directory renames?

Conclusion

Using Emacs as a Word Processor

Problems with Wordperfect 8

JOE Editor (Jstar)

What Should a Word Processor Do?

RFC 1896 The text/enriched MIME Content-type

Using Emacs Enriched Mode

Plain Text Benefits

Conclusion

Footnotes

Microsoft Edit vs GNU Nano

Conclusion

See Also

Footnotes

Why I use GNU Nano as my Default Editor

The Backstory

My Editor Requirements

Vital

Nice to Have’s

Where I Looked Closer at Nano

What I Needed / Wanted

My Tweaks

My .nanorc

Why Not Just Use…

Conclusion

Footnotes

Trying Out OpenBSD Game of Trees (Got)

Background & Motivations

Installation

The Got Way

Creating the Repo and importing code

Creating & Listing Tags

Branching & Merging

Conclusion

See Also

First Look at OpenBSD Game of Trees (Got)

Git, but simpler (to use, maybe…)

Features I’d love to see

Conclusion

Using GNU RCS for Managing Configuration Files

The Problem

The Solution

The New Workflow

Running bottle.py with systemd

Using GNU RCS in 2023

Old School Version Control

Manage Invoices with GNU Recutils

Conclusion

A Guide to Choosing, Starting and Finishing Software Projects

Intro

Step One

Step Two

Step Three

Step Four

Summary and Conclusion

Send SMS from Linux Computer via ADB

On the Pros and Cons of Software Testing

My Workflow

Example

Conclusion

Why I still Use Windows XP

Why?

Tips

Getting Started with GnuPG (gpg)

Using WordPerfect 6.2 in DOSBox (Part 1)

Why?

My Reasons

Add Colemak Keyboard Layout to Dosbox

Obtain the Software

WordPerfect Tips

Interesting UNIX, BSD, Linux programs and Packages

Mail

Mercurial for the Lone Coder

Background

My Thoughts about Fossil

My New Favourite Text Editor