simonh.uk - web

Topic Time Vampires

2025-02-07T00:00:00+00:00

There is a problem that web surfers have always had to deal with but has become even more pervasive. The topic time vampires I call them. TTV’s for short. I’ve recently discovered that the topic of photography has enticed many of them away from whatever they were wasting peoples time with previously.

What is a TTV?

They are Youtubers / bloggers who are trying to build or grow their niche. They target keywords / phrases that web surfers are typing (or speaking) in to search engines and hope we will land on their site or channel. Their aim is not to give useful or original information, it’s to convince you they’re an expert. Very rarely will you encounter an expert. Most of them are imposters. Imitators. They just copy / rehash each others content. Or worse, steal other peoples content and pass it off as there own.

A History Lesson

TTV’s were rampant back in the mid to late 2000’s especially in the “make money online” niche (as they called it). Man, I wasted hours, going round in circles. Everyone pretended they were an expert. Everyone had a link to an affiliate program that would teach you “all the secrets”. They would do whatever they could to waste as much of your time on purpose. Really. You see, back then (not sure about today), the search engines rewarded pages that kept a visitor reading. It was in the TTV’s interest to waffle on for screenfuls of content, while actually writing nothing of any value. All that was required was to pepper the article with the keywords and boom, they’d climb a bit higher towards the coveted number one spot.

Enter the “Influencers”

Since the web has shifted from static pages of text based results to multimedia content, chasing the top spot on any particular search engine is no longer enough. The TTV’s potentially lucrative audience could be one of the many intranets: Instagram, Facebook etc. Or, the searcher might prefer to watch a video instead of reading an article. So now the TTV’s must have a presence on every platform. Or at least all the big ones.

How to Detect a TTV?

If it’s a website, popups (of any kind)¹ should usually be enough to encourage usage of your browsers back button. Youtubers who have sponsored videos and pester you to “click that subscribe button” should set off alarm bells. If in doubt, watch or read a couple of their offerings and ask yourself, “did I get what the title suggested?”

They’re Just Trying to Make a Living!

I know some people will object to my characterising all these people as TTV’s, but as the old saying goes “if it walks like a duck…”. I mean seriously, how many Youtubers do we need to be making photography vids?

Find the Source(s)

Searching for “take better photos” on Youtube returns the following result on the first “page”:

5 Ways To Take Better Photos… (2022)

The source:

Photo Tip: Five for Five (2011)

This of course may not be the source, but I’m sure the Youtuber discovered the concept from elsewhere. Shame he didn’t bother to mention it in his video. It took me 10 seconds to find the above article, by the way.

I’m sure many, many more easy examples could be found if needed.

There aren’t any Shortcuts

The bottom line, and the point of this post, is that there aren’t any shortcuts to learning / mastering / becoming something. The most efficient way to get what you want may be to be invest in a book, or go on a course that will cover the fundamentals. It may be tempting to do a search on “become a better photographer” or “take better photos” but be aware, those phrases will lead you to TTV’s. Every time. Your time is their paycheck.

Footnotes

¹ With the exception of those highly annoying cookie consent dialogues.

Interesting Websites & Blogs January 2025

2025-01-13T00:00:00+00:00

Intro

This is a series I’ve been planning for a few months (but not got round to until now). My aim is to give a shout out to other websites and blogs that I think you might find interesting. I’ve deliberately avoided including any professional bloggers, influencers, those types.

John Matthews

Writes on a wide range of topics — programming, photography, money, current affairs. Something for everyone here. Very nicely designed site, clean and minimal(ish).

The photo at the top of the page is Dog Walker from John’s site.

Syndicate Unofficial Site

I loved the game Syndicate Wars on the Playstation. Actually completed it too! Sites like this one used to all over the net in the 90’s / 00’s. Not so much now. Have a look at how much work went it this.

The Raider.Net

Indiana Jones. Nothing else! Looks like this hasn’t been updated since about 2008 but it’s still got an active messageboard / forum. I’m no superfan (though I’ve seen all the films apart from Dial of Destiny), but there’s an enormous amount of content here that you can only marvel at.

Contains Moderate Peril

TV, music, movies, gaming. A fine example of a guy with obviously wide ranging interests publishing whatever takes his fancy.

Alvan Rahimli

Found this site from the site above. Specifically, the blog questions challenge page. Yes, I too will be completing it. Anyway, I digress…

Alvan’s site is very well laid out and clean. His blog just has whatever he feels like writing about (same here). A good example of a personal blog that just exists because he can.

Herman Martinus

Creator of the bear blog platform (which I just found). Up until today I’ve advised people to just use Blogger if they want to quickly get a blog up and running. Hmmm. Maybe I was wrong.

Afterword

If you know of a site I should include next time, please email me below! If I included your site, but you want it removed, also send me an email.

The Little to Big GoAccess Tutorial

2025-01-02T00:00:00+00:00

README

Hello internet surfers! The goal of this post is to set up GoAccess for server side reporting of our website visitors (and other stuff eventually). Here’s the program description from their website:

GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser. It provides fast and valuable http statistics for system administrators that require a visual server report on the fly.

The plan is to start with the vanilla (default) configuration and slowly tweak settings to end up with a lean and accurate reporting system for our site. I’ll publish new parts as time allows. My server information is:

Debian 12.8
Apache 2.4.62
GoAccess 1.7 (from Debian repository)

Why Are We Doing This?

Client-side javascript analytics scripts have a big drawback. Look at the screenshot below:

See the problem? Simple Analytics, which I’ve been trying out, is massively under reporting my visitors. That might have something to do with over 900 million people (including me) are using ad blockers! Luckily we have access to our server logs, and that’s going to give us far more accurate statistics. Let’s dive in!

We’ll start by dealing with crawlers.

Part 1 – Crawlers

simon@server:~ [ssh] $ cd /etc/goaccess/

simon@server:/etc/goaccess [ssh] $ ll
total 48K
drwxr-xr-x   2 root root 4.0K Aug 20  2023 .
drwxr-xr-x 119 root root  12K Jan  2 16:06 ..
-rw-r--r--   1 root root 3.3K Mar  1  2021 browsers.list
-rw-r--r--   1 root root  21K Jan  4  2023 goaccess.conf
-rw-r--r--   1 root root 1.7K Mar  1  2021 podcast.list

Let’s become root, so we don’t have to bother typing sudo all the time

simon@server:/etc/goaccess [ssh] $ sudo -s
[sudo] password for simon:
root@server:/etc/goaccess#

Let’s find out how many lines are in goaccess.conf as generally the more lines, the more options we can fiddle with

root@server:/etc/goaccess# cat goaccess.conf | wc -l
757

757 lines, so quite a few. As always, I’ll keep track of changes I make to the file with GNU RCS. I’ve written a previous post about using RCS for managing configuration files, available here

root@server:/etc/goaccess# mkdir RCS
root@server:/etc/goaccess# ci -l goaccess.conf
RCS/goaccess.conf,v  <--  goaccess.conf
enter description, terminated with single '.' or end of file:
NOTE: This is NOT the log message!
>> Initial version
>> .
initial revision: 1.1
done
root@server:/etc/goaccess#

Right, now we’ve safely got the unedited file checked in, let’s run goaccess with the default configuration. All we’re interested in today is how many visitors this site has had today. Running the command below will output a few screenfuls of information. I want section 9 — Virtual Hosts¹

root@server:/etc/goaccess# goaccess /var/log/apache2/other_vhosts_access.log -c

   9 - Virtual Hosts

 Hits     h% Vis.     v% Tx. Amount Data
 ---- ------ ---- ------ ---------- ----
  847 46.93%  172 69.92%  13.55 MiB www.simonh.uk

So, we’re told I’ve had 172 visitors today. Good luck getting more than that, Google!² But, before I officially announce this as the most popular site on the whole internet, I better double check my numbers…

Open /etc/goaccess/goaccess.conf with your editor of choice, and find the following section:

# Ignore crawlers from being counted.
# This will ignore robots listed under browsers.c
# Note that it will count them towards the total
# number of requests, but excluded from any of the panels.
#
ignore-crawlers true

# Parse and display crawlers only.
# This will ignore all hosts except robots listed under browsers.c
# Note that it will count them towards the total
# number of requests, but excluded from any of the panels.
#
crawlers-only false

# Unknown browsers and OS are considered as crawlers
#
unknowns-as-crawlers true

I’m going to first change ignore-crawlers from false to true, and then checkin the change.

root@server:/etc/goaccess# ci -l goaccess.conf
RCS/goaccess.conf,v  <--  goaccess.conf
new revision: 1.2; previous revision: 1.1
enter log message, terminated with single '.' or end of file:
>> Ignore-crawlers = true
>> .
done

Now we’ll run goaccess again and see how that change has affected our visitors

root@server:/etc/goaccess# goaccess /var/log/apache2/other_vhosts_access.log -c

   9 - Virtual Hosts

 Hits     h% Vis.     v% Tx. Amount Data
 ---- ------ ---- ------ ---------- ----
  636 41.79%   90 69.23%   7.37 MiB www.simonh.uk

So by just ignoring crawlers, my visitors have been nearly halved. I don’t know whether to be happy or sad about that. But lets keep going. The next obvious thing to do is to set that visitors that are using unknown browsers and OS’s are also probably crawlers (makes sense). So let’s throw them away as well! And checkin our change

root@server:/etc/goaccess# ci -l goaccess.conf
RCS/goaccess.conf,v  <--  goaccess.conf
new revision: 1.3; previous revision: 1.2
enter log message, terminated with single '.' or end of file:
>> unknowns-as-crawlers = true
>> .
done

Let’s run goaccess yet again to see if I’ve lost any more visitors

  9 - Virtual Hosts

 Hits     h% Vis.     v% Tx. Amount Data
 ---- ------ ---- ------ ---------- ----
  550 78.57%   58 68.24%   6.36 MiB www.simonh.uk

So by changing just those two options from false to true, I’ve wiped out another third of my visitors! You’ll have to bear with me while I ascend a mountain to contemplate whether ignorance is bliss, or, the truth shall set you free.

See you in the next part…

Part 2 – Emailing the Report

Turns out there are no mountains near me (plus it’s cold) so I’ll continue. I wrote a post back in 2021 about getting a daily email of visitors using goaccess, which I’ll slightly modify here.

Here is the script we’ll use which we’ll call apache_report:

#!/bin/bash
now=`date +%Y-%m-%d_%H%M%S`
goaccess /var/log/apache2/other_vhosts_access.log --log-format=VCOMBINED -o report_$now.html
gzip report_$now.html
echo "Report" | mail -s "Apache Report - $now" -A report_$now.html.gz <you@youremail.address>

Remember to change you@youremail.address to your email address!

If you haven’t got round to setting up a mail server yet, you can follow my tutorial for OpenSMTPD, or find one elsewhere on the net.

First thing to do is save apache_report on the server and check it works.

simon@server:~/tmp [ssh] $ nano apache_report

Copy the script above and paste it into your editor. With that saved, let’s run it.

simon@server:~/tmp [ssh] $ sudo bash apache_report 
 [PARSING /var/log/apache2/other_vhosts_access.log] {256} @ {0/s}
Cleaning up resources...
simon@server:~/tmp [ssh] $

No errors. Good. Let’s check our email:

I’ve sent a few to myself, as you can see. Below is how the email looks on mobile. I’ve formatted it like this to make it easier to view here on this page:

Awesome! The final thing to do in part 2, is set up a cron job so we’ll get a daily email with our stats.

simon@server:~/tmp [ssh] $ sudo cp apache_report /etc/cron.daily/
[sudo] password for simon: 
simon@server:~/tmp [ssh] $ ls /etc/cron.daily/
00logwatch     apt-compat                dpkg       mod-pagespeed
apache2        aptitude                  logrotate  ntp
apache_report  bsdmainutils.dpkg-remove  man-db     popularity-contest

Make our file executable

simon@server:~/tmp [ssh] $ sudo chmod+x /etc/cron.daily/apache_report

One thing to warn you about: each email is currently 148kB in size (gzipped). Extracted they are coming out at about 450KB. I know we aren’t in 1980 anymore, but make sure you’ve got plenty of storage with your mail provider!

The next day @ 06:25:55 my report arrives in my inbox (table below is the vhosts section):

Part 3 – ?

Footnotes

¹ Note to self: trim all that trailing whitespace with sed -i 's/[[:space:]]*$//' <file>

² I’m informed that Google shares numbers in the millions or billions. Maybe trillions. Still, 172 should put me in the top 10 I expect.

Updates

2024-01-04: Add why are we doing this? section to part 1

2024-01-03: Add Part 2

It's Time to Get Back on The Internet

2024-12-31T00:00:00+00:00

An Intranet is not The Internet

Facebook, Twitter/X, Linkedin, Tiktok et al. are all intranets.

From Wikipedia

An intranet is a computer network for sharing information, easier communication, collaboration tools, operational systems, and other computing services within an organization, usually to the exclusion of access by outsiders. The term is used in contrast to public networks, such as the Internet, but uses the same technology based on the Internet protocol suite.

Pay particular attention to easier communication as all these intranets are predominantly concerned with their users interacting with each other. Invite your friends! They’ll love it here.

No, I’m Not On […]

Quite often through my work, I’ll be asked “can you whatsapp that photo to to me?” I reply that I cannot, as I don’t use it. I never have used it and I never will. Why would I a messaging intranet? I have loads of email addresses, and a phone that can make and receive calls and messages.

“I’ll share it with you on Facebook!” I’m not on there. Just email it mate.

I think they sometimes think I’m a bit odd (I am) for not having accounts that they presume everyone has. But here’s the thing…

It’s Not Cool Anymore

When these intranets started popping up, it was new and cool. “Are you on Friends Reunited?” Of course I am, we would reply back in about 2000.

I’m just going on my recollections but as I remember it, you would find people you hadn’t seen in donkeys, send them a message and then if you wanted to, you’d exchange emails and the site had fufilled it’s purpose.

It was more of a directory of people than anything else. And that was fine. Better, actually.

But then, other companies wanted you on their intranet all the time. Forget about that old internet thing. We’ve got everything you’ll ever need right here. You like games? Check. Got a group that likes to meet up and talk about trains, planes or automobiles? Bring them here! None of you will ever need to stand up again. It’ll be awesome. Like shopping? We’ve got loads of ads that’ll catch your eye. And they’ll deliver it to you as well.

The Kids Get Angry

One day, the young will rediscover what the internet was once like and how it slowly got worse. How the adults wrung their hands about the rampant grooming and exploitation of children on various hugely profitable intranets (that were delighted to have those children on their “platforms”). And they’ll be angry. Quite rightly. They’ll (re)invent their own internet.

Unless the adults lead by example and ditch these awful places. But, they won’t.

Afterword

Tnanks to these dreadful intranets, here in the UK we now have the Online Safety Act 2023 (303 pages)!

From the NSPCC (UK charity):

The Act means that tech companies running social networking sites or search engines must promote online safety by tackling illegal material and content that is harmful to children, conducting regular risk assessments, and properly enforcing age limits.

Sounds good. Except that the legislation has attracted considerable from criticism from, for example, Article 19, the FSF and many other sites.

Why I Prefer Textile Over Markdown

2024-12-15T00:00:00+00:00

Plain Text > Textile Parser > HTML

I write a lot of HTML by hand. Have done for years. But when I started this blog, I knew I’d stay a lot saner using a tool such as Markdown, Textile or reStructuredText to generate the final markup for me. I know most people use Markdown but as a contrarian, I chose Textile. I’ve found that Textile gets the balance between ease-of-use and power just right.

Let’s dive right in and have a look at how you write textile documents.

The Basics

Headings

Headings are simply h1. to h4. followed by your heading (all on the same line):

h1. This is my heading

will produce

This is my heading

h4. This is my tiny heading

This is my tiny heading

Paragraphs

Paragraphs are blocks of plain text separated by two newlines. You just type as you would in a text editor. The Textile parser will wrap your paragraphs in <p> tags.

Text formatting

Underscores are used to _emphasise_ text. Bold text needs an *asterisk* on either side of what you want to be bold. If you want emphasised, bold text, just use both _*together*_:

really important you understand this. Easy.

Lists

Unordered (bullet) lists use asterisks as well. If you need sublists, add asterisks for each sub-level:

Here is my shopping list
* Milk
* Bread
* Ingredients for cake:
** Eggs
** Chocolate
** Sugar

Personally, I think that is very easy to remember and use. This will output:

Here is my shopping list

Milk
Bread
Ingredients for cake:
- Eggs
- Chocolate
- Sugar

Ordered (numbered) lists use the hash character # for each list item:

How to make a cuppa
# Get a cup
## Place teabag and sugar (if desired) into cup
# Boil a kettle
# When boiled add water to cup
# etc.

Which renders to:

How to make a cuppa

Get a cup
1. Place teabag and sugar (if desired) into cup
Boil a kettle
When boiled add water to cup
etc.

The Textile parser will keep track of numbering for you.

Links

here is a link to my "website":https://simonh.uk

here is a link to my website

Link text in double quotes, immediately followed by a colon, then the URL

Additional Textile Features

This is where Textile beats the competition (in my opinion).

p>. This text is right aligned.
p=. This text is centred.
p<. Back to left aligned (the default).

Which produces:

This text is right aligned.

This text is centred.

Back to left aligned (the default).

Tables

Example below copied from the Textile website.

| A | simple | table | row |
| And | another | table | row |
| With an | | empty | cell |

| A | simple | table | row |
| And | another | table | row |
| With an | | empty | cell |

There’s a lot more you can do with tables. Consult the Textile documentation for more info.

Footnotes

This line of text has a footnote[1]

This line of text has a footnote¹

At the bottom of your Textile document, you simply write

fn1. This is the footnote text.

Increment the footnote for each one you created in the document

fn2. Here's the second one.

Easily Reference CSS Classes and Styles

I love this one and use it throughout my blog and in other projects. Textile allows you to do this:

p(some-class). Here is my text

which will apply whatever you’ve defined for some-class. Below is what I use to put icons before headings and paragraphs (in paper.css):

.info::before {
  content: url('/img/info.svg');
  width: 18px;
  vertical-align: middle;
  margin-right: 10px;
  display: inline-block;
}

By simply putting this into my textile template:

p(info). This paragraph now has an icon placed on the left. Useful for letting a reader know something!

that will be rendered to:

This paragraph now has an icon placed on the left. Useful for letting a reader know something!

Textile also helpfully lets you apply inline styles.

Do you like the color %{color:yellow;}yellow%?

produces this:

Do you like the color yellow?

I don’t use the inline styling very much at all. But if I / you need to, Textile has you covered.

Conclusion

Dean Allen, the guy who created Textile for a CMS he was writing explained why he created the format (from Wikipedia):

Textile was developed by Dean Allen in 2002, which he billed as “a humane web text generator” that enabled you to “simply write”.

I think he did a great job.

Footnotes

¹ I told you there was a footnote down here!

Updates

2025-01-08: Add Easily Reference CSS Classes and Styles section.

Why CGI Programming is Still Useful in 2023

2023-08-19T00:00:00+01:00

What is CGI?

Common Gateway Interface has been around since the mid 1990’s. It was the one of the first standards to specify how a server and a program could serve requests. From the RFC:

The Common Gateway Interface (CGI) allows an HTTP server and a CGI script to share responsibility for responding to client requests.

CGI allows any script or program to be called to serve dynamic content on a website. Needless to say, this was very popular and helped the web to explode from the 90’s onwards (pretty much everything was written using CGI). One of the amazing things about CGI is that you can use any programming language. C, Perl, Python, Ruby, even Bash. As long as the language has stdin and stdout, it will work. Here is a Bash script showing some system information:

#!/bin/bash

echo "Content-type: text/html"
echo ""

echo '<html>'
echo '<head>'
echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">'
echo '<title>CGI Bash</title>'
echo '</head>'
echo '<body>'
echo '<p>Demo of using GNU Bash with CGI</p>'

echo '<h2>Hostname</h2>'
echo '<pre>'$(hostname)'</pre>'

echo '<h2>Bash Version</h2>'
echo '<pre>Using '$(bash --version | head -1)'</pre>'

echo '<h2>System Uptime</h2>'
echo '<pre>'$(uptime)'</pre>'

echo '<h2>Current Date</h2>'
echo '<pre>'$(date)'</pre>'

echo '</body>'
echo '</html>'

exit 0

And the output of above script is:

Pretty awesome I think. Especially for a few lines of bash. And I’m sure you’ve noticed that the output could be anything. The sky is the limit.

Disadvantages of CGI

A lot of people like to hate CGI. Here are some of the common complaints they will shriek if someone asks a question about using CGI in 2023

A new process is started for each request which is horribly inefficient!

Yes it is, and for a helluva lot of websites / web services, that’s not a problem. If is a problem, FastCGI might help you out. Or, switching to a compiled language like C or Go.

It’s not suitable for creating a full blown site or service!

Maybe not. But what if you just want a simple contact form? Or a way of quickly uploading a file to a server (see below).

It’s been superseded by better alternatives and is now obsolete!

This seems to be the main objection that I see. Perl has dropped CGI.pm from the standard distribution. Python is removing CGI support in version 3.13.

I think this in the same mindset that says that people should never use GNU RCS now that Git has is a thing. But as I wrote in another post on this site, RCS is perfect for version controlling single files. Like a CGI script for example! Yep, I use RCS for my CGI scripts.

Advantages of CGI

You don’t have to choose between a static or a dynamic site

I have quite a few static sites and quite a few dynamic ones. Usually, I have to decide up front which type I want to make as I’ll have to use a language that works on the web if I want to generate pages dynamically. The beauty of CGI is that you can have almost everything served statically, and just a few pages served dynamically. As usual with CGI, I don’t know of another technology that offers this flexibility.

Use whatever language you want
Set up is minimal

This is what I use with Apache:

# Enable CGI scripts in cgi-bin directory
ScriptAlias "/cgi-bin/" "/var/www/my-website/cgi-bin/"
<Directory /var/www/my-website/cgi-bin/*>
    Options ExecCGI
	SetHandler cgi-script
	DirectoryIndex
</Directory>

Done.

You can get a service running in minutes

Just have a cgi-bin directory properly configured and dump your scripts in there. No “frameworks”. No config files. Just your CGI scripts. And the HTML files calling those scripts. It makes even the most minimal of minimal frameworks look complicated.

It’s the right tool for some jobs

Most of the web software I write is made with bottle.py. But sometimes, even bottle is overkill. A recent example: my son wanted to send me some files. Even though I have accounts for Google Drive, Onedrive, Dropbox etc I thought it was crazy that even though I have multiple web servers, I had no easy way for someone to quickly upload a file to one of them. Well, thanks to CGI, I now have a (very) simple form that lets someone upload a file (10MB maximum) and then sends me an email notifying me. And it was done in about 10 minutes. To set up an equivalent service with Bottle would have probably taken hours.

Security Considerations

As with any service running on the internet, you will need to read some documentation to be aware of security risks. This is not just true for CGI, but for anything. The Apache website linked below has some tips. A search engine will no doubt give you more tips.

Conclusion

There is a reason that in 2023, CGI is still in use and refuses to die:

1. It’s relatively simple to learn and use
2. You can use whichever programming language you want
3. It’s an elegant protocol that solves a real problem
4. With CGI you can get something up and running quicker than any other technology I’m aware of.
5. You can mix static and dynamic content as you please

So please, ignore the CGI-MUST-DIE-TODAY crowd and at least keep it in mind as a solution to some of your programming problems. Like many technologies from the “early web” (yes. I’m looking at you WebDAV) CGI solved a problem pretty much as soon as it was discovered and was used for many years. Some of us still use it happily today and quietly laugh to ourselves at the “guru’s” having nervous breakdowns at the idea that such an old standard could still be used in 2023.

References and Recommended Reading

Apache CGI
The Magic of cgi-bin (Video)
Why I recommend CGI instead of web frameworks
Wikipedia FastCGI

A Guide to Setting up WebDAV with Apache HTTPD

2021-12-25T00:00:00+00:00

Credits: Digital Ocean for the WebDAV tutorial. hexeract for his post about multiple WebDAV users. Raymond Camden for the responsive directory index.

For a project I’ve been working on, I needed some place for users to upload and access files. I started off using Google Drive, Sharepoint etc, but I also wanted to have the option of hosting the files myself.

So, I looked at all sorts of options (Pydio, Seafile, Owncloud, FTP, WebDAV). I decided to have a stab at WebDAV, mainly due to it being included with Apache through mod_dav. Initially, I was using file based storage, but the Apache docs advise to use a dbm database instead due to being far faster. However, the same docs weren’t very helpful when it came to using dbm for group management / authorisation.

The following is what I did to get everything working. If you want to do something similar yourself, hopefully it will help. I’m using Debian 11, Apache 2.4.51.

First, create a directory for the credentials to live

# mkdir -p /usr/local/apache2/var

cd into the new directory and then create the dbm file that Apache will use for auth. I’ll add myself and also what groups I belong to. After running command below, you’ll be prompted for a password. I use pwgen for generating passwords.

# htdbm -ct users.dbm me@mydomain.com "project1,project2"

The -c is needed to create the database. -t is used to add a comment which Apache uses for the group credentials (it would have been nice if the Apache docs stated that!). The most important things to remember are: no spaces in group names, and separate each group with commas. Just stick with alphanumeric characters or an underscore, and you should be golden.

Next, we’ll add a second user who we only want to access project1

# htdbm -t users.dbm someone@somewhere.com "project1,"

The next user can only access project2

# htdbm -t users.dbm user@somewhere.com "project2,"

If you need to delete a user you can do that like so:

# htdbm -x users.dbm simon

I think you can see why setting up access control in this way is such a good idea. You’ve only got one file to manage. Adding, modifying or deleting users is just one line using htdbm. And the lookups are super fast. I’ve been using dbm for session management for years. You might want to install db-util as well for managing the database.

List database members:

htdbm -l users.dbm

Apache setup

You’ll need to enable the required modules.

# a2enmod dav
# a2enmod dav_fs
# a2enmod authn_dbm 
# a2enmod authz_dbm

The following is what I’m using for my apache.conf

DavLockDB /usr/local/apache2/var/DavLock
<IfModule mod_ssl.c>
<VirtualHost *:443>
    DocumentRoot "/var/www/files.yoursite.com"
    ServerName files.yoursite.com

    SSLCertificateFile /etc/letsencrypt/live/files.yoursite.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/files.yoursite.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

    # Directory Listing Display Options
    IndexOptions +FancyIndexing
    IndexOptions +VersionSort
    IndexOptions +HTMLTable
    IndexOptions +FoldersFirst
    IndexOptions +IconsAreLinks
    IndexOptions +IgnoreCase
    IndexOptions +XHTML
    IndexOptions +NameWidth=*
    IndexOptions +SuppressDescription
    IndexOptions +SuppressHTMLPreamble
    IndexOptions +Charset=UTF-8
    # Make it responsive
    IndexHeadInsert "<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">"

    # project1 
    Alias /project1 /var/www/webdav/project1
    <Directory /var/www/webdav/project1>
        DAV                On
        SSLRequireSSL
        AuthType           Basic
        AuthName           "webdav"
        AuthBasicProvider  dbm
        AuthDBMUserFile    "/usr/local/apache2/var/users.dbm"
        AuthDBMGroupFile   "/usr/local/apache2/var/users.dbm"
        Require dbm-group  project1
    </Directory>

    # project2
    Alias /project2 /var/www/webdav/project2
    <Directory /var/www/webdav/project2>
        DAV                On
        SSLRequireSSL
        AuthType           Basic
        AuthName           "webdav"
        AuthBasicProvider  dbm    	
        AuthDBMUserFile    "/usr/local/apache2/var/users.dbm"
        AuthDBMGroupFile   "/usr/local/apache2/var/users.dbm"
        Require dbm-group  project2
    </Directory>

</VirtualHost>
</IfModule>

As always check your config is correct

# apachectl -t
Syntax OK

Restart Apache

# systemctl restart apache2

Each time you have a new project, you can just copy the Alias and directory block, at the bottom like so.

DavLockDB /usr/local/apache2/var/DavLock
<IfModule mod_ssl.c>
<VirtualHost *:443>
    DocumentRoot "/var/www/files.yoursite.com"
    ServerName files.yoursite.com

    SSLCertificateFile /etc/letsencrypt/live/files.yoursite.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/files.yoursite.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

    # Directory Listing Display Options
    IndexOptions +FancyIndexing
    IndexOptions +VersionSort
    IndexOptions +HTMLTable
    IndexOptions +FoldersFirst
    IndexOptions +IconsAreLinks
    IndexOptions +IgnoreCase
    IndexOptions +XHTML
    IndexOptions +NameWidth=*
    IndexOptions +SuppressDescription
    IndexOptions +SuppressHTMLPreamble
    IndexOptions +Charset=UTF-8
    # Make it responsive
    IndexHeadInsert "<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">"

    # project1 
    Alias /project1 /var/www/webdav/project1
    <Directory /var/www/webdav/project1>
        DAV                On
        SSLRequireSSL
        AuthType           Basic
        AuthName           "webdav"
        AuthBasicProvider  dbm
        AuthDBMUserFile    "/usr/local/apache2/var/users.dbm"
        AuthDBMGroupFile   "/usr/local/apache2/var/users.dbm"
        Require dbm-group  project1
    </Directory>

    # project2
    Alias /project2 /var/www/webdav/project2
    <Directory /var/www/webdav/project2>
        DAV                On
        SSLRequireSSL
        AuthType           Basic
        AuthName           "webdav"
        AuthBasicProvider  dbm    	
        AuthDBMUserFile    "/usr/local/apache2/var/users.dbm"
        AuthDBMGroupFile   "/usr/local/apache2/var/users.dbm"
        Require dbm-group  project2
    </Directory>

    # project3
    Alias /project3 /var/www/webdav/project3
    <Directory /var/www/webdav/project3>
        DAV                On
        SSLRequireSSL
        AuthType           Basic
        AuthName           "webdav"
        AuthBasicProvider  dbm    	
        AuthDBMUserFile    "/usr/local/apache2/var/users.dbm"
        AuthDBMGroupFile   "/usr/local/apache2/var/users.dbm"
        Require dbm-group  project3
    </Directory>

</VirtualHost>
</IfModule>

Hopefully, this will save someone a few hours of digging around Apache mailing lists. Or looking through the source code for comments.

In a future post, I’ll tackle another item on my todo list – making the directory listing look a bit better. It’s not unbearable, but it’s a bit disappointing that in 2021, the default page is not even responsive. Hence the IndexHeaderInsert override in the apache conf.

October 2023 Update

I forgot to mention I’m now using mod_macro which simplifies setting up new shares:

<Macro WebDAVShare $alias>
    Alias /$alias /var/www/myserver.tld/$alias
    <Directory /var/www/myserver.tld/$alias>
        DAV                On
        SSLRequireSSL
        AuthType           Basic
        AuthName           "webdav"
        AuthBasicProvider  dbm
        AuthDBMUserFile    "/usr/local/apache2/var/users.dbm"
        AuthDBMGroupFile   "/usr/local/apache2/var/users.dbm"
        Require dbm-group  admin $alias
    </Directory>

    <Directory /var/www/myserver.tld/$alias/cgi-bin/*>
        Options ExecCGI
        SetHandler cgi-script
        DirectoryIndex
    </Directory>
</Macro>

Use WebDAVShare simon
Use WebDAVShare bob
Use WebDAVShare alice
Use WebDAVShare some_other_site

I’ve also started using an admin group. This group can access any webdav share (as shown in the conf). I add myself to that group, so if there’s a problem with a share, I can access it and troubleshoot any problems.

$ htdbm -t users.dbm simon@mydomain.com "admin"

Using Pelican with Invoke

2021-05-04T00:00:00+01:00

I’m probably very late to the Invoke party, and if you are too, it’s worth a few minutes of your time to start taking advantage of Invoke for your Pelican administration.

The Old Way

Since I got back into blogging, I was getting sick of:

pelican content

Check everything looks good locally:

pelican --listen

Publish the site ready for upload to server:

pelican content -s publishconf.py

Send changes to server

rsync -avc --delete output/ myserver.com:/var/www/simonh.uk/html

The New Way

# pip3 install invoke

Open up tasks.py in the root of your site and check the config section:

CONFIG = {
    'settings_base': SETTINGS_FILE_BASE,
    'settings_publish': 'publishconf.py',
    # Output path. Can be absolute or relative to tasks.py. Default: 'output'
    'deploy_path': SETTINGS['OUTPUT_PATH'],
    # Remote server configuration
    'ssh_user': 'you',
    'ssh_host': 'yoursite.com',
    'ssh_port': '22',
    'ssh_path': '/var/www/yoursite/html',
    # Port for `serve`
    'port': 8000,
}

Be sure to check the ssh lines in particular. The ssh_path was not right for mine, probably from when I ran pelican-quickstart.

Once you start using invoke, you can simply type:

invoke reserve which builds, then serves.
invoke publish which builds using publishconf.py, then runs rsync to your server.

I’ll have to see if I can automate my Mercurial committing and pushing using invoke, next.

Here is the relevant section from the pelican documentation: invoke

Get a Daily Email Report of Your Website Visitors

2021-05-01T00:00:00+01:00

It’s too easy not to!

If you haven’t already:

# apt install goaccess

Then, create /etc/cron.daily/apache_report with the following contents

#!/bin/bash
goaccess /var/log/apache2/other_vhosts_access.log --log-format=VCOMBINED -o report.html
gzip report.html
echo "Report" | mail -s "Apache Report" -A report.html.gz you@yourdomain.com

Remember to change the email address to your email address

tips

Instead of waiting a day to find out if the script is working correctly, you can copy apache_report to /etc/cron.hourly and then cd into it and run:

# run-parts .

If all went well, you should have a new email with a nicely formatted html report.

Remember to delete the script from /etc/cron.hourly or you’ll have a full inbox!

Also, I’m assuming that you’re using virtual hosts in Apache. If not, you can reference access.log instead of other_vhosts_access.log

Tips for Your First Blog

2021-04-26T00:00:00+01:00

Don’t get distracted

When you use a static site generator or a CMS for your blog, you’ll quickly find there are enormous opportunities for theming and customising. In fact, if you’re not careful, you won’t have time for anything else. Themes in particular, can have you downloading half the internet.

My advice (if you’re using Pelican) is to click this link on github. This will show you all Pelican themes that have been recently updated. This is how I found the theme I’m using here. Thanks, by the way to Anthony Bouvier who converted this theme from Hugo. It’s simple, responsive and fast. And that’s all I (or you) need.

Focus on writing content

Since you’ve decided to start blogging, I assume you’ve got things to say?! Personally, I think a blog has value even if nobody ever knows it, or you, exist. Writing is a skill which only gets better with practice. So if you want to write about something but aren’t sure if anyone else will be interested — just write it and throw it out there.

Don’t obssess about SEO

Let’s face it, neither one of our sites will ever be massive. We’ve got some things to say and we’re saying them. Occasionally, we might write a tutorial that gets a bit of traction, but generally things are quiet. Fine by us! After some time blogging, we might set up another one which is focused on making money in some way, but for the moment, we’re just writing and posting stuff. By all means go to one of the many sites for a SEO check, Bing and Google both do this. But you can always start writing and fix the technical issues when you’ve got more knowledge.

I have Analytics on this site, but it’s not really needed. I’ll probably start parsing the Apache logs directly at some point.

Provide value

A short aside. Last year, my partner got very ill and nearly died. She has a Facebook account (I don’t), and so I asked people to pray for her. Out of courtesy I let them know her status the following day and got lots of comments asking me to keep them updated. So I did. Every day I made a post, often long and detailed about her progress and what I was doing, how the kids were doing, etc.

What amazed me was how people responded. After posting, I’d get messages saying “I’ve been looking forward to your daily updates. Keep it up!” The support was incredible.

My point? Even though I had no idea, our story was somehow interesting to other people. Lots of people. So even if you think something might be boring, you might be completely wrong.

Stay away from controversial topics

It’s really not worth your trouble. Not yet anyway. Keep things nice and cheerful. Professional, almost. We all have strong opinions about certain things, but let’s keep our blog cheerful. Remember, what you write could be here forever.

Writing tips

Writing can be hard or easy. It may depend on your level of interest and knowledge of the topic. When I’m writing a post like this one which is an informative post, I’ll start off with all my headings. That’s how my mind works; from broad to specific. Your mind may be set up differently though, and you might want to start writing until you’re done. Only time will tell what works best for you.

Ultimately, you must sit down and start writing though. I like pretty short and direct writing as I assume anyone reading my stuff is no fool. Their time is valuable and I don’t want to waste it. So keep waffle to a minimum, please.

Be organised

I started emailing myself a while ago if I have a good idea that I can’t deal with at the time. Not only for blogging, but in general. You may prefer a notebook or a dictaphone (voice recorder). Now that you’re a writer, you need to capture those thoughts as they tend to evaporate within a few minutes.

Good luck!

And as always, my email is below if you’ve got something to say.

Fix Your Email Address by Getting a Domain Name

2021-04-25T00:00:00+01:00

Hi simon_harrison189@myemailprovider.com

Ugh.

It still amazes me when someone gives me an email address like the above. For the sake of a tenner a year, you can have your own memorable, personalised email.

A site that is really good for searching multiple domains is netim.com

Here’s a partial result for European domains for simonharrison:

simonharrison too long? Of course it is. Let’s try simonh:

These are only a small proportion of the results. The domains are split into geographical regions. Here is what we get for Oceania:

I think you get the point.

mail@simonh.be is far better than simon_harrison189@myemailprovider.com

The other major benefit is that once you’ve registered a domain name, you can forward emails to your awful simon_harrison189@myemailprovider.com mail account. This way, you’ll only ever need one email address. I’ve been doing this since 2002!

As to which domain registrar to use, I use ovh and fasthosts. Make sure to shop around and compare prices before you sign up.

Email for Static Website Commenting

2021-04-24T00:00:00+01:00

A few months ago, when I resurrected this blog, I decided to put a link at the bottom of each post to let interested readers send me a comment by email. Nothing fancy, just a mailto with the post title as the subject. I haven’t had any replies yet, nor any spam, but I’m surprised it isn’t more widespread what with the uptake of static site generators.

Have I missed something?

Benefits

No need for a new account or a new service. Just utilise the email address everyone has

There are loads of programs for handling email, detecting spam etc.

It wouldn’t be too hard to have some email processing scripts on your server to regenerate the page when a comment comes in. A sort of static / dynamic website hybrid

For sites interested in having / building a mailing list, surely this would be the easiest way to get email addresses (with the users permission, obviously)

I’m honestly surprised that if I search for ‘static site commenting’ the first result is not:

Seriously, send me an email if you disagree!

Pelican Static Paths for Extra / Other Content

2021-04-24T00:00:00+01:00

I like to have a few directories on my web server for easy access from anywhere. Most importantly, /files/

Of course, Pelican can handle such things. All you need to do is set STATIC_PATHS in your pelicanconf.py

STATIC_PATHS = ['files', 'share', 'public', 'docs']

And make sure your static directory is within content like below:

content
├── first post.textile
├── second post.textile
├── number three.textile
├── files
│   └── some document.pdf
├── pages
│   ├── about.textile
│   ├── contact.textile
│   ├── links.textile

Pelican will nicely copy these static files from content to output without processing them in any way.

See here for more details: Pelican Docs

Why You Should Always have a Static Personal Website

2021-04-22T00:00:00+01:00

Where I show that only a static website gives you

Speed
Simplicity
Security
Portability
Control

But, I’m on Facebook, Linkedin etc…

Sorry to have to inform you, but the corps don’t care about you. At all.

Say the wrong thing, write the wrong thing, even like the wrong thing, and you could be deleted. Many people have. Or, one day they could go the way of the dodo. Along with all your content. Look up Myspace or Google+ if you didn’t live through those messes.

Also, ask yourself — what value do you get from these platforms?

If you do use these types of sites, be sure to drive visitors to your website. Use them as a bypass to your website. Linger on those sites for as little time as possible.

I use Wordpress, Wix, Medium etc…

These all have advantages — lots of options, commenting, seo optimizations (all the bells & whistles).

But, do you really need all that for a personal site? And remember, if you use a companies platform, you no longer have control over your content. Effectively, you’re at their mercy as you are on the social media platforms! Self hosting a CMS is different of course, but you still have the complexity problem (setting everything up, security, site maintenance, upgrades).

Tell me about static sites then

Glad you asked! Static websites are simply html documents served by your http server (Apache, Nginx). For the last few years, most of us use Static Site Generators, such as pelican that I’m using for this site. These tools generate the html on our behalf and create rss / atom feeds, sitemaps (loved by search engines), handle pagination and lots of other things.

But, at it’s core, it’s just html. So, if you fall out with your web hosting provider, you just dump your entire site (a directory) on another server. No faffing about with databases, and minimal configuration of any kind.

Plus, static sites work great with revision control tools like CVS, Mercurial or Git. I may do a post on this another day.

Are they suitable for business use?

Lots of companies have (re)realised the benefits of static websites. hugo has a showcase of sites created using it, as does jekyll. So, yes. Definately sutiable for business use.

Which static site generator should I use?

The primary reason why someone chooses one over another is the language they’re written in. I’m a Python guy primarily, so I use Pelican.

Honestly, they all do the job and having an abundance of choice is a good thing. I believe most of them allow you to import an existing site made by another site generator; similar to converting a Microsoft Word .docx document to a Libreoffice Writer .odt document. No big deal.

Conclusion

I wrote at the top that only static websites give you the following desirable features. I’ll summarise them now:

Speed

Nothing is faster than just serving html. It’s the original purpose of the web — request a page, deliver the page.

Simplicity & Security

No databases, no logging users in and out. No caching. No need for https (even though it’s best practice everywhere nowadays).

Portability

Your entire website is a single directory (usually). Back it up, move it from one host to another. It’s just a folder.

Control

Your website, your platform. Host it where you want, even from home if your ISP has ports 80 / 443 open.

The End

Thanks for reading, and as always, email me a comment using the link below if you’ve something to say.

Pre-compress gzip files with Apache

2020-02-05T00:00:00+00:00

Taken from: desgrange.net

Remember to

sudo a2enmod rewrite
sudo a2enmod headers
sudo apachectl configtest

in your <VirtualHost>:

<Directory */path/to/my/blog/*>
    RewriteEngine on
    # Gzip
    # If the web browser accept gzip encoding… 
    RewriteCond %{HTTP:Accept-Encoding} gzip
    # …and the web browser is fetching a probably pre-compressed file…
    RewriteCond %{REQUEST_URI} .*\.(css|html|js)
    # …and a matching pre-compressed file exists… 
    RewriteCond %{REQUEST_FILENAME}.gz -s
    # …then rewrite the request to deliver the gzip file
    RewriteRule ^(.+) $1.gz
    # For each file format set the correct mime type (otherwise gzip mime type is returned) and prevent Apache for recompressing the files
    RewriteRule "\.css\.gz$" "-" [T=text/css,E=no-brotli,E=no-gzip]
    RewriteRule "\.html\.gz$" "-" [T=text/html,E=no-brotli,E=no-gzip]
    RewriteRule "\.js\.gz$" "-" [T=application/javascript,E=no-brotli,E=no-gzip]
    <FilesMatch "\.(css|html|js)\.gz$">
        # Serve correct encoding type
        Header set Content-Encoding gzip
        # Force proxies to cache gzip & non-gzip files separately
        Header append Vary Accept-Encoding
    </FilesMatch>
</Directory>

Whilst scouring the web for useful pelican plugins, I came across gzip_cache

The idea is that you have a precompressed html.gz / css.gz / js.gz files alongside your html / css / js files.

The above apache2 config should get Apache to serve these already compressed files instead of compressing on the fly.

Clunky way to use Rsync to update Static Site

2020-02-03T00:00:00+00:00

A temporary fix

In your working directory

rsync -avc output/ example.com:~/tmp/simonh

In the remote dir from above:

sudo rsync -r * /var/www/simonh.uk/html

It ain’t pretty but it’s working (at least)

Where is my name

2016-01-04T00:00:00+00:00

Note: This post is from an old Pelican blog I used to have at simonh.me, which I found on a backup drive on 2025-01-07. It's probably of no use, but reposted for posterity!

Background

I haven't done much website stuff for quite a few years. So I set myself the goal of creating a few websites to get back into the swing of things. After a bit of research, I decided to use Pelican as it's fairly easy to get going with.

I don't really have a plan for this site (yet), but I was pleasantly surprised to see my site listed on the fourth page of Google for the search term 'simonh' (after submitting it, of course). This was yesterday. Today, I'm the third result! I wondered how I'm doing for the term 'simon harrison'...

After going through fifteen(!) pages of search results, I gave up. I can only conclude, I don't exist. Boo hoo. Now, I know that certain organisations that provide results for web searches have changed things to avoid sites from appearing disproportionately high, but my name actually is, simon harrison. So why am I nowhere to be seen? It probably doesn't help that this domain doesn't have simonharrison in it. I noticed that the first four results do have the search term in the domain.

The Plan

I'm sure the many other Simon Harrison's are just chuffed to say to their friends "Google Me". Hopefully, soon, I'll be saying the same thing :)