How this site is built

DISCLAIMER 2023-09-15: The site doesn't work like this and hasn't for years. I'm leaving this up though because it's fun. Very briefly, the site is now a static site generated by shell scripts and php files.

DISCLAIMER 2024-05-06: The site also doesn't work like the above anymore. Now it is all done in lua, except deploying, which is still a shell script.

Obviously most sites in 2020 are enourmous beasts, running megabytes of javascript on both the client and server. This one isn’t, and I thought I’d take some time to explain how it does work. I want to write an article critical of the web as a whole at some point so I haven’t put any reasoning for why it works this way in this post.

This is no ordinary site

This is both a blog and a phlog. All of the posts are available on both a website and a gopher server, both at shtanton.com. If you haven’t heard of gopher, go have a look! I hadn’t until recently but it is a lot of fun. I got started here.

Server Side: HTTP

There is basically nothing on the http server. It is a rust executable using actix that serves files from a directory. All the pages on this site are just html files. The only interesting thing it does is redirect / to /index.html.

Server Side: Gopher

Even simpler, I didn’t need a gopher server library since the protocol is wonderfully simple. Again a rust executable is serving static files, and / serves a file called index.

Server Hosting

These are hosted on a debian buster server with vultr. I wrote a couple of systemd unit files for my executables and they run 24/7 and autorestart should they fail.

Client Side

The phlog (gopher log) just serves the raw markdown files that I write, but the blog has some additional styling. A tiny bit of css which uses css-grid to layout the pages and make them responsive, as well as gruvbox colours.

Directory structure

The project root has:

• config.json - list of posts with metadata
• gopher - built gopher files
• gopher_src - scripts and templates for building gopher files
• html - built html files
• html_src - scripts and templates for building html files
• posts - post markdown files
• gopher-server - rust gopher server
• web-server - rust web server
• publish.sh - publish everything
• Tupfile - tupfile (see later)

Build process

I’ve hinted that I write markdown posts but obviously they are published as html. To generate all the pages for the site I’ve used a bunch of python scripts connected together with tup, which I hadn’t used before but I quite like now. So far there are 4 python scripts:

• gopher_src/index.py which generates the gopher index
• html_src/index_html.py which generates index.html
• html_src/nav_html.py which generates the nav menu snippet of html (not a whole page) from a json list of posts.
• html_src/post_html.py which generates the post pages.

These are all pretty simple but html_src/post_html.py is the most complex and interesting so I’ll break that down briefly.

#!/usr/bin/python3.8
import argparse, json, sys, os, subprocess, shutil

parser = argparse.ArgumentParser(description="Generate html for post from markdown")
parser.add_argument("post", help="Post markdown file name")
args = parser.parse_args()

with open(os.path.join("posts", args.post+".md"), encoding="utf-8") as post_file, open("config.json", encoding="utf-8") as config_file, open("html_src/post_template.html", encoding="utf-8") as template_file, open("html_src/nav.html", encoding="utf-8") as nav_file:
    config = json.load(config_file)
    post_metadata = next((p for p in config["posts"] if p["file"] == args.post), None)
    if not post_metadata:
        print("Post not found")
        sys.exit(1)
    for line in template_file:
        if line == "{content}\n":
            sys.stdout.flush()
            subprocess.call(["markdown"], stdin=post_file, stdout=sys.stdout)
            sys.stdout.flush()
        elif line == "{header}\n":
            sys.stdout.write("""
            <h2>{title}</h2>
            <em>{date}</em>
            """.format(title=post_metadata["title"], date=post_metadata["date"]))
        elif line == "{nav}\n":
            shutil.copyfileobj(nav_file, sys.stdout)
        else:
            sys.stdout.write(line)

This isn’t a python tutorial so I’ll be quick. It takes an argument with the name of the post (not including the extension). It finds the metadata for the post in the json file, then reads the template line by line looking for substitution points which is substitutes accordingly.

The Tupfile that runs all of these scripts is:

: foreach html_src/*.css |> cp %f %o |> html/%B.css
: |> ./html_src/nav_html.py > %o |> html_src/nav.html
: foreach posts/*.md | html_src/nav.html |> ./html_src/post_html.py %B > %o |> html/%B.html
: foreach posts/*.md |> cp %f %o |> gopher/%b
: | html_src/nav.html |> ./html_src/index_html.py > %o |> html/index.html
: |> ./gopher_src/index.py > %o |> gopher/index

Finally I have a publish.sh that runs tup and then rsync to copy all the updates to the server.

The git repo with all this stuff in can be accessed using git clone git://shtanton.com/phlog.

Conclusion

I was pretty pleased with myself once I got this working so thought I’d write this, but it’ll probably improve further. Maybe I’ll make this a series. As always my email is shtanton at -this domain-