Shit, I better make this server work.

On the layout of folders and files!

Basically I'm not sure how I should lay out this website. The current organization looks something like this:

This has a number of problems. First and foremost, barely any thought has gone into naming of folders and files - which is because not much thought has gone into their placement and hierarchy.

The server is written using Flask, and this is the rundown on how it works when a user goes to http://example.com/some-path/:

  1. If the path requested is the root of the website, render main_page.html.
  2. If the path requested begins with static/, render a file from the static content directory.
  3. Else, convert the path (some-path) into the filename (pages/some_path.html) assuming it's an article. If it exists, render it.
  4. If there's no article by that name, display the miscellaneous articles page. Or 404. The current behavior isn't very high on my priorities.

There are a bunch of problems with this system - it's kind of brittle, file paths have the meaningless-to-the-user static/ at the beginning, and it means that when I include a resource in http://example.com/tamari/ (the file pages/tamari.html) I have to use a path like ../static/tamari/figure_1.png - when I should be able to reference files related to the Tamari article by a closely-related URL: ./figure_1.png.

All this points to the idea of having my articles be in folders, so it would look something like this:

And hell, at this point having the filenames match the folders (auriga/auriga.html) is just a pain, so we could name the main page in each folder index.html, and suddenly this looks very standard. So why don't I do this?

Well... I don't know. But something always bothered me about the index.html standard. Maybe it's just that my article about Tamari lattices isn't an index, so I don't think it should be called one. That's certainly part of it. Also, some of my articles and pages are going to be very short - in theory I'd like them to be able to be tiny snippets, which could be included on other pages. It'd be annoying to have a subfolder for every single article I write! But... maybe not too annoying. I'll have to think about that.

But what are we doing?

What would my ideal web framework be? Well, something like Flask, really. I like flask because this is easy:

@app.route("/some-convoluted/url/<options>/with-stuff/")
def render_shoofle_page(options=None):
  if options == "bananas":
return render_page("bananas.html")
  else:
return render_page("some_other_template.html", options=options)

And, lo and behold, I can go to example.com/some-convoluted/url/bananas/with-stuff/ and see the right generated page! I like having that capacity at my fingertips. In particular, it makes it easy to respond to various other kinds of requests. That's what I really like about Flask - I feel like I can do everything a webserver can do, but it all takes the same very small amount of work. This compares to, say, PHP, where it's easy to make pages... But more complicated behavior, especially anything involving dynamic URL paths, involves jumping through hoops.It might seem like an overblown complaint, but the integrity of my mental models is important to me - and in PHP, I don't understand what gets sent to the requestor. If I make a Flask route returning JSON, I know it's exactly what a script wants. With PHP, I would hope that it wouldn't have weird headers or content type or additional markup. Plus, you have to work with PHP, which I would wish on no one.

I don't like Flask because I need to specify all the behaviors, and I'm worried that I'll get it wrong (thus, this article).

My ideal world solution would be something that lets me pass requests for subpaths to other python modules:

Aaaand how it works would be simple:

  1. A request comes in to http://example.com/tamari/, and it is handled by server.py.
  2. server.py says "Okay, I don't have any specific handlers that do anything with this URL, but I think articles/articles.py does!" and sends the request to articles.py.
  3. articles.py receives a request for /tamari/ and has a rule for handling that, and so it renders the appropriate HTML file.

The basic idea is that requests can be routed to other python modules. If those modules can be reloaded easily (or automatically!), this would mean I could write complicated URL handling in subdirectories of my website - without having to touch any of the code that handles anything else. It means I can modularize - and that's awesome.

  1. An HTTP GET request comes in to http://example.com/games/points, and server.py takes it.
  2. server.py doesn't know what to do with it, but it knows that requests for URLs starting with /games should be handled by games/games.py.
  3. games.py receives an HTTP GET request for /games/points, and so it can respond to this with a JSON object that tells the requestor how many points they got.

This is the main point of the tiered handlers system - that I can write handlers for all kinds of requests, without having to interact with other parts of my website.

Basically, I want to combine the simple directory structure of PHP websites with the principles of Flask I really like - separating content from server logic, keeping things simple, and having control.

Here's how I imagine this would work:

from flask import Flask, render_template, redirect

app = Flask(__name__)

def main_page(): return render_template("main.html")
app.add_url_rule("/", main_page)

import games
app.add_module_url_rule("/games/<path:rest>", games)

As usual, don't think too hard about the syntax I just proposed. The point is to distribute URL routing logic throughout multiple files, so that my entire website isn't controlled by one brutalistic overlord server.py file that decrees the entire path of every request. I want things split up, dammit!

Now I'm reading up on the Flask docs, and it turns out that it already has the capacity for this. Unfortunately, it's complicated and I'm whiiiny.

It seems that Flask provides the developer with three options:

Okay, but... how do they score up? I have some requirements:

Organization
I want to define the behavior for example.com/articles/tamari/ in the same place as example.com/articles/language-for-games/ is defined, but in a different file from where example.com/games/hunt-the-wumpus is defined.
Simplicity
Defining a URL route and endpoint for each subfolder of my application shouldn't be much more complicated than defining behaviors in a flat system. Ideally, registering all the behaviors for the articles/ subpath is no more complicated than app.route("/articles/", articles_handler)
Ease of Extension
I'm writing a family of pages. They're starting to get a little too complicated. It should be really easy to go from that situation to having a nicely-delineated subsection of my website.
Do I Need to Restart the Server?
I hate restarting the server. Maybe this is a little thing, but gah! I want the server to detect changes automatically! This shouldn't be that hard. Of course, this is a much thornier topic when it comes to subpages that might store their own state... So it might be out the window unless I want to get my hands way more dirty.

How do they fare up?
Organization Simplicity Ease of Extension No Restart Required?
Current
Organize as a Package
Blueprints
Application Dispatch

In light of this chart, it seems like Blueprints are the clear option. Also, "organize as package" is really vague. What they suggested didn't sound very helpful - but making python packages (finally learning how to use __init__.py files!) will probably get used in the solution I'm thinking I'll use, which looks something like this:

If there's static content that is globally important, with simple routing rules, it can - of course - be stored in the server root's static/ directory. Static file handlers don't take much configuration, so this can go in the root.

All the files related to articles are contained in the articles/ directory, or subdirectories. article_routing_blueprint.py contains a blueprint definition that tells the server how to route calls that result in the rendering of the various articles I've written.

All files related to the little javascript games I've made, on the other hand, go in the games/ directory. State for this application is managed by games_blueprint.py, which is sort of like a very tiny application. I think it's possible to keep track of state, if you remember to keep variables global... This is tricksy, but should be doable - and if it gets larger, it should almost certainly be developed as a separate application entirely. At that point, I should figure out how to include multiple applications in one website!

I want to finish this bit by writing down some code for the base server.py logic:

from flask import Flask

# app configuration and initialization
app = Flask(__name__)
app.debug = True

# content in articles/
import articles
app.register_blueprint(articles.blueprint)

# content in games/
import games
app.register_blueprint(games.blueprint, url_prefix="/games")

Okay, one last thing.

This has been a lot about how to get some basic extensible functionality working. But wait, someone cries in the distance! Why are we going to such lengths to do all this in the first place, when you're just serving static HTML files? Isn't this way more work than you need?In case you're reading this article long after I've finished these changes (I hope so, 'cause I should get to work on this immediately), at this point all the articles were static HTML files. That voice speaks the truth.

Thanks for asking. The answer is that I want to change how this site is structured, and the articles are the specific thing I want to change. See, as it is, each article has basically the same header at the top, and basically the same footer at the bottom, and... Well, they're just all enclosed in the same stuff, and so when I change the styling of the website - or, horror of horrors, try to add a sidebar across the hwole site - I have to change all of them! And, as I think we've been over before, I'm lazy.

Conveniently, as I've been obsessively reading discussions of standards-compliant HTML5 and document formats and such lately, this laziness messes with my indomitable enthusiasm for conceptually pleasing solutions. Here's what I've got:

And ta-da! We now have a page which has those three articles in a row.

This, of course, also means we have a neat format for how to display things in general. Someone requested access to a URL! Find the file it points to. If it's an HTML fragment, then point the general page template to that file, and render. The general page template is just a template that adds the header, footer, sidebar, universal styling, and so on, as required, for universality.

So if you go to example.com/tamari/, the server says "Okay, you want to look at the article in the HTML fragment file articles/tamari.html." It renders the general page template around that article, and returns that to the user.

Because I like having the option to look at these things many ways, I'm probably also going to allow for a user to do python articles.py --generate [path], which will simply spit out the the rendered results of going to [path]. I guess that might just be indistinguishable from using wget, but hey. It can't hurt to give more functionality!

It remains to be seen precisely how many pages on this site will use the general page template. I don't want to rely on it too much, but, man, is there any reason not to? Basically, it means that I don't have to worry about writing the headers of anything on my website. I guess I'll need to be able to throw it off when I'm making subpages that are intended to be stylistically distinct from the rest of my site.