FeedPipe Design

I am (gradually) making progress on the implementation of FeedPipe, and I have recently run into a difficult decision. In an attempt to keep things simple, the initial design used an configuration file per index, and sites with multiple indexes required multiple configuration files. After trying it out in practice, however, I do not like it for various reasons. Most importantly, I do not like the feel of it. A concrete issue with the design is that common configuration needs to be maintained across multiple files, which is error-prone.

The Haskell Books index on this website is essentially an initial prototype of FeedPipe, implemented as part of my website software. The website software is much more complicated than FeedPipe as it is more general and has many more features, but I realized that FeedPipe would probably be much better with a similar design. I have been working on an updated design, and I started to wonder if I should just abort FeedPipe and focus on the website software instead, since the website software can be used to achieve the same goals as FeedPipe…

After much deliberation, I have decided to continue with FeedPipe, using a design based on my website software. FeedPipe will provide an easy way to maintain static indexes and RSS feeds. My website software, which will be released as open source when it is ready, will include the same functionality in addition to many other features and options.

At a very high-level, my website software consists of two types of components: builders and servers. A builder prepares web content according to a well-defined interface that a server uses to serve the content. Different builder implementations can provide different ways to organize and manage the source content, and different server implementations allow users to use the technology stack of their choice to implement custom server components that work alongside the built-in components.

The builder that I use to manage this website is what I call a filesystem builder. It uses a hierarchy of directories and files to organize and configure the source content, all managed using Git, not unlike many other static site generators. The new FeedPipe design is based on this filesystem builder design.

Design Notes

FeedPipe source files and configuration is stored within a single directory, named src in the examples below. The names of files and directories in the source directory determine the path of the built content. For example, file src/foo.md renders the content for path /foo/ and src/example/bar.md renders the content for path /example/bar/. Since FeedPipe supports building web content to add to existing sites, an arbitrary path prefix may also be specified in the configuration. For example, when the path prefix is configured as /hoge, file src/foo.md renders content for path /hoge/foo/.

When traversing the source directory, files and directories with names that begin with an underscore (_) are handled specially.

• File src/foo/_index.md renders the content for path /foo/. Use of src/foo.md may be preferred for simple pages while src/foo/_index.md may be preferred in order to group the file with related source content. The user can decide how to organize the source files.
• Directory _templates contains the templates used to render the output content. For example, directory src/_templates may contain the default templates for the site while directory src/foo/_templates contains special templates for path /foo/.
• Directory _static contains static files that should be copied to the build directory without processing. For example, an image included in path /foo/ can be stored in the src/foo/_static directory.
• Directory _items contains source files that are indexed. Information about the source files is passed to the template context used to render the HTML index as well as RSS feeds.
• Configuration is loaded from files named _feedpipe.yaml, and configuration in subdirectories can be used to override previously loaded configuration. For example, a src/_feedpipe.yaml file can be used to define the default/common configuration for the site, while files src/books-en/_feedpipe.yaml and src/books-ja/_feedpipe.yaml can be used to specify configuration that is specific to the English and Japanese book indexes. Only the configuration that is different needs to be specified, of course.

I will of course have more extensive documentation in the manual as well as example sites that demonstrate usage. The conventions look a bit complicated in a manual, but they are easy to use in practice!

Update

I changed the configuration file name to _feedpipe.yaml. Since FeedPipe has such a narrow focus, I initially planned on just using feedpipe.yaml, but I realized that such a decision could cause problems in the future as the software evolves. Since the file is handled specially, it should start with an underscore.