FeedPipe Design
I am (gradually) making progress on the implementation of FeedPipe, and I have recently run into a difficult decision. In an attempt to keep things simple, the initial design used an configuration file per index, and sites with multiple indexes required multiple configuration files. After trying it out in practice, however, I do not like it for various reasons. Most importantly, I do not like the feel of it. A concrete issue with the design is that common configuration needs to be maintained across multiple files, which is error-prone.
The Haskell Books index on this website is essentially an initial prototype of FeedPipe, implemented as part of my website software. The website software is much more complicated than FeedPipe as it is more general and has many more features, but I realized that FeedPipe would probably be much better with a similar design. I have been working on an updated design, and I started to wonder if I should just abort FeedPipe and focus on the website software instead, since the website software can be used to achieve the same goals as FeedPipe…
After much deliberation, I have decided to continue with FeedPipe, using a design based on my website software. FeedPipe will provide an easy way to maintain static indexes and RSS feeds. My website software, which will be released as open source when it is ready, will include the same functionality in addition to many other features and options.
At a very high-level, my website software consists of two types of components: builders and servers. A builder prepares web content according to a well-defined interface that a server uses to serve the content. Different builder implementations can provide different ways to organize and manage the source content, and different server implementations allow users to use the technology stack of their choice to implement custom server components that work alongside the built-in components.
The builder that I use to manage this website is what I call a filesystem builder. It uses a hierarchy of directories and files to organize and configure the source content, all managed using Git, not unlike many other static site generators. The new FeedPipe design is based on this filesystem builder design.
Design Notes
FeedPipe source files and configuration is stored within a single
directory, named src
in the examples below. The names of
files and directories in the source directory determine the path of the
built content. For example, file src/foo.md
renders the
content for path /foo/
and src/example/bar.md
renders the content for path /example/bar/
. Since FeedPipe
supports building web content to add to existing sites, an arbitrary
path prefix may also be specified in the configuration. For example,
when the path prefix is configured as /hoge
, file
src/foo.md
renders content for path
/hoge/foo/
.
When traversing the source directory, files and directories with
names that begin with an underscore (_
) are handled
specially.
- File
src/foo/_index.md
renders the content for path/foo/
. Use ofsrc/foo.md
may be preferred for simple pages whilesrc/foo/_index.md
may be preferred in order to group the file with related source content. The user can decide how to organize the source files. - Directory
_templates
contains the templates used to render the output content. For example, directorysrc/_templates
may contain the default templates for the site while directorysrc/foo/_templates
contains special templates for path/foo/
. - Directory
_static
contains static files that should be copied to the build directory without processing. For example, an image included in path/foo/
can be stored in thesrc/foo/_static
directory. - Directory
_items
contains source files that are indexed. Information about the source files is passed to the template context used to render the HTML index as well as RSS feeds. - Configuration is loaded from files named
_feedpipe.yaml
, and configuration in subdirectories can be used to override previously loaded configuration. For example, asrc/_feedpipe.yaml
file can be used to define the default/common configuration for the site, while filessrc/books-en/_feedpipe.yaml
andsrc/books-ja/_feedpipe.yaml
can be used to specify configuration that is specific to the English and Japanese book indexes. Only the configuration that is different needs to be specified, of course.
I will of course have more extensive documentation in the manual as well as example sites that demonstrate usage. The conventions look a bit complicated in a manual, but they are easy to use in practice!
Update
I changed the configuration file name to _feedpipe.yaml
.
Since FeedPipe has such a narrow focus, I initially planned on just
using feedpipe.yaml
, but I realized that such a decision
could cause problems in the future as the software evolves. Since the
file is handled specially, it should start with an underscore.