Today is Thanksgiving in the United States. It also happens to be Hanukkah. There’s been news going around that Thanksgiving and Hanukkah will not coincide again for about 80,000 years. This sounded somewhat unbelievable to me because the Gregorian repeats every 400 years. I decided to compute it for myself to double-check this figure.
I’m not Jewish and I know very little about Hanukkah, so I had to look it up. After learning that Hanukkah is based on the Hebrew calendar, the rumors were sounding more believable. The Hebrew calendar repeats every 689,472 Hebrew years. This means the correspondence between Gregorian and Hebrew calendars is about 14 billion years. That 80,000 seems lowball.
Since I decided to use Emacs Lisp for the computation, I fortunately was able to ignore all the unfamiliar, complicated rules for the Hebrew calendar: Emacs knows how to compute Hebrew dates. It can be accessed through the function calendar-hebrew-date-string.
Hanukkah begins on the 25th of Kislev, so I can write a quick-and-dirty function to detect if a date is the first day of Hanukkah.
(defunhanukkah-p(date)"Return non-nil if DATE is Hanukkah."(string-match-p"^Kislev 25"(calendar-hebrew-date-stringdate)))
Next I need a function to compute Thanksgiving, which is really simple. Thanksgiving falls on the fourth Thursday of November.
(defunthanksgiving(year)"Return the date of Thanksgiving for YEAR."(loopfordayfrom1upto7when(=4(calendar-day-of-week`(11,day,year)))return`(11,(+day21),year)))
If there was no calendar-day-of-week I could compute it using Zeller’s algorithm, which I already happen to have implemented,
(defuncal/day-of-week(yearmonthday)"Return day of week number (0-7)."(let*((Y(if(<month3)(1-year)year))(m(1+(mod(+month9)12)))(y(modY100))(c(/Y100)))(mod(+day(floor(-(*26m)2)10)y(/y4)(/c4)(*-2c))7)))
Now for each year find Thanksgiving and test it for Hanukkah. I started with 1942 because that’s when the fourth-Thursday-of-November rule was established. Presumably due to the regexp part, this expression takes a moment to compute.
This past weekend I had some questions from next-user-here (NUH) on my original Elfeed post about changing some of Elfeed’s behavior. NUH is an Elisp novice so accomplishing some of the requested modifications wasn’t obvious. A novice is mostly limited to setting variables, not defining advice or using hooks. I’ve also been using Elfeed daily for about three months now as my sole web feed reader and along the way I’ve developed some best practices. In addition to responding to some of NIH’s questions here, I’d like to share some tips and tricks.
Custom Entry Launchers
Currently you can press “b” to launch one or more entries in your browser. You can use “y” to copy an single entry to the clipboard. What if you want to make another action.
In my configuration I have a fancy binding that sends the entry URLs in the selected region to youtube-dl for downloading the videos. It’s too large to share as a snippet so here’s a small example of something similar using a program called xcowsay.
Now when I hit “x” over an entry in Elfeed I’m greeted by a cow announcing the title.
Entry Listing Customization
The search buffer you see when starting Elfeed, where entries are listed, can be customized a few different ways. First, this buffer does grow dynamically. After re-sizing the window/frame horizontally you just have to refresh the view by pressing g (an Emacs convention). How it fills out depends on the settings of these variables,
elfeed-search-title-max-width
elfeed-search-title-min-width
elfeed-search-trailing-width
They control how wide the different columns should be as the window size changes. An important caveat to this is that the cache stored in elfeed-search-cachemust be cleared before the changes will be reflected in the display. This cache exists because building the display, assembling all the special faces, is actually quite CPU-intensive. It was an optimization I established early on.
(clrhashelfeed-search-cache)
If you set these variables in your start-up configuration you don’t need to worry about clearing the cache because it will already be empty. It’s only a concern when playing with the settings.
Date Display
Another question was about adding time to the entry listing. Elfeed only displays the entry’s date. Dates are formatted by the function elfeed-search-format-date. This can be redefined to display dates differently.
Feeds and entries in the database can be manipulated to become whatever you want them to be. Because Elfeed is regularly modifying the database, the trick is to perform the manipulation at just the right time.
Feed Title Changes
Say you want to change a feed title because you don’t like the title supplied by the feed. For example, the title to my blog’s feed is “null program” but instead you think it should be “Seriously Handsome Programmer” (head injury, remember?). The function elfeed-db-get-feed can be used to fetch a feed’s data structure from the database, given it’s exact URL as listed in your elfeed-feeds.
Hold it, that didn’t work. First, that display cache is getting in the way again. Feed titles change very infrequently so they’re cached aggressively. More importantly, next time you update your feeds Elfeed will re-synchronize the feed title with the official title. It’s going to fight against your intervention.
The solution is to do it with a little bit of advice just before the title is displayed. Advise the function elfeed-search-update with some “before” advice.
Automatic entry modification should happen immediately upon discovery so that it looks like the entry arrived that way. This is done through the elfeed-new-entry-hook. Generally this would be used for applying custom tags. These examples are from the documentation:
;; Mark all YouTube entries(add-hook'elfeed-new-entry-hook(elfeed-make-tagger:feed-url"youtube\\.com":add'(videoyoutube)));; Entries older than 2 weeks are marked as read(add-hook'elfeed-new-entry-hook(elfeed-make-tagger:before"2 weeks ago":remove'unread));; Building subset feeds(add-hook'elfeed-new-entry-hook(elfeed-make-tagger:feed-url"example\\.com":entry-title'(not"something interesting"):add'junk:remove'unread))
Due to a feature I recently ported from my personal configuration, this tagger helper function is less necessary. You can put lists in your elfeed-feeds list to supply automatic tags.
(setqelfeed-feeds'(("http://nullprogram.com/feed/"blogemacs)"http://www.50ply.com/atom.xml"; no autotagging("http://nedroid.com/feed/"webcomic)))
The same trick could be used to remove advertising, change the date, change the title, etc. The elfeed-deref and elfeed-ref parts are needed to fetch and store content in the content database. Only a reference is stored on the structure. You can actually use these functions at any time outside of Elfeed, but they’ll eventually get garbage collected if Elfeed doesn’t know about them.
A question that’s been asked few times is if entries can be deleted. To start off, the answer to that question is “no.” There is no function provided to remove entries from the database. If you want to remove entries you’re probably taking the wrong approach.
The main problem with removal is that Elfeed needs to keep track of what it’s seen before. If an entry is removed and then rediscovered, it will reappear as unread. There are better ways to “remove” entries, such as tagging them specially.
On a moderately-powerful computer Elfeed can easily handle at least several tens of thousands of database entries. If “too many entries” ever becomes a performance problem I’d rather solve it by making the database faster than by removing information from the database. It’s already very date-oriented so that older entries are infrequently touched.
If storage is a concern, you shouldn’t get too worked up about that. As of this post I have about 6,000 entries in my database and the index file is only 3.5 MB. The content database after garbage collection, which is the data/ directory under ~/.elfeed/, with these 6k entries is 17MB. When I run M-x elfeed-db-compact, currently an experimental feature, it drops down to 1.8MB. That’s less than 1 kB per entry. It’s also less than my personal Liferea database of roughly the same amount of content (~15MB) before I wrote Elfeed.
If even this storage is still too much you can always blow away your data/ content database directory. This is safe to do even while Emacs is running. You’ll still see all of the entries listed in the search buffer but won’t be able to read them within Emacs until after the next database update (when it re-fetches the most recent entry content).
You can also clear out the content database from within Elisp by visiting every entry and clearing its content field.
The same sort of expression can be used to run over all known entries to perform other changes. If there was a delete function you might use it here to remove entries older than a certain date, then hope they’re not rediscovered.
If you never want to store entry content (you never read entries within Emacs), you can use a hook to always drop it on the floor as it arrives,
If you have any questions or suggestions about how to make Elfeed do what you want it to do, feel free to ask. Some things may actually require that I make changes to Elfeed to support it, though I hope I’ve anticipated your particular need well enough to avoid that.
My GitHub activity, including this blog, has really slowed down for the past month because I’ve spent a lot of free time grading homework for a design patterns class, taught by a colleague at the Whiting School of Engineering. Conveniently for me, all of my interaction with the students is through e-mail. It’s been a great exercise of my new e-mail setup, which itself has definitely made this job easier. It’s kept me very organized through the whole process.
Each assignment involves applying two or three design patterns to a crude (in my opinion) XML parsing library. Students are given a tarball containing the source code for the library, in both Java and C++. They pick a language, modify the code to use the specified patterns, zip/archive up the result, and e-mail me their zipfile/tarball.
It took me the first couple of weeks to work out an efficient grading workflow, and, at this point, I can accurately work my way through most new homework submissions rapidly. On my end I already know the original code base. All I really care about is the student’s changes. In software development this sort of thing is expressed a diff, preferably in the unified diff format. This is called a patch. It describes precisely what was added and removed, and provides a bit of context around each change. The context greatly increases the readability of the patch and, as a bonus, allows it to be applied to a slightly different source. Here’s a part of a patch recently submitted to Elfeed:
I’d really prefer to receive patches like this as homework submissions but this is probably too sophisticated for most students. Instead, the first thing I do is create a patch for them from their submission. Most students work off of their previous submission, so I just run diff between their last submission and the current one. While I’ve got a lot of the rest of the process automated with scripts, I unfortunately cannot script patch generation. Each student’s submission follows a unique format for that particular student and some students are not even consistent between their own assignments. About half the students also include generated files alongside the source so I need to clean this up too. Generating the patch is by far the messiest part of the whole process.
I grade almost entirely from the patch. 100% correct submissions are usually only a few hundred lines of patch and I can spot all of the required parts within a few minutes. Very easy. It’s the incorrect submissions that consume most of my time. I have to figure out what they’re doing, determine what they meant to do, and distill that down into discrete discussion items along with point losses. In either case I’ll also add some of my own opinions on their choice of style, though this has no effect on the final grade.
For each student’s submission, I commit to a private Git repository the raw, submitted archive file, the generated patch, and a grade report written in Markdown. After the due date and once all the submitted assignments are graded, I reply to each student with their grade report. On a few occasions there’s been a back and forth clarification dialog that has resulted in the student getting a higher score. (That’s a hint to any students who happen to read this!)
Even ignoring the time it takes to generate a patch, there are still disadvantages to not having students submit patches. One is the size: about 60% of my current e-mail storage, which goes all the way back to 2006, is from this class alone from the past one month. It’s been a lot of bulky attachments. I’ll delete all of the attachments once the semester is over.
Another is that the students are unaware of the amount of changes they make. Some of these patches contain a significant number of trivial changes – breaking long lines in the original source, changing whitespace within lines, etc. If students focused on crafting a tidy patch they might try to avoid including these types of changes in their submissions. I like to imagine this process being similar to submitting a patch to an open source project. Patches should describe a concise set of changes, and messy patches are rejected outright. The Git staging area is all about crafting clean patches like this.
If there was something else I could change it would be to severely clean up the original code base. When compiler warnings are turned on, compiling it emits a giant list of warnings. The students are already starting at an unnecessary disadvantage, missing out on a very valuable feature: because of all the existing noise they can’t effectively use compiler warnings themselves. Any new warnings would be lost in the noise. This has also lead to many of those trivial/unrelated changes: some students are spending time fixing the warnings.
I want to go a lot further than warnings, though. I’d make sure the original code base had absolutely no issues listed by PMD, FindBugs, or Checkstyle (for the Java version, that is). Then I could use all of these static analysis tools on student’s submissions to quickly spot issues. It’s as simple as using my starter build configuration. In fact, I’ve used these tools a number of times in the past to perform detailed code reviews for free (1, 2, 3). Providing an extensive code analysis for each student for each assignment would become a realistic goal.
I’ve expressed all these ideas to the class’s instructor, my colleague, so maybe some things will change in future semesters. If I’m offered the opportunity again – assuming I didn’t screw this semester up already – I’m still unsure if I would want to grade a class again. It’s a lot of work for, optimistically, what amounts to the same pay rate I received as an engineering intern in college. This first experience at grading has been very educational, making me appreciate those who graded my own sloppy assignments in college, and that’s provided value beyond the monetary compensation. Next time around wouldn’t be as educational, so my time could probably be better spent on other activities, even if it’s writing open source software for free.
From working on Elfeed, I’ve recently become fairly intimate with the Atom and RSS specifications. I needed to write a parser for each that would properly handle valid feeds but would also reasonably handle all sorts of broken feeds that it would come across. At this point I’m quite confident in saying that Atom is by far the better specification and I really wish RSS didn’t exist. This isn’t surprising: Atom was created specifically in response to RSS’s flawed and ambiguous specification.
One consequence of this realization is that I’ve added an Atom feed to this blog and made it the the primary feed. Because so many people are still using the RSS feed, it will continue to be supported even though there are no longer links to it (Ha, try to find it now!). You may have noticed that I also started including the full post body in my feed entries. Now that my feed usage habits have changed, I felt that truncating content was actually rather rude. There’s still the issue that it contains relative URLs, but I’m not aware of any way to fix this with Jekyll. I also got a lot more precise with dates. Until recently, all posts occurred at midnight PST on the post date.
For reference, here are the specifications. Just these two documents cover about 99% of the web feeds out there.
Not that it matters too much, but it’s unfortunate that RSS has sort of “won” this format war. Of the feeds that I follow, about 75% are RSS and 25% are Atom. That’s still a significant number of web feeds and Atom is well-supported by all the clients that I’m aware of, so it’s in no danger of falling out of use. The broken (but still valid) RSS feeds I’m come across probably wouldn’t be broken if they were originally created as Atom feeds. Atom is a stricter standard and, therefore, would have guided these authors to create their feeds correctly from the start. RSS encourages authors to do the wrong thing.
The Flaws of RSS
For reference, here’s a typical, friendly RSS 2.0 feed.
Two of the biggest RSS flaws – flaws that forced me to make a major design compromise when writing Elfeed – have to do with the guid tag. That’s GUID, as in Global Unique Identifier. Not only did it not appear until RSS 2.0, but the guid tag is not required. In practice an RSS client will be rereading the same feed items over and over, so it’s critical that it’s able to identify what items it’s seen before.
Without a guid tag it’s up to the client to guess what items have been seen already, and there’s no guidance in the specification for doing so. Without a guid tag, some clients use contents of the link tag as an identifier (Elfeed, The Old Reader). In practice it’s very unlikely for two unique items to have the same link. Other clients track the entire contents of the item, so when any part changes, such as the description, it’s treated as a brand new item (Liferea). Some guid-less feeds regularly change their description (advertising, etc.), so they’re not handled well by the latter clients. It’s a mess.
In contrast, Atom’s id element is required. If someone doesn’t have one you can send them angry e-mails for having an invalid feed.
The bigger flaw of the guid tag is that, by default, guid tag content is not actually a GUID! This was a huge oversight by the specification’s authors. By default, the content of the guid tag must be a permanent URL. Only if the isPermalink attribute is set to false can it actually be a GUID (but even that’s unlikely). If two different feeds contain items that link to content with the same permalink then that “GUID” is obviously no longer unique. Two unique items have the same “unique” ID. Doh! Even if the guid tag was required, I still couldn’t rely on it in Elfeed.
In contrast, Atom’s id element must contain an Internationalized Resource Identifier (IRI). This is guaranteed to be unique.
Unlike Atom, RSS feeds themselves also don’t have identifiers. Due to RSS guids never actually being GUIDs, in order to uniquely identify feed entries in Elfeed I have to use a tuple of the feed URL and whatever identifier I can gather from the entry itself. It’s a lot messier than it should be.
In a purely Atom world, the GUID alone would be enough to identify an entry and the feed URL wouldn’t matter for identification: I wouldn’t care where the feed came from, just what it’s called. If the same feed was hosted at two different URLs, a user could list both, the second appearance acting as a backup mirror, and Elfeed would merge them effortlessly.
pubDate, the incorrectly specified
RSS didn’t have any sort of date tag until version 2.0! A standard specifically oriented around syndication sure took a long time to have date information. Before 2.0 the workaround was to pull in a date tag from another XML namespace, such as Dublin Core.
In contrast, Atom has always had published and updated tags for communicating date information.
Finally, in RSS 2.0, dates arrived in the form of the pubDate tag. For some reason the name “date” wasn’t good enough so they went with this ugly camel-case name. Despite all the extra time, they still screwed this part up. The specification says that dates must conform to the outdated RFC 822, then provides examples that aren’t RFC 822 dates! Doh! This is because RFC 822 only allows for 2-digit years, so no one should be using it anymore. The RSS authors unwittingly created yet another date specification – a mash-up between these two RFCs. In practice everyone just pretends RSS uses RFC 2822, which superseded RFC 822.
In contrast, Atom consistently uses RFC 3339 dates, along with a couple of additional restrictions. These dates are much simpler to parse than RFC 2822, which is complex because it attempts to be backwards compatible with RFC 822.
RSS 1.0, the problem child
RSS changed a lot between versions. There was the 0.9x series, several of which were withdrawn. Later on there was version 1.0 (2000) and 2.0 (2002). The big problem here is that RSS 1.0 has very little in common with 0.9x and 2.0. It’s practically a whole different format. In order to officially support RSS, a client has to be able to parse all of these different formats. In fact, in Elfeed I have an entirely separate parser for RSS 1.0.
What’s so weird about RSS 1.0? If you thought the name “pubDate” was ugly you might want to skip this part. In practice it’s namespace hell. For example, look at this Gmane RSS 1.0 feed. Unlike the other RSS versions, the top level element is rdf:RFD. That’s not a typo.
Remember, if you want dates you’ll need to import another namespace.
Notice the completely redundant items tag. It’s not like you’re going to download a partial feed and use the items tag to avoid grabbing full content. It’s just noise.
Even more important: notice that the items are outside the channel tag! Why would they completely restructure everything in 1.0? It’s madness. Fortunately everything here was dumped in RSS 2.0 and, except for a very small number of feeds, it’s almost just a bad memory.
channel, the vestigial tag
Notice in the example RSS feed it goes rss -> channel -> item*. Having a channel tag suggests a single feed can have a number of different channels. Nope! Only one channel is allowed, meaning the channel tag serves absolutely no purpose. It’s just more noise. Why was this ever added?
The good news is that RSS has a category tag which serves this purpose much better anyway. Tagging is preferable to hierarchies – e.g. an item could only belong to one channel but it could belong to multiple categories.
Atom
Atom is a much cleaner specification, with much clearer intent, and without all the mistakes and ambiguities. It’s also more general, designed for the syndication of many types and shapes of content. This is what made it popular for use with podcasts. Everything I listed above I discovered myself while writing Elfeed. There are surely many other problems with RSS I haven’t noticed yet.
If I only had to support Atom, things would have been significantly simpler. At the moment I have no complaints about Atom. It’s given me no trouble.
Someday if you’re going to create a new feed for some content, please do the web a favor and choose Atom! You’re much more likely to get things right the first time and you’ll make someone else’s job a lot easier. As the author of a web feed client you can take my word for it.
The design of Elfeed’s database took some experimentation before any part of it was settled. A major design constraint was Emacs’ very limited file input/output. There’s no random access and, without the aid of an external program, files must always be read and written wholesale. That’s not database-friendly at all! In the end I settled on a design that minimized the size of the frequently rewritten parts, an index with two different data models, by storing immutable data in a loose-file, content-addressable database.
At the moment there really aren’t any pure-Elisp database solutions for Emacs. This is almost certainly due to the aforementioned I/O limitations. I ran into this same problem last year when I created an Emacs pastebin server. I attempted, and failed, to interface with a SQLite database through it’s command line program. Nic Ferrier has published a generic database interface, but it lacks concrete implementations.
As a bit of good news, as far as I know Emacs does properly handle atomic file updates across all platforms, so a pure-Elisp database developer would never have to worry about only writing half the database. It’s always a safe operation. Worst case scenario you’re left with an old version of data rather than no data at all.
A real possibility for a database would be connecting to an established database server via TCP with an Emacs network process. If the server has a specified wire protocol Elisp could talk to it efficiently. In fact, there’s exists pg.el that does exactly this for PostgreSQL. Unfortunately I was not able to get this working with my pastebin, nor is this solution appropriate for Elfeed. It would be unreasonable to require users to first set up a PostgreSQL server just to read web feeds!
Ultimately it would seem that any efficient Emacs database requires the help of an external program. The notmuch mail client, which inspired Elfeed, does this. To access the notmuch database a command line program is run once for each request. A query is passed as a program argument and the output of the program is parsed into the result.
The Early Database
For the first few days of its existence Elfeed only had an in-memory database. Closing Emacs would lose everything. For my personal usage patterns, where I read, or at least address, all entries that arrive – and especially because I use Elfeed on a couple of different computers – I don’t really need to track things long term. I could easily mark everything after a certain date as read and forget about them. However, it would be nice to have and, more importantly, many people wouldn’t use Elfeed without persistence between Emacs sessions.
So, for the first database I did what I always do: dumped the data structure to a file using the printer and parsed it back in later using the reader. This is dead simple in Lisp, it’s very fast, and it even works for circular data structures. It’s something I missed so much with the much-less-capable JSON format earlier this year that I wrote a JavaScript library to do it.
(defunsave-data(filedata)(with-temp-filefile(let((standard-output(current-buffer))(print-circlet)); Allow circular data(prin1data))))(defunload-data(file)(with-temp-buffer(insert-file-contentsfile)(read(current-buffer))))(save-data"demo.dat"'(abc["1"23]))(load-data"demo.dat");; => (a b c ["1" 2 3])
Anything with a printed representation can be serialized and stored this way, including symbols, string, numbers, lists, vectors (structs, objects), hash tables, and even compiled functions (.elc files). Basically every Emacs library that stores data on disk uses this technique.
Unfortunately, this is where I hit another serious database constraint: print-circle is broken in Emacs 24.3, the current stable release. This means Elfeed cannot take advantage of this useful feature, at least not for a long time, as I had been counting on. The final database is slightly slower and larger than strictly required as a result.
The Content Database
After breaking the circular references of the in-memory database I finally had persistence for the first time. With the naive printer/reader approach it was slow, almost 1 second to write just a few thousand entries on my 6-year-old laptop (my minimum requirements target machine). I wanted Elfeed to support hundreds of thousands of entries, if not millions, so this was much too slow.
The big slowdown was writing out all the entry content each time the database is saved. These large strings containing HTML that rarely change. There’s no reason to write these out every time, nor is there a reason to even keep them in memory all the time, as it’s rarely accessed. The solution is a loose-file, content-addressable database, very similar to an unpacked Git object database.
The content database stores immutable sequences of characters – not just raw bytes, but rather multibyte strings – using an unspecified coding system (right now it’s UTF-8 for all platforms). The filename for the content is the content hashed with SHA-1 (“content-addressable”). To limit the number of files per directory, these files are stored in subdirectories named by the first hex-encoded byte of the hash (just like Git). A database of 4 items might look like this:
Something really neat about the content database is that it’s completely agnostic about Elfeed. If it weren’t for Elfeed’s garbage collector, anyone could use it to store arbitrary content. The function elfeed-ref accepts a string and returns a reference into the database. Because of the hash, providing the same string in the future will return the same reference without actually performing a write. References are dereferenced with elfeed-deref.
With content stored elsewhere, entries are a struct containing only some small metadata: title, link, date, and a content database reference. Writing out many of them at once is much, much faster.
I don’t expect it happens often, but this also means content is de-duplicated. If two entries happen to have the same content they’ll share content database storage. A small savings.
At this point it’s really tempting to get fancier and really put this content database to use. The core index itself could be stored as raw content, and the root to accessing the database would be a single SHA-1 hash referencing it – again, very similar to Git. If an index stores a reference to the previously written index, then the the Elfeed database would be an immutable structure tracking its entire history. Such a change would cost virtually nothing in performance, just disk space.
Multiple Representations
With all the content out of the way, the database is now just a lean index. At this point it’s a hash table mapping feed IDs to feeds. Feeds contain a list of its entries. To build the entry listing for the elfeed-search buffer, Elfeed needs to visit each feed in the hash table, gather its entries into one giant list, then finally sort that list by date. At around O(n log n), that sort operation is a real performance killer. Completely unacceptable. To fix this we need to think about how the data is updated and used.
First, entries are always viewed in date order, no exceptions. From my experience of using web feeds for the last six years I never had a reason to list feed entries by any other order. The vast majority of the time, newer entries are most relevant, and if I need to look for something specific I can search for it.
We definitely want to store entries in date-order so we can create entry listings without performing a sort: something around O(n) or so. Inserting new entries into this structure should also be efficient.
Second, entries are never removed from the database. This isn’t e-mail. Even if a user doesn’t want to see an entry again, we have to keep track of it. Otherwise it will show up as new if it’s discovered in a feed again, which is likely. Things are added to the database and never removed. In Elfeed, I use a junk tag to completely hide entries I don’t want to see, and I always have a -junk element in my filter.
There’s an important caveat to this one that I had missed until after the public release: entry dates can change! When a previously discovered entry is read from a feed, Elfeed updates (read: mutates) the entry struct to reflect the new state. This includes the date. It’s very likely that a date-sorted representation won’t tolerate date changes underneath it since it’s keying off of them. Either we refuse to update the entry date, or we remove the entry, update the date, and then re-insert it (how it currently works).
Third, entries are generally added with a recent date. After the database is initially populated, it’s only picking up new items. We should prefer adding recently-dated entries be faster than adding older entries. I didn’t get a chance to take advantage of this, but it’s something to keep in mind.
Fourth, entries need to be keyed by an ID string. Each entry has a unique, unchanging identifier string, either provided by the feed itself (RSS’s guid or Atom’s id) or generated intelligently by Elfeed. Especially because of the print-circle bug, we need to be able to talk about feeds in terms of their ID – an indirect pointer.
(Actually, even when RSS guid tags are present, they’re permalinks by default. So, unfortunately, RSS IDs are not at all resistant to collisions across feeds. To work around this, entry identifiers are a pair of strings: feed ID and entry ID. Atom doesn’t have this problem, but we’re stuck with the lowest common denominator.)
A date-oriented representation would be unable to efficiently look up an entry by its ID, so it needs to be supplemented by an ID-oriented representation. This means we need two representations in our database: date-oriented and ID-oriented.
So what do we use? Well, for keeping entries sorted by date we want some sort of balanced tree. A B-tree is probably a good choice. Rather than write one I went with an AVL tree since Emacs comes with a library for it (avl-tree). It’s already debugged and optimized! The bad news is that the internal structure is unspecified, so there are no guarantees that it can be serialized. A future update to the library may break the Elfeed database. I also had to hack into it to work around a security issue. The comparison function is embedded in the tree. After deserializing the database, Elfeed needs to ensure that no one stuck a malicious function in there.
The choice for an ID database was super-easy: a hash table. Due to the print-circle bug, this is actually the main representation. The AVL tree only stores IDs and it has to reach into the hash table to do any date comparisons. If print-circle was working I could store the same exact entry objects in the AVL tree as the hash table, so mutating them would update them in all representations. However, with print-circle off, on deserialization these would become unique objects and updates would break.
The Future
That’s where the database is today. I put in a few extra fields that aren’t actually used yet, so that there’s room to make a few changes without breaking the database. Perhaps someday I’ll work out a whole new database structure, or maybe a proper database library will come into existence, and this post will simply document the old database.
Unsatisfied with my the results of recent search for a new web feed reader, I created my own from scratch, called Elfeed. It’s built on top of Emacs and is available for download through MELPA. I intend it to be highly extensible, a power user’s web feed reader. It supports both Atom and RSS.
The design of Elfeed was inspired by notmuch, which is my e-mail client of choice. I’ve enjoyed the notmuch search interface and the extensibility of the whole system – a side-effect of being written in Emacs Lisp – so much that I wanted a similar interface for my web feed reader.
The search buffer
Unlike many other feed readers, Elfeed is oriented around entries – the Atom term for articles – rather than feeds. It cares less about where entries came from and more about listing relevant entries for reading. This listing is the *elfeed-search* buffer. It looks like this,
This buffer is not necessarily about listing unread or recent entries, it’s a filtered view of all entries in the local Elfeed database. Hence the “search” buffer. Entries are marked with various tags, which play a role in view filtering – the notmuch model. By default, all new entries are tagged unread (customize with elfeed-initial-tags). I’ll cover the filtering syntax shortly.
From the search buffer there are a number of ways to interact with entries. You can select an single entry with the point, or multiple entries at once with a region, and interact with them.
b: visit the selected entries in a browser
y: copy the selected entry URL to the clipboard
r: mark selected entries as read
u: mark selected entries as unread
+: add a specific tag to selected entries
-: remove a specific tag from selected entries
RET: view selected entry in a buffer
(This list can be viewed within Emacs with the standard C-h m.)
The last action uses the Simple HTTP Renderer (shr), now part of Emacs, to render entry content into a buffer for viewing. It will even fetch and display images in the buffer, assuming your Emacs has been built for it. (Note: the GNU-provided Windows build of Emacs doesn’t ship with the necessary libraries.) It looks a lot like reading an e-mail within Emacs,
The standard read-only keys are in action. Space and backspace are for page up/down. The n and p keys switch between the next and previous entries from the search buffer. The idea is that you should be able to hop into the first entry and work your way along reading them within Emacs when possible.
Configuration
Elfeed maintains a database in ~/.elfeed/ (configurable). It will start out empty because you need to tell it what feeds you’d like to follow. List your feeds elfeed-feeds variable. You would do this in your .emacs or other initialization files.
Once set, hitting G (capitalized) in the search buffer or running elfeed-update will tell Elfeed to fetch each of these feeds and load in their entries. Entries will populate the search buffer as they are discovered (assuming they pass the current filter), where they can be immediately acted upon. Pressing g (lower case) refreshes the search buffer view without fetching any feeds.
Everything fetched will be added to the database for next time you run Emacs. It’s not required at all in order to use Elfeed, but I’ll discuss some of the details of the database format in another post.
The search filter
Pressing s in the search buffer will allow you to edit the search filter in action.
There are three kinds of ways to filter on entries, in order of efficiency: by age, by tag, and by regular expression. For an entry to be shown, it must pass each of the space-delimited components of the filter.
Ages are described by plain language relative time, starting with @. This component is ultimately parsed by Emacs’ time-duration function. Here are some examples.
@1-year-old
@5-days-ago
@2-weeks
Tag filters start with + and -. When +, entries must be tagged with that tag. When -, entries must not be tagged with that tag. Some examples,
+unread: show only unread posts.
-junk +unread: don’t show unread “junk” entries.
Anything else is treated like a regular expression. However, the regular expression is applied only to titles and URLs for both entries and feeds. It’s not currently possible to filter on entry content, and I’ve found that I never want to do this anyway.
Putting it all together, here are some examples.
linu[xs] @1-year-old: only show entries about Linux or Linus from the last year.
-unread +youtube: only show previously-read entries tagged with youtube.
Note: the database is date-oriented, so age filtering is by far the fastest. Including an age limit will greatly increase the performance of the search buffer, so I recommend adding it to the default filter (elfeed-search-search-filter).
Tagging
Generally you don’t want to spend time tagging entries. Fortunately this step can easily be automated using elfeed-make-tagger. To tag all YouTube entries with youtube and video,
Any functions added to elfeed-new-entry-hook are called with the new entry as its argument. The elfeed-make-tagger function returns a function that applies tags to entries matching specific criteria.
This tagger tags old entries as read. It’s handy for initializing an Elfeed database on a new computer, since I’ve likely already read most of the entries being discovered.
Tagging is also really handy for fixing some kinds of broken feeds or otherwise filtering out unwanted content. I like to use a junk tag to indicate uninteresting entries.
There are a few feeds I’d like to follow but do not because the entries lack dates. This makes them difficult to follow without a shared, persistent database. I’ve contacted the authors of these feeds to try to get them fixed but have not gotten any responses. I haven’t quite figured out how to do it yet, but I will eventually create a function for elfeed-new-entry-hook that adds reasonable dates to these feeds.
Custom actions
In my own .emacs.d configuration I’ve added a new entry action to Elfeed: video downloads with youtube-dl. When I hit d on a YouTube entry either in the entry “show” buffer or the search buffer, Elfeed will download that video into my local drive. I consume quite a few YouTube videos on a regular basis (I’m a “cord-never”), so this has already saved me a lot of time.
Adding custom actions like this to Elfeed is exactly the extensibility I’m interested in supporting. I want this to be easy. After just a week of usage I’ve already customized Elfeed a lot for myself – very specific customizations which are not included with Elfeed.
Web interface
Elfeed also includes a web interface! If you’ve loaded/installed elfeed-web, start it with elfeed-web-start and visit this URL in your browser (check your httpd-port).
http://localhost:8080/elfeed/
Elfeed exposes a RESTful JSON API, consumable by any application. The web interface builds on this using AngularJS, behaving as a single-page application. It includes a filter search box that filters out entries as you type. I think it’s pretty slick, though still a bit rough.
It still needs some work to truly be useful. I’m intending for this to become the “mobile” interface to Elfeed, for remote access on a phone or tablet. Patches welcome.
Try it out
After Google Reader closed I tried The Old Reader for awhile. When that collapsed under its own popularity I decided to go with a local client reader. Canto was crushed under the weight of all my feeds, so I ended up using Liferea for awhile. Frustrated at Liferea’s lack of extensibility and text-file configuration, I ended up writing Elfeed.
Elfeed now serving 100% of my personal web feed reader needs. I think it’s already far better than any reader I’ve used before. Another case of “I should have done this years ago,” though I think I lacked the expertise to pull it off well until fairly recently.
At the moment I believe Elfeed is already the most extensible and powerful web feed reader in the world.
For the last 8 years I have been using Gmail as my e-mail provider, during which 3 years I was a student. It was very convenient, inexpensive (cost-free!), and especially suitable for a student using various lab computers and stuck behind a fairly restrictive firewall (an anti-file-sharing measure). I didn’t have to worry about filtering spam or maintaining or paying for a server. Easy, easy, easy. This has finally come to an end.
That convenience kept me as a user after college up until now. As of two weeks ago I changed my e-mail address. You can find it listed below my portrait on this page (if you’re reading this on my website). Not only am I using my own domain name, but I’m running my own e-mail server. After attempting, and failing, at creating a decent setup with mu4e, I ended up following this guide:
It’s built around the superb notmuch mail indexer, which includes a powerful, fast Emacs e-mail client. Just from its technical superiority I’m wishing I switched to notmuch years ago. I’m now understanding for the first time why all those old fogey hackers like to use e-mail for everything: mailing lists, software patches, bug reporting, etc. E-mail a user interface agnostic system, giving everyone their own choice. The tricky part is setting up a decent interface to it.
Prompting the change
As you know, Google Reader shut down this past July. This left me using only two Google services: Talk and Mail. Not only is Talk easily replaceable – it has no significant data on Google’s servers – it will be shut down soon as well. I would need to move to a different XMPP server anyway. If I could move off of Gmail, I would finally be able to discontinue my Google account for good!
Why would I want to do this? It’s become increasingly apparent, especially this year, that there is very little privacy to be had when logged into a Google account. Various intelligence and law enforcement organizations have easy – likely automated – access to user data, especially e-mail. I’d really like to take that privacy back.
There are also the technical reasons. It’s a bit embarrassing to admit, but the final straw that pushed me to finally leaving Gmail was the new compose interface – a technical issue rather than a privacy issue – which was pushed out just a few days before I left. I found it to be very unpleasant and, worse, completely incompatible with Pentadactyl, as there is no longer a plain-text option (the one listed is a fake). A huge technical step backwards, towards the layman and away from the power user. I could do so much better than this.
Also embarrassing was being unable to have any meaningful use of PGP with e-mail all these years. That’s something I’ve always wanted to fix.
Brian was a guinea pig for me, because, between the two of us, he was actually the first one to move to his own e-mail platform. Seeing his success was a big encouragement for me. Not only could it be done fairly easily, but the results would be a huge improvement. This was all within my grasp!
A daunting task
Switching everything about my e-mail – provider, client, spam filter, server, domain – and running it myself would be a daunting task. There’s 8 years of archived e-mail to manage, though I kept it trimmed to a relatively light 1 GB of storage (which I cut down to 200 MB before exporting). I’ve made backups of all this e-mail on occasion, but I never had to worry about searching or actually using it. The backups were done “just in case.”
Gmail has excellent spam filtering, an advantage of having so many samples available, and I’m a complete newbie at dealing with it myself. Despite having my e-mail address published on this blog for the last 6 years, I surprisingly only receive about one spam message per day, so this wasn’t actually a huge risk for me.
Then there’s the issue of not looking like spam myself. My e-mail server needs to sit in a friendly IP neighborhood. I need to have a proper PTR record (reverse DNS). I need to generally look legitimate. No showing up to deliver mail to other mail servers in just a t-shirt. This is actually something I’m still struggling with right now. In fact, if I’ve sent you personal e-mail in the past and you’re using Gmail, you should check your spam folder right now, because something I’ve sent recently may likely have been caught in it. I don’t know why yet.
I would also need to learn the ropes of a new e-mail client. I used Eudora from 1995 to 2005, then Gmail up until now. Now, I’m the last person to be reluctant about learning how to operate a new piece of software. I’m constantly on the lookout for better software. The problem is that I use e-mail for a lot of very important things. I can’t afford mistakes. I need to hit the ground running on this one.
notmuch vs mu4e
Since I decided early on to go with an Emacs-based e-mail client, learning a new e-mail client got a lot easier. I know Emacs Lisp pretty darn well. In the worst case of getting stuck, I could very easily study the client’s source code and work out for myself whatever is going on. I could even monkey patch it in my configuration if it was causing problems for me (and I’ve already done exactly that).
I wanted to use the Maildir format, something I could hack on if needed. The two obvious choices for this were mu4e and notmuch, both started in 2009. I initially reached for mu4e. Compared to notmuch, it follows Emacs idioms more closely. For example, the e-mail listing is oriented around a mark and execute paradigm, like a dired buffer. After an initial glace, it felt more integrated.
Unfortunately, mu4e is still not mature enough for real productive use, making it far too risky for e-mail. I found out the hard way that the database format has varied regularly between versions. Worse, mu4e is not suitable for remote access. Not only does it assume the Maildir directory is on the local host, it uses absolute paths to access it, so it won’t work over sshfs as I had hoped. Bummer.
In contrast, the notmuch client is specifically designed to be operated remotely. Emacs doesn’t realize that it runs the notmuch client over SSH. Emacs doesn’t need to touch the Maildir directly. It’s a beautiful setup, one very friendly to versioning dotfiles. I’ve already done some pretty heavy configuration to get it exactly the way I want it. On top of this, notmuch is incredibly fast and stable. It’s been a very enjoyable client. So much so that it inspired me to build a web feed reader with a similar interface (to be described in my next post).
The mail server
My early plan was to run an e-mail server on a Raspberry Pi. It’s low-powered, making it very inexpensive and quiet to operate. It’s also very portable, so I wouldn’t need to lug a server around if I needed to move it – no more difficult than moving a cell phone charger. I could run it from my nightstand next to my bed if I wanted to. On the other hand, I would be a little nervous running my mail server on a residential connection. The downtime would be a product of my ISP, my power company, and my router. It’s not bad, but it’s risky when I’m worried about receiving important e-mail. If I was in the middle of job hunting I probably wouldn’t attempt it at all. Fortunately e-mail servers will retry over the course of several days, so I think this would generally be manageable.
This plan was struck down by Comcast’s network policies. The good news is that my IP address has not changed in years. The bad news is that they block port 25 both incoming and outgoing as an anti-spam measure. This makes it impossible to run an e-mail server, because e-mail must be received on port 25 (MX records were misdesigned feature). For outgoing, I would need to send e-mail through Comcast’s smarthost server, which brings up the privacy issue again. I assume the same organizations have this tapped as well. Even if port 25 wasn’t blocked, I wouldn’t be able to set a PTR record and my IP neighborhood would be suspicious.
I ended up going with Digital Ocean, as the linked guide suggested. The smallest, cheapest offering is more than suitable for my needs both as an e-mail server and an XMPP server. It will probably be handy for other short-lived, experimental servers too.
I used getmail to get all of my old mail onto the mail server in the Maildir format. It was completely straightforward and probably the easiest part of it all.
PGP
I can finally use GnuPG with my e-mail. An important factor in this setup is that encryption and decryption is done locally, not on my e-mail server. I don’t need to trust the server with my private keys. However, verification of signatures is done on the server, which is slightly less than ideal, but manageable.
I thought I might need to generate a fresh PGP key for this new e-mail address, but instead I learned something new about PGP. A key can have multiple identities attached to it, so all I needed to do was edit the key and add the new e-mail address. The PGP designers had already thought of this problem two decades ago! The updated key is linked next to my portrait, as well as distributed on the public keyservers.
I look forward to making more use of cryptography with my e-mail.
The old address
It will take me awhile, a year or more, to move everything off of my old e-mail address. I have a lot of accounts associated with it, many of which won’t allow me to change my e-mail address. So, in the meantime, anything sent there will continue to be forwarded to me. That’s now the only purpose of my Google account.
I’m a little worried about using my new e-mail address as my Digital Ocean account because it presents a circularity problem. I could easily get myself locked out of everything. I’ll have to figure out how I’m going to handle that situation. (Use my work e-mail address? So long as I don’t get locked out of my e-mail and get fired all at the same time.)
If you’re a hacker, I encourage you to run your own e-mail server if you’re not doing so already. It’s been extremely liberating for me.
The JavaScript Function constructor is useful metaprogramming feature of JavaScript. It works like eval, treating the contents of a string as code, but without the quirkiness. The constructor’s API looks like this,
new Function([arg1[, arg2[, ... argN]],] functionBody)
For example, creating a 2-argument add function at run-time,
varadd=newFunction('x','y','return x + y');add(3,5);// => 8
In all of the JavaScript engines I’m aware of, functions created this way are fully optimized and JITed just like any other, except that this is done later. The function isn’t established at compile-time but at some point during run-time. The constructor could be implemented in pure JavaScript using eval,
/* Notice: not 100% correct, but close. */functionFunction(){varargs=Array.prototype.slice.call(arguments,0);varbody=args.pop();returneval('(function('+args.join(', ')+') { '+body+' })');}
Constructor Misuse
Misusing the Function constructor has the risk that you may invoke compilation repeatedly. For example, both of these functions return an array of adder functions.
functionliteral(){varout=[];for(vari=0;i<10;i++){out.push(function(x,y){returnx+y;});}returnout;}functionconstructor(){varout=[];for(vari=0;i<10;i++){out.push(newFunction('x','y','return x + y'));}returnout;}
The literal function creates 10 unique closure objects.
varx=literal();x[0]===x[1];// => false
While these appear to be unique objects, they’re all backed by the same code in memory. Since the function has no free variables, it doesn’t actually capture anything. These closures are really just empty handles to the same function. This function will be recognized and compiled ahead of time before literal is ever executed.
On the other hand, short of any serious optimization magic, the constructor function version of this creates 10 unique backing functions, invoking the compiler for each one individually at run-time. If they’re used enough to warrant it, each will also be optimized separately. This is a misuse of the Function constructor, as bad as misusing eval.
What’s it for?
As stated before, the Function constructor is useful for metaprogramming. Use it to generate source code programmatically. For example, this function generates 64 new functions by assembling source code from strings.
Writing out all these functions explicitly would take 66 lines of code instead of just 16, and it would be error prone and more difficult to maintain. Metaprogramming is a win here.
The opfuncs function should be called exactly once. These functions shouldn’t be generated multiple times because the benefits of the metaprogramming approach would be lost. To ensure that, I’m replacing the function with its result in this example,
The final metaprogrammed opfuncs object should completely indistinguishable from the longer, explicit version.
The Design Flaw
The primary flaw with the Function constructor is that it’s variadic. The whole purpose of this constructor is to dynamically generate functions at run-time, but there’s (generally) no straightforward way to call a constructor with a variable number of arguments. The apply method can’t directly compose with new.
Say we want to write a version of opfuncs where, rather than generate 64 4-argument functions, it generates 4^(n-1) n-argument functions, taking an argument n. Now new Function needs to be applied to a variable number of arguments (n + 1).
If rather than take argument names as individual arguments, Function took them as an array, this would be straightforward. It would be like having apply built-in. This is how I would have designed Function to work.
new Function(argNames, functionBody)
Used like this,
varadd=newFunction(['x','y'],'return x + y');
Fortunately there’s a simple workaround. The built-in constructors do something useful for the most part when used without new. My personal favorite is the Boolean constructor. Without new it returns a primitive boolean based on the truthiness of its argument. It can be used to remove falsy values from an array.
In the case of Function, new isn’t actually needed at all! This way, apply can be used with the constructor. This works as expected,
varadd=Function.apply(null,['x','y','return x + y']);
Here it is being used to generate functions of n arguments.
/** Return a function of n-args that sums its arguments. */functionaddN(n){varargs=[];for(vari=0;i<n;i++){args.push('a'+i);}args.push('return '+args.join(' + '));returnFunction.apply(null,args);}addN(5)(3,4,5,6,7);// => 25
A single variadic function that uses the arguments special variable to sum its arguments could be used in place of these individual functions, but generating a function with a specific arity and using it many times will have much better performance than the variadic version.
I mentioned a thought to a co-worker about taking a smart phone back in time 50 years to the 1960’s, the very early days of computing. A consumer smart phone, a ubiquitous battery-powered device today, has many orders of magnitude more computing power than was available to all humankind at this time. Not to mention the amount of digital storage would be unimaginably huge. It would likely be the single most valuable device on Earth. If it goes back just a couple more decades it would significantly impact the events of World War II. What would the engineers of a half-century ago do with it?
I like to assume that it had no useful software installed, so it’s just the advanced hardware. I guess time-traveling erases digital storage in this scenario. Suppose the engineers of the day could work out a way to interface with it electronically using the technology of the time – something that I believe would be very difficult considering the bandwidth and speed of the interfaces – and use that interface to program it. Outside of study, some of the hardware would be pretty useless, such as the H.264 decoder/encoder, wireless radios (what would it talk to?), and probably the graphics chip, but the CPU would definitely be useful.
This was a time before mice, monitors, conventional operating systems, and all the modern computing paradigms. Not only was the computer hardware primitive but so were the concepts about computing. I imagine they would build and run the same sort of timesharing operating system that existed at the time. They would probably attach thousands of typewriter-based consoles to this one smart phone so that several buildings worth of people could make use of this one smart phone. Picturing in my head all these old-timey typewriters attached to a cell phone is amusing.
My co-worker said that there needs to be a word for this. Specifically, if you use time-travel to benefit yourself in some other time period. For example, something like the word “nepotism”: favoring relatives because of their relationship rather than their ability. I came up with the word autoism for this.
Crony - ism: The word “crony” means “friend.” Favoring friends regardless of their ability.
Nepot - ism: The Latin “nephos” means “nephew,” which was also a euphemism for “natural son.” Favoring family regardless of their ability.
Auto - ism: “Auto” comes from ancient Greek for “self.” Favoring yourself regardless of your own ability.
Fortunately or unfortunately, the technology needed in order to actually practice autoism, time travel, doesn’t currently exist, so the word will only be useful in science fiction for the time being.
The personal live system I configured a couple months ago has turned out to be quite handy. I’ve used it almost every day, especially as I’ve learned more about the features of live-boot and live-config, the runtime components of live-build.
At work I use a number of “scratch” computers whose only real purpose is to temporarily sit at particular points on an experimental, isolated network. It doesn’t matter what operating system is installed on them; they really just need to be able to communicate on the network and, occasionally, run Wireshark.
For awhile these systems just had a base Ubuntu install from a couple of years ago. This works fine, of course, but it was missing all of my dotfiles. It didn’t feel like home. I could put my dotfiles on these systems, but there are a number of these machines and I’d have to keep my dotfiles up to date on each, without any Internet access, as they changed.
More so, Unity, Ubuntu’s default desktop environment, isn’t all that friendly to a keyboard-only setup. I don’t have any mice attached to these systems so I have to navigate without one. Sure, It has keyboard shortcuts for the basic things, but that’s not enough. On the other hand, my personal configuration is entirely built around avoiding the mouse. Except for a few specific, reasonable cases (graphics programs like Gimp, Inkscape), I can do anything just as comfortably without as I can with a mouse.
Here’s where my live system comes to the rescue. The generated ISO is a “hybrid” image. This means it can be both burned into a CD as a standard ISO 9660 filesystem and written directly to a hard disk as a read-only disk image. After dding the ISO onto a thumb drive, I could plug the drive into these machines and boot directly into a comfortable development environment. When I update my dotfiles significantly I just build a new ISO (takes about 5 minutes) and copy the new version onto the thumb drive.
Running a live system from RAM
So the next problem is that there are a number of these systems but I have only one thumb drive. Once I boot a machine from the thumb drive I can’t remove the drive and continue to use the system. I could burn a bunch of CDs, one for each system, but, fortunately, live-boot has a wonderful solution for this: toram.
That’s right, just add toram to the system boot arguments and during boot the entire system is lifted into memory. Editing the system arguments before booting can be done by hitting tab at the isolinux/syslinux boot menu. Once the system is up I’m free to remove the thumb drive and use it to boot another system. Additionally, the system will also be much faster – faster than even a normal system – since it’s running entirely in RAM.
The disadvantage is that there’s now less memory available to the system for other things. This can be mitigated by making use of a swap partition on a local disk.
Persistence
These live systems use an interesting filesystem called SquashFS. It’s like a compressed tar archive, but it has two significant advantages:
The contents are available for efficient random access.
It can be mounted as a first-class filesystem.
SquashFS is read-only. This presents a big problem if you want to use it as your root filesystem. Luckily there’s another cool trick to fix this: a union filesystem, aufs. A writable filesystem can be mounted over top the read-only SquashFS filesystem, unioning the contents of the two filesystems, providing the illusion of a writable SquashFS. It’s effectively copy-on-write.
Typically for a live system the writable filesystem is tmpfs, a filesystem backed by RAM. This means all changes are lost when the operating system halts. This is useful for a throwaway operating system, but sometimes data persistence between boots is required. This time live-config has a solution: persistence. Add this word to the system boot arguments and, rather than a RAM filesystem, a partition on real media will be used to back changes to the SquashFS filesystem.
The next question would be, “Which partition?” That’s part of a discovery phase. During boot, the various storage media are scanned looking for a filesystem labeled “persistence”. If one is found, it looks for a file in the root of that filesystem called persistence.conf, which explains how to map this partition with the live system. Full persistence can be achieved with this one-liner configuration,
/ union
This means the root of this filesystem should be union mounted with the live system. However, this will capture some extraneous directories like /tmp, which I don’t want to write to storage media. I prefer a more nuanced configuration,
/etc union
/home union
/opt union
/sbin union
/usr union
/var union
This captures all the important stuff while leaving /tmp to a RAM filesystem (where it belongs!). Not only is my home directory persistent, but so is any additional software I install. Effectively, it is now a normal, non-live system.
If the live system is installed on a thumb drive, the space after the image can be used for this persistence. After dding the ISO, use fdisk/parted to add a second partition, format a fresh partition labeled “persistence” in it, then drop in config file.
mkfs.ext4 -L persistence /dev/sdb2
mount /dev/sdb2 /mnt/sdb2
cp peristence.conf /mnt/sdb2
Furthermore, the persistence partition can be encrypted with LUKS, providing a portable, persistent, encrypted operating system. The live-config will automatically find it, prompt for the passphrase on boot, and handle all the rest on its own like normal. (Well, once this bug fix is released.)
Persistence as an installation
Remember how I had said the exact operating system for these scratch computers wasn’t important? While using one of these systems a lightbulb went off in my head. After booting the live system, I could just dd the contents of the thumb drive directly onto the hard disk, including the persistence partition, make “persistence” a default boot argument, and I’ll have installed a fully configured operating system with all my goodies within a few minutes! It’s the ultimate anti-thin client. This is what I’m using now.
Since it’s possible to build a live system image within a live system (“Yo dawg …”), I can use the toram boot option to update and/or reinstall the entire system from scratch, in-place, without the help of external storage.
Live systems are a powerful tool, at least for my line of work.
