Lua Packages

I’m starting to look into how to add pipelined or asynchronous support to my luaimap4 project. It had been awhile since I’d looked at the code so I started the task of refreshing my understanding of the code. In the course of doing so, I opted to take the original source file and break out some of the functionality into separate support modules. After doing so, I didn’t like that I now had multiple source files directly in my install directory, so I opted to create an imap4 subdirectory and put all the related modules under that directory.

And that’s when the fun began.

Before I get going, a quick disclaimer- the explanation that follows is my understanding of what goes on underneath the hood of lua. I’ve likely got some details incorrect, but my final solution does work and I arrived at it through a series of trial and error. So it’s not completely off base.

In order to understand my final solution, it’s necessary to understand how lua searches for modules that are imported using require. Lua uses path search patterns and loaders to bring in the code. It does this in order to leverage the already present internal resources such as tables, and code loading functions such as loadstring or loadfile. All of this stuff is configured under a global table called package.

The first part is the package.path settings. This is a series of patterns that configures lua to search in specific directories for a module. The module name is determined by the name argument passed to require. Here’s what mine looks like, straight from the lua shell:

./?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/local/lib/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua

For the above pattern, the ‘?’ character is replaced with the module name during the search. The first path to yield a match is then compiled and assigned to packages.loaded[modname] so further calls to require don’t unnecessarily recompile the module. In particular, note the init.lua lines. They are important for my final solution.

A second means available for loading a lua module is to use the package.preload table. This table is a dictionary of loader functions for specific modules. The function needs to return a compiled chunk which require will assign to packages.loaded[modname]. The packages.preload table is consulted prior to the search path explained previously.

It’s useful to keep in mind that my goal is for the user to simply be able to enter a line like local imaplib = require("imap4") into their source to make the imap4 module available. So the package needs to take care of it’s own module requirements.

So with all that in mind, on to my solution:

local path = "/usr/local/share/lua/5.1/imap4/"

local function loader(modulename)
    local filename = path..modulename..".lua"
    return assert(loadfile(filename)())
end

package.preload['auth'] = loader
package.preload['utils'] = loader

return loader(...)

This code chunk goes into a file called init.lua and is placed in the package directory with the rest of the package modules: imap4.lua, auth.lua, utils.lua.

Here’s what happens: because of the search path order, when lua executes the function call require("imap4") the first file that returns a match will be the file /usr/local/share/lua/5.1/imap4/init.lua (assuming the install directory for the package is /usr/local/share/lua/5.1). So init.lua has to handle the task of setting everything up for the package in the lua environment.

The loader function is a simple function that returns a compiled chunk of lua code, in this case from a file. The require function calls the loader with the modulename as it’s sole argument which is then passed to the loader function. A detail here is that loadfile returns an anonymous function, which isn’t quite enough to make everything work. In order for the function code to be usable as a module it has to be defined. So the returned function from loadfile is called for this purpose. That value is then returned by the loader.

The loader is used for the imap4 support modules auth.lua and utils.lua as well as the main module imap4.lua. I use the package.preload table for these support modules because they are in a fixed location that the normal search path will not find. They are required by the imap4.lua module and these settings will allow lua to find them.

Also, because this init.lua module is being executed as a result of a call to require("imap4"),initneeds to return the code chunk forimap4sorequirecan make the appropriate assignment in thepackages.loadedtable. Thus the final return statement. The…idiom
grabs the module name passed in byrequire`.

The upshot of all this is a user of the imap4 package only has to worry about requiring the imap4 module. But there’s a fair amount of shenanigans that make it possible.