Extending an Application with Lua Plugins

Introduction

A very common use of the Lua language (which is a very versatile) is using Lua to extend an application via plugins. Many popular games use Lua for this very purpose. Adding a plugin framework to an existing application is trivial with Lua. Also, Lua provides a very capable and easy to use language for writing plugins.

To demonstrate using Lua for plugins I’m going to make very basic text editor and allow features to be added via Lua scripts (user installable plugins).

The plugins

All plugins will need to be .lua files in a directory called “plugins”. The plugins directory must be in the same location as the application binary.

Plugins are tables with attributes and function. They must use specific attributes to identify themselves as usable plugins for our application:

  • name: Unique name identifying the plugin.
  • ptype: Plugin type. Can be “text” or “report”.
  • function run: Main function of the plugin. Takes a single argument which is text from the editor.

The plugins can have an optional attribute “pver” which is a string version number for the plugin.

From the ptype description you can see that two plugin types are supported; text and report. Text plugins modify the text in the editor. The text given to the run function will either be the selected text (if there is a selection) or all the text in the editor (if there is no selection). The return for text plugins is text which will replace the text in the editor (selected or full depending on what was provided to the plugin). The report plugin will be given all text in the editor but it does not modify the text. Instead it returns data to be displayed in a report dialog (should return HTML).

Here are three example plugins. Two text plugins (upper and lower case manipulations) and one report plugin (word count).

lowercase.lua

local M = {}

M.name  = "Lower"
M.ptype = "text"
M.pver  = "1.0"

function M.run(text)
    return tostring(text):lower()
end

return M

uppercase.lua

local M = {}

M.name  = "Upper"
M.ptype = "text"
M.pver  = "1.3"

function M.run(text)
    return tostring(text):upper()
end

return M

wordcount.lua

local M = {}

M.name  = "Word Count"
M.ptype = "report"
M.pver  = "2.0"

function M.run(text)
    local words  = {}
    local sorted = {}
    local out    = {}
    local word
    local num
    local total  = 0

    for w in text:gmatch("%w+") do
    	if words[w] then
    		words[w] = words[w] + 1
        else
            words[w] = 1
            sorted[#sorted+1] = w
        end
    	total = total + 1
    end
    table.sort(sorted)

    out[#out+1] = "<table>"
    out[#out+1] = "<tr><th>Word</th><th>Count</th></tr>"
    for _,v in ipairs(sorted) do
    	out[#out+1] = string.format("<tr><td>%s</td><td>%s</td></tr>", v, words[v])
    end
    out[#out+1] = "</table>"
    out[#out+1] = "<br />"
    out[#out+1] = string.format("<b>total: </b>%s", total);

    return table.concat(out, "\n")
end

return M

Remember these need to be in a “plugins” directory.

Text Editor Application

The editor will use Qt for the Gui (so it’s cross platform). I’m not going to use Qt’s .ui files or QML simply to reduce the number of auxiliary files. The application is very basic, it’s literally a text edit widget in a window with a menu bar for interacting with plugins. There is also a report dialog for showing report output.

Application (Auxiliary) Files

Here are the auxiliary files that are necessary for the application. These don’t deal with plugins or Lua themselves but they’re still necessary for a complete application.

main.cpp

#include <QApplication>
#include "mainwindow.h"

int main(int argc, char **argv)
{
	QApplication a(argc, argv);
    MainWindow m;
    m.show();
	return a.exec();
}

reportdialog.h

#ifndef __REPORT_DIALOG_H__
#define __REPORT_DIALOG_H__

#include <QDialog>
#include <QDialogButtonBox>
#include <QTextEdit>
#include <QVBoxLayout>

class ReportDialog : public QDialog
{
    Q_OBJECT

public:
    ReportDialog(QWidget *parent=0, Qt::WindowFlags f=0);

    void setText(const QString &text);

private:
    QVBoxLayout      *m_layout;
    QTextEdit        *m_edit;
    QDialogButtonBox *m_bb;

};

#endif // __REPORT_DIALOG_H__

reportdialog.cpp

#include "reportdialog.h"

ReportDialog::ReportDialog(QWidget *parent, Qt::WindowFlags f)
    : QDialog(parent, f),
      m_layout(new QVBoxLayout(this)),
      m_edit(new QTextEdit()),
      m_bb(new QDialogButtonBox())
{
    setText("Report");

    m_edit->setReadOnly(true);

    m_bb->addButton(QDialogButtonBox::Ok);

    m_layout->addWidget(m_edit);
    m_layout->addWidget(m_bb);

    connect(m_bb, SIGNAL(accepted()), this, SLOT(accept()));
}

void ReportDialog::setText(const QString &text)
{
    m_edit->setHtml(text);
}

The Main Window

The main window consists of two parts. An editor widget and a menu bar. The menu bar has an “Actions” menu with load, unload, and info actions. Once plugins are loaded they are added to a “Plugins” menu which can run the individual plugins.

mainwindow.h

#ifndef __MAINWINDOW_H__
#define __MAINWINDOW_H__

#include <lua.hpp>

#include <QAction>
#include <QHash>
#include <QMainWindow>
#include <QMenu>
#include <QMenuBar>
#include <QPlainTextEdit>

class MainWindow : public QMainWindow
{
    Q_OBJECT

public:
    MainWindow(QWidget *parent=0, Qt::WindowFlags flags=0);
    ~MainWindow();

private slots:
    void loadPlugins();
    void unloadPlugins();
    void infoPlugins();
    void runPluginText(QAction *action);
    void runPluginReport(QAction *action);

private:
    void createMenu();

    lua_State      *m_L;

    QPlainTextEdit *m_edit;
    QMenuBar       *m_menuBar;
    QMenu          *m_menuActions;
    QMenu          *m_menuPlugins;
    QMenu          *m_menuPluginsText;
    QMenu          *m_menuPluginsReport;
};

#endif // __MAINWINDOW_H__

mainwindow.cpp

#include <QDir>
#include <QMessagebox>

#include "mainwindow.h"
#include "reportdialog.h"

MainWindow::MainWindow(QWidget *parent, Qt::WindowFlags flags)
    : QMainWindow(parent, flags),
      m_L(NULL),
      m_edit(new QPlainTextEdit(this)),
      m_menuBar(menuBar()),
      m_menuActions(NULL),
      m_menuPlugins(NULL),
      m_menuPluginsText(NULL),
      m_menuPluginsReport(NULL)
{
    setCentralWidget(m_edit);

    setWindowTitle("Lua Plugin Example");

    createMenu();
}

MainWindow::~MainWindow()
{
    if (m_L)
        lua_close(m_L);
}

void MainWindow::loadPlugins()
{
    QDir        dir("plugins");
    QStringList files;
    QString     path;
    QString     ptype;
    QString     pname;
    QString     pver;
    QHash<QString, QString> pluginInfo;

    unloadPlugins();

    /* Create a new state and load the standard libraries. */
    m_L = luaL_newstate();
    luaL_openlibs(m_L);

    /* Iterate though all .lua files in the plugins directory. */
    Q_FOREACH (QString plugin, dir.entryList(QStringList("*"), QDir::Files)) {
        path = QDir::toNativeSeparators(QString("plugins/%1").arg(plugin));

        /* Load the plugin. */
        if (luaL_dofile(m_L, path.toUtf8().constData())) {
            QMessageBox::critical(this, "Error", QString("Could not load file %1: %2").arg(plugin).arg(lua_tostring(m_L, -1)));
            continue;
        }

        /* Get and check the plugin has a ptype */
        lua_getfield(m_L, -1, "ptype");
        if (lua_isnil(m_L, -1)) {
            QMessageBox::critical(this, "Error", QString("Could not load file %1: ptype missing").arg(plugin));
            lua_pop(m_L, 2);
            continue;
        }
        ptype = lua_tostring(m_L, -1);
        if (ptype != "text" && ptype != "report") {
            QMessageBox::critical(this, "Error", QString("Could not load file %1: ptype is not supported").arg(plugin));
            lua_pop(m_L, 2);
            continue;
        }
        lua_pop(m_L, 1);

        /* Get and check the plugin has a name*/
        lua_getfield(m_L, -1, "name");
        if (lua_isnil(m_L, -1)) {
            QMessageBox::critical(this, "Error", QString("Could not load file %1: name missing").arg(plugin));
            lua_pop(m_L, 2);
            continue;
        }
        pname = lua_tostring(m_L, -1);
        lua_pop(m_L, 1);

        /* Get the plugin version (optional attribute). */
        lua_getfield(m_L, -1, "pver");
        pver = lua_tostring(m_L, -1);
        lua_pop(m_L, 1);

        /* Set the loaded plugin to a global using it's name. */
        lua_setglobal(m_L, pname.toUtf8().constData());

        /* Create a plugins menu if necessary. */
        if (m_menuPlugins == NULL)
            m_menuPlugins = m_menuBar->addMenu("Plugins");

        /* Create any menus needed and put an action in the approprate menu. */
        if (!pluginInfo.contains(pname)) {
            if (ptype == "text") {
                if (m_menuPluginsText == NULL) {
                    m_menuPluginsText = m_menuPlugins->addMenu("Text");
                    connect(m_menuPluginsText, SIGNAL(triggered(QAction *)), this, SLOT(runPluginText(QAction *)));
                }
                m_menuPluginsText->addAction(pname);
            } else {
                /* ptype == "report" */
                if (m_menuPluginsReport == NULL) {
                    m_menuPluginsReport = m_menuPlugins->addMenu("Report");
                    connect(m_menuPluginsReport, SIGNAL(triggered(QAction *)), this, SLOT(runPluginReport(QAction *)));
                }
                m_menuPluginsReport->addAction(pname);
            }
        }

        /* Insert the plugin name and version into a hahstable. */
        pluginInfo.insert(pname, pver);
    }

    /* Create a global table with name = version of loaded plugins. */
    lua_createtable(m_L, 0, pluginInfo.size());
    Q_FOREACH (pname, pluginInfo.keys()) {
        lua_pushstring(m_L, pname.toUtf8().constData());
        lua_pushstring(m_L, pluginInfo.value(pname, "").toUtf8().constData());
        lua_settable(m_L, -3);
    }
    lua_setglobal(m_L, "plugins");
}

void MainWindow::unloadPlugins()
{
    if (m_menuPlugins != NULL) {
        m_menuPlugins->clear();
        m_menuBar->removeAction(m_menuPlugins->menuAction());
        delete m_menuPlugins;
    }
    m_menuPlugins       = NULL;
    m_menuPluginsText   = NULL;
    m_menuPluginsReport = NULL;
    if (m_L != NULL)
        lua_close(m_L);
    m_L = NULL;
}

void MainWindow::infoPlugins()
{
    QStringList   p;
    ReportDialog  rd(this);
    lua_State    *L;
    QString       pname;
    QString       pver;

    p << "<html><body><table>";
    p << "<tr><th>Name</th><th>Version</th></td>";

    /* Loop though each plugin in the plugins table and
     * pull out the name and version. */
    if (m_L != NULL) {
        L = lua_newthread(m_L);

        lua_getglobal(L, "plugins");
        lua_pushnil(L);
        while (lua_next(L, -2)) {
            if (!lua_isstring(L, -2)) {
                lua_pop(L, 1);
                continue;
            }
            pname = lua_tostring(L, -2);

            if (lua_isstring(L, -1)) {
                pver = lua_tostring(L, -1);
            } else {
                pver = "?";
            }

            p << QString("<tr><td>%1</td><td>%2</td></tr>").arg(pname).arg(pver);

            lua_pop(L, 1);
        }

        lua_settop(L, 0);
        lua_pop(m_L, 1);
    }

    p << "</table></body></html>";

    rd.setText(p.join("\n"));
    rd.exec();
}

void MainWindow::runPluginText(QAction *action)
{
    lua_State  *L;
    QString     text;
    QTextCursor tc;
    int         pos;

    L = lua_newthread(m_L);

    /* Get the plugin. */
    lua_getglobal(L, action->text().toUtf8().constData());
    if (lua_isnil(L, -1)) {
        QMessageBox::critical(this, "Error", "Fatal: Could not run plugin");
        lua_pop(m_L, 1);
        return;
    }

    /* Get the text (selected or all if no selection). */
    tc = m_edit->textCursor();
    if (tc.hasSelection()) {
        text = tc.selectedText();
    } else {
        text = m_edit->toPlainText();
        pos  = tc.position();
    }

    /* Run the plugin's run function providing it with the text. */
    lua_getfield(L, -1, "run");
    lua_pushstring(L, text.toUtf8().constData());
    if (lua_pcall(L, 1, LUA_MULTRET, 0) != 0) {
        QMessageBox::critical(this, "Error", QString("Fatal: Could not run plugin: %1").arg(lua_tostring(L, -1)));
        lua_pop(m_L, 1);
        return;
    }

    if (lua_gettop(L) == 3) {
        QMessageBox::critical(this, "Error", QString("Fatal: plugin failed: %1").arg(lua_tostring(L, -1)));
        lua_pop(m_L, 1);
        return;
    } else {
        /* Replace the text in the editor with the text returned from the run function. */
        if (tc.hasSelection()) {
            tc.insertText(lua_tostring(L, -1));
        } else {
            m_edit->setPlainText(lua_tostring(L, -1));
            tc.setPosition(pos);
        }
        m_edit->setTextCursor(tc);
    }

    lua_pop(m_L, 1);
}

void MainWindow::runPluginReport(QAction *action)
{
    lua_State   *L;
    ReportDialog rd(this);
    QStringList  p;

    L = lua_newthread(m_L);

    /* Get the plugin. */
    lua_getglobal(L, action->text().toUtf8().constData());
    if (lua_isnil(L, -1)) {
        QMessageBox::critical(this, "Error", "Fatal: Could not run plugin");
        lua_pop(m_L, 1);
        return;
    }

    /* Run the plugin's run function using the text from the editor as its argument. */
    lua_getfield(L, -1, "run");
    lua_pushstring(L, m_edit->toPlainText().toUtf8().constData());
    if (lua_pcall(L, 1, LUA_MULTRET, 0) != 0) {
        QMessageBox::critical(this, "Error", QString("Fatal: Could not run plugin: %1").arg(lua_tostring(L, -1)));
        lua_pop(m_L, 1);
        return;
    }

    p << "<html><body>";
    p << lua_tostring(L, -1);
    p << "</body></html>";

    /* Show the text returned by the plugin. */
    rd.setText(p.join("\n"));
    rd.exec();

    lua_pop(m_L, 1);
}

void MainWindow::createMenu()
{
    m_menuActions = m_menuBar->addMenu("Actions");
    m_menuActions->addAction("Load", this, SLOT(loadPlugins()));
    m_menuActions->addAction("Unload", this, SLOT(unloadPlugins()));
    m_menuActions->addAction("Info", this, SLOT(infoPlugins()));
}

The MainWindow Explained

loadPlugins

Before loading any plugins we unload them all. A new lua_State (m_L) is created and the standard libraries are loaded. All plugins will be loaded into and use this single state. This allows plugins to interact with one another.

One way the plugin system could be extended is to add “library” plugins which are plugins that aren’t run directly but provide functionality for other plugins. This would allow plugins to be used in a restricted environment. Such as disallowing the use of require and instead forcing libraries to be loaded as “library” plugins.

Each plugin could be created and assigned to its own independent state as well. This would allow a level of protection and separation. A plugin for example couldn’t modify the global state in a malicious or unexpected way that would negatively impact other plugins.

The basic sequence for loading plugins is:

  1. Iterate over all files in the plugins directory and filter on files that end with .lua.
  2. Verify the Lua file is a plugin based on presence of attributes such as name and ptype.
  3. Add the plugin to the state as a global.
  4. Add the plugin to the appropriate menu.
  5. Add the plugin name and version to a global plugins table.

unloadPlugins

Unloading the plugins is very simple. Remove/destroy the menu entries then destroy the lua_State that holds our loaded plugins.

Being able to unload (and then reload) plugins is useful because it allows you to add plugins after the application has started. It also makes developing plugins easier because you don’t have to restart the application itself to test changes to plugins.

Using lua_newthread

The infoPlugins function and the runPLugin* functions use lua_newthread to create a separate lua_State. This essentially creates a coroutine via the C API. We’re not going to actually use the features of a coroutine. What we’re doing is using this to provide a clean and independent stack. States created with lua_newthread share the global environment with the state they were created from but have their own stack. This means we don’t have to worry about the state of the main stack. This is completely optional as we could just use the main state.

When using lua_newthread you’ll notice lua_close is never called using the state returned by lua_newthread. This is correct; you can’t use lua_close with a state created via lua_newthread. This state is owned by Lua and will be garbage collected when necessary.

In all functions where lua_newthread is used the thread state is removed from the main stack after all processing is completed. We can’t leave it on the main stack otherwise we’d A) eventually run out of stack slots and B) the thread would never be garbage collected because it would never go out of use.

infoPlugins

This function generates a report of loaded plugins and their version. It does so by reading the global “plugins” table created during plugin loading. The report dialog is leveraged to display this information.

runPlugin*

All of the runPlugin* functions take text from the editor as an argument. These plugins don’t necessarily need to work in this way. Right now the plugin design is for plugins to be provided data as an argument (text only at this point). Since the current plugins only do text processing this design suffices.

A more robust plugin system would could allow plugins to access specific parts of the application instead of relying on limited data via arguments. Look at my “Using Lua as a Templating Engine” post for details on doing this.

Building

CMakeLists.txt

cmake_minimum_required (VERSION 3.0)
project (lua_plugin CXX)

find_package(Lua REQUIRED)
find_package(Qt5 COMPONENTS Core Widgets)
set(CMAKE_AUTOMOC ON)

include_directories (
    ${CMAKE_CURRENT_BINARY_DIR}
    ${CMAKE_CURRENT_SOURCE_DIR}
    ${LUA_INCLUDE_DIR}
)

set (SOURCES
    main.cpp
    mainwindow.cpp
    reportdialog.cpp
)

add_executable (${PROJECT_NAME} ${SOURCES} ${LUA_LIBRARIES})
target_link_libraries (${PROJECT_NAME} lua)
qt5_use_modules(${PROJECT_NAME} Widgets)

Build and run

$ mkdir build
$ cd build
$ cmake ..
$ make
$ cp lua_plugin ..
$ cd ..
$ ./lua_plugin

Load the plugins, add some text to the text edit, and try running some or all of the plugins.