John's Blog

Most this week was spent turning PML input and output. I spent a bit of work bug tracking and enhancing FB2 output as well.

The changes for PML input are as follows. Pass along the included cover as the cover when converting (also applies to eReader PDB). Allow for images to be in top level, archivename_img or images directory for PMLZ. Based on that order it will check for images and if they are not found move onto the next location. For PML, images can be in pmlname_img or images directory. Footnotes and sidebars now display cleaner. They are separated better and EPUB puts them on individual pages. They also include a return link which goes back to the place in the text they are referenced. This assumes one footnote and sidebar per entry in the text, so if it’s referenced multiple times the return link will go back to the return reference.

PML output now creates \a and \U codes only for supported characters. All characters that are not supported and that cannot be turned into a \a or \U code will be replaced with a ?.

Along with the changes for PML input reading the cover they are now read as part of the metadata. This applies to both PML, PMLZ and eReader PDB files.

I’ve created a PML2PMLZ FileType plugin which will run when ever PML is imported into the GUI. It takes a PML file looks for images in the above mentioned locations, takes it all and puts it into a PMLZ archive. The PMLZ archive is them added to the library.

When I went to test the PML2PMLZ plugin I found that the GUI on my system was horribly broken. After a bit of work with Kovid, I found that calibre-parallel had to be in the path if calibre was installed in a non standard location. I install into my home directory using the develop command. Kovid has committed a fix that writes the install path to the launcher for these instances.

FB2 output now turns h1 tags into <section><title> tags to allow for TOC generation. As far as I can tell FB2 has not set TOC and instead readers dynamically generate the TOC based on looking at all of the body and sections and sets the text using the title tag. Right now the FB2 output is limited to only turning h1 tags and cannot use the user defined TOC based on an XPATH expression. I plan to fix this limitation in the future.

Tags: calibre, eReader, fb2, GUI, pdb, pml, pmlz.

Robert B, who is Astak’s Director of Bus. Devl., posted a blog entry about user replaceable batteries. I mostly agree with him that they are a benefit to consumer electronics. I mostly agree because I don’t see them as a positive in every case. I posted the following on his blog as a comment but I wanted to post it here as well. This is in response to the statement that reviewers don’t get the idea of user replaceable batteries.

It’s not that most reviewers do not get the idea of a user replaceable battery, it’s that it really isn’t a selling point to most people. There are three reasons I can think of as to why a user replaceable batter does not matter.

1) Sealed in causes the device to be cheaper to produce and thus cheaper for the consumer. This leads into point two.

2) The device is not seen as a long term investment. This is very reminiscent of how Apple positions the iPod by inciting consumers to upgrade to the latest release. In one or two years the device will be replaced with a newer model. As someone who is looking to buy my third ebook reader for the third year in a row I haven’t had to worry about the battery wearing out and needing to be replaced.

3) Worries of availability. While it is very easy to buy a spare battery now what about in 5 years from now. Chances are the product will not longer be produced as the company has moved on to better and cheaper technology. 5 years from now obtaining a replacement battery can easily be impossible or cost prohibitive.

Tags: batteries, ebook, electronics, gadgets.

I spent the past week fixing as many bugs with the new PML input parser and cleaning up as much of the output as I could. I really need to thank WayneD for helping find bugs with the new code. Also, Kevin Hendricks who has been working on his own parser based on code from a tool that does some work on eReader files. He helped me formulate what output should be derived in certain cases. The new PML input parser has been released as part of calibre 0.6.25.

Tags: calibre, eReader, pml.

PML input had some major changes this week. Thank the user WayneD for helping me out and getting me to actually do the work I’ve been putting off since I introduced PML/eReader as an input format.

There is now a metadata reader for PML and PMLZ. WayneD provided me with a set of regular expressions that can extract the metadata from a metatdata comment within a PML document. I took those regexes and created a metatdata plugin that supports both straight PML files as well as the PMLZ archive file.

The other major change to PML is, I’ve re-written the input parser. It is not longer based on a set of regular expressions. It is now a line oriented simple state machine. When I created the regex parser I intended to replace it at some point in the future with a true parser. The regex based one was simply a quick and dirty way to get PML supported. The new parser is much faster, produces cleaner and more accurate HTML output. It also has the added benefit of reading \CX codes and turns them into table of contents entries for PML and PMLZ input. The new parser is much better and I’m not completely finished with it. I still need to add support for \v comments (they are currently removed), \n codes, and implement font attribute tracking to condense changes (this is how \n will be handled).

WayneD did provide me with his Perl based line oriented simple state machine for PML to HTML conversion. I did use one idea from it. Turning footnote and sidebar xml syntax into custom PML tags. I had intended to port his parser to python and use it as a base but when I started looking at it I remembered I don’t know Perl at all and I can’t make heads or tails of Perl code. I have no desire or need to actually learn Perl, so I ended up writing my own parser.

Tags: calibre, eReader, pml, pmlz.

Large parts of the application have been restructured. For instance QtSingleApplication is now being used. The major new feature is the window manager decoration close button (X at the top right) can now optionally iconify the window when clicked. The -c command line option has also changed. It no longer creates a borderless window it now disables the iconify behavior of the decoration close button.

Note: There is one known issue with this release. Drag and Drop is partly broken for docked windows.

Tags: KDocker.

In my X11 Intercept Window Close Event post I left out one very important piece of information. How to actually close the embedded window after intercepting the WM_DELETE_WINDOW message. The best thing to do is send our own WM_DELETE_WINDOW message to the embedded window. This will keep the embedded window embedded until it actually closes. Meaning any question dialogs that might stop the window from closing (Gedit asking to save for instance) can be answered and it keeps the window embedded if the user decides not to close.

In the previous post we overrode the WM_DELETE_WINDOW message. All we need to do is send it on to the embedded window. This will keep it embedded until it actually closes. Sending a this message entails setting the type to ClinetMessage, message type to the WM_PROTOCOLS Atom, the first l data value to the WM_DELETE_WINDOW Atom, and the second l data value to CurrentTime. The format of course if 32. Send the message to the embedded window and it will try to close.

#include <QX11Info>
#include <X11/XLib.h>
 
Display *display = QX11Info::display();
XEvent ev;
 
memset(&ev, 0, sizeof (ev));
 
ev.xclient.type = ClientMessage;
ev.xclient.window = window;
ev.xclient.message_type = XInternAtom(display, "WM_PROTOCOLS", true);
ev.xclient.format = 32;
ev.xclient.data.l[0] = XInternAtom(display, "WM_DELETE_WINDOW", false);
ev.xclient.data.l[1] = CurrentTime;
XSendEvent(display, window, False, NoEventMask, &ev);

Tags: c++, KDocker, x11, xlib.

***Note Nov, 8 2009: a few additions have been made to the code samples to make them more complete. Specifically subscribing to X11 events.

A feature request for KDocker was made a few days ago for docking when closed. Basically the person wants the window to iconify when the X button on the window decoration is clicked. It’s something I’ve been thinking about for quite some time but it’s not the easiest feature to implement. Girish thankfully pointed me onto the right path when he suggested looking into using a QX11EmbedContainer. I’ve gotten it working and it will be in the 4.3 release.

There isn’t a whole lot of documentation out there on achieving this so I’m going to detail how I’ve implemented the feature in KDocker. Do be awhare that there are a few short comings. Versions <= 4.2 will iconify all windows associated with the docked one. Meaning Gimp and Audacious (XMMS) which have multiple windows will not have the extra windows iconify when the main one is. Also, decorationless windows like audacious (possibly Google Chrome) are no longer movable by clicking on the top of the window. Alt+Left Mouse Click still works and is currently the only way to move them.

The basic overview of the implementation is: Take the user started application window and get info about it (title, size, location, window decorations …). Create a container window. Use XEmbed to put the window into the container. Set the container properties to those of the window. Intercept the WM_DELETE_WINDOW message. Forward other changes like title and icon from the window to the container.

Getting info about the window requires Xlib calls. I’m using the following to get the minimum size, current size, position, width, height and decorations. window is the window id for the application window. The decoration code requires mwmutil.h which can be found in libmotif and LessTif.

Display *display = QX11Info::display();
XSizeHints sizeHint;
long dummy;
Window root;
int x, y;
unsigned int width, height, border, depth;
 
XGetWMNormalHints(display, window, &sizeHint, &dummy);
XGetGeometry(display, window, &root, &x, &y, &width, &height, &border, &depth);
 
Atom wm_hints_atom = XInternAtom(display, _XA_MOTIF_WM_HINTS, false);
unsigned char *wm_data;
Atom wm_type;
int wm_format;
unsigned long wm_nitems, wm_bytes_after;
 
XGetWindowProperty(display, window, wm_hints_atom, 0, sizeof (MotifWmHints) / sizeof (long), false, AnyPropertyType, &wm_type, &wm_format, &wm_nitems, &wm_bytes_after, &wm_data);

Next create the container. I’m using QX11EmbedContainer because it is is a Qt provided XEmbed container widget.

QX11EmbedContainer *container = new QX11EmbedContainer();
container->embedClient(window);
container->show();

We need to tell X that we want to receive certain events about the container and embedded window. If we don’t set the appropriate masks the events we are interested in will never be sent to our X11EventFilter.

long mask_container = StructureNotifyMask | PropertyChangeMask | VisibilityChangeMask | FocusChangeMask;
long mask_embed = PropertyChangeMask;
 
XWindowAttributes attr_container;
XWindowAttributes attr_embed;
 
XGetWindowAttributes(display, m_container->winId(), &attr_container);
XGetWindowAttributes(display, window, &attr_embed);
 
if ((attr_container.your_event_mask & mask_container) != mask_container)) {
    XSelectInput(display, m_container->winId(), attr_container.your_event_mask | mask_container);
}
if ((attr_container.your_event_mask & mask_embed) != mask_embed)) {
    XSelectInput(display, window, attr_embed.your_event_mask | mask_embed);
}

Now we want to register WM_DELETE_WINDOW so that instead of closing it will be directed to us as a client message. We will check for the message in the X11EventFilter and take appropriate action. This will only work because we have reparented the window by embedding it into the container we control. Simply setting this on the window will not work.

Atom wm_delete = XInternAtom(display, "WM_DELETE_WINDOW", False);
XSetWMProtocols(display, container->winId(), &wm_delete, 1);

Take all the info we obtained from the window and apply it to our container. This way it will look and behave the same as if it was not embedded.

m_container->setMinimumSize(m_sizeHint.min_width, m_sizeHint.min_height);
m_container->setGeometry(x, y, width, height);
 
if (wm_type == None) {
    MotifWmHints hints;
    memset(&hints, 0, sizeof (hints));
    hints.flags = MWM_HINTS_DECORATIONS;
    hints.decorations = MWM_DECOR_ALL;
    XChangeProperty(display, m_container->winId(), wm_hints_atom, wm_hints_atom, 32, PropModeReplace, (unsigned char *) & hints, sizeof (MotifWmHints) / sizeof (long));
} else {
    MotifWmHints *hints;
    hints = (MotifWmHints *) wm_data;
    if (!(hints->flags & MWM_HINTS_DECORATIONS)) {
        hints->flags |= MWM_HINTS_DECORATIONS;
        hints->decorations = 0;
    }
    XChangeProperty(display, m_container->winId(), wm_hints_atom, wm_hints_atom, 32, PropModeReplace, (unsigned char *) hints, sizeof (MotifWmHints) / sizeof (long));
}
XFree(wm_data);

Thats it for constructing the window but there are a few important pieces we still need to worry about. Within QApplication’s x11EventFilter we need to processes some events.

bool x11EventFilter(XEvent *ev) {
    XAnyEvent *event = (XAnyEvent *) ev;
    if (event->window == m_container->winId()) {
        if (ev->type == ClientMessage && (ulong) ev->xclient.data.l[0] == XInternAtom(QX11Info::display(), "WM_DELETE_WINDOW", False)) {
            if (m_iconifyOnClose) {
                iconifyWindow();
            } else {
                close();
            }
            return true;
        }
    }
 
    if (event->window == window) {
        if (event->type == PropertyNotify) {
            return propertyChangeEvent(((XPropertyEvent *) event)->atom);
        }
    }
    return false
}
 
bool propertyChangeEvent(Atom property) {
    Display *display = QX11Info::display();
    static Atom WM_NAME = XInternAtom(display, "WM_NAME", True);
    if (property == WM_NAME) {
        updateTitle();
        return true;
    }
}
 
void updateTitle() {
    Display *display = QX11Info::display();
    char *windowName = 0;
    QString title;
 
    XFetchName(display, window, &windowName);
    title = windowName;
    if (windowName) {
        XFree(windowName);
    }
 
    container->setWindowTitle(title);
}

Tags: c++, KDocker, x11, xlib.

A while back I made a post about ASCIIizing Text. With it was a simple python application that would convert Unicode characters to ASCII equivalents. It doesn’t do a basic conversion but also Latinizes the characters when they are outside of the ASCII range.

The uni2ascii package I made has a few short comings I’ve decided to fix. The three major problems with it are: 1) Very basic permission checking, 2) Only accepts one file, 3) Required all input to be UTF8 encoded, 4) The decoder was a very literal port of a the ruby version.

To fix these issues I’ve written an entirely new script. Problems 1, 2 and 3 are fixed. It has robust error checking, can handle an arbitrary number of files, and the file encoding can be specified. Number 4 is fixed by using the Python port created by Tomaz Solc.

I’ve put the source code for the new decoder into a Launchpad branch:

$ bzr branch lp:~user-none/+junk/unidecoder

Tags: ebook, python, txt.

Mostly bug fixes this week. The majority of them were centered around eReader PDB output and PML generation. eReader PDB output now marks the first image as the cover image if a cover image is not explicitly set. PMLZ got images named properly in the output. PML generation now has .png added to the end of image names. I also fixed a bug where excessive new lines were not being properly removed. PML, TXT, RB, FB2 output all got excessive space removal tones down so instances were spaces were completely removed will stop happening. Regex header and footer matching was tweaked to match at a later stage in the conversion pipeline. This should ease issues of expressions not matching properly. Finally, at Kovid’s request I’ve added some info about header / footer regexes and converting TXT and PDF files to the documentation.

Tags: calibre, eReader, fb2, pdb, pml, pmlz, rb, txt.

Like every week there were miscellaneous bug fixes. However, this week I did a bit more. TCR input and output. Do be warned that the output supports multiple compression levels; the higher levels being slower than the lower. For instance a 200K TXT file as input will take around 25 seconds on the lowest level and 3.5 minutes at the highest.

TCR is an compressed text format used mainly by the Psion 3 and 5 series PDAs that were produced in the 90s. The compression used by TCR files is very interesting. It doesn’t have as high a compression ratio as say zlib but that is a trade off for being decompressable starting at any point in the stream. The history and more information about the format can be found at Andrew Giddings’ TCR page.

Tags: calibre, ebook, tcr.