John's Blog

I spent a bit of time working on calibre this week. I worked on profiles, devices and a bug fix here and there.

A few new profiles were added. Specifically profiles for the Sony PRS 300 and 900. They have a different screen size and resolution than the 6″ models so they warranted their own profiles.

I added support for two new devices. The Airis Dbook and the Binatone Readme. Along with supporting new devices I also reorganized and renamed a few. The BeBook device interface is not called Hanlin. The BeBook is a rebraned Hanlin. The EZReader, LBook, Eco Reader are also rebranded Hanlins. They all were using the BeBook interface. So I renamed it to be a little more generic and avoid confusion.

One big bug fix this week. There was a typo in the tag map for PML output. It causes italics to be ignore. This has been corrected.

Tags: calibre, device interfaces, pml, prs300, prs900.

Not much on the calibre front for this past week from me. The only thing I’ve worked on was adding support to the Nook driver for the cover image to be sent to the device with the book. Also, if there is no cover associated with the book a default cover with the title authors and calibre library image are used. Similar to what already happens with Cybook Gen 3 and Opus.

Tags: calibre, cover, cybook, ebook, nook.

FB2 output has been improved. It no longer generated very invalid markup. The output generator still isn’t where I want it to be though. The changes are mostly cleanup and fix long standing issues with the output. One major change is I reverted having <h1> tags work as section and title markers. I don’t like having this hard coded into the generator.

As far as I can tell FB2 does not support a true table of contents (TOC). What seems to happen is reader software will dynamically generate the TOC based on the appearance of <section><title><p>text</p></title> within th e text. If I’m wrong about this and FB2 really does support an external TOC I would love to know. The <h1> differentiation causes the files to have these sections. I do not like how this is hard coded and dependent on this single tag. Especially since calibre’s conversion process allows for an XPath expression to be specified to generate TOC point.

I would love to use the TOC sent to the FB2 output generator but this does not seem to be feasible. The problem with the TOC that is sent to the FB2 output generator is how it corresponds to locations in the document. The OEB TOC points are text which may or my not appear in the text and an anchor id. The anchor id is a set point in the document. FB2 section titles are part of the text itself. I cannot use the text from the OEB TOC because it may or may not actually appear in the text. This also prevents me from determining what the text in the document is supposed to be associated with the TOC point. The anchor id points to an anchor in the document but often that point is something like <a id=”blah” />. In this case there is no text associated with the anchor in the document. While I can assume the text following is part of the title I have no fool proof way to determine where it stops.

At this point the only TOC associated with the FB2 output is the inline TOC that can be optionally generated.

Support for two new readers has been added. Ganaxa’s GeR2 and Nokia’s 770 internet tablet. The N770 should have been support a long time ago and I apologize for how long this request has been unfulfilled. I put it at the bottom of my todo list and at some point it simply fell off and I forgot about it.

The GeR2 reader was a bit of a challenge to get supported. This reader and some models of the Cybook Gen 3 have the same vendor, product and revision (BCD) ids. On Windows and OS X this is not an issue because once the ids are matched further matches are done based on the plug and play (PNP) strings. However, on Linux only the ids are matched.

To solve this problem, matching on Linux needed some further checks. Kovid added support for libusb-1 which provides the vendor and product strings. He also added a call back that can be implemented in the device interface to implement platform and device specific checks. We did run into a few problems. The first was an easy to solve 32 vs 64 bit issue with the Python to C interface Kovid wrote for libusb-1. Once that was sorted out we ran into a larger problem. libusb-1 on Ubuntu by default is denied access to the vendor and product strings.

libusb-1, after appearing in only run release (0.6.27), has been dropped. Kovid has now written a custom device scanner for Linux that will parser the devices in /sys/bus/usb to determine if a reader is connected. libusb-1 is supposed to be an easy to use library capable of providing this functionality but unfortunately this turned out not to be the case. The custom scanner works and allowed me to implement differentiation between the GeR2 and the Cybook Gen 3 so both readers can be properly supported without conflict and with the correct device interface being used.

Tags: calibre, cybook, device interfaces, fb2, GeR2, Linux, N770, TOC.

Most this week was spent turning PML input and output. I spent a bit of work bug tracking and enhancing FB2 output as well.

The changes for PML input are as follows. Pass along the included cover as the cover when converting (also applies to eReader PDB). Allow for images to be in top level, archivename_img or images directory for PMLZ. Based on that order it will check for images and if they are not found move onto the next location. For PML, images can be in pmlname_img or images directory. Footnotes and sidebars now display cleaner. They are separated better and EPUB puts them on individual pages. They also include a return link which goes back to the place in the text they are referenced. This assumes one footnote and sidebar per entry in the text, so if it’s referenced multiple times the return link will go back to the return reference.

PML output now creates \a and \U codes only for supported characters. All characters that are not supported and that cannot be turned into a \a or \U code will be replaced with a ?.

Along with the changes for PML input reading the cover they are now read as part of the metadata. This applies to both PML, PMLZ and eReader PDB files.

I’ve created a PML2PMLZ FileType plugin which will run when ever PML is imported into the GUI. It takes a PML file looks for images in the above mentioned locations, takes it all and puts it into a PMLZ archive. The PMLZ archive is them added to the library.

When I went to test the PML2PMLZ plugin I found that the GUI on my system was horribly broken. After a bit of work with Kovid, I found that calibre-parallel had to be in the path if calibre was installed in a non standard location. I install into my home directory using the develop command. Kovid has committed a fix that writes the install path to the launcher for these instances.

FB2 output now turns h1 tags into <section><title> tags to allow for TOC generation. As far as I can tell FB2 has not set TOC and instead readers dynamically generate the TOC based on looking at all of the body and sections and sets the text using the title tag. Right now the FB2 output is limited to only turning h1 tags and cannot use the user defined TOC based on an XPATH expression. I plan to fix this limitation in the future.

Tags: calibre, eReader, fb2, GUI, pdb, pml, pmlz.

Robert B, who is Astak’s Director of Bus. Devl., posted a blog entry about user replaceable batteries. I mostly agree with him that they are a benefit to consumer electronics. I mostly agree because I don’t see them as a positive in every case. I posted the following on his blog as a comment but I wanted to post it here as well. This is in response to the statement that reviewers don’t get the idea of user replaceable batteries.

It’s not that most reviewers do not get the idea of a user replaceable battery, it’s that it really isn’t a selling point to most people. There are three reasons I can think of as to why a user replaceable batter does not matter.

1) Sealed in causes the device to be cheaper to produce and thus cheaper for the consumer. This leads into point two.

2) The device is not seen as a long term investment. This is very reminiscent of how Apple positions the iPod by inciting consumers to upgrade to the latest release. In one or two years the device will be replaced with a newer model. As someone who is looking to buy my third ebook reader for the third year in a row I haven’t had to worry about the battery wearing out and needing to be replaced.

3) Worries of availability. While it is very easy to buy a spare battery now what about in 5 years from now. Chances are the product will not longer be produced as the company has moved on to better and cheaper technology. 5 years from now obtaining a replacement battery can easily be impossible or cost prohibitive.

Tags: batteries, ebook, electronics, gadgets.

I spent the past week fixing as many bugs with the new PML input parser and cleaning up as much of the output as I could. I really need to thank WayneD for helping find bugs with the new code. Also, Kevin Hendricks who has been working on his own parser based on code from a tool that does some work on eReader files. He helped me formulate what output should be derived in certain cases. The new PML input parser has been released as part of calibre 0.6.25.

Tags: calibre, eReader, pml.

PML input had some major changes this week. Thank the user WayneD for helping me out and getting me to actually do the work I’ve been putting off since I introduced PML/eReader as an input format.

There is now a metadata reader for PML and PMLZ. WayneD provided me with a set of regular expressions that can extract the metadata from a metatdata comment within a PML document. I took those regexes and created a metatdata plugin that supports both straight PML files as well as the PMLZ archive file.

The other major change to PML is, I’ve re-written the input parser. It is not longer based on a set of regular expressions. It is now a line oriented simple state machine. When I created the regex parser I intended to replace it at some point in the future with a true parser. The regex based one was simply a quick and dirty way to get PML supported. The new parser is much faster, produces cleaner and more accurate HTML output. It also has the added benefit of reading \CX codes and turns them into table of contents entries for PML and PMLZ input. The new parser is much better and I’m not completely finished with it. I still need to add support for \v comments (they are currently removed), \n codes, and implement font attribute tracking to condense changes (this is how \n will be handled).

WayneD did provide me with his Perl based line oriented simple state machine for PML to HTML conversion. I did use one idea from it. Turning footnote and sidebar xml syntax into custom PML tags. I had intended to port his parser to python and use it as a base but when I started looking at it I remembered I don’t know Perl at all and I can’t make heads or tails of Perl code. I have no desire or need to actually learn Perl, so I ended up writing my own parser.

Tags: calibre, eReader, pml, pmlz.

Large parts of the application have been restructured. For instance QtSingleApplication is now being used. The major new feature is the window manager decoration close button (X at the top right) can now optionally iconify the window when clicked. The -c command line option has also changed. It no longer creates a borderless window it now disables the iconify behavior of the decoration close button.

Note: There is one known issue with this release. Drag and Drop is partly broken for docked windows.

Tags: KDocker.

In my X11 Intercept Window Close Event post I left out one very important piece of information. How to actually close the embedded window after intercepting the WM_DELETE_WINDOW message. The best thing to do is send our own WM_DELETE_WINDOW message to the embedded window. This will keep the embedded window embedded until it actually closes. Meaning any question dialogs that might stop the window from closing (Gedit asking to save for instance) can be answered and it keeps the window embedded if the user decides not to close.

In the previous post we overrode the WM_DELETE_WINDOW message. All we need to do is send it on to the embedded window. This will keep it embedded until it actually closes. Sending a this message entails setting the type to ClinetMessage, message type to the WM_PROTOCOLS Atom, the first l data value to the WM_DELETE_WINDOW Atom, and the second l data value to CurrentTime. The format of course if 32. Send the message to the embedded window and it will try to close.

#include
#include 

Display *display = QX11Info::display();
XEvent ev;

memset(&ev, 0, sizeof (ev));

ev.xclient.type = ClientMessage;
ev.xclient.window = window;
ev.xclient.message_type = XInternAtom(display, "WM_PROTOCOLS", true);
ev.xclient.format = 32;
ev.xclient.data.l[0] = XInternAtom(display, "WM_DELETE_WINDOW", false);
ev.xclient.data.l[1] = CurrentTime;
XSendEvent(display, window, False, NoEventMask, &ev);

Tags: c++, KDocker, x11, xlib.

***Note Nov, 8 2009: a few additions have been made to the code samples to make them more complete. Specifically subscribing to X11 events.

A feature request for KDocker was made a few days ago for docking when closed. Basically the person wants the window to iconify when the X button on the window decoration is clicked. It’s something I’ve been thinking about for quite some time but it’s not the easiest feature to implement. Girish thankfully pointed me onto the right path when he suggested looking into using a QX11EmbedContainer. I’ve gotten it working and it will be in the 4.3 release.

There isn’t a whole lot of documentation out there on achieving this so I’m going to detail how I’ve implemented the feature in KDocker. Do be awhare that there are a few short comings. Versions <= 4.2 will iconify all windows associated with the docked one. Meaning Gimp and Audacious (XMMS) which have multiple windows will not have the extra windows iconify when the main one is. Also, decorationless windows like audacious (possibly Google Chrome) are no longer movable by clicking on the top of the window. Alt+Left Mouse Click still works and is currently the only way to move them.

The basic overview of the implementation is: Take the user started application window and get info about it (title, size, location, window decorations …). Create a container window. Use XEmbed to put the window into the container. Set the container properties to those of the window. Intercept the WM_DELETE_WINDOW message. Forward other changes like title and icon from the window to the container.

Getting info about the window requires Xlib calls. I’m using the following to get the minimum size, current size, position, width, height and decorations. window is the window id for the application window. The decoration code requires mwmutil.h which can be found in libmotif and LessTif.

Display *display = QX11Info::display();
XSizeHints sizeHint;
long dummy;
Window root;
int x, y;
unsigned int width, height, border, depth;

XGetWMNormalHints(display, window, &sizeHint, &dummy);
XGetGeometry(display, window, &root, &x, &y, &width, &height, &border, &depth);

Atom wm_hints_atom = XInternAtom(display, _XA_MOTIF_WM_HINTS, false);
unsigned char *wm_data;
Atom wm_type;
int wm_format;
unsigned long wm_nitems, wm_bytes_after;

XGetWindowProperty(display, window, wm_hints_atom, 0, sizeof (MotifWmHints) / sizeof (long), false, AnyPropertyType, &wm_type, &wm_format, &wm_nitems, &wm_bytes_after, &wm_data);

Next create the container. I’m using QX11EmbedContainer because it is is a Qt provided XEmbed container widget.

QX11EmbedContainer *container = new QX11EmbedContainer();
container->embedClient(window);
container->show();

We need to tell X that we want to receive certain events about the container and embedded window. If we don’t set the appropriate masks the events we are interested in will never be sent to our X11EventFilter.

long mask_container = StructureNotifyMask | PropertyChangeMask | VisibilityChangeMask | FocusChangeMask;
long mask_embed = PropertyChangeMask;

XWindowAttributes attr_container;
XWindowAttributes attr_embed;

XGetWindowAttributes(display, m_container->winId(), &attr_container);
XGetWindowAttributes(display, window, &attr_embed);

if ((attr_container.your_event_mask & mask_container) != mask_container)) {
    XSelectInput(display, m_container->winId(), attr_container.your_event_mask | mask_container);
}
if ((attr_container.your_event_mask & mask_embed) != mask_embed)) {
    XSelectInput(display, window, attr_embed.your_event_mask | mask_embed);
}

Now we want to register WM_DELETE_WINDOW so that instead of closing it will be directed to us as a client message. We will check for the message in the X11EventFilter and take appropriate action. This will only work because we have reparented the window by embedding it into the container we control. Simply setting this on the window will not work.

Atom wm_delete = XInternAtom(display, "WM_DELETE_WINDOW", False);
XSetWMProtocols(display, container->winId(), &wm_delete, 1);

Take all the info we obtained from the window and apply it to our container. This way it will look and behave the same as if it was not embedded.

m_container->setMinimumSize(m_sizeHint.min_width, m_sizeHint.min_height);
m_container->setGeometry(x, y, width, height);

if (wm_type == None) {
    MotifWmHints hints;
    memset(&hints, 0, sizeof (hints));
    hints.flags = MWM_HINTS_DECORATIONS;
    hints.decorations = MWM_DECOR_ALL;
    XChangeProperty(display, m_container->winId(), wm_hints_atom, wm_hints_atom, 32, PropModeReplace, (unsigned char *) & hints, sizeof (MotifWmHints) / sizeof (long));
} else {
    MotifWmHints *hints;
    hints = (MotifWmHints *) wm_data;
    if (!(hints->flags & MWM_HINTS_DECORATIONS)) {
        hints->flags |= MWM_HINTS_DECORATIONS;
        hints->decorations = 0;
    }
    XChangeProperty(display, m_container->winId(), wm_hints_atom, wm_hints_atom, 32, PropModeReplace, (unsigned char *) hints, sizeof (MotifWmHints) / sizeof (long));
}
XFree(wm_data);

Thats it for constructing the window but there are a few important pieces we still need to worry about. Within QApplication’s x11EventFilter we need to processes some events.

bool x11EventFilter(XEvent *ev) {
    XAnyEvent *event = (XAnyEvent *) ev;
    if (event->window == m_container->winId()) {
        if (ev->type == ClientMessage && (ulong) ev->xclient.data.l[0] == XInternAtom(QX11Info::display(), "WM_DELETE_WINDOW", False)) {
            if (m_iconifyOnClose) {
                iconifyWindow();
            } else {
                close();
            }
            return true;
        }
    }

    if (event->window == window) {
        if (event->type == PropertyNotify) {
            return propertyChangeEvent(((XPropertyEvent *) event)->atom);
        }
    }
    return false
}

bool propertyChangeEvent(Atom property) {
    Display *display = QX11Info::display();
    static Atom WM_NAME = XInternAtom(display, "WM_NAME", True);
    if (property == WM_NAME) {
        updateTitle();
        return true;
    }
}

void updateTitle() {
    Display *display = QX11Info::display();
    char *windowName = 0;
    QString title;

    XFetchName(display, window, &windowName);
    title = windowName;
    if (windowName) {
        XFree(windowName);
    }

    container->setWindowTitle(title);
}

Tags: c++, KDocker, x11, xlib.