While it’s not coming out today I have been working on a major feature. Metadata caching for devices. Basically a file that specifies all of the metadata for the books on the reader will be created. It will be used when the device is connected and if any files have changed their metadata will be updated. This speeds up device detection dramatically. Also, takes up less space than a book so it won’t use too much space to store it. Right now it’s storing the data using json so other application can have easy access to the data too. Other than faster device detection this is the ground work that is required to implement syncing.

There were two reasons for combining the interfaces. I was unable to find a way to differentiate the Gen 3 and the Opus with the 2.0 firmware. A Gen 3 with the 2.0 firmware reports itself as an Opus. I was unable to find a way around this and felt it was better to have a Cybook interface. Also, I have the unfortunate pleasure of owning a Gen 3 which, even before the 2.0 firmware, used the same device ids of the Opus. It however did not use the Opus device strings. Due to this I am unable to determine if the 2.0 firmware changes the ids or just the strings. Combining the two interfaces keeps all Gen 3′s working and keeps some Gen 3′s from being detected as the Opus.

$ calibre-debug -d
Version: 0.6.32
USB devices on system:
...
['0xbda', '0x703', '0x110', 'Bookeen', 'Cybook Opus', '600AB038CB09484'],
...
Looking for CYBOOKG3
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Looking for CYBOOK_OPUS
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Devices possibly connected: Cybook Opus Device Interface,
Trying to open Cybook Opus Device Interface ... OK
Main memory: '/media/Cybook Gen3/'
Total space: (495251456L, 0, 0)

It seems Bookeen has once again screwed up the firmware release. They wanted one firmware so now every Gen 3 that has updated now has the product identifier set to Opus. The Gen 3 models use a number of different hex product and vendor ids and only the later ones are the same as the opus. Anyone on Linux or OS X with an earlier Gen 3 who is using the 2.0 (possibly 1.5 but I didn’t test it) firmware will not have their device detected by calibre any longer. This is because once the hex ids are matched there is further matching based on the text ids for Linux and OS X. This isn’t going to be fun to work around.

I did a little bit of work on TXT and PML output. Now they both honor the “Remove spacing between paragraphs” option. Previously TXT output had TXT specific options for this behavior. I’ve removed them and just use that look and feel option. PML output previously ignored it but now it honors it. So you can have both look more like a printed book than a web page.

A few new profiles were added. Specifically profiles for the Sony PRS 300 and 900. They have a different screen size and resolution than the 6″ models so they warranted their own profiles.

I added support for two new devices. The Airis Dbook and the Binatone Readme. Along with supporting new devices I also reorganized and renamed a few. The BeBook device interface is not called Hanlin. The BeBook is a rebraned Hanlin. The EZReader, LBook, Eco Reader are also rebranded Hanlins. They all were using the BeBook interface. So I renamed it to be a little more generic and avoid confusion.

One big bug fix this week. There was a typo in the tag map for PML output. It causes italics to be ignore. This has been corrected.

As far as I can tell FB2 does not support a true table of contents (TOC). What seems to happen is reader software will dynamically generate the TOC based on the appearance of <section><title><p>text</p></title> within th e text. If I’m wrong about this and FB2 really does support an external TOC I would love to know. The <h1> differentiation causes the files to have these sections. I do not like how this is hard coded and dependent on this single tag. Especially since calibre’s conversion process allows for an XPath expression to be specified to generate TOC point.

I would love to use the TOC sent to the FB2 output generator but this does not seem to be feasible. The problem with the TOC that is sent to the FB2 output generator is how it corresponds to locations in the document. The OEB TOC points are text which may or my not appear in the text and an anchor id. The anchor id is a set point in the document. FB2 section titles are part of the text itself. I cannot use the text from the OEB TOC because it may or may not actually appear in the text. This also prevents me from determining what the text in the document is supposed to be associated with the TOC point. The anchor id points to an anchor in the document but often that point is something like <a id=”blah” />. In this case there is no text associated with the anchor in the document. While I can assume the text following is part of the title I have no fool proof way to determine where it stops.

At this point the only TOC associated with the FB2 output is the inline TOC that can be optionally generated.

Support for two new readers has been added. Ganaxa’s GeR2 and Nokia’s 770 internet tablet. The N770 should have been support a long time ago and I apologize for how long this request has been unfulfilled. I put it at the bottom of my todo list and at some point it simply fell off and I forgot about it.

The GeR2 reader was a bit of a challenge to get supported. This reader and some models of the Cybook Gen 3 have the same vendor, product and revision (BCD) ids. On Windows and OS X this is not an issue because once the ids are matched further matches are done based on the plug and play (PNP) strings. However, on Linux only the ids are matched.

To solve this problem, matching on Linux needed some further checks. Kovid added support for libusb-1 which provides the vendor and product strings. He also added a call back that can be implemented in the device interface to implement platform and device specific checks. We did run into a few problems. The first was an easy to solve 32 vs 64 bit issue with the Python to C interface Kovid wrote for libusb-1. Once that was sorted out we ran into a larger problem. libusb-1 on Ubuntu by default is denied access to the vendor and product strings.

libusb-1, after appearing in only run release (0.6.27), has been dropped. Kovid has now written a custom device scanner for Linux that will parser the devices in /sys/bus/usb to determine if a reader is connected. libusb-1 is supposed to be an easy to use library capable of providing this functionality but unfortunately this turned out not to be the case. The custom scanner works and allowed me to implement differentiation between the GeR2 and the Cybook Gen 3 so both readers can be properly supported without conflict and with the correct device interface being used.

The new feature, which Kovid helped me to implement, is ejecting the reader from within the GUI. When you mouse over the reader icon in the location list it will show an eject button next to the icon. Clicking the eject button will do just that, safely remove the device from the system and remove it from the location listing. I did the GUI work, created the interface for the device driver and added the Linux ejection code. Kovid added the OS X code and we both came up with a Window solution but his was more robust to it was used.

One minor and one major change to PDF processing this week. The minor change was a fix for bug 2342. German umlauts are now displayed correctly in the output. The major change is PDF output now supports comics. cbz, cbr, cbc are some of the input formats for comics that are support and now you can turn them into a PDF. The huge advantage is for people (like me) who have a Cybook. A comic can be turned into one PDF file sized for the device keeping down the amount of clutter in the library view.

I also worked on the device framework and have pluginized all of the device interfaces (I like the term interfaces better than drivers because it reduces confusion as Windows device drivers are very different). They also sport a new configuration system (though they didn’t have configuration before at all). The user will be able to specify their preferred format order for sending to the device. As well as disable certain formats from being sent to the device at all. I said will because while the configuration code is done there is currently no way to call it in the preferences dialog. However, this will be rectified before 0.6 is released.

eReader output has been put on hold for the foreseeable future. eReader input is complete and working but due to the undocumented nature of the eReader format I have not been able to produce a working output plugin. The main issue I’ve run into is the eReader header (record 0 within the pdb container) is a 132 byte package with 66 sections. There are to many unknown sections. Even with the inspector script I wrote to see what the values are in working eReader files I have not been able to understand how all of the sections interact with the file itself. My guesses have all resulted in files that are not readable by the eReader Pro software.

eReader files uses the PML markup language and while I couldn’t get eReader output working I have added support for PML input and PML output. The PML output can be taken and put into either MakeBook or DropBook to produce a working eReader file.

Two things to note about the the PML support is input can take either a straight .pml file or it can take a zip archive filled with .pml files and PNG images (the images must be in PNG format). The zip archive must have the extension changed to .pmlz for this to work. PML output will produce a zip archive with the extension .pmlz. Within this archive will be all of the image files in PNG format and the produced .pml files.

.pmlz is simply an easy way to group the files and ensure that there is not issues with including missing files or not being able to find referenced files.