John's Blog

It’s been a while since I’ve made any posts. Mainly work and other matters have kept me away. I haven’t been working on very much recently.

This isn’t to say my participation with calibre has been abandoned. I’m still making regular bug fixes. I’ve also been working on a rather large change that is coming along much slower that I would like. Mainly metadata caching and muli way syncing. I was shooting for the end of March to complete the changes but it might not be done until the end of April.

I’ve also moved this blog off of the shared hosting package I was using. It’s now hosted on my own virtual private server. Attachments, images and the now reading list did not make it over in the transition. I’m still using WordPress but instead of having multiple separate installs of it all the blogs here are powered by one WordPress MU installation.

Tags: calibre.

Kovid just tagged the 0.6.33 release so it should be out the door later today. Only one major feature from me this week. The PRS driver can new customize what gets turned into a collection. The default was and is series and tags. Now under the device interface configuration you can specify any metadata field. The main use for this change is so people who have a large number of tags can disable collections being created from them.

While it’s not coming out today I have been working on a major feature. Metadata caching for devices. Basically a file that specifies all of the metadata for the books on the reader will be created. It will be used when the device is connected and if any files have changed their metadata will be updated. This speeds up device detection dramatically. Also, takes up less space than a book so it won’t use too much space to store it. Right now it’s storing the data using json so other application can have easy access to the data too. Other than faster device detection this is the ground work that is required to implement syncing.

Tags: calibre, device interfaces, json, prs.

I’ve committed a fix for the Cybook 2.0 firmware id issue I posted yesterday. In calibre I’ve combined the Gen 3 and Opus device interfaces into one generic Cybook interface. Both were very similar and the Opus was just a subclass of the Gen 3. Also, with the 2.0 firmware being for both the Gen 3 and Opus there should be no difference in the capabilities of the two devices. If you own an Opus you will still want to use the Opus profile however, when you connect the device it will just show as using the Cybook interface. Also, with the interfaces being combined into one, the device specific configuration is now Cybook / Opus.

There were two reasons for combining the interfaces. I was unable to find a way to differentiate the Gen 3 and the Opus with the 2.0 firmware. A Gen 3 with the 2.0 firmware reports itself as an Opus. I was unable to find a way around this and felt it was better to have a Cybook interface. Also, I have the unfortunate pleasure of owning a Gen 3 which, even before the 2.0 firmware, used the same device ids of the Opus. It however did not use the Opus device strings. Due to this I am unable to determine if the 2.0 firmware changes the ids or just the strings. Combining the two interfaces keeps all Gen 3′s working and keeps some Gen 3′s from being detected as the Opus.

Tags: calibre, cybook, device interfaces.

On Christmas eve Bookeen finally released the long awaited 2.0 firmware for the Cybook Gen 3 and Cybook Opus. As soon as I returned from my Christmas vacation I upgraded my Gen 3 to the 2.0 version. Today while debugging a device issue for calibre I noticed that my Gen 3 kept being detected as an Opus. Doing a little digging I found this:

$ calibre-debug -d
Version: 0.6.32
USB devices on system:
...
['0xbda', '0x703', '0x110', 'Bookeen', 'Cybook Opus', '600AB038CB09484'],
...
Looking for CYBOOKG3
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Looking for CYBOOK_OPUS
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Devices possibly connected: Cybook Opus Device Interface,
Trying to open Cybook Opus Device Interface ... OK
Main memory: '/media/Cybook Gen3/'
Total space: (495251456L, 0, 0)

It seems Bookeen has once again screwed up the firmware release. They wanted one firmware so now every Gen 3 that has updated now has the product identifier set to Opus. The Gen 3 models use a number of different hex product and vendor ids and only the later ones are the same as the opus. Anyone on Linux or OS X with an earlier Gen 3 who is using the 2.0 (possibly 1.5 but I didn’t test it) firmware will not have their device detected by calibre any longer. This is because once the hex ids are matched there is further matching based on the text ids for Linux and OS X. This isn’t going to be fun to work around.

Tags: bookeen, calibre, cybook, device interfaces.

There was a major change this week to the device infrastructure. Kovid merged (with some modification) my changes to allow “Send to device” to use custom device paths just like “Save to disk”. Kovid’s major change to my implementation are having a separate save template for “Send to device” and allowing for per device overrides of the template. Kovid and I spent yesterday testing and it is working well. Expect it in the next release. Oh, News and the / tags still work as expected.

I did a little bit of work on TXT and PML output. Now they both honor the “Remove spacing between paragraphs” option. Previously TXT output had TXT specific options for this behavior. I’ve removed them and just use that look and feel option. PML output previously ignored it but now it honors it. So you can have both look more like a printed book than a web page.

Tags: calibre, device interfaces, pml, txt.

I spent a bit of time working on calibre this week. I worked on profiles, devices and a bug fix here and there.

A few new profiles were added. Specifically profiles for the Sony PRS 300 and 900. They have a different screen size and resolution than the 6″ models so they warranted their own profiles.

I added support for two new devices. The Airis Dbook and the Binatone Readme. Along with supporting new devices I also reorganized and renamed a few. The BeBook device interface is not called Hanlin. The BeBook is a rebraned Hanlin. The EZReader, LBook, Eco Reader are also rebranded Hanlins. They all were using the BeBook interface. So I renamed it to be a little more generic and avoid confusion.

One big bug fix this week. There was a typo in the tag map for PML output. It causes italics to be ignore. This has been corrected.

Tags: calibre, device interfaces, pml, prs300, prs900.

Not much on the calibre front for this past week from me. The only thing I’ve worked on was adding support to the Nook driver for the cover image to be sent to the device with the book. Also, if there is no cover associated with the book a default cover with the title authors and calibre library image are used. Similar to what already happens with Cybook Gen 3 and Opus.

Tags: calibre, cover, cybook, ebook, nook.

FB2 output has been improved. It no longer generated very invalid markup. The output generator still isn’t where I want it to be though. The changes are mostly cleanup and fix long standing issues with the output. One major change is I reverted having <h1> tags work as section and title markers. I don’t like having this hard coded into the generator.

As far as I can tell FB2 does not support a true table of contents (TOC). What seems to happen is reader software will dynamically generate the TOC based on the appearance of <section><title><p>text</p></title> within th e text. If I’m wrong about this and FB2 really does support an external TOC I would love to know. The <h1> differentiation causes the files to have these sections. I do not like how this is hard coded and dependent on this single tag. Especially since calibre’s conversion process allows for an XPath expression to be specified to generate TOC point.

I would love to use the TOC sent to the FB2 output generator but this does not seem to be feasible. The problem with the TOC that is sent to the FB2 output generator is how it corresponds to locations in the document. The OEB TOC points are text which may or my not appear in the text and an anchor id. The anchor id is a set point in the document. FB2 section titles are part of the text itself. I cannot use the text from the OEB TOC because it may or may not actually appear in the text. This also prevents me from determining what the text in the document is supposed to be associated with the TOC point. The anchor id points to an anchor in the document but often that point is something like <a id=”blah” />. In this case there is no text associated with the anchor in the document. While I can assume the text following is part of the title I have no fool proof way to determine where it stops.

At this point the only TOC associated with the FB2 output is the inline TOC that can be optionally generated.

Support for two new readers has been added. Ganaxa’s GeR2 and Nokia’s 770 internet tablet. The N770 should have been support a long time ago and I apologize for how long this request has been unfulfilled. I put it at the bottom of my todo list and at some point it simply fell off and I forgot about it.

The GeR2 reader was a bit of a challenge to get supported. This reader and some models of the Cybook Gen 3 have the same vendor, product and revision (BCD) ids. On Windows and OS X this is not an issue because once the ids are matched further matches are done based on the plug and play (PNP) strings. However, on Linux only the ids are matched.

To solve this problem, matching on Linux needed some further checks. Kovid added support for libusb-1 which provides the vendor and product strings. He also added a call back that can be implemented in the device interface to implement platform and device specific checks. We did run into a few problems. The first was an easy to solve 32 vs 64 bit issue with the Python to C interface Kovid wrote for libusb-1. Once that was sorted out we ran into a larger problem. libusb-1 on Ubuntu by default is denied access to the vendor and product strings.

libusb-1, after appearing in only run release (0.6.27), has been dropped. Kovid has now written a custom device scanner for Linux that will parser the devices in /sys/bus/usb to determine if a reader is connected. libusb-1 is supposed to be an easy to use library capable of providing this functionality but unfortunately this turned out not to be the case. The custom scanner works and allowed me to implement differentiation between the GeR2 and the Cybook Gen 3 so both readers can be properly supported without conflict and with the correct device interface being used.

Tags: calibre, cybook, device interfaces, fb2, GeR2, Linux, N770, TOC.

Most this week was spent turning PML input and output. I spent a bit of work bug tracking and enhancing FB2 output as well.

The changes for PML input are as follows. Pass along the included cover as the cover when converting (also applies to eReader PDB). Allow for images to be in top level, archivename_img or images directory for PMLZ. Based on that order it will check for images and if they are not found move onto the next location. For PML, images can be in pmlname_img or images directory. Footnotes and sidebars now display cleaner. They are separated better and EPUB puts them on individual pages. They also include a return link which goes back to the place in the text they are referenced. This assumes one footnote and sidebar per entry in the text, so if it’s referenced multiple times the return link will go back to the return reference.

PML output now creates \a and \U codes only for supported characters. All characters that are not supported and that cannot be turned into a \a or \U code will be replaced with a ?.

Along with the changes for PML input reading the cover they are now read as part of the metadata. This applies to both PML, PMLZ and eReader PDB files.

I’ve created a PML2PMLZ FileType plugin which will run when ever PML is imported into the GUI. It takes a PML file looks for images in the above mentioned locations, takes it all and puts it into a PMLZ archive. The PMLZ archive is them added to the library.

When I went to test the PML2PMLZ plugin I found that the GUI on my system was horribly broken. After a bit of work with Kovid, I found that calibre-parallel had to be in the path if calibre was installed in a non standard location. I install into my home directory using the develop command. Kovid has committed a fix that writes the install path to the launcher for these instances.

FB2 output now turns h1 tags into <section><title> tags to allow for TOC generation. As far as I can tell FB2 has not set TOC and instead readers dynamically generate the TOC based on looking at all of the body and sections and sets the text using the title tag. Right now the FB2 output is limited to only turning h1 tags and cannot use the user defined TOC based on an XPATH expression. I plan to fix this limitation in the future.

Tags: calibre, eReader, fb2, GUI, pdb, pml, pmlz.

I spent the past week fixing as many bugs with the new PML input parser and cleaning up as much of the output as I could. I really need to thank WayneD for helping find bugs with the new code. Also, Kevin Hendricks who has been working on his own parser based on code from a tool that does some work on eReader files. He helped me formulate what output should be derived in certain cases. The new PML input parser has been released as part of calibre 0.6.25.

Tags: calibre, eReader, pml.