John's Blog

I did some work with PDF output. Mainly I refactored some of the output generation code to reduce redundant sections. Over all there won’t be any user visible changes.

The main reason I dove back into PDF output was because a user on OS X noted that PDF produced were not searchable. Windows users are getting searchable PDFs and on Kovid’s Gentoo Linux machine he was able to get a searchable PDF. I looked into the issue and cannot get searchable PDFs on OS X. However, I can get searchable PDFs when using the ebook-viewers print feature via print to file. I’m not sure why this happens because the ebook-viewer and PDF output use the same technique for generating a PDF. I’ve decided not to peruse the matter further because PDF output on OS X is pretty much broken due to Qt bugs. See this for an example.

TCR compression was something I added a while ago. I was never fully happy with it because it was slow, and produced low quality output. I spent a few days completely rewriting the compressor and now it performs beautifully. The new compressor is cleaner, an order of 10x faster, compresses to a much smaller size, and I would say is on par with the output of Andrew Giddings’ TCR implementation. However, calibre’s TCR compressor is a pure python implementation is still considerably slower than Andrew’s C implementation.

A minor bug in FB2 output was brought to my attention and fixed. Basically JPG images in the input document were not being written to the FB2 output file. This has been corrected.

Tags: calibre, fb2, pdf, tcr.

I’ve become active in contributing to calibre again. So far I’ve been focusing on fixing issues related to the output formats I maintain. I’ve been focusing on FB2 and TXT output at the moment.

With FB2 output my goals were to fix as many bugs with it as possible and to produce 100% valid output. The first goal corresponds nicely to the second because most of the open bugs dealt with invalid markup.

FB2 output underwent some very large changes with a large amount of code being re-written. Also, I’ve removed a number of options. The idea is to simplify the code while working toward valid output and to remove options that were really just work arounds for invalid output in certain cases.

Overall I’m pleased with the FB2 output changes. It’s 100% valid (at least with the test book I ran though it) and the code is simpler. As always if any issues are found with the output a ticket would be appreciated.

TXT output had one small bug fix and one major change. TXT output can now produce Markdown formatted text. However, I’m not fully satisfied with the markdown generation. I didn’t spend much time with it and as of now it doesn’t appear to be taking css styling into account. I only pushed the xhtml from the OEB intermediate stage into html2markdown. I need to spend some more time with it. My fear is I may have to abandon the use of html2markdown if it’s unable to cope with css.

One other change with me getting back into calibre development is my working branch. I’ve changed it to lp:~user-none/calibre/dev because of some issues relating to my previous branch and some failures with upgrading the branch format.

Tags: calibre, fb2, markdown, txt.

It’s been a while since I’ve made any posts. Mainly work and other matters have kept me away. I haven’t been working on very much recently.

This isn’t to say my participation with calibre has been abandoned. I’m still making regular bug fixes. I’ve also been working on a rather large change that is coming along much slower that I would like. Mainly metadata caching and muli way syncing. I was shooting for the end of March to complete the changes but it might not be done until the end of April.

I’ve also moved this blog off of the shared hosting package I was using. It’s now hosted on my own virtual private server. Attachments, images and the now reading list did not make it over in the transition. I’m still using WordPress but instead of having multiple separate installs of it all the blogs here are powered by one WordPress MU installation.

Tags: calibre.

Teleread has published my Beginner’s Guide to Calibre.

Tags: calibre.

Kovid just tagged the 0.6.33 release so it should be out the door later today. Only one major feature from me this week. The PRS driver can new customize what gets turned into a collection. The default was and is series and tags. Now under the device interface configuration you can specify any metadata field. The main use for this change is so people who have a large number of tags can disable collections being created from them.

While it’s not coming out today I have been working on a major feature. Metadata caching for devices. Basically a file that specifies all of the metadata for the books on the reader will be created. It will be used when the device is connected and if any files have changed their metadata will be updated. This speeds up device detection dramatically. Also, takes up less space than a book so it won’t use too much space to store it. Right now it’s storing the data using json so other application can have easy access to the data too. Other than faster device detection this is the ground work that is required to implement syncing.

Tags: calibre, device interfaces, json, prs.

I’ve committed a fix for the Cybook 2.0 firmware id issue I posted yesterday. In calibre I’ve combined the Gen 3 and Opus device interfaces into one generic Cybook interface. Both were very similar and the Opus was just a subclass of the Gen 3. Also, with the 2.0 firmware being for both the Gen 3 and Opus there should be no difference in the capabilities of the two devices. If you own an Opus you will still want to use the Opus profile however, when you connect the device it will just show as using the Cybook interface. Also, with the interfaces being combined into one, the device specific configuration is now Cybook / Opus.

There were two reasons for combining the interfaces. I was unable to find a way to differentiate the Gen 3 and the Opus with the 2.0 firmware. A Gen 3 with the 2.0 firmware reports itself as an Opus. I was unable to find a way around this and felt it was better to have a Cybook interface. Also, I have the unfortunate pleasure of owning a Gen 3 which, even before the 2.0 firmware, used the same device ids of the Opus. It however did not use the Opus device strings. Due to this I am unable to determine if the 2.0 firmware changes the ids or just the strings. Combining the two interfaces keeps all Gen 3′s working and keeps some Gen 3′s from being detected as the Opus.

Tags: calibre, cybook, device interfaces.

On Christmas eve Bookeen finally released the long awaited 2.0 firmware for the Cybook Gen 3 and Cybook Opus. As soon as I returned from my Christmas vacation I upgraded my Gen 3 to the 2.0 version. Today while debugging a device issue for calibre I noticed that my Gen 3 kept being detected as an Opus. Doing a little digging I found this:

$ calibre-debug -d
Version: 0.6.32
USB devices on system:
...
['0xbda', '0x703', '0x110', 'Bookeen', 'Cybook Opus', '600AB038CB09484'],
...
Looking for CYBOOKG3
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Looking for CYBOOK_OPUS
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Devices possibly connected: Cybook Opus Device Interface,
Trying to open Cybook Opus Device Interface ... OK
Main memory: '/media/Cybook Gen3/'
Total space: (495251456L, 0, 0)

It seems Bookeen has once again screwed up the firmware release. They wanted one firmware so now every Gen 3 that has updated now has the product identifier set to Opus. The Gen 3 models use a number of different hex product and vendor ids and only the later ones are the same as the opus. Anyone on Linux or OS X with an earlier Gen 3 who is using the 2.0 (possibly 1.5 but I didn’t test it) firmware will not have their device detected by calibre any longer. This is because once the hex ids are matched there is further matching based on the text ids for Linux and OS X. This isn’t going to be fun to work around.

Tags: bookeen, calibre, cybook, device interfaces.

There was a major change this week to the device infrastructure. Kovid merged (with some modification) my changes to allow “Send to device” to use custom device paths just like “Save to disk”. Kovid’s major change to my implementation are having a separate save template for “Send to device” and allowing for per device overrides of the template. Kovid and I spent yesterday testing and it is working well. Expect it in the next release. Oh, News and the / tags still work as expected.

I did a little bit of work on TXT and PML output. Now they both honor the “Remove spacing between paragraphs” option. Previously TXT output had TXT specific options for this behavior. I’ve removed them and just use that look and feel option. PML output previously ignored it but now it honors it. So you can have both look more like a printed book than a web page.

Tags: calibre, device interfaces, pml, txt.

I spent a bit of time working on calibre this week. I worked on profiles, devices and a bug fix here and there.

A few new profiles were added. Specifically profiles for the Sony PRS 300 and 900. They have a different screen size and resolution than the 6″ models so they warranted their own profiles.

I added support for two new devices. The Airis Dbook and the Binatone Readme. Along with supporting new devices I also reorganized and renamed a few. The BeBook device interface is not called Hanlin. The BeBook is a rebraned Hanlin. The EZReader, LBook, Eco Reader are also rebranded Hanlins. They all were using the BeBook interface. So I renamed it to be a little more generic and avoid confusion.

One big bug fix this week. There was a typo in the tag map for PML output. It causes italics to be ignore. This has been corrected.

Tags: calibre, device interfaces, pml, prs300, prs900.

Not much on the calibre front for this past week from me. The only thing I’ve worked on was adding support to the Nook driver for the cover image to be sent to the device with the book. Also, if there is no cover associated with the book a default cover with the title authors and calibre library image are used. Similar to what already happens with Cybook Gen 3 and Opus.

Tags: calibre, cover, cybook, ebook, nook.