John's Blog

Not much happened this week. A few bug fixes and a new output format, RTF. It produces acceptable results. It also embeds images into the file. The output could use some tweaking, but this will come with time. The only caveat is the output is ascii only. This is to keep compatibility with Cailbre’s RTF intput which can only accept ascii rtf files.

Pluginize has been merged back into trunk. Once a bit of testing is done by Kovid, he will be rolling out a beta for the 0.6 release. For those of you, like me, who use Ubuntu and build Calibre from source, there is a little change you will need to make in order to have it build. Open the file /usr/lib/python2.6/dist-packages/PyQt4/uic/Compiler/qtproxies.py and modify _qwidgets on line 238 to include “QWizardPage”.

Tags: calibre, pyqt, python, qt, rtf.

A lot of work went into eReader and PML to have it supported better. Also, a new format has been added.

The XHTML to PML parser has been completely rewritten. It is based on the XHTML to FB2 parser I wrote for FB2 output. It produces much better looking PML markup and the displayed output looks very close to the original XHTML source. One major advantage of the new parser is that it accounts for XHTML style information and translates that into PML tags. For example if text is set to bold by CSS then the text will get the bold PML tag.

eReader also got another very important addition. Support for Makebook (202 bye header) file input. Makebook and Dropbook are the two applications provided by eReader (the company) for producing eReader files. Makebook is the older application that is no longer supported. Makebook and Dropbook produce very different record 0 headers. This header has information about where the text, images and other things contained in the file are located. It took a while but I’ve been able to understand enough of the Makebook header to add input support for these files.

Makebook produces a 202 byte header while Dropbook produces a 132 byte header. After comparing header values and section sizes I was able to determine that the 2 byte int at offset 0×08 contained the start of the non-text offset. Just like the 132 byte header files, everything before this offset is text.

Images in the 202 byte header files were easy to find because they are in the same format as the Dropbook produced files. However, I didn’t bother to determine if there was a header value. Since all images are in PNG format and the their section start with the text PNG, I simply loop though all non-text sections and see if they start with PNG. If they do I know it’s an image and extract it.

The hardest part of the 202 byte header files was the text itself. Even though I knew which sections contained the text I didn’t know how it was compressed. This is where Google came to the rescue. On the homepage for the Z-DOC PalmPilot application I found there was some work to reverse engineer this older format. This page gave me the information I was looking for. Text is PalmDoc compressed and then xored with 0xA5. It looks like this xor is an attempt to obfuscate the compression used to make it harder to decompress. It isn’t for copy protection because the Makebook application only produces non-DRM files. DRM eReader files from that time would be created in a different manner.

Syncing news now supports auto convert in the GUI. It’s just like auto convert with sending email and sending an eBook to a device. If the book is not in a format supported by the device it will be auto converted to a supported format based on user preference.

The final bit of work this week was support for the RocketBook (RB) format. Both input and output are working. Though they both do need testing. Output in particular as I don’t have a device that supports these files so I can only guess based on my input code that the RB files produced are 100% correct. If someone has a device that reads RB files please let me know if the output files are read correctly.

Tags: calibre, eReader, GUI, pml, rb.

This week comes with some great new additions. It also comes with some great new challenges. Not to mention more work for next week.

eReader output is complete and working. The files produced are the same format as those produced by Dropbook (more on this in a bit). In addition to eReader output I’ve added metadata writing for eReader files.

There is one major issue I’ve come across concerning the eReader format. The files produced by Makebook are significantly different than those produced by Dropbook. The files I’ve been using for reverse engineering the format have all been produced by Dropbook. My implementation for eReader input only works with Dropbook format files. As such, Makebook produced files will not work and are currently not supported. They will be unsupported for the foreseeable future because of how different the format is to Dropbook, and because Makebook is not supported nor being developed. It was replaced by Dropbook some time ago.

The PDB container format also got a metadata writer. However, the PDB wrapper itself only supports setting the book title. So that’s all that gets written to PDB files that either don’t support metadata in their internal format or that don’t have their own specific metadata writer.

eReader wasn’t the only format to get some output work. FB2 output has been added as well. However, there is no metadata writer for it yet.

The final bit of work I did for this week was to add auto convert to sending by email in the GUI. It’s the same idea as auto convert for sending to a device. If the file that is being sent is not in a format that is accepted by the email address (this is configurable setting) the file will be auto converted to a suitable format before being sent.

Tags: calibre, eReader, fb2, GUI.

As a person living in the USA I highly prefer double quotation marks to single quotes when denoting speech. Some authors use the single quote for effect but mostly it’s just a style choice. I find UK authors generally use the two interchangeably. Tolkien books are a good example. I have The Hobbit, The Lord of the Rings, and The Children of Hurin. The Hobbit use double quotes while the other two uses single quotes.

Following is some simple python code that will take the book (named th.txt) and change the single quotes into double quotes for the books in question. Both use ’ and ’ for the opening and closing quotes. Also ’ is used for contractions. The regexes take the opening and closing characters into account as well as change the contractions to the non-unicode ‘ character.

>>> th = open('th.txt', 'rb+wb')
>>> th_t = th.read()
>>> th_t = re.sub('(?u)(?>> th_t = re.sub('(?u)‘', '"', th_t)
>>> th_t = re.sub('(?u)’', '"', th_t)
>>> th.seek(0)
>>> th.truncate(0)
>>> th.write(th_t)

I do realize that the listed regexes could be combined a bit especially the opening and closing quotes. However, that would reduce their readability.

Tags: ebook, python, regex.

Device interfaces can now be configured in the GUI. Also, there is a simple framework for creating plugin configuration widgets.

I’ve added a metadata reader for the eReader format. However, eReader supports 3 ways to set the metadata in the file. 1) In the pdb header (only supports setting a short title). 2) In the metadata section of the file (supports the most information: title, author, publisher, copyright, isbn). 3) Embedded in the text as a comment. 2 and 3 are only accessible if the book does not contain DRM (or has been unlocked, but Calibre does not support this). 3 is not supported at all with this metadata reader. The reader first tires 2 then falls back to 1 if the book is DRMed or if the metadata section is non-existent.

Two new input and output formats have been added. ztxt and palmdoc. They are both pdb formats like eReader. For input the pdb input plugin will automatically determine the internal format and call the appropriate code path. For output the default is palmdoc but there is an option –format that can be used to change it to any other supported pdb output format (ztxt is the only other currently). The format option is also available in the conversion dialog in the GUI.

Speaking of conversion in the GUI. It now works. There are all new dialogs for single and bulk conversion. Pretty much anything that can be done using the command line ebook-convert can be done in the GUI. Bulk, single and auto conversion are all complete and working. Auto conversion will also honor a users preferences for formats set for the device interface plugin.

Tags: calibre, eReader, GUI, palmdoc, pdb, ztxt.

It seems that PDF is becoming the never ending format for me. Maybe I should start naming the posts PDF Work instead of Calibre Week in Review…

One minor and one major change to PDF processing this week. The minor change was a fix for bug 2342. German umlauts are now displayed correctly in the output. The major change is PDF output now supports comics. cbz, cbr, cbc are some of the input formats for comics that are support and now you can turn them into a PDF. The huge advantage is for people (like me) who have a Cybook. A comic can be turned into one PDF file sized for the device keeping down the amount of clutter in the library view.

I also worked on the device framework and have pluginized all of the device interfaces (I like the term interfaces better than drivers because it reduces confusion as Windows device drivers are very different). They also sport a new configuration system (though they didn’t have configuration before at all). The user will be able to specify their preferred format order for sending to the device. As well as disable certain formats from being sent to the device at all. I said will because while the configuration code is done there is currently no way to call it in the preferences dialog. However, this will be rectified before 0.6 is released.

eReader output has been put on hold for the foreseeable future. eReader input is complete and working but due to the undocumented nature of the eReader format I have not been able to produce a working output plugin. The main issue I’ve run into is the eReader header (record 0 within the pdb container) is a 132 byte package with 66 sections. There are to many unknown sections. Even with the inspector script I wrote to see what the values are in working eReader files I have not been able to understand how all of the sections interact with the file itself. My guesses have all resulted in files that are not readable by the eReader Pro software.

eReader files uses the PML markup language and while I couldn’t get eReader output working I have added support for PML input and PML output. The PML output can be taken and put into either MakeBook or DropBook to produce a working eReader file.

Two things to note about the the PML support is input can take either a straight .pml file or it can take a zip archive filled with .pml files and PNG images (the images must be in PNG format). The zip archive must have the extension changed to .pmlz for this to work. PML output will produce a zip archive with the extension .pmlz. Within this archive will be all of the image files in PNG format and the produced .pml files.

.pmlz is simply an easy way to group the files and ensure that there is not issues with including missing files or not being able to find referenced files.

Tags: calibre, device interfaces, eReader, pdf, pml, pmlz.