John's Blog

This weeks review of what I’ve been working on is a little late. Overall it wasn’t as productive as last week looking at what was accomplished but I spent just as much time coding as last. With projects like this you can’t judge output by the number of features add or bugs fixed.

The GUI received context sensitive treatment for the device menu. It will only have send to device, when a device is connected and send to card A and B will only enabled when they are available as well. A simple change but one that will reduce confusion.

I’ve spent a lot of time working with Lee Dolsen (ldolse from mobileread) on pdftohtml processing rules. They are nearly complete and the output is looking really good. I know I’ve been saying that for a while now but each week it just keeps getting better. However, PDF is still not an ebook format and should not be treated as such. This simply helps to get content out of the PDF format and into a more manageable one.

One big thing I spent most of my time this week on was eReader input. Yep, eReader pdb files can now be converted to any supported output format. Metadata reading of eReader files is not yet supported. That is on my todo list. The html it produces could probably use some work but that will come as people report issues once 0.6 is released.

The other big thing that has taken up my Calibre time is eReader output. Sadly, it does not work. Also, it will not be working for the foreseeable future. The issue I’ve run into is I don’t know enough about the format to produce a file that can be read by eReader’s reading software. The main problem I face is there are around 66 “sections” to the eReader format header (not the pdb header, this is record 0 of an eReader file). I know what 10 of those sections are and what values they should have as they are used for my reader. Around 40ish of the sections should have a value of 0. However, that leaves 26ish sections that I don’t know what they are, what they do or what value they should have and how it relates to the rest of the file. Suffice it to say until I know more about the format I won’t be able to complete the output plugin.

Oh, I did write an inspector script (it’s in the eReader directory in the Calibre source tree) to help understand the eReader format. If anyone is interested in analyzing the format they can use it to help them see what is in the header.

Tags: calibre, eReader, GUI, pdf.

This has been a busy week for me on the Calibre front. All of my changes were to pluginize and the first three I talk about also made it into trunk and will be appearing in the next release.

I re-worked the mobi metadata reader so that it does not read the entire file into memory. It only reads the parts of the file that hold the metadata. The advantage is reading the metadata is now about five times faster. These results are from unscientific testing by a the bug reporter. Basically he said that listing the books on his Kindle went from 5 minutes to about 1 minute.

The metadata writer for pdf files has been re-worked and is now enabled. Kovid did some work to my initial work so that it won’t lock up the GUI when working with large pdf files.

I (a bit of help from Kovid on this too) was able to fix bug 2112 (last few pdf files held open). Calibre relies on Python’s garbage collector and object scope for closing files. It does not explicitly close them. The bug as caused by pyPdf which is a Python library Calibre uses to read and write pdfs. For some reason pyPdf’s file reader wan’t allowing the files to be closed. They were no longer in use and the object went out of scope but the garbage collector didn’t close the file immediately. It would close it eventually. A wrapper object was created and is used so that pyPdf doesn’t have a direct reference to the open file and it now gets closed properly.

The GUI in the releases only supports displaying one storage card from a device. Not all device support two storage cards but the Sony PRS devices do. Support for the GUI to display two storage cards has been added.

To go along with the GUI supporting displaying two storage cards, Almost all device drivers have been made to support up to two storage cards. The USBMS base class supports two cards and as most device drivers use this base they all get support for it without much work. However, this doesn’t mean that a device that doesn’t physically a storage card or two storage card slots won’t magically support two cards. All except the PRS drivers don’t have any user visible changes. For anyone looking to write a device driver using USBMS if the device supports two cards USBMS has you covered.

The PRS505 and PRS700 drivers both received the two card treatment. They also received a bit of work. They have been moved to use the USBMS base class. This removed a lot of redundant code and puts them on the same code path as the other (except PRS500) drivers. Overall this change is to reduce work in finding and fixing bugs and maintenance.

Internal work on the PRS505 and PRS700 drives wasn’t all I did to them. They no longer dump all books into a single directory. Books are stored in author/title/book hierarchy. News items are stored in a news/title hierarchy. They also support the USBMS / tag as a custom layout path.

Earlier I said almost all device driver got two storage card support. The PRS500 driver did not. It still only supports one storage card. Due to the way the driver works I will not be touching it.

I’ve been working with ldolse from mobileread and with his help the processing rules for pdftohtml (used for pdf input) have been improved.

Tags: calibre, ebook, mobi, pdf, prs505, prs700.

Other than little fixes here and there I’ve mainly focused this week on getting PDF output complete. It now supports profiles as well as custom page sizes. A little work is still needed on the processing rules for the html produced by the PDFInput. Otherwise, PDF Input/Output is complete. Though, pluginize is still in flux so what is complete now might not be complete next week as new requirements and interfaces are added. Or if I get some suggestions about what could or should be added.

Tags: calibre, pdf.

This has been a busy week for Calibre. The new conversion pipeline is complete. Part of this change is there is a new framework for command line options.

I’ve spent most of this past week moving the PDF input/output and TXT input/output over to use the new framework.

The other major work I’ve completed is moving the pdfmanipulte program and it’s commands over to the new command line option framework. I’ve also added a few new commands to pdfmanipulate. The current commands it supports are [crop, decrypt, encrypt, info, merge, reverse, split]. The trim command is now crop. It has also been cleaned up a bit.

Tags: calibre, pdf, txt.