John's Blog

Barns and Noble has announced and is pre-selling the new Nook second generation Simple Touch Reader (STR). Looking at the specifications the eReader PDB format is not listed as being supported. Only EPUB and PDF are listed. My concern is the fact that I still get books in the PDB format from B&N and this new Nook isn’t going to be able to read them.

I sent and email to B&N’s customer service asking for clarification. The email is as follows:

The new Nook STR has EPUB and PDF listed as the only supported ebook formats. How will I be able to read eReader PDB formatted books distributed by you on this device? Just the other day I purchased Toll the Hounds by Steven Erikson and received this book as an eReader PDB file. Will I not be able to read large parts (including this title) of the B&N sold books on the new Nook?

B&N’s response is as follows:

Dear John Schember,

Thank you for writing to us about NOOK books.

We truly appreciate your taking the time to share your feedback
concerning NOOK 2nd edition’s compatibility with .pdb NOOK Books.

Unfortunately, the 2nd edition of our NOOK is designed only in EPUB and
PDF format NOOK Books.

Customer feedback is critical to the success of our business, and we
rely on suggestions such as yours to help us determine what our
customers value most. We have forwarded your suggestion to the
individuals responsible for NOOK Books for consideration and possible
implementation.

Once again, thank you for your email.

Visit www.bn.com and click on the options that appear in the upper
right-hand corner to view information about your order.

We look forward to your next visit.

Sincerely,

Maik
Customer Service Representative – Digital Support
Barnes & Noble

http://www.bn.com/

Visit our NOOK Support site for the latest updates and downloads at:

http://www.barnesandnoble.com/nook/support/

The Nook STR does not in fact support the eReader PDB format.

This in itself shouldn’t be much of an issue as I’ve heard B&N has moved their offerings to be 100% EPUB and any PDB books can be re-downloaded from your B&N library as EPUB. This would be fine except I’m still receiving books (new purchases) in the PDB format!

I started a thread on MobileRead about this issue. The user Jane12 figured out what’s happening.

Apparently eReader PDB files are still sent to users who download to their computer using a Mac. Jane12 suggested I try changing my user agent (in my web browser) to say I’m on Windows. A re-download of Toll of the Hounds and I get an EPUB file. Change my user agent back to say I’m using Apple’s OS X and a re-download give me an eReader PDB file.

For what ever reason B&N gives Mac users eReader PDB files and Windows user EPUB file. On top of that their new device doesn’t read eReader PDB files. I foresee a lot of upset Mac user when they find out they can’t read books they’ve purchased from B&N on their B&N device. Especially when B&N defaults to giving Mac users books in a format they are not willing to support in their latest Nook (but support in the first generation model).

Tags: Apple, B&N, eReader, Mac, nook, OS X, pdb.

Most this week was spent turning PML input and output. I spent a bit of work bug tracking and enhancing FB2 output as well.

The changes for PML input are as follows. Pass along the included cover as the cover when converting (also applies to eReader PDB). Allow for images to be in top level, archivename_img or images directory for PMLZ. Based on that order it will check for images and if they are not found move onto the next location. For PML, images can be in pmlname_img or images directory. Footnotes and sidebars now display cleaner. They are separated better and EPUB puts them on individual pages. They also include a return link which goes back to the place in the text they are referenced. This assumes one footnote and sidebar per entry in the text, so if it’s referenced multiple times the return link will go back to the return reference.

PML output now creates \a and \U codes only for supported characters. All characters that are not supported and that cannot be turned into a \a or \U code will be replaced with a ?.

Along with the changes for PML input reading the cover they are now read as part of the metadata. This applies to both PML, PMLZ and eReader PDB files.

I’ve created a PML2PMLZ FileType plugin which will run when ever PML is imported into the GUI. It takes a PML file looks for images in the above mentioned locations, takes it all and puts it into a PMLZ archive. The PMLZ archive is them added to the library.

When I went to test the PML2PMLZ plugin I found that the GUI on my system was horribly broken. After a bit of work with Kovid, I found that calibre-parallel had to be in the path if calibre was installed in a non standard location. I install into my home directory using the develop command. Kovid has committed a fix that writes the install path to the launcher for these instances.

FB2 output now turns h1 tags into <section><title> tags to allow for TOC generation. As far as I can tell FB2 has not set TOC and instead readers dynamically generate the TOC based on looking at all of the body and sections and sets the text using the title tag. Right now the FB2 output is limited to only turning h1 tags and cannot use the user defined TOC based on an XPATH expression. I plan to fix this limitation in the future.

Tags: calibre, eReader, fb2, GUI, pdb, pml, pmlz.

I spent the past week fixing as many bugs with the new PML input parser and cleaning up as much of the output as I could. I really need to thank WayneD for helping find bugs with the new code. Also, Kevin Hendricks who has been working on his own parser based on code from a tool that does some work on eReader files. He helped me formulate what output should be derived in certain cases. The new PML input parser has been released as part of calibre 0.6.25.

Tags: calibre, eReader, pml.

PML input had some major changes this week. Thank the user WayneD for helping me out and getting me to actually do the work I’ve been putting off since I introduced PML/eReader as an input format.

There is now a metadata reader for PML and PMLZ. WayneD provided me with a set of regular expressions that can extract the metadata from a metatdata comment within a PML document. I took those regexes and created a metatdata plugin that supports both straight PML files as well as the PMLZ archive file.

The other major change to PML is, I’ve re-written the input parser. It is not longer based on a set of regular expressions. It is now a line oriented simple state machine. When I created the regex parser I intended to replace it at some point in the future with a true parser. The regex based one was simply a quick and dirty way to get PML supported. The new parser is much faster, produces cleaner and more accurate HTML output. It also has the added benefit of reading \CX codes and turns them into table of contents entries for PML and PMLZ input. The new parser is much better and I’m not completely finished with it. I still need to add support for \v comments (they are currently removed), \n codes, and implement font attribute tracking to condense changes (this is how \n will be handled).

WayneD did provide me with his Perl based line oriented simple state machine for PML to HTML conversion. I did use one idea from it. Turning footnote and sidebar xml syntax into custom PML tags. I had intended to port his parser to python and use it as a base but when I started looking at it I remembered I don’t know Perl at all and I can’t make heads or tails of Perl code. I have no desire or need to actually learn Perl, so I ended up writing my own parser.

Tags: calibre, eReader, pml, pmlz.

Mostly bug fixes this week. The majority of them were centered around eReader PDB output and PML generation. eReader PDB output now marks the first image as the cover image if a cover image is not explicitly set. PMLZ got images named properly in the output. PML generation now has .png added to the end of image names. I also fixed a bug where excessive new lines were not being properly removed. PML, TXT, RB, FB2 output all got excessive space removal tones down so instances were spaces were completely removed will stop happening. Regex header and footer matching was tweaked to match at a later stage in the conversion pipeline. This should ease issues of expressions not matching properly. Finally, at Kovid’s request I’ve added some info about header / footer regexes and converting TXT and PDF files to the documentation.

Tags: calibre, eReader, fb2, pdb, pml, pmlz, rb, txt.

I haven’t had one of these for quite some time. I’ve been working on other projects and on the calibre font I’ve only dealing with small bug fixes. However, this past week I’ve done a bit of work that is worth mentioning.

I’ve cleaned up the FB2 output. It fixes some invalid markup. Fixes some issues with text not being displayed by FBReader. It also fixes some issues with invalid characters making there way into hrefs.

eReader PDB output also got some love. Some kind people have been working on the reverse engineering of the file format and have filled in a number of the blanks I left. All of the additional information that has been discovered has been added to the files produced. The two main things that have been added are chapter and link indexes. The chapter indexes give the nice names at the top of the eReader viewer application. The link index allows links to work in the eReader viewer application.

To coincide with the eReader PDB output changes, PML input and output had some cleanup. It looks better now and replaces unicode characters with the \UXXXX equivalent.

Tags: calibre, ebook, eReader, fb2, pml.

A lot of work went into eReader and PML to have it supported better. Also, a new format has been added.

The XHTML to PML parser has been completely rewritten. It is based on the XHTML to FB2 parser I wrote for FB2 output. It produces much better looking PML markup and the displayed output looks very close to the original XHTML source. One major advantage of the new parser is that it accounts for XHTML style information and translates that into PML tags. For example if text is set to bold by CSS then the text will get the bold PML tag.

eReader also got another very important addition. Support for Makebook (202 bye header) file input. Makebook and Dropbook are the two applications provided by eReader (the company) for producing eReader files. Makebook is the older application that is no longer supported. Makebook and Dropbook produce very different record 0 headers. This header has information about where the text, images and other things contained in the file are located. It took a while but I’ve been able to understand enough of the Makebook header to add input support for these files.

Makebook produces a 202 byte header while Dropbook produces a 132 byte header. After comparing header values and section sizes I was able to determine that the 2 byte int at offset 0×08 contained the start of the non-text offset. Just like the 132 byte header files, everything before this offset is text.

Images in the 202 byte header files were easy to find because they are in the same format as the Dropbook produced files. However, I didn’t bother to determine if there was a header value. Since all images are in PNG format and the their section start with the text PNG, I simply loop though all non-text sections and see if they start with PNG. If they do I know it’s an image and extract it.

The hardest part of the 202 byte header files was the text itself. Even though I knew which sections contained the text I didn’t know how it was compressed. This is where Google came to the rescue. On the homepage for the Z-DOC PalmPilot application I found there was some work to reverse engineer this older format. This page gave me the information I was looking for. Text is PalmDoc compressed and then xored with 0xA5. It looks like this xor is an attempt to obfuscate the compression used to make it harder to decompress. It isn’t for copy protection because the Makebook application only produces non-DRM files. DRM eReader files from that time would be created in a different manner.

Syncing news now supports auto convert in the GUI. It’s just like auto convert with sending email and sending an eBook to a device. If the book is not in a format supported by the device it will be auto converted to a supported format based on user preference.

The final bit of work this week was support for the RocketBook (RB) format. Both input and output are working. Though they both do need testing. Output in particular as I don’t have a device that supports these files so I can only guess based on my input code that the RB files produced are 100% correct. If someone has a device that reads RB files please let me know if the output files are read correctly.

Tags: calibre, eReader, GUI, pml, rb.

This week comes with some great new additions. It also comes with some great new challenges. Not to mention more work for next week.

eReader output is complete and working. The files produced are the same format as those produced by Dropbook (more on this in a bit). In addition to eReader output I’ve added metadata writing for eReader files.

There is one major issue I’ve come across concerning the eReader format. The files produced by Makebook are significantly different than those produced by Dropbook. The files I’ve been using for reverse engineering the format have all been produced by Dropbook. My implementation for eReader input only works with Dropbook format files. As such, Makebook produced files will not work and are currently not supported. They will be unsupported for the foreseeable future because of how different the format is to Dropbook, and because Makebook is not supported nor being developed. It was replaced by Dropbook some time ago.

The PDB container format also got a metadata writer. However, the PDB wrapper itself only supports setting the book title. So that’s all that gets written to PDB files that either don’t support metadata in their internal format or that don’t have their own specific metadata writer.

eReader wasn’t the only format to get some output work. FB2 output has been added as well. However, there is no metadata writer for it yet.

The final bit of work I did for this week was to add auto convert to sending by email in the GUI. It’s the same idea as auto convert for sending to a device. If the file that is being sent is not in a format that is accepted by the email address (this is configurable setting) the file will be auto converted to a suitable format before being sent.

Tags: calibre, eReader, fb2, GUI.

Device interfaces can now be configured in the GUI. Also, there is a simple framework for creating plugin configuration widgets.

I’ve added a metadata reader for the eReader format. However, eReader supports 3 ways to set the metadata in the file. 1) In the pdb header (only supports setting a short title). 2) In the metadata section of the file (supports the most information: title, author, publisher, copyright, isbn). 3) Embedded in the text as a comment. 2 and 3 are only accessible if the book does not contain DRM (or has been unlocked, but Calibre does not support this). 3 is not supported at all with this metadata reader. The reader first tires 2 then falls back to 1 if the book is DRMed or if the metadata section is non-existent.

Two new input and output formats have been added. ztxt and palmdoc. They are both pdb formats like eReader. For input the pdb input plugin will automatically determine the internal format and call the appropriate code path. For output the default is palmdoc but there is an option –format that can be used to change it to any other supported pdb output format (ztxt is the only other currently). The format option is also available in the conversion dialog in the GUI.

Speaking of conversion in the GUI. It now works. There are all new dialogs for single and bulk conversion. Pretty much anything that can be done using the command line ebook-convert can be done in the GUI. Bulk, single and auto conversion are all complete and working. Auto conversion will also honor a users preferences for formats set for the device interface plugin.

Tags: calibre, eReader, GUI, palmdoc, pdb, ztxt.

It seems that PDF is becoming the never ending format for me. Maybe I should start naming the posts PDF Work instead of Calibre Week in Review…

One minor and one major change to PDF processing this week. The minor change was a fix for bug 2342. German umlauts are now displayed correctly in the output. The major change is PDF output now supports comics. cbz, cbr, cbc are some of the input formats for comics that are support and now you can turn them into a PDF. The huge advantage is for people (like me) who have a Cybook. A comic can be turned into one PDF file sized for the device keeping down the amount of clutter in the library view.

I also worked on the device framework and have pluginized all of the device interfaces (I like the term interfaces better than drivers because it reduces confusion as Windows device drivers are very different). They also sport a new configuration system (though they didn’t have configuration before at all). The user will be able to specify their preferred format order for sending to the device. As well as disable certain formats from being sent to the device at all. I said will because while the configuration code is done there is currently no way to call it in the preferences dialog. However, this will be rectified before 0.6 is released.

eReader output has been put on hold for the foreseeable future. eReader input is complete and working but due to the undocumented nature of the eReader format I have not been able to produce a working output plugin. The main issue I’ve run into is the eReader header (record 0 within the pdb container) is a 132 byte package with 66 sections. There are to many unknown sections. Even with the inspector script I wrote to see what the values are in working eReader files I have not been able to understand how all of the sections interact with the file itself. My guesses have all resulted in files that are not readable by the eReader Pro software.

eReader files uses the PML markup language and while I couldn’t get eReader output working I have added support for PML input and PML output. The PML output can be taken and put into either MakeBook or DropBook to produce a working eReader file.

Two things to note about the the PML support is input can take either a straight .pml file or it can take a zip archive filled with .pml files and PNG images (the images must be in PNG format). The zip archive must have the extension changed to .pmlz for this to work. PML output will produce a zip archive with the extension .pmlz. Within this archive will be all of the image files in PNG format and the produced .pml files.

.pmlz is simply an easy way to group the files and ensure that there is not issues with including missing files or not being able to find referenced files.

Tags: calibre, device interfaces, eReader, pdf, pml, pmlz.