John's Blog

Barns and Noble has announced and is pre-selling the new Nook second generation Simple Touch Reader (STR). Looking at the specifications the eReader PDB format is not listed as being supported. Only EPUB and PDF are listed. My concern is the fact that I still get books in the PDB format from B&N and this new Nook isn’t going to be able to read them.

I sent and email to B&N’s customer service asking for clarification. The email is as follows:

The new Nook STR has EPUB and PDF listed as the only supported ebook formats. How will I be able to read eReader PDB formatted books distributed by you on this device? Just the other day I purchased Toll the Hounds by Steven Erikson and received this book as an eReader PDB file. Will I not be able to read large parts (including this title) of the B&N sold books on the new Nook?

B&N’s response is as follows:

Dear John Schember,

Thank you for writing to us about NOOK books.

We truly appreciate your taking the time to share your feedback
concerning NOOK 2nd edition’s compatibility with .pdb NOOK Books.

Unfortunately, the 2nd edition of our NOOK is designed only in EPUB and
PDF format NOOK Books.

Customer feedback is critical to the success of our business, and we
rely on suggestions such as yours to help us determine what our
customers value most. We have forwarded your suggestion to the
individuals responsible for NOOK Books for consideration and possible
implementation.

Once again, thank you for your email.

Visit www.bn.com and click on the options that appear in the upper
right-hand corner to view information about your order.

We look forward to your next visit.

Sincerely,

Maik
Customer Service Representative – Digital Support
Barnes & Noble

http://www.bn.com/

Visit our NOOK Support site for the latest updates and downloads at:

http://www.barnesandnoble.com/nook/support/

The Nook STR does not in fact support the eReader PDB format.

This in itself shouldn’t be much of an issue as I’ve heard B&N has moved their offerings to be 100% EPUB and any PDB books can be re-downloaded from your B&N library as EPUB. This would be fine except I’m still receiving books (new purchases) in the PDB format!

I started a thread on MobileRead about this issue. The user Jane12 figured out what’s happening.

Apparently eReader PDB files are still sent to users who download to their computer using a Mac. Jane12 suggested I try changing my user agent (in my web browser) to say I’m on Windows. A re-download of Toll of the Hounds and I get an EPUB file. Change my user agent back to say I’m using Apple’s OS X and a re-download give me an eReader PDB file.

For what ever reason B&N gives Mac users eReader PDB files and Windows user EPUB file. On top of that their new device doesn’t read eReader PDB files. I foresee a lot of upset Mac user when they find out they can’t read books they’ve purchased from B&N on their B&N device. Especially when B&N defaults to giving Mac users books in a format they are not willing to support in their latest Nook (but support in the first generation model).

Tags: Apple, B&N, eReader, Mac, nook, OS X, pdb.

Once again this is a weeks in review instead of once. I’ve been focusing more on new features than blogging. Calibre 0.7.57 is out and is also the beta for 0.8. Adding the tweak “test_eight_code = True” will enable the 0.8 features.

Get Books aka Stores

For quite some time I’ve been working on integrating support for searching and connecting to third party stores to make it easier for users to find and acquire books they’re interested in. This is a very large feature and one I’m very excited about. There are two pieces: The individual store plugins that connect the user to a given store and a meta-store search that searches all of the store plugins at once.

For 0.8 I have support for 14 stores. They are a mix of big name, independent, paid, free, and public domain. There is something for everyone. The majority of the stores are implemented though an embedded web browser. This is because the majority of stores are only accessible via their web site. MobileRead is the one exception but I’ll talk about that later.

By default accessing the stores is done though the embedded web browser but each store can be configured to open in the system web browser instead. One major befit of this approach is I’m able to detect ebook downloads. When an ebook is downloaded it is automatically added to the currently open library.

MobileRead is the exception and opens in it’s own search window. Right now it opens to the specific book’s entry in the embedded web browser so you can see details and download the book.

The meta-store search (along with MobileRead’s search dialog) allow for full boolean and field logic. Just like the main calibre window. The search gets results from every store and shows them in one easy to sort list. Title, Author, Price, DRM status, Store, and Formats are all listed.

PDB – Plucker Input

Not much to say about this but it’s been a long time coming. Plucker is now supported as an input format. Not all features are supported (tables for instance). However, plucker files for pretty much every source will have the main content come though.

Tags: calibre, ebook, pdb, plucker, store.

This week saw massive improvements to TXT input. I started the week with a slew of changes and as soon as I had implemented the first of them Lee Dolsen contacted me. We’ve worked together before improving PDF input. Since then he’s done a lot of work with preprocessing of PDF and other not so clean input.

TXT input now auto detects the character encoding of the file. It isn’t 100% accurate but should work for the majority of cases. It’s using chardet for the detection. Unfortunately, cp1252 is the most common encoding that gives people issues and unless you’re using things like smart quotes and curly apostrophes it doesn’t always detect properly.

I started getting TXT input to detect the document structure. Mainly, are the paragraphs arranged in block, single line, or print fashion. Lee saw the detection code and modifying some of his preprocessing code he was able to greatly increased the detection accuracy over my initial work. He’s also added an unformatted type that assumes the text is one big blob and tries to determine paragraphs in much the same way PDF input tries to determine them. By unwrapping based upon punctuation and other factors.

In addition to detecting the paragraph style used in the document, TXT input now tries to detect the formatting of the text content. Markdown formatted text is detected. I’ve also added a heuristic processor which runs by default if either Markdown is not detected or if the user has not specified the formatting as none (which disables any type of formatting processing).

The heuristic processor uses some ideas from GutenMark. Specifically italicizing common words and certain contentions used in Project Gutenberg texts that denote italics. I started working on a set of heuristics to detect chapter headings but Lee quickly pointed out he had already created something similar using regular expressions in his preprocessing code. I quickly began using it in my heuristic processor and it’s working well. Chapter headings and subheadings are now formatted with the appropriate h tags. He has some plans to enhance the detection further using a word list.

TCR, PDB PalmDoc and PDB zTXT inputs all pass the extracted text to the TXT input plugin for processing. This allows them to take advantage of all the work that’s gone into TXT input. Also, with auto detection now being part of TXT input it should allow for one time conversion instead of convert, check, tweak some options, convert again. Especially since these formats don’t make it easy to see how the text is structured within the file without first converting.

TXT input wasn’t the only part of TXT support that was touched. I spent some time cleaning up the TXT output. Consistant spacing is now created around headings. Also, when using the –remove-paragraph-spacing option, headings are not indented with a tab. The output now looks much cleaner and I consider it acceptable for reading.

Not to be left out FB2 output got a small bug fix. With all the work rewriting it I broke having it read covers. If you were converting an EPUB for instance that specified the cover (or title page) in the guide rather than the spine it would not be included. Also, the –cover option was being ignored. Now that’s fixed and external covers are inserted properly.

Tags: calibre, fb2, heuristic, palmdoc, pdb, tcr, txt, ztxt.

I have been taking a short break from blogging again. The pressure at work has only increased and is eating into a lot of my time. I haven’t been motivated to work on personal projects because well they are work. However, this has recently changed a bit.

I’ve started a Qt based library for reading ebooks in a generic manner. It is called lebookread! It is it’s early stages. So far I have it supporting epub, palmdoc pdb, ztxt pdb, tcr, and rb files. I plan to support ereader pdb, mobi, and plucker files in the near future.

The main goal of this project is to make reading ebooks easy for Qt based projects. I’ve chose to write the library in C++. This is also my first attempt at writing a library and it shows. I hope that it will be used by Sigil.

The real motivation of writing lebook read is I really want a good light weight ebook reader. The current offering have issues. I want something that is a bit more advanced in it’s rendering than FBReader. I also didn’t want anything with as large a dependency list as calibre. So, I plan on using lebookread to write my own ebook viewer.

Tags: c++, epub, lebookread, palmdoc, pdb, qt, rb, ztxt.

Most this week was spent turning PML input and output. I spent a bit of work bug tracking and enhancing FB2 output as well.

The changes for PML input are as follows. Pass along the included cover as the cover when converting (also applies to eReader PDB). Allow for images to be in top level, archivename_img or images directory for PMLZ. Based on that order it will check for images and if they are not found move onto the next location. For PML, images can be in pmlname_img or images directory. Footnotes and sidebars now display cleaner. They are separated better and EPUB puts them on individual pages. They also include a return link which goes back to the place in the text they are referenced. This assumes one footnote and sidebar per entry in the text, so if it’s referenced multiple times the return link will go back to the return reference.

PML output now creates \a and \U codes only for supported characters. All characters that are not supported and that cannot be turned into a \a or \U code will be replaced with a ?.

Along with the changes for PML input reading the cover they are now read as part of the metadata. This applies to both PML, PMLZ and eReader PDB files.

I’ve created a PML2PMLZ FileType plugin which will run when ever PML is imported into the GUI. It takes a PML file looks for images in the above mentioned locations, takes it all and puts it into a PMLZ archive. The PMLZ archive is them added to the library.

When I went to test the PML2PMLZ plugin I found that the GUI on my system was horribly broken. After a bit of work with Kovid, I found that calibre-parallel had to be in the path if calibre was installed in a non standard location. I install into my home directory using the develop command. Kovid has committed a fix that writes the install path to the launcher for these instances.

FB2 output now turns h1 tags into <section><title> tags to allow for TOC generation. As far as I can tell FB2 has not set TOC and instead readers dynamically generate the TOC based on looking at all of the body and sections and sets the text using the title tag. Right now the FB2 output is limited to only turning h1 tags and cannot use the user defined TOC based on an XPATH expression. I plan to fix this limitation in the future.

Tags: calibre, eReader, fb2, GUI, pdb, pml, pmlz.

Mostly bug fixes this week. The majority of them were centered around eReader PDB output and PML generation. eReader PDB output now marks the first image as the cover image if a cover image is not explicitly set. PMLZ got images named properly in the output. PML generation now has .png added to the end of image names. I also fixed a bug where excessive new lines were not being properly removed. PML, TXT, RB, FB2 output all got excessive space removal tones down so instances were spaces were completely removed will stop happening. Regex header and footer matching was tweaked to match at a later stage in the conversion pipeline. This should ease issues of expressions not matching properly. Finally, at Kovid’s request I’ve added some info about header / footer regexes and converting TXT and PDF files to the documentation.

Tags: calibre, eReader, fb2, pdb, pml, pmlz, rb, txt.

There was no week in review last week because I went on vacation this past week. So this week in review combines everything since the last week in review.

I’ve made a few bug fixes to some output formats, PDB metadata and FB2 output mainly. The major things I’ve been working on is a bit of restructuring for the GUI and fixing some small bugs.

The GUI has had the button in the status bar (jobs, tags, cover flow) moved to a side bar on the right hand side. The version information and device connected information has moved to the status bar. The donate button was moved to the side bar. The status bar is now collapsible. When collapsed it shows less information (the list of formats for the selected book). When expanded it shows the book info the same as it does currently. The location view that lists the library and the connected device is hidden when no device is connected. I’ve added an About button to the new sidebar that will show some information about calibre. Overall these changes have two major benefits. It makes the interface a lot more netbook friendly and makes the book table larger so more information can be seen.

These changes to the GUI will be part of the 0.6 series but I can’t say for certain if it will be included in the initial 0.6.0 release.

Tags: calibre, fb2, GUI, pdb.

Device interfaces can now be configured in the GUI. Also, there is a simple framework for creating plugin configuration widgets.

I’ve added a metadata reader for the eReader format. However, eReader supports 3 ways to set the metadata in the file. 1) In the pdb header (only supports setting a short title). 2) In the metadata section of the file (supports the most information: title, author, publisher, copyright, isbn). 3) Embedded in the text as a comment. 2 and 3 are only accessible if the book does not contain DRM (or has been unlocked, but Calibre does not support this). 3 is not supported at all with this metadata reader. The reader first tires 2 then falls back to 1 if the book is DRMed or if the metadata section is non-existent.

Two new input and output formats have been added. ztxt and palmdoc. They are both pdb formats like eReader. For input the pdb input plugin will automatically determine the internal format and call the appropriate code path. For output the default is palmdoc but there is an option –format that can be used to change it to any other supported pdb output format (ztxt is the only other currently). The format option is also available in the conversion dialog in the GUI.

Speaking of conversion in the GUI. It now works. There are all new dialogs for single and bulk conversion. Pretty much anything that can be done using the command line ebook-convert can be done in the GUI. Bulk, single and auto conversion are all complete and working. Auto conversion will also honor a users preferences for formats set for the device interface plugin.

Tags: calibre, eReader, GUI, palmdoc, pdb, ztxt.