John's Blog

It’s been a while since I’ve made any posts. Mainly work and other matters have kept me away. I haven’t been working on very much recently.

This isn’t to say my participation with calibre has been abandoned. I’m still making regular bug fixes. I’ve also been working on a rather large change that is coming along much slower that I would like. Mainly metadata caching and muli way syncing. I was shooting for the end of March to complete the changes but it might not be done until the end of April.

I’ve also moved this blog off of the shared hosting package I was using. It’s now hosted on my own virtual private server. Attachments, images and the now reading list did not make it over in the transition. I’m still using Wordpress but instead of having multiple separate installs of it all the blogs here are powered by one Wordpress MU installation.

Tags: calibre.

The new epub thumbnail files (.epub.thn) are what Bookeen calls t4b files. They are very similar to the older t2b thumbnail files they were using in earlier versions of the Cybook firmware. As the name suggests instead of using 2 bits to represent color values 4 bits are now used. This increases the number of colors from 4 to 16. In addition to the increased color range the t4b files now require a header of “t4bp” without the quotes.

The image’s dimensions are 96×144. The bits representing 0, 1, 2, 3… are written directly to the file. it is very similar to a pgm file in this regard. Each 4 bit sequence represents a pixel color. Only black, white and shades of gray are supported.

Every t4b file will have 13,824 pixels. The file size will always be 6,916 bytes. The formula to determine this is: (height x width x 2 bits per pixel) / 8 bits per byte. ((96 * 144 * 4) / 8 ) + 4 = 6,916. The + 4 is the header.

Following are two python scripts for converting an image to a t4b file and for converting a t4b file into a pgm image.

image2t4b.py

#!/usr/bin/env python

import sys, Image

REDUCE_MARKS = [16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240]

def reduce_color(c):
    val = 0
    for mark in REDUCE_MARKS:
        if c > mark:
            val += 1
        else:
            break
    return val

def main():
    if len(sys.argv) != 3:
        raise Exception('Must have 2 arguments. %s input.image output.epub.thn' % sys.argv[0])

    outf = open(sys.argv[2], 'wb')

    im = Image.open(sys.argv[1]).convert("L")
    im.thumbnail((96, 144))

    newim = Image.new('L', (96, 144), 'white')

    x,y = im.size
    newim.paste(im, ((96-x)/2, (144-y)/2))

    outf.write('t4bp')

    px = []
    for p in newim.getdata():
        px.append(p)
        if len(px) == 2:
            byte_val = bin(reduce_color(px[0]))[2:].zfill(4) + bin(reduce_color(px[1]))[2:].zfill(4)
            outf.write(chr(int(byte_val, 2)))
            px = []
        elif len(px) > 2:
            raise Exception('Fatal error px length increased past 2.')

    outf.close()

if __name__ == '__main__':
    main()

t4b2pgm.py

#!/usr/bin/env python

import sys, os

def get_greys(b):
    if not b:
        return 0, 0

    b = bin(int(ord(b)))
    b = b[2:].zfill(8)

    w = str(int(b[0:4], 2))
    x = str(int(b[4:8], 2))

    return w, x

def main():
    if len(sys.argv) != 3:
        raise Exception('Must have 2 arguments. %s input.epub.thm output.pgm' % sys.argv[0])

    t4bfile = open(sys.argv[1], 'rb')
    pgmfile = open(sys.argv[2], 'w')

    pgmfile.write('P2\n96 144\n15\n')

    # Read past the t4b header
    t4bfile.read(4)

    for i in range(144):
        for j in range(48):
            b = t4bfile.read(1)

            vals = get_greys(b)
            pgmfile.write('%s %s ' % (vals[0], vals[1]))
        pgmfile.write('\n')

    pgmfile.close()
    t4bfile.close()

if __name__ == '__main__':
    main()

Tags: cybook, image, pgm, python, t2b, t4b, thumbnail.

The 2.0 fimware for the Cybook and Opus have new thumbnails for epub files. They use the .thn extension and it is append after the .epub extension. This is unlike the _6090.t2b thumbnails which use the book name without the extension and _6090.t2b appended to it. I have yet to start figuring out this new format but at first glance it looks to be similar to the _6090.t2b files.

What I have found is, if a _6090.t2b file is present that will be used and the .thn file will not be generated and the _6090.t2b will be used as the thumbnail. However, if both the _6090.t2b and .thn are present then the .thn will be used.

Tags: cybook, gadgets, image, thumbnail.

Teleread has published my Beginner’s Guide to Calibre.

Tags: calibre.

Kovid just tagged the 0.6.33 release so it should be out the door later today. Only one major feature from me this week. The PRS driver can new customize what gets turned into a collection. The default was and is series and tags. Now under the device interface configuration you can specify any metadata field. The main use for this change is so people who have a large number of tags can disable collections being created from them.

While it’s not coming out today I have been working on a major feature. Metadata caching for devices. Basically a file that specifies all of the metadata for the books on the reader will be created. It will be used when the device is connected and if any files have changed their metadata will be updated. This speeds up device detection dramatically. Also, takes up less space than a book so it won’t use too much space to store it. Right now it’s storing the data using json so other application can have easy access to the data too. Other than faster device detection this is the ground work that is required to implement syncing.

Tags: calibre, device interfaces, json, prs.

I’ve committed a fix for the Cybook 2.0 firmware id issue I posted yesterday. In calibre I’ve combined the Gen 3 and Opus device interfaces into one generic Cybook interface. Both were very similar and the Opus was just a subclass of the Gen 3. Also, with the 2.0 firmware being for both the Gen 3 and Opus there should be no difference in the capabilities of the two devices. If you own an Opus you will still want to use the Opus profile however, when you connect the device it will just show as using the Cybook interface. Also, with the interfaces being combined into one, the device specific configuration is now Cybook / Opus.

There were two reasons for combining the interfaces. I was unable to find a way to differentiate the Gen 3 and the Opus with the 2.0 firmware. A Gen 3 with the 2.0 firmware reports itself as an Opus. I was unable to find a way around this and felt it was better to have a Cybook interface. Also, I have the unfortunate pleasure of owning a Gen 3 which, even before the 2.0 firmware, used the same device ids of the Opus. It however did not use the Opus device strings. Due to this I am unable to determine if the 2.0 firmware changes the ids or just the strings. Combining the two interfaces keeps all Gen 3’s working and keeps some Gen 3’s from being detected as the Opus.

Tags: calibre, cybook, device interfaces.

On Christmas eve Bookeen finally released the long awaited 2.0 firmware for the Cybook Gen 3 and Cybook Opus. As soon as I returned from my Christmas vacation I upgraded my Gen 3 to the 2.0 version. Today while debugging a device issue for calibre I noticed that my Gen 3 kept being detected as an Opus. Doing a little digging I found this:

$ calibre-debug -d
Version: 0.6.32
USB devices on system:
...
['0xbda', '0x703', '0x110', 'Bookeen', 'Cybook Opus', '600AB038CB09484'],
...
Looking for CYBOOKG3
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Looking for CYBOOK_OPUS
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Devices possibly connected: Cybook Opus Device Interface,
Trying to open Cybook Opus Device Interface ... OK
Main memory: '/media/Cybook Gen3/'
Total space: (495251456L, 0, 0)

It seems Bookeen has once again screwed up the firmware release. They wanted one firmware so now every Gen 3 that has updated now has the product identifier set to Opus. The Gen 3 models use a number of different hex product and vendor ids and only the later ones are the same as the opus. Anyone on Linux or OS X with an earlier Gen 3 who is using the 2.0 (possibly 1.5 but I didn’t test it) firmware will not have their device detected by calibre any longer. This is because once the hex ids are matched there is further matching based on the text ids for Linux and OS X. This isn’t going to be fun to work around.

Tags: bookeen, calibre, cybook, device interfaces.

Teleread published an article of mine about converting e-books.

.

There was a major change this week to the device infrastructure. Kovid merged (with some modification) my changes to allow “Send to device” to use custom device paths just like “Save to disk”. Kovid’s major change to my implementation are having a separate save template for “Send to device” and allowing for per device overrides of the template. Kovid and I spent yesterday testing and it is working well. Expect it in the next release. Oh, News and the / tags still work as expected.

I did a little bit of work on TXT and PML output. Now they both honor the “Remove spacing between paragraphs” option. Previously TXT output had TXT specific options for this behavior. I’ve removed them and just use that look and feel option. PML output previously ignored it but now it honors it. So you can have both look more like a printed book than a web page.

Tags: calibre, device interfaces, pml, txt.

Background

My wife bought me an Astak EZ Reader Pocket Pro For Christmas this year. This device isn’t my first or even second ebook reader. It is now my third. The first a Sony PRS-505 having been commandeered by my wife. The second is a Cybook Gen 3 which due to the firmware update shortly before Christmas might stay my primary reading device.

Price

The Pocket Pro (PP) retails for 199 USD and US residents can purchase it at the the EZ Reader website. Mine came with a 2GB SD card, serviceable leather cover, usb cable, AC adapter, and the usual marketing / user materials. All this makes it the best deal I’ve found for ebook readers in the 5″ size.

Hardware

The PP is a 5″ device and uses an eInk screen like most ebook readers. I did not find the 5″ screen to be too small. It is a good balance between portability and readablility. It does cause a few more page turns than with the larger devices but it was not cumbersome in any way. Overall found the size to my liking.

It comes in a variety of colors and feels good in your hands. The paint gives it a rubberized texture. It’s light while still feeling solid and sturdy. Visually it isn’t the best looking device but the buttons along the bottom work well enough for navigation. It is very similar to how the Sony (non-touch) readers work. However, I do think Sony, having put the buttons next to where they correspond to the screen, makes it a bit more intuitive than matching the number to the button as is required by the PP.

One area where I felt the PP’s hardware design was problem is with the thumb wheel along the right hand side. I had issues using it to turn pages. It would often turn more than one page. It is also a hard plastic nub and after using it for awhile my finger started to hurt. I soon stopped using it and only turned the page using the buttons.

Firmware

One thing that I really like about the PP is how easy it is to change the firmware. There are a number of companies selling branded versions of the device. It is really a Hanlin V5 made by Jinke. The various companies that sell the device all have their own versions of the firmware that deviate to different degrees from what is produced by Jinke. The LBook has one of the more customized firmwares. I have tried it and found that it is a bit on the usable side because the majority of it is not in English. While there are a number of firmware options I’m going to focus the remainder of this review on the firmware available from Astak as of this writing.

Bookshelf

It works. That’s the nicest thing I can say about it. The only thing it does is list all folders in the storage location. You select the folder and it opens it. When you get to a book you want to read you select it and it opens. On the surface this doesn’t sound so bad but compared to other devices (the Cybook and PRS-505) it is terrible.

When listing folders it lists all folders. Even system folders that should and do not contain books. Also, it does not read any metadata such as author or title. You only have the folder and filename to go by. There are no tags, collections, genre views or custom sorting. Selecting books is a slow, cumbersome and painful process.

Another issue I have with the bookshelf is it tries to force the use of an SD card. It displays the SD card and the main memory separately. It also defaults to the SD card when ever the device is turned on. I’ve gotten used to the combined view other readers offer and I don’t care if the book is on the SD card or the main memory. I just want to be able to get to my book quickly.

Fonts are another thing I have an issue with. I like the fact that users can include their own fonts. You can also set a font as your default font so you don’t have to change it every time you open a book (this doesn’t actually work, see the EPUB Rendering section). However, the only way you can add your own fonts is with an SD card. They can only be read from an SD card. There is no way to add your own fonts by putting them in the main memory. I have no idea why this is the case but it is an annoyance.

The bookself falls flat but it’s not the main place a person will be. Reading books is the main purpose of the device. My ebook library is mainly in two formats. .txt and .epub. Lets talk about how well it works with reading these formats.

TXT Rendering

One major thing it gets right, in my opinion, is justified text. It does a pretty accurate representation of the text. Another thing it does well is you can change the font size easily. Page turns happed very quickly. Much quicker than my other readers which was a pleasant surprise.

However, it does do some fancy auto detection of components and renders them differently. Words are often hyphenated and span two lines. This is without regard to where or what comes on the second line. Many times the last two letters and the period will appear alone on the second line because it is the end of the paragraph. This causes the text to become disjointed and ugly.

EPUB Rendering

Just like with TXT rendering it is very accurate and just like TXT rendering this is also a problem. It’s so accurate that you cannot change the font. Only the font size can be changed. I tried reading two EPUB files with it and found both to be unreadable.

Harry Potter’s Bookshelf by John Granger was the first EPUB I tried. Upon initially opening it, the text was too small to read. Increasing the text size to a reasonable level made it possible to read the text. However. the margins increased as well. The book has small margins included but to have the text at a reasonable level the margins ended up taking up a quarter of the page, each top, bottom, left, and right. This made the screen essentially a small little window with text. The text being justified only allowed for a few words per line with large spaces between them.

Page turning with Harry Potter’s Bookshelf was completely contrary to how wonderful it is for TXT files. With this book, turning the page was very slow and it didn’t always work. 3 out of 5 button presses wouldn’t register. The light lit up and nothing happened. To make the page turn I had to start holding down the button until the page changed and if it didn’t change after a few seconds I would let go and hold the button down again.

The second book I tried was Word War Z by Max brooks. It doesn’t even open. This is not an issue with DRM because it had been removed.

So far the two books I have as EPUB that I want to read cannot be read on the PP. Those same books open and render beautifully on both the Cybook and the PRS-505.

Overall

The PP is disappointing. The hardware is nice; I really like the 5″ size. However, the poor bookshelf, the poor rendering, and the inability to even open some books makes it pretty much unusable. I’m going to keep looking into new firmware releases but until I can actually use it to read books it’s not much more than a poor substitute for a paper weight.

Tags: ebook, epub, ezreader pocket pro, gadgets, pocket pro, reveiw, txt.