John's Blog

The new epub thumbnail files (.epub.thn) are what Bookeen calls t4b files. They are very similar to the older t2b thumbnail files they were using in earlier versions of the Cybook firmware. As the name suggests instead of using 2 bits to represent color values 4 bits are now used. This increases the number of colors from 4 to 16. In addition to the increased color range the t4b files now require a header of “t4bp” without the quotes.

The image’s dimensions are 96×144. The bits representing 0, 1, 2, 3… are written directly to the file. it is very similar to a pgm file in this regard. Each 4 bit sequence represents a pixel color. Only black, white and shades of gray are supported.

Every t4b file will have 13,824 pixels. The file size will always be 6,916 bytes. The formula to determine this is: (height x width x 2 bits per pixel) / 8 bits per byte. ((96 * 144 * 4) / 8 ) + 4 = 6,916. The + 4 is the header.

Following are two python scripts for converting an image to a t4b file and for converting a t4b file into a pgm image.

image2t4b.py

#!/usr/bin/env python

import sys, Image

REDUCE_MARKS = [16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240]

def reduce_color(c):
    val = 0
    for mark in REDUCE_MARKS:
        if c > mark:
            val += 1
        else:
            break
    return val

def main():
    if len(sys.argv) != 3:
        raise Exception('Must have 2 arguments. %s input.image output.epub.thn' % sys.argv[0])

    outf = open(sys.argv[2], 'wb')

    im = Image.open(sys.argv[1]).convert("L")
    im.thumbnail((96, 144))

    newim = Image.new('L', (96, 144), 'white')

    x,y = im.size
    newim.paste(im, ((96-x)/2, (144-y)/2))

    outf.write('t4bp')

    px = []
    for p in newim.getdata():
        px.append(p)
        if len(px) == 2:
            byte_val = bin(reduce_color(px[0]))[2:].zfill(4) + bin(reduce_color(px[1]))[2:].zfill(4)
            outf.write(chr(int(byte_val, 2)))
            px = []
        elif len(px) > 2:
            raise Exception('Fatal error px length increased past 2.')

    outf.close()

if __name__ == '__main__':
    main()

t4b2pgm.py

#!/usr/bin/env python

import sys, os

def get_greys(b):
    if not b:
        return 0, 0

    b = bin(int(ord(b)))
    b = b[2:].zfill(8)

    w = str(int(b[0:4], 2))
    x = str(int(b[4:8], 2))

    return w, x

def main():
    if len(sys.argv) != 3:
        raise Exception('Must have 2 arguments. %s input.epub.thm output.pgm' % sys.argv[0])

    t4bfile = open(sys.argv[1], 'rb')
    pgmfile = open(sys.argv[2], 'w')

    pgmfile.write('P2\n96 144\n15\n')

    # Read past the t4b header
    t4bfile.read(4)

    for i in range(144):
        for j in range(48):
            b = t4bfile.read(1)

            vals = get_greys(b)
            pgmfile.write('%s %s ' % (vals[0], vals[1]))
        pgmfile.write('\n')

    pgmfile.close()
    t4bfile.close()

if __name__ == '__main__':
    main()

Tags: cybook, image, pgm, python, t2b, t4b, thumbnail.

The 2.0 fimware for the Cybook and Opus have new thumbnails for epub files. They use the .thn extension and it is append after the .epub extension. This is unlike the _6090.t2b thumbnails which use the book name without the extension and _6090.t2b appended to it. I have yet to start figuring out this new format but at first glance it looks to be similar to the _6090.t2b files.

What I have found is, if a _6090.t2b file is present that will be used and the .thn file will not be generated and the _6090.t2b will be used as the thumbnail. However, if both the _6090.t2b and .thn are present then the .thn will be used.

Tags: cybook, gadgets, image, thumbnail.

I’ve committed a fix for the Cybook 2.0 firmware id issue I posted yesterday. In calibre I’ve combined the Gen 3 and Opus device interfaces into one generic Cybook interface. Both were very similar and the Opus was just a subclass of the Gen 3. Also, with the 2.0 firmware being for both the Gen 3 and Opus there should be no difference in the capabilities of the two devices. If you own an Opus you will still want to use the Opus profile however, when you connect the device it will just show as using the Cybook interface. Also, with the interfaces being combined into one, the device specific configuration is now Cybook / Opus.

There were two reasons for combining the interfaces. I was unable to find a way to differentiate the Gen 3 and the Opus with the 2.0 firmware. A Gen 3 with the 2.0 firmware reports itself as an Opus. I was unable to find a way around this and felt it was better to have a Cybook interface. Also, I have the unfortunate pleasure of owning a Gen 3 which, even before the 2.0 firmware, used the same device ids of the Opus. It however did not use the Opus device strings. Due to this I am unable to determine if the 2.0 firmware changes the ids or just the strings. Combining the two interfaces keeps all Gen 3’s working and keeps some Gen 3’s from being detected as the Opus.

Tags: calibre, cybook, device interfaces.

On Christmas eve Bookeen finally released the long awaited 2.0 firmware for the Cybook Gen 3 and Cybook Opus. As soon as I returned from my Christmas vacation I upgraded my Gen 3 to the 2.0 version. Today while debugging a device issue for calibre I noticed that my Gen 3 kept being detected as an Opus. Doing a little digging I found this:

$ calibre-debug -d
Version: 0.6.32
USB devices on system:
...
['0xbda', '0x703', '0x110', 'Bookeen', 'Cybook Opus', '600AB038CB09484'],
...
Looking for CYBOOKG3
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Looking for CYBOOK_OPUS
	(3034, 1795, 272, 'Bookeen', 'Cybook Opus', '600AB038CB09484')
...
Devices possibly connected: Cybook Opus Device Interface,
Trying to open Cybook Opus Device Interface ... OK
Main memory: '/media/Cybook Gen3/'
Total space: (495251456L, 0, 0)

It seems Bookeen has once again screwed up the firmware release. They wanted one firmware so now every Gen 3 that has updated now has the product identifier set to Opus. The Gen 3 models use a number of different hex product and vendor ids and only the later ones are the same as the opus. Anyone on Linux or OS X with an earlier Gen 3 who is using the 2.0 (possibly 1.5 but I didn’t test it) firmware will not have their device detected by calibre any longer. This is because once the hex ids are matched there is further matching based on the text ids for Linux and OS X. This isn’t going to be fun to work around.

Tags: bookeen, calibre, cybook, device interfaces.

Not much on the calibre front for this past week from me. The only thing I’ve worked on was adding support to the Nook driver for the cover image to be sent to the device with the book. Also, if there is no cover associated with the book a default cover with the title authors and calibre library image are used. Similar to what already happens with Cybook Gen 3 and Opus.

Tags: calibre, cover, cybook, ebook, nook.

FB2 output has been improved. It no longer generated very invalid markup. The output generator still isn’t where I want it to be though. The changes are mostly cleanup and fix long standing issues with the output. One major change is I reverted having <h1> tags work as section and title markers. I don’t like having this hard coded into the generator.

As far as I can tell FB2 does not support a true table of contents (TOC). What seems to happen is reader software will dynamically generate the TOC based on the appearance of <section><title><p>text</p></title> within th e text. If I’m wrong about this and FB2 really does support an external TOC I would love to know. The <h1> differentiation causes the files to have these sections. I do not like how this is hard coded and dependent on this single tag. Especially since calibre’s conversion process allows for an XPath expression to be specified to generate TOC point.

I would love to use the TOC sent to the FB2 output generator but this does not seem to be feasible. The problem with the TOC that is sent to the FB2 output generator is how it corresponds to locations in the document. The OEB TOC points are text which may or my not appear in the text and an anchor id. The anchor id is a set point in the document. FB2 section titles are part of the text itself. I cannot use the text from the OEB TOC because it may or may not actually appear in the text. This also prevents me from determining what the text in the document is supposed to be associated with the TOC point. The anchor id points to an anchor in the document but often that point is something like <a id=”blah” />. In this case there is no text associated with the anchor in the document. While I can assume the text following is part of the title I have no fool proof way to determine where it stops.

At this point the only TOC associated with the FB2 output is the inline TOC that can be optionally generated.

Support for two new readers has been added. Ganaxa’s GeR2 and Nokia’s 770 internet tablet. The N770 should have been support a long time ago and I apologize for how long this request has been unfulfilled. I put it at the bottom of my todo list and at some point it simply fell off and I forgot about it.

The GeR2 reader was a bit of a challenge to get supported. This reader and some models of the Cybook Gen 3 have the same vendor, product and revision (BCD) ids. On Windows and OS X this is not an issue because once the ids are matched further matches are done based on the plug and play (PNP) strings. However, on Linux only the ids are matched.

To solve this problem, matching on Linux needed some further checks. Kovid added support for libusb-1 which provides the vendor and product strings. He also added a call back that can be implemented in the device interface to implement platform and device specific checks. We did run into a few problems. The first was an easy to solve 32 vs 64 bit issue with the Python to C interface Kovid wrote for libusb-1. Once that was sorted out we ran into a larger problem. libusb-1 on Ubuntu by default is denied access to the vendor and product strings.

libusb-1, after appearing in only run release (0.6.27), has been dropped. Kovid has now written a custom device scanner for Linux that will parser the devices in /sys/bus/usb to determine if a reader is connected. libusb-1 is supposed to be an easy to use library capable of providing this functionality but unfortunately this turned out not to be the case. The custom scanner works and allowed me to implement differentiation between the GeR2 and the Cybook Gen 3 so both readers can be properly supported without conflict and with the correct device interface being used.

Tags: calibre, cybook, device interfaces, fb2, GeR2, Linux, N770, TOC.

I unplugged my Cybook from the computer without first unmounting the volume. It was still in the process of deleting a few files. When I turned it on the files showed in the library but were unable to be viewed. After I plugged the Cybook back into the computer the file system was mounted as read only and dmesg spit out a large number of IO errors.

The Cybook’s file system was damaged. Thankfully the Cybook uses Fat32. All that was needed to fix the errors was to run the following:

sudo fsck.vfat /dev/sdb -artvVw

/dev/sdb is the device id for the Cybook’s memory. If you are unsure of what it is, plug in the Cybook, wait for it to be detected and run dmesg in the console. At the end of the output there will be information about the device being connected and what device id the system has assigned to it.

Tags: cybook, Linux.

The Cybook Gen 3 uses it’s own image file for thumbnails. The t2b file is generated by the device based upon the image found in the ebook. If there is no image a default one with the file name written across it is created.

One of the feature request for Cybook support in Calibre is for the t2b thumbnail files to be generated on the computer and moved to the device. This is much faster than having the Cybook generate the thumbnail itself.

The t2b file used by the Cybook is a 2-bit image. Meaning 2 bits represent 1 pixel. 2 bits can have a total of four combinations giving the image a total of four colours. There is no header or footer. The bits representing 0, 1, 2, 3 are written directly to the file. The image’s dimensions are 96×144.

Every t2b file will have 13,824 pixels. The file size will always be 3,456 bytes. The formula to determine this is: (height x width x 2 bits per pixel) / 8 bits per byte. (96 * 144 * 2) / 8 = 3,456.

Following are two python scripts for converting an image to a t2b file and for converting a t2b file into a pgm image.

image2t2b.py

#!/usr/bin/env python

import sys, Image

def reduce_color(c):
    if c  64 and c  128 and c > y) & 1) for y in range(1, -1, -1)])

def main():
    if len(sys.argv) != 3:
        raise Exception('Must have 2 arguments. %s input.image output.t2b' % sys.argv[0])

    outf = open(sys.argv[2], 'wb')

    im = Image.open(sys.argv[1]).convert("L")
    im.thumbnail((96, 144))

    newim = Image.new('L', (96, 144), 'white')

    x,y = im.size
    newim.paste(im, ((96-x)/2, (144-y)/2))

    px = []
    pxs = newim.getdata()
    for i in range(len(pxs)):
        px.append(pxs[i])
        if len(px) >= 4:
            binstr = i2b(reduce_color(px[0])) + i2b(reduce_color(px[1])) + i2b(reduce_color(px[2])) + i2b(reduce_color(px[3]))
            outf.write(chr(int(binstr, 2)))
            px = []

    outf.close()

if __name__ == '__main__':
    main()

t2b2pgm

#!/usr/bin/env python

import sys, os

def get_greys(b):
    b.zfill(8)
    b = "".join([str((ord(b) >> y) & 1) for y in range(7, -1, -1)])

    w = str(int(b[0:2],2))
    x = str(int(b[2:4],2))
    y = str(int(b[4:6],2))
    z = str(int(b[6:8],2))

    return [w, x, y, z]

def main():
    if len(sys.argv) != 3:
        raise Exception('Must have 2 arguments. %s input.t2b output.pgm' % sys.argv[0])

    t2bfile = open(sys.argv[1], 'rb')
    pgmfile = open(sys.argv[2], 'w')

    pgmfile.write('P2\n96 144\n3\n')

    for i in range(144):
        for j in range(24):
            b = t2bfile.read(1)

            if b != '':
                vals = get_greys(b)
                pgmfile.write('%s %s %s %s ' % (vals[0], vals[1], vals[2], vals[3]))
        pgmfile.write('\n')

    pgmfile.close()
    t2bfile.close()

if __name__ == '__main__':
    main()

Tags: calibre, cybook, ebook, python.