John's Blog

Here is a Qt4 widget written in Python that allows for line numbers next to a QTextEdit. Similar to what is seen in a number of text editors such as gedit and kate.

from PyQt4.Qt import QFrame, QWidget, QTextEdit, QHBoxLayout, QPainter
 
class LineTextWidget(QFrame):
 
    class NumberBar(QWidget):
 
        def __init__(self, *args):
            QWidget.__init__(self, *args)
            self.edit = None
            # This is used to update the width of the control.
            # It is the highest line that is currently visibile.
            self.highest_line = 0
 
        def setTextEdit(self, edit):
            self.edit = edit
 
        def update(self, *args):
            '''
            Updates the number bar to display the current set of numbers.
            Also, adjusts the width of the number bar if necessary.
            '''
            # The + 4 is used to compensate for the current line being bold.
            width = self.fontMetrics().width(str(self.highest_line)) + 4
            if self.width() != width:
                self.setFixedWidth(width)
            QWidget.update(self, *args)
 
        def paintEvent(self, event):
            contents_y = self.edit.verticalScrollBar().value()
            page_bottom = contents_y + self.edit.viewport().height()
            font_metrics = self.fontMetrics()
            current_block = self.edit.document().findBlock(self.edit.textCursor().position())
 
            painter = QPainter(self)
 
            line_count = 0
            # Iterate over all text blocks in the document.
            block = self.edit.document().begin()
            while block.isValid():
                line_count += 1
 
                # The top left position of the block in the document
                position = self.edit.document().documentLayout().blockBoundingRect(block).topLeft()
 
                # Check if the position of the block is out side of the visible
                # area.
                if position.y() > page_bottom:
                    break
 
                # We want the line number for the selected line to be bold.
                bold = False
                if block == current_block:
                    bold = True
                    font = painter.font()
                    font.setBold(True)
                    painter.setFont(font)
 
                # Draw the line number right justified at the y position of the
                # line. 3 is a magic padding number. drawText(x, y, text).
                painter.drawText(self.width() - font_metrics.width(str(line_count)) - 3, round(position.y()) - contents_y + font_metrics.ascent(), str(line_count))
 
                # Remove the bold style if it was set previously.
                if bold:
                    font = painter.font()
                    font.setBold(False)
                    painter.setFont(font)
 
                block = block.next()
 
            self.highest_line = line_count
            painter.end()
 
            QWidget.paintEvent(self, event)
 
 
    def __init__(self, *args):
        QFrame.__init__(self, *args)
 
        self.setFrameStyle(QFrame.StyledPanel | QFrame.Sunken)
 
        self.edit = QTextEdit()
        self.edit.setFrameStyle(QFrame.NoFrame)
        self.edit.setAcceptRichText(False)
 
        self.number_bar = self.NumberBar()
        self.number_bar.setTextEdit(self.edit)
 
        hbox = QHBoxLayout(self)
        hbox.setSpacing(0)
        hbox.setMargin(0)
        hbox.addWidget(self.number_bar)
        hbox.addWidget(self.edit)
 
        self.edit.installEventFilter(self)
        self.edit.viewport().installEventFilter(self)
 
    def eventFilter(self, object, event):
        # Update the line numbers for all events on the text edit and the viewport.
        # This is easier than connecting all necessary singals.
        if object in (self.edit, self.edit.viewport()):
            self.number_bar.update()
            return False
        return QFrame.eventFilter(object, event)
 
    def getTextEdit(self):
        return self.edit

Tags: GUI, python, qt.

This time I missed last weeks week in review because I simply forgot. I’m hoping to keep this to a minimum in the future.

The big news is calibre 0.6 has been released. Kovid is now back to his regular (weekly at the least) bug fix releases too. As of now the latest version is 0.6.4 and I get the feeling that 0.6.5 is right around the corner. For a listing of what’s gone into the 0.6 release take a look here.

Some features I’ve been working on that are included in the current release are: asciiize text, and iRex Iliad support.

The asciiize feature is one I’ve been wanting for a while and I’ve finally implemented it. It’s based on the ASCIIize text post I made last week. There is now an –asciiize option for the ebook-convert command and an option for the conversion dialog in the GUI. The basic premise of this feature is unicode characters often are not displayed correctly by ebook readers. My Cybook Gen 3 exhibits this behavior. ASCIIize transliterates the unicode text to an ASCII representation. Meaning it takes “Михаил Горбачёв” and converts it to “Mikhail Gorbachiov”.

The iRex Iliad is now as supported as I can get it. calibre detects it as a device, displays the list of books on the device and you can send books to it using send to device. The part I’m running into issues with is the manifest.xml file. It looks like it’s similar to the Sony PRS’s media.xml file, meaning it is a quick store for metadata. However, I don’t really know what goes into this file or should I say files (there are more than one). It also doesn’t look like the device updates it in any way because I had a user send me the files off of their Iliad and they were empty even though the user has put 20 or so books on the device with the Mobipocket desktop software. On the bright side it works well enough that I don’t think anyone will notice.

On the GUI tweaks front (these won’t be in a release until some point in the future) I’ve added a history drop down to the search field. There is a swap button for authors and title in the metadata bulk dialog, and you can hide the toolbars in the ebook viewer.

The remainder of what I accomplished over the past two weeks was bug fixes and code refactoring. Just th boring stuff that I would rather put off versus actually doing it.

Tags: calibre, ebook, GUI.

There was no week in review last week because I went on vacation this past week. So this week in review combines everything since the last week in review.

I’ve made a few bug fixes to some output formats, PDB metadata and FB2 output mainly. The major things I’ve been working on is a bit of restructuring for the GUI and fixing some small bugs.

The GUI has had the button in the status bar (jobs, tags, cover flow) moved to a side bar on the right hand side. The version information and device connected information has moved to the status bar. The donate button was moved to the side bar. The status bar is now collapsible. When collapsed it shows less information (the list of formats for the selected book). When expanded it shows the book info the same as it does currently. The location view that lists the library and the connected device is hidden when no device is connected. I’ve added an About button to the new sidebar that will show some information about calibre. Overall these changes have two major benefits. It makes the interface a lot more netbook friendly and makes the book table larger so more information can be seen.

These changes to the GUI will be part of the 0.6 series but I can’t say for certain if it will be included in the initial 0.6.0 release.

Tags: calibre, fb2, GUI, pdb.

This week has been a productive one. I’ve made a lot of small GUI enhancements and did some work on PDF input as well. All of these changes have not made it into trunk yet. This is mainly because Kovid has been away this week.

I’ve added auto complete to a number of the input control on the GUI. Authors, Publisher, and Tags all auto complete pretty much everywhere now. The Tags will even auto complete in the table view in the main window. However, Authors, Series and Publisher do not auto complete in the main windows as of yet.

I’ve also been working with the GUI’s search. ISBN, Rating, Cover fields are all included in the default search. They are also search field identifiers. Meaning you can do isbn:123 to search just isbn numbers. Searching for empty and filled fields has been implemented as well. Use field:false and field:true respectively.

PDF input, either is or last week, got the ability to specify an unwrap factor for unwrapping lines. Previously this was a fixed value. Now it can be changed by the user. I have some ideas to enhance this further but I’m not going to to into detail because they may not materialize. Use the option –unwrap-factor with a decimal value 0 – 1. It is used by the regular expression that determines the minimum line length required for unwrapping.

PDF input had another highly requested change. The ability to remove headers and footers. However, it’s not as user friendly as I would like. There are four new options in total. –remove-header, –remove-footer, –header-regex, and –footer-regex. If the the –remove-* options are used then a regular expression that can be customized by using –*-regex is used to match headers and footers. The header and footer matching happens before all other processing rules. Use the ebook-convert’s –debug-input option to see the HTML that the regex will be matched against.

$ ebook-convert input.pdf .epub --debug-input output_dir/

Tags: calibre, GUI, pdf.

This week hasn’t seen very much in the way of new features from me. I’ve only added one. This is mainly because I’ve been doing small bug fixes leading up to the beta for 0.6.

The new feature, which Kovid helped me to implement, is ejecting the reader from within the GUI. When you mouse over the reader icon in the location list it will show an eject button next to the icon. Clicking the eject button will do just that, safely remove the device from the system and remove it from the location listing. I did the GUI work, created the interface for the device driver and added the Linux ejection code. Kovid added the OS X code and we both came up with a Window solution but his was more robust to it was used.

Tags: calibre, device interfaces, ebook, GUI.

A lot of work went into eReader and PML to have it supported better. Also, a new format has been added.

The XHTML to PML parser has been completely rewritten. It is based on the XHTML to FB2 parser I wrote for FB2 output. It produces much better looking PML markup and the displayed output looks very close to the original XHTML source. One major advantage of the new parser is that it accounts for XHTML style information and translates that into PML tags. For example if text is set to bold by CSS then the text will get the bold PML tag.

eReader also got another very important addition. Support for Makebook (202 bye header) file input. Makebook and Dropbook are the two applications provided by eReader (the company) for producing eReader files. Makebook is the older application that is no longer supported. Makebook and Dropbook produce very different record 0 headers. This header has information about where the text, images and other things contained in the file are located. It took a while but I’ve been able to understand enough of the Makebook header to add input support for these files.

Makebook produces a 202 byte header while Dropbook produces a 132 byte header. After comparing header values and section sizes I was able to determine that the 2 byte int at offset 0×08 contained the start of the non-text offset. Just like the 132 byte header files, everything before this offset is text.

Images in the 202 byte header files were easy to find because they are in the same format as the Dropbook produced files. However, I didn’t bother to determine if there was a header value. Since all images are in PNG format and the their section start with the text PNG, I simply loop though all non-text sections and see if they start with PNG. If they do I know it’s an image and extract it.

The hardest part of the 202 byte header files was the text itself. Even though I knew which sections contained the text I didn’t know how it was compressed. This is where Google came to the rescue. On the homepage for the Z-DOC PalmPilot application I found there was some work to reverse engineer this older format. This page gave me the information I was looking for. Text is PalmDoc compressed and then xored with 0xA5. It looks like this xor is an attempt to obfuscate the compression used to make it harder to decompress. It isn’t for copy protection because the Makebook application only produces non-DRM files. DRM eReader files from that time would be created in a different manner.

Syncing news now supports auto convert in the GUI. It’s just like auto convert with sending email and sending an eBook to a device. If the book is not in a format supported by the device it will be auto converted to a supported format based on user preference.

The final bit of work this week was support for the RocketBook (RB) format. Both input and output are working. Though they both do need testing. Output in particular as I don’t have a device that supports these files so I can only guess based on my input code that the RB files produced are 100% correct. If someone has a device that reads RB files please let me know if the output files are read correctly.

Tags: calibre, eReader, GUI, pml, rb.

This week comes with some great new additions. It also comes with some great new challenges. Not to mention more work for next week.

eReader output is complete and working. The files produced are the same format as those produced by Dropbook (more on this in a bit). In addition to eReader output I’ve added metadata writing for eReader files.

There is one major issue I’ve come across concerning the eReader format. The files produced by Makebook are significantly different than those produced by Dropbook. The files I’ve been using for reverse engineering the format have all been produced by Dropbook. My implementation for eReader input only works with Dropbook format files. As such, Makebook produced files will not work and are currently not supported. They will be unsupported for the foreseeable future because of how different the format is to Dropbook, and because Makebook is not supported nor being developed. It was replaced by Dropbook some time ago.

The PDB container format also got a metadata writer. However, the PDB wrapper itself only supports setting the book title. So that’s all that gets written to PDB files that either don’t support metadata in their internal format or that don’t have their own specific metadata writer.

eReader wasn’t the only format to get some output work. FB2 output has been added as well. However, there is no metadata writer for it yet.

The final bit of work I did for this week was to add auto convert to sending by email in the GUI. It’s the same idea as auto convert for sending to a device. If the file that is being sent is not in a format that is accepted by the email address (this is configurable setting) the file will be auto converted to a suitable format before being sent.

Tags: calibre, eReader, fb2, GUI.

Device interfaces can now be configured in the GUI. Also, there is a simple framework for creating plugin configuration widgets.

I’ve added a metadata reader for the eReader format. However, eReader supports 3 ways to set the metadata in the file. 1) In the pdb header (only supports setting a short title). 2) In the metadata section of the file (supports the most information: title, author, publisher, copyright, isbn). 3) Embedded in the text as a comment. 2 and 3 are only accessible if the book does not contain DRM (or has been unlocked, but Calibre does not support this). 3 is not supported at all with this metadata reader. The reader first tires 2 then falls back to 1 if the book is DRMed or if the metadata section is non-existent.

Two new input and output formats have been added. ztxt and palmdoc. They are both pdb formats like eReader. For input the pdb input plugin will automatically determine the internal format and call the appropriate code path. For output the default is palmdoc but there is an option –format that can be used to change it to any other supported pdb output format (ztxt is the only other currently). The format option is also available in the conversion dialog in the GUI.

Speaking of conversion in the GUI. It now works. There are all new dialogs for single and bulk conversion. Pretty much anything that can be done using the command line ebook-convert can be done in the GUI. Bulk, single and auto conversion are all complete and working. Auto conversion will also honor a users preferences for formats set for the device interface plugin.

Tags: calibre, eReader, GUI, palmdoc, pdb, ztxt.

This weeks review of what I’ve been working on is a little late. Overall it wasn’t as productive as last week looking at what was accomplished but I spent just as much time coding as last. With projects like this you can’t judge output by the number of features add or bugs fixed.

The GUI received context sensitive treatment for the device menu. It will only have send to device, when a device is connected and send to card A and B will only enabled when they are available as well. A simple change but one that will reduce confusion.

I’ve spent a lot of time working with Lee Dolsen (ldolse from mobileread) on pdftohtml processing rules. They are nearly complete and the output is looking really good. I know I’ve been saying that for a while now but each week it just keeps getting better. However, PDF is still not an ebook format and should not be treated as such. This simply helps to get content out of the PDF format and into a more manageable one.

One big thing I spent most of my time this week on was eReader input. Yep, eReader pdb files can now be converted to any supported output format. Metadata reading of eReader files is not yet supported. That is on my todo list. The html it produces could probably use some work but that will come as people report issues once 0.6 is released.

The other big thing that has taken up my Calibre time is eReader output. Sadly, it does not work. Also, it will not be working for the foreseeable future. The issue I’ve run into is I don’t know enough about the format to produce a file that can be read by eReader’s reading software. The main problem I face is there are around 66 “sections” to the eReader format header (not the pdb header, this is record 0 of an eReader file). I know what 10 of those sections are and what values they should have as they are used for my reader. Around 40ish of the sections should have a value of 0. However, that leaves 26ish sections that I don’t know what they are, what they do or what value they should have and how it relates to the rest of the file. Suffice it to say until I know more about the format I won’t be able to complete the output plugin.

Oh, I did write an inspector script (it’s in the eReader directory in the Calibre source tree) to help understand the eReader format. If anyone is interested in analyzing the format they can use it to help them see what is in the header.

Tags: calibre, eReader, GUI, pdf.