John's Blog

For some time now Sigil has had two different help forums. One at MobileRead and the other as a Google Group. This has caused quite a bit of confusion because people don’t know the best place to go for help.

I’ve decided to close (it’s already done so don’t ask me to reconsider) the Google Group. This was an easy decision because 1) MobileRead’s sub forum gets more traffic, 2) I use MobileRead and I don’t use the Google Gorup, 3) Most posts on the Google Group were unanswered making it a poor place to go for help.

Tags: Sigil.

I’m happy to announce the release of Sigil version 0.5.0.

0.5.0 comes with a number of bug fixes and some major new features:

• Inline spell check in code view
• Support for PCRE in search and replace
• Translations into 15 languages

Please see the changelog for a full list of changes in this release.

One smaller change is I’ve decided to drop OpenCandy. Surprisingly I’ve only encountered one complaint about OpenCandy and it was directed at bundling offers for other software inside of an installer not at OpenCandy in particular. I want to make it clear that this change is not due to user request or opposition to OpenCandy but my own decision.

While I respect what OpenCandy the company is doing and I don’t see anything wrong with the offerings they provide I don’t think their system is right for Sigil. The big thing I don’t like is OpenCandy’s installer components are distributed as closed source binary modules. While my understanding is this can be used without running afoul of the GPL I fully believe it goes against the spirit of the GPL and open source in general.

Thank you to everyone who provided feedback and helped during the beta process.

Tags: release, Sigil.

Introduction

A question I often get asked is what kind of workflow is best for creating ebooks. The short answer is there is no best, perfect, or standard way to create ebooks. There are as many workflows as there are people. I have regular contact with a comercial company (Booknook.biz) that provides digital file to ebook creation servies (I am not affiliated with or work for this company). The company is broken down into teams and each team has a very different work flow, uses very different tools and they all produce the same quality (high) ebooks. Suffice it to say there is no one way that works best. What I want to outline is a few different ways that I have seen work well for people in common situations.

The big thing to think about is what you want to do with the final look of the ebook. There are applications like Jutoh and Atlantis Word Processor that bill themselves as the equivalent of Word for ebook creation. Atlantis even goes as far as being able to directly open Word documents. These two are not free but compared to the cost of Word itself they are not over priced. These are tools with strong WYSIWYG formatting support.

Sigil fits into the ebook creation picture in cases where a high degree of control is desired. While Sigil does have some WYSIWYG features, it excels at working on code that comprises an ePub internally. Sigil is very much for manual, do it yourself, or fine grain control needs. That said, many users of Sigil use it solely for its WYSIWYG features. Sigil also has the advantage of being free (price) and open source.

In all cases I recommend getting your book into the ePub format. As noted there are many tools for working with ePub. The internals of the format are HTML and CSS which means it’s easy to learn how to do advanced formatting. Also, ePub is an open, fully documented, and standardized format. Further, it’s very easy to convert an ePub into a variety of other ebook formats such as mobi for use with Amazon’s Kindle.

Authors starting with a Word document who don’t want to know about HTML or CSS that makes up the inside of an ePub

Often times an author has already written their book. They have it saved as a Word document or in some other similar format. The challenge in this case is going from their favored text editor to an ebook. In this case I recommend either copy and pasting the text from Word into Jutoh, Sigil or opening the Word document with Atlantis Word Processor. Use the inbuilt features to add the desired formatting and save as an ePub.

This works and can produce a functional ebook. However you are limited by what you can do with the formatting and dealing with quirks in various reading systems.

The case for manually working on an ePub’s internals

Using Sigil to actually work on the internals of the ePub allow for quirks in different reading systems to be accommodated. Two major examples I can think of that illustrate this are thumbnail images on the Nook Touch and the Kobo iPad/iPod apps. Both have very specific requirements in order for them to appear properly. Calibre, for example, takes issues like these into account when it creates an ePub but this doesn’t mean calibre handles every quirk in every reader out there. Manual intervention is sometimes required. Also, this goes back to professional digitization services (like I mentioned earlier) because they know about the little quirks of the most popular reading devices and can ensure an ebook looks the same across all devices.

Three stage ebook creation

What I can say is the best processes in my experience when using Sigil is three stage: write, convert, finish.

1. Write the book in what ever format you’re most confortable using what ever tools you’re most confortable with. The caveat is you really need to be working in a digital medium. At some point you will have to digitize and scanning in pages of paper then correcting errors is going to be a long and arduous process. In most cases simply typing your book into Word is going to save you time after you’ve finished writing if you go the pen and paper route. Most authors are going to use Word which lets them, write, edit and do basic formatting. At this point leave the formatting pretty basic.
2. Use an automated conversion tool like calibre to get your book into a format that is a bit more editable as an ebook. Word is a great editor but Word files are complex (and older versions do not have official format specifications). For Word the best advise is to have Word save as HTML. This will lead to it’s own issues but they can be worked around. HTML produced by Word will have basic formatting intact and the beauty of using an automated conversion tool is a lot will come through when converting to ePub.
3. Take that mostly formatted ePub put it into Sigil and make the necessary changes to make it look perfect. Since Sigil allows for direct access to the XHTML and CSS you can use a number of different techniques to ensure it looks exactly how you want. Also, if you are working on a number of book and want to ensure consistant formatting you can do things like easily use a default stylesheet. The best way I can sum up how the majority of people use and how I view Sigil is, Sigil is for people who know XHTML, CSS, and regular expressions and want to use their knowledge.

Three stage variation: Markdown and Textile with plain text

I personally use a slight variation of the above. I prefer to use plain text with either Markdown or Textile for basic formatting. Then conversion with calibre and final adjustments with Sigil. I use both depending on what I want to accomplish as I find Markdown more intuitive but Textile allows for things like margins and text alignment. Even without using Markdown or Textile converting using calibre with heuristic processing will get most novels about 75-90% formatted.

Three stage variation: Starting with HTML and CSS

The biggest difference with this variation is it skips right to step three. It is possible to still use calibre to take the HTML and produce an ePub but in this case I would advise against doing so. calibre is an automated tool and will make changes to the HTML and CSS. Visually it should look the same but the HTML and CSS will be different. This can be good if you have HTML and CSS that is so horrible you need it cleaned as calibre does some normalization but if you’re using your own hand crafted HTML and CSS you probably don’t want it modified.

Once you have all of your files ready, open Sigil and have a blank document open. Right click in the book browser on the text folder. Choose the Add Existing Files. Select your HTML files and import them. You should now see the Text section includes all of those files. Do the same with your CSS.

You many need to edit the HTML files to ensure that the CSS is referenced from the correct location. All stylesheets are located in the Styles directory which is one step above the Text directory. For example:

<head>
<link href="../Styles/stylesheet.css" rel="stylesheet" type="text/css">
</head>

If you have any images add them in the same way you added the HTML and CSS files. Also, make sure they’re referenced properly just like you did with the CSS.

Check the look in the book view to make sure everything looks correct. Also don’t forget to delete the Section0001.xhtml that is left over from it being a new file.

As you added each file the filename is perserved and it is added to the TOC based on the file itself. You have two options for working with the TOC. If you want to leave it file based you’re pretty much done. If you don’t like the text that displays in the TOC you can edit the toc.ncx and change the text.

You’re other option is to generate the TOC from headings. In the TOC browser (should be on the right hand side) click generate TOC from headings. Automatic TOC generation is based on h1 – h3 tags. h2 are nested under h1 and h3 are nested under h2. The only other way to deal with the TOC is to manually edit the toc.ncx by hand.

Conclusion

What I would recommend to authors is take the easiest path. If you’re  happy with the results of using Jutoh or Atlantis then your done. If you want to get a bit more involved use a variation of the three stage work flow I outlined. If all this seems like too much to deal with, pay someone to do it for you.

.

The first beta for 0.5 (0.4.902) is now available.

There are a few new features I’m most interested in getting feedback on. Inline spell check, translations, and the new PCRE engine. Of course crashes and major issues will be looked into and hopefully fixed before the final release.

.

The majority of the data loss issues have been mitigated at this point. With a work flow of open, save as after major changes and saving after minor ones, catastrophic data loss can be worked around to the point that Sigil can and is being used on a day to day basis.

That said, there are issues with data loss in Sigil and they are a priority. I’m currently finishing up the 0.5 release (I do not have a set release date at this point) which is mainly a feature release and only addresses some of the the data loss issue. For example you can still have everything in an entire XHTML document removed by putting a malformed XML header in the document.

The issue has three components that require major work to fix. I hope to have it all completed for the 0.6 release but it’s going to be some time it’s ready.

The issues are:

1) Sigil currently uses Tidy to clean all XHTML to ensure it conforms (as much as it can) to the XHTML spec. I have seen Tidy remove tags it thinks are empty when they influence how the document is rendered. I want to keep Tidy as part of Sigil but I believe it should only be run when the user asks for it and any changes it makes the user should be able to revert.

2) An intermediate data store is used that requires valid XML is used. This store shuffles data between the book and code view. Due to this store requiring valid XML (valid XHTML conforms) there is the potential for data loss if it has to auto correct the XHTML. If you are in code view and have malformed structural issues with the XHTML and move out of it there is a warning dialog. This only appears when you are working on one file at a time. If you are replacing across multiple files auto correction is used and this can lead to data loss. This data store needs to be replaced with one that does not require valid XML.

3) Putting malformed content into the book view will cause the book view to try to correct it. Again auto correction can lead to data loss. This is mitigated by the malformed error dialog but many users just disable it and find that sections of their document are missing after looking at it in book view. Also, the book view is a WYSIWYG tool so it does make structural changes to the document and these may or may not be what the user expects. As with Tidy changes made by the book view need to be able to be reverted. I am thinking about ways to make the fact that the book view more obvious that it makes changes to the document. This way the user is aware that they need to use undo (doesn’t currently work for book view changes) to revert the changes if they don’t like them. I’m thinking about using a preview mode by default that doesn’t make any changes and an edit mode to make this distinction obvious.

The above issues can be fixed but they are not quick or easy changes. I plan on making them for the 0.6 release as part of the changes necessary to support EPUB 3. However, there is the possibility that they will slip to 0.7 due to how large they are. Unfortunately, all I can say right now is I’m aware of the issue, I know what the cause is, and I have an idea of how to correct it but it’s not going to happen tomorrow.

Tags: Sigil.

I have been working on adding inline spell check to Sigil recently and ran into a quirk on Qt that isn’t immediately obvious. I ended up having to look though the Qt source code to understand exactly what was happening.

When dealing with a QPlainTextEdit you can get the QTextCursor and use the charFormat() function to retrieve the QTextCharFormat for the character before the cursor. This does not work when the formatting is set by a QSyntaxHighlighter!.

charFormat retrieves the character format that has explicitly been set on the QPlainTextEdit. QSyntaxHighlighter does not directly set the formatting on the QPlainTextEdit. Instead QSyntaxHighlighter sets the format in additionalFormats as part of the block layout. All formatting for the block the cursor is currently in can be accessed by using QPlainTextEdit::textCursor().block().layout()->additionalFormats().

QTextLayout::additionalFormats() returns a list of FormatRange objects. A FormatRange gives the start of the formatting (relative to the block not the full text in the QPlainTextEdit), the length and the formatting (as set by the QSyntaxHighlighter). Simply loop over all of the FormatRange objects and check if the cursor is within a range to determine what formatting is applied to a particular part of the block’s text. Use QTextCursor::positionInBlock() to determine the relative position of the cursor within the block.

Here is an example from Sigil that I use for spell checking. It determines if a particular segment of text has the misspelled word style applied to it. It then selects the text.

QTextCursor c = textCursor();
int pos = c.positionInBlock();
foreach (QTextLayout::FormatRange r, textCursor().block().layout()->additionalFormats()) {
    if (pos >= r.start && pos <= r.start + r.length && r.format.underlineStyle() == QTextCharFormat::SpellCheckUnderline) {
        c.setPosition(c.block().position() + r.start);
        c.movePosition(QTextCursor::Right, QTextCursor::KeepAnchor, r.length);
        setTextCursor(c);
        break;
    }
}

*Note: QTextEdit can be substituted any place QPlainTextEdit is used. This applies to both not just QPlainTextEdit.

Tags: c++, qt.

One of the the new features that has been implemented for 0.5 (release date yet to be determined) is support for Translations. For Sigil’s first supported language Grzegorz Wolszczak has provided a Polish translation. Currently translations are loaded based upon the current system locale. There no support for choosing the language via preferences. This may come at a later time but for now I believe that using the system locale will handle the majority of user needs.

I’ve put together a wiki page with instructions for creating translations. This first revision is a bit basic but as people have questions I plan to update it to make it more robust.

Tags: internationalization, language, qt, Sigil, translations.

Thanks to Grzegorz Wolszczak Sigil now (will be part of the 0.5 release) allows users to change keyboard shortcuts for many actions. Grzegorz has been helping out a lot and helped to introduce a preferences dialog and provided user configurable keyboard shortcuts.

Tags: GUI, Sigil.

About Formatting Tips.

This is a very easy formatting type and is similar to doing a drop cap. The big difference is a raised initial the letter is on the base line and is higher than the other letters. Only the first letter of the first paragraph of a chapter should be raised.

Simply wrap the first letter in a span tag referencing the appropriate CSS class like so.

<p><span class="ri">L</span>orem...</p>

The CSS for a raised initial is very easy. Simply make the font size larger than normal and set it to bold.

span.ri {
    font-size: 4em;
    font-weight: bold;
}

The best way to illustrate this concept is with an example. Download ft-raised-initial.epub. Opening the file with Sigil you will see the example chapters and the external CSS that is referenced by each XHTML file.

Tags: book, calibre, ebook, epub, formatting, FT, print, tips.

About Formatting tips.

Many non-fiction books utilize footnotes or endnotes. Footnotes do not work very well with ebooks because footnotes need to be placed at the bottom of a page. It is possible with EPUB to specify content to be shown at the bottom of a page using the following:

<div style="display: oeb-page-foot">...</div>

I highly recommend against using the oeb-page-foot display style. Many reading devices and software do not support this and just ignore this text. My recommendation is to use endnotes.

All endnotes should be collected into a single page located at the end of the ebook. I recommend using either a * or increasing numbers. If you use numbers the number in the text should correspond and be displayed infront of the endnote. Use the sup tag and link to the end note.

<sup id="ra"><a href="c2.xhtml#enda">*</a></sup>

One thing to keep in mind is not all ebook reading device and software support a back button. So it’s a good idea to include a return link to take the reader back to the exact place in the text the endnote is referenced by. Make the return link subtle but not something the reader will over look. I just make the font smaller.

<sub><a class="return" href="c1.xhtml#ra">return</a></sub>

.return {
    font-size: x-small;  
}

The best way to illustrate this concept is with an example. Download ft-endnotes.epub. Opening the file with Sigil you will see the example chapters and the external CSS that is referenced by each XHTML file.

Tags: book, ebook, epub, formatting, FT, print, tips.