John's Blog

Thanks to Grzegorz Wolszczak Sigil now (will be part of the 0.5 release) allows users to change keyboard shortcuts for many actions. Grzegorz has been helping out a lot and helped to introduce a preferences dialog and provided user configurable keyboard shortcuts.

Tags: GUI, Sigil.

Calibre

This week I focused on PDF output. There was a bug introduced in 0.8.17 that broke PDF output which has now been fixed. I was also able to fix PDF output on OS X. The PDF output engine on OS X is now using OS X’s internal PDF engine instead of Qt’s. Page sizes other than A4 are now possible and the PDFs produced are no longer large image based monstrosities. Meaning, text is now selectable and can be copied.

Sigil

I am currently working on Pearl compatible regular expression (PCRE) support. An initial version has been put into git. I have an enhanced version that allows for case changes in the replacement text working. Right now I’m working caching the results of a search to improve performance.

Tags: calibre, pcre, pdf, Sigil.

Introduction

With 0.4 my focus has been on getting the existing features in a stable state. I foresee 0.4 being around for quite some time as development shifts to accommodate new features. I wanted to be sure a relatively bug free version is available for people to use. If data loss is a constant then there wouldn’t be any point in using Sigil. Now that 0.4 is done it’s time to start working on what’s next.

Just what is next? For the time being I’ve marked a number of issues on the issue tracker as Milestone-0.5. My plan is to have 0.5 just implement the most commonly requested and most interesting features. 0.5 has no vision and is just a stop gap while I familiarize myself with Sigi’s code base. 0.5 is my short term plan. It’s not grand but it’s functional and sufficient.

Recently I posted the conclusion of my Sigil user study. The findings are Sigil is most used and most useful to power users and small professional ebook creating houses. Also, the overlap between the two is significant. Thus I want to target these two group and make Sigil even more useful for them. Keep this in mind because these two groups are who is going to shape my views of where I want to take Sigil.

Please realize that not everything I’m going to talk about is set in stone. A lot of it probably will never happend. Also, this is part plans, part what I want to do, and part rant about what Sigil does that I don’t like. This is what my ideal Sigil would look like and it is what I’m going to work toward. However, nothing is set in stone.

Plugins

If you’ve ever used calibre or Firefox you will know that plugins are amazing. They allow for easy and quick changes and additions to be made without having to change the main application. Both calibre and Firefox have large third party plugin communities. I would like to bring this to Sigil and I want a framework where all book manipulation is available over a plugin interface.

My feelings with Sigil are plugins should make small self contained changes. Similar to calibre’s heuristic processing. For instance, italicize common cases, up / down shift headings, and normalize CSS. To make plugins really useful I want to have a system where multiple plugins can be chained together and run in sequence. This would be super basic internal script functionality.

For plugins themselves I’m undecided about how they should be implemented. I don’t mean API wise because that isn’t even a thought at this point. I’m talking about what languages they should be able to be written in. C++ as a shared library will of course be supported because Sigil is written in C++. However, I want to Sigil to be able to load plugins written in scripting languages.

My first thought is Python because I’m very familiar with it and love to work with it. I’m also thinking about Lua and QtScript (Javascript without DOM). I don’t support frameworks for every one of these languages due to the amount of maintenance required. So I want to support only one scripting language. Python is big and slow. Lua is small but doesn’t have the advanced text manipulation libraries Python offers. QtScript is Javascript with is an abomination of a language. Added size of Sigil’s install, execution speed, ease of supporting, knowledge by contributors and text manipulation support are all major considerations.

Editor

Currently Sigil does not respect the structure of existing files. When you open an EPUB in Sigil it restructures the file layout. It even goes as far as to rewrite each XHTML file by running it through Tidy. With 0.4.0 cleaning with Tidy can be disabled but pretty printing is still used and alters the XHTML. I absolutely hate this! If I want my XHTML or file structure changed I’ll do it myself.

I want to change Sigil to not be as automatic. Restructuring and cleaning of the XHTML should be moved to plugins and run when the user requests it. This way a user can open Sigil, change the metadata, save, and the only thing that changes is the OPF with the metadata changes. Not every single piece of the EPUB.

I also hate WYSIWYG editing because it inherently must make drastic changes to the underlying code. I don’t think it’s a good idea to remove it though. I would prefer to have the book view default to a preview mode that is read only. There wouldn’t be any changes made to the code by using book view. Read only is the default but the user should be able to have an edit toggle that will set the book view to edit mode which will work like it already does. This way a user can make changes that may not be valid or work, check them, see there is an error (say a missing </a> tag) without losing any work. They can see the issue fix it and still be able to use WYSIWYG editing when they want.

Data Store

Right now XML (XHTML included) data is stored as a Xerces DOMDocument. This is then loaded into the book or code view depending on which one is focused. The use of a DOMDocument often leads to data loss. Putting malformed XML into a DOMDocument can have unintended consequences. Especially when then loading that into a QWebView and getting back a string.

I want to replace the DOMDocument with a plain string as the data store. This will prevent a lot of data loss, especially combined with the book view defaulting to read only. Further, this combined with not making automatic changes to the code will make the well-formed error warning unnecessary.

Not auto processing with Tidy and checking for errors automatically will allow Sigil to produce invalid EPUBs. I really don’t care that this can happen. The tools (FlightCrew) will still be there to check that the file conforms to the spec. It’s up to the author to ensure they’re publishing valid EPUBs. An EPUB that is being actively edited doesn’t have to be valid at all times. I’d rather put the onus on the person using Sigil to ensure their EPUB is correct before publishing versus having Sigil force validity at every moment.

Undo

Undo is terrible right now. Some actions cannot be undone, some can. The book view’s undo is completely separate from the code view. You can’t undo a replacement when doing it across all HTML files on files that aren’t open in a tab. I want to see a unified single undo that allows for setting back out of any change.

Further along this line I would like some graphical display where you can look at the changes that have been made to make it easy to find exactly how far back to undo. Something like Apple’s Time Machine but for the state of the book.

Conclusion

Here is where I want to take Sigil: less hand holding, less automatic changes and more advanced text manipulation though a plugin interface. The big question is, should I skip putting out a 0.5.0 release with just the Milestone-0.5.0 marked changes and get started on the above now?

Tags: future, plans, Sigil.

The official Linux packages for Sigil are generic packages. They’re bundled in an InstallJammer installer and contain a number of libraries that Sigil depends on. This is not ideal but it’s not possible to provide Linux packages for every distro.

I’ve created a wiki page which I’m putting together a list of Linux distributions that have their own Sigil packages. These are the best packages for users to install because they’re smaller and tailored.

If your distro isn’t listed and it has Sigil packages let me know and I’ll add it to the list. If your distro doesn’t package Sigil let them know you would like to see them package it. I’m always willing to lend a hand to get Sigil in more Linux distros.

Tags: distros, Linux, packages, Sigil.

Sigil 0.4.1 is complete and available. This is mainly a maintenance release and fixes a number of bugs. Specifically a few bugs related to data loss. There was one major user visible change. The well-formed error dialog can be toggled not to show. This will cause errors to be auto fixed. Use this with care because the auto fix Sigil makes might not be what you want. As always see the changelog for a complete list of changes.

Tags: release, Sigil.

Introduction

Since taking over as the maintainer of Sigil I have spent some time reaching out to specific people in the ebook community to ask them about Sigil. Specifically if they use Sigil? Why or why not? What do they see as Sigil’s shortcomings? How do they use Sigil in their work flow? Why doesn’t Sigil work in their work flow. Basically, their thoughts and opinions on Sigil.

I asked specific people privately because I didn’t want to be inundated with responses. The people can be broken down into three different groups: self publishers, power users, and professionals. After talking to professionals I’ve come to realize that they can be broken down into small and large. The size relating to the size of the company and production volume. I spoke with about 8 people total and I tried to keep it even between the various groups.

I wanted to find out who is using Sigil, who isn’t using Sigil and why so I can determine where I want to take Sigil in the future. The only ebook editing I do is cleaning up a few books here and there. Learning how people use Sigil will help me to determine the best direction to take the project.

Self Publishers

Self publishers are authors. These are people who write their book and then want to sell it as an ebook themselves. Typically these people are using Word for writing. they export their work as HTML, then import into an ebook editor for final adjustments and savings as an ebook file. The two biggest things self publishers are looking for are easy and high quality .doc or .docx import and one click send to store functionality.

Self publishers are also interested in WYSIWYG editing and don’t want to know about the internals of ebooks. They are primarily writers who see ebooks one of many distribution methods. They don’t care about the intricacies of EPUB for instance, they just want their work to look good and be readable by their audience.

The typical tools I hear being used by self publishers are calibre for format shifting. Atlantis Word Processor and Jutoh for formatting and base ebook creation. Atlantis and Jutoh both provide very easy to use WYSIWYG interaction and you can use these without ever seeing a line of code.

Power Users

These are people who prepare works in their spare time as a hobby. They are not motivated by money and do not sell the works they publish. Typically the works power users deal with are public domain such as Shakespeare. This group also encompasses people who do not distribute works covered by copyright but spend their time cleaning and reformatting their favorite books strictly for their own enjoyment and personal use.

Power users are comfortable using either WYSIWYG and code editors. The biggest feature requested and talked about by power users is robust regular expression support for search and replace. Many of the books power users work with have terrible and often non-existant formatting. These works typically started life as either a scanned copy of a print book or a PDF file. Both of which typically leave broken paragraphs and misspellings thought the document. Which leads to spell check being the next most common request from this group. They are trying to take a jumble of half sentences and put them back together into a visually appealing layout.

The tools used by power users are Sigil, calibre, Word or Open Office macros, and many custom scripts. Also an advanced text editor like BBEdit and Notepad++ are must have tools.

Professionals

Professionals format ebooks for one purpose, money. This is what they do for a living. An author comes to them and pays to have the company turn their work into an ebook. For a modest fee an author can have a beautiful ebook produced without any headaches or hassle. Many authors prefer paying someone to do this portion of publishing for them just like they will pay an editor to edit, a print house to print, cover artist to design a cover and so forth. Authors write and typically want to concentrate solely on writing. Many self publishers format their own ebooks out of necessity because of the cost of hiring a professional.

With both small and large professionals I’m specifically talking about ebook publishing and digitization services. I’m not talking about huge publishers like Macmillan that do everything. However, the larger publishers I talked to makes me believe their process is the same as the huge publishers. The big difference between small and large professionals are the tools they use.

 Small

Small professionals tend to use either Sigil or Adobe’s InDesign for a good portion of their work. Both fill a very similar role in ebook creation. The big draw of InDesign over Sigil is InDesign supports print book layout creation. It’s an all in one tool. This type of professional tends to use off the shelf tools that are readily available. Sigil and InDesign are not the only exclusive tools they use but one or the other tends to be a heavily used tool in their tool box.

Large

Large professionals tend to use custom tools. They staff people who’s sole job is to develop and maintain ebook creation and formatting tools. They can afford to have custom tools that integration directly into their process. They don’t use off the shelf or vanilla tools. This group is all about custom everything. This allows them to quickly adapt to changes.

Professional Tools

Sigil or InDesign and custom tools are all I know. Many professionals are vague about their process and tools. Some even declined to talk to me at all. They use tools in some way that works for them but their methods and implementation are proprietary.

What Does This Mean For Sigil?

Out of all of these groups I have little desire to target self publishers. There are existing tools that do a great job of meeting this groups needs. Sigil has a WYSIWYG editor and it can certainly be improved but I don’t want to tie Sigil to a particular store or stores like Amazon or B&N. Also, I want to keep Sigil as an EPUB editor and not a generic ebook editor. I believe that Sigil’s strength lies in being able to manipulate the internals of the EPUB format itself. I want to target this aspect more.

Power users are the major group I want to target. Out of all of the people I spoke with power users use Sigil the most and get the most out of it. Advanced editing of an EPUB’s structure and code is where I want to take Sigil. That along with advanced text manipulation. Think expansion of calibre’s heuristic processing.

Small professionals are major users of Sigil and I do not want to discount them. I believe that their use of Sigil overlaps with power users enough that targeting power users will also target small publishers. I do not want to alienate small professionals and will continue to take their needs seriously. From what I’ve learned about small professionals tools that make code manipulation easier will be a benefit and hopefully reduce their need for other formatting tools.

The last group, large professionals, do not use Sigil. I don’t believe that changing Sigil to accommodate this group will get them to use Sigil. They use their own custom tools and Sigil doesn’t fit into their work flow and I don’t see it ever doing so. Thus I don’t see it being worth while to work toward making Sigil “the tool” for this group.

Tags: future, plans, Sigil.

Sigil 0.4.1 is complete and available. This is mainly a maintenance release and fixes a number of bugs. There are a few new features mostly around the code view. As always see the changelog for a complete list of changes.

Tags: release, Sigil.

Last night I moved Sigil’s SCM from Mercurial (hg) to Git (git). The change was completed successfully and without any issues. 0.4.0 had locations for things like the user manual pointing to the git locations. These are now live and working again. These links are now broken in any release before 0.4.0. This is unfortunate but unavoidable.

Tags: git, hg, scm, source code.

The long awaited Sigil 0.4.0 release is now out. Along with Sigil is a new release of FlightCrew. FlightCrew version 0.7.2 is the latest version bundled with Sigil.

If you’re an OS X user then one thing need to be aware of is these two releases bump the minimum OS X version to 10.6 (Snow Leopard). The Linux builds (x86 and x64) are built on Ubuntu 11.04.

A lot of work went into 0.4.0. New features and bug fixes galore. See the ChangeLog for full details. I have to thank Strahinja Marković (the original creator) for leaving the 0.4.0 in a nearly finished state. Also, Charles King for being a bug fixing monster and helping make this release great.

One major change that is going to take place next week is, I am switching the source code management (SCM) system from Mercurial (hg) to Git. This will break the update checker in previous versions. Hence waiting a week before making the change. I have already converted FlightCrew (needed to test) so FlightCrew’s updater in previous version won’t inform you of an update. This is unfortunate but unavoidable.

Due to the change to Git the links for the manual in Sigil are not going to work for a week. This is because I’ve updated the locations in Sigil to point to the new locations that won’t be available until the switch to Git.

Tags: release, Sigil.

I’ve updated the donation links they now point to my PayPal account. Going forward donations will be directed toward me instead of Strahinja. This is something we had talked about and planned. I just want to warn people who have donated before because you will see a different email address than least time.

Tags: donations, Sigil.