After a break of several months, I’m getting back to working on Press. Status is pretty much the same as last time I posted about it. (It’s actually even a little more behind than that, since I had HarfBuzz Python bindings working then, but now — after upgrading to macOS Sierra — I’m running into issues with PyGObject’s introspection module. I may end up having to write my own HarfBuzz bindings with CFFI. We’ll see.)
The high-level roadmap right now: get font embedding to work correctly, add support for embedding images (which should be fairly easy, I think), integrate ICU for language analysis and HarfBuzz for shaping, and add color space support.
As of now, I plan to use Press for making language charts (which I’ve been using PlotDevice for) and picture books. Once it’s to the point where I can do that, then I’ll start on Ink (low-level typesetting engine, intended for typesetting books, and higher-level rule-based engine for making it easier to work with).
Progress on Press has been a bit slower lately. I’ve fixed most of the errors I discovered by running the exported PDFs through the 3-Heights PDF validator. I also refactored the code and reorganized the package per Kenneth Reitz’s advice.
I’ve implemented initial support for embedding subsetted fonts (doing the subsetting via fontTools.subset), and while the fonts (including uninstalled fonts) display fine on my macOS box, the PDFs don’t validate properly and the fonts don’t show at all on iOS, which means the embedding isn’t actually working right. Current suspects include the /Differences array (which I’m not generating properly yet) and the CMap (which I haven’t implemented at all yet). I still have to implement ToUnicode as well, so that copying and pasting does what it should, but I’m fairly certain that isn’t what’s causing the fonts to not embed properly.
I’m also trying to figure out color spaces. In general I believe I want the output to be either DeviceRGB or DeviceCMYK, with some way of specifying an output intent, and also an option for the user to embed an ICC profile if they want. I’m part of the way there.
Anyway, the font stuff is far more complicated than I expected going in, but I’m still making progress, and I’m learning a lot.
Rather than starting work on Ink with the low-level typesetting engine, I’m thinking it’ll be worthwhile instead to start with a processor that goes through Ink rules and outputs TeX and/or SILE code. More to come later.
Font maps are admittedly extra work, but they do have some advantages as well: you can use fonts you haven’t installed, for example, and you can specify exactly which font files you want to use. And I can’t see any good way around the lack of a cross-platform font selection mechanism (meaning, a way to pass in ‘Minion Pro’ with specific weight and styles, and get a font filename in return).
Anyway, I’m in the middle of reading the PDF spec on CIDFonts and CMaps. It’s … complicated. It makes my head hurt. But it’ll be awesome when it’s done.
The plan for Ink took a bit of a turn a few nights ago. Erlang’s pattern matching was on my mind (having read about it earlier that evening) when I came across a passage from Mitchell’s Book Typography on house rules:
The following are examples of the authors’ own house rules:
Speech to be indicated by single quotation marks (‘quote’ not “quote”)
Circa shortened to italic c. with no word-space (c.1895 not c. 1895)
Use multiplication symbol, not ‘x’ for dimensions (24 × 36 not 24 x 36)
Letter-space strings of capital letters (ABCDnot ABCD)
The two ideas came together and I saw that declarative typesetting (rule-based typesetting) could be a much nicer way to typeset.
For example: if you want to use the multiplication symbol for dimensions (× instead of x), you usually have to edit your source file (InDesign, TeX, etc.), find matching instances, and change them. It’s a one-time thing, a permanent transformation.
If, on the other hand, you had a rule that said “find any ‘x’ characters between numbers and transform them to ‘×’), then you could leave your source file alone and let the rule do the work for you instead. Using rules like this — textual and stylistic transformations applied at compile time — seems far more reusable, shareable, and easier to use.
From there, the Ink language morphed almost completely from how I was envisioning it earlier (TeX with nicer syntax, basically) to this new thing, inspired by Erlang, XSLT/XPath, CSS, Inform, and more.
A few quick notes before we get to the examples:
I’m leaning very much towards a template/data separation, like in Django and Mustache and other template engines popular in web frameworks.
For this rule-based thing to work well, you have to be able to set general rules but also fix specific cases where the rule doesn’t apply, or where it doesn’t make sense to write a general rule. At the moment I’m leaning towards having those specific fixes be rules as well, rather than tagging the source file. See the second rule listed in Exceptions below for an initial stab at this idea.
Ink will be three languages — High Ink (or just Ink), which is the rule-based language shown below; Medium Ink, a tagged version of the source text with all the rules applied; and Low Ink (har har), a page description language that gets compiled to PDF.
Splitting it up like this allows for extra flexibility — it would be relatively easy, for example, to write a compiler that takes Medium Ink and outputs HTML/CSS or EPUB or what have you. I don’t know that that would actually be a good idea, but it’s more possible this way. Splitting it up also makes it more manageable.
Rephrased, the Ink-to-Medium-Ink compilation involves applying the rules intelligently. Medium-Ink-to-Low-Ink compilation involves the typesetting itself — line breaks, page breaks, etc. Low-Ink-to-PDF compilation will be easiest, translating the Low Ink code to PDF code.
This morning I came across Jon Gold’s post on declarative design tools, with somewhat similar ideas. I like the direction he’s gone in with the combinations — it’s a nice workflow. We could do something in that vein here, with syntax to output a bunch of variations (typefaces, sizes, leading, etc.) with minimal effort.
Note: these are all first-draft thoughts on how to do this kind of a thing. Syntax is very much not set in stone at all — rule/endrule vs. rule { }, selector syntax, whether to use regular expressions or something simpler, etc.
rule
size 6x9";
# alternates
size letter;
size 210x297mm;
size a4;
font Arno Pro, 10/13pt;
margin 1";
inner-margin .75"; # overrides earlier margin value
endrule
# Named rule (for use later)
rule @times
find \dx\d : replace \1 × \1
endrule @times
rule @year-labels
# Find "a.d." and turn on the smcp OpenType feature
find a.d. : feature smcp;
# alternate way
find [b.c. | b.c.e. | c.e. | a.d.] : feature smcp;
endrule
# Exceptions
rule
# If a.d. is found in heading style text, don't run @year-labels on it
find a.d. (style=heading) : ignore @year-labels;
# Find a specific "m.a.d." and don't run @year-labels on it
# This selector language needs a lot of work
find /chapter:4/paragraph:2/word:[m.a.d.] : ignore @year-labels;
# Usually, though, you'd want to revise the general rule like this
# Only run the rule if it's by itself (word boundaries) and not a heading
find \wa.d.\w (style!=heading) : feature smcp;
endrule
rule @paragraph-indents
# Indent first line of all paragraphs 1.25em;
find %paragraph : initial-indent 1.25em;
# Override for first paragraph of a section/chapter
find %paragraph:nth(1) : initial-indent 0;
# Paragraphs following tables aren't indented
find %table + %paragraph : initial-indent 0;
# Hanging indents for paragraphs with hanging tag
find %paragraph.hanging : hanging-indent .125";
endrule
# Tracking/kerning
rule
# Find sequential uppercase and bump tracking up
find [A-Z]+ | tracking 50;
# Find V followed by a and kern -25
find Va | kern -25;
endrule
# Coptic (named Unicode range)
range @coptic
u+2c80 .. u+2cff;
u+03e2 .. u+03ee;
u+03ef; # separated for demo purposes
endrange
rule @coptic-font
# Any characters in the Coptic range should be set in Antinoou
find range @coptic : font Antinoou, 24/28pt, dlig;
endrule
# Unicode properties
rule @numbers
# Replace any numbers with old-style figures
find [ unicode.Nd | unicode.No ] : feature onum;
endrule
# Masters
master @a
frame ...; # incomplete, but this part would have text frames
endmaster
# Apply masters
rule
# All pages get master @a by default
page * : master @a;
# Remove master for pages i-iv
page i-iv : master none;
# Ignore @paragraph-indents rule on page 9
page 9 : ignore @paragraph-indents;
# Set data to be put into masters
data @main, @index;
# Variable used for running heads
$title War and Peace;
# Set running heads
# (@inside-header etc. are frames in the master)
page.odd : @inside-header $pagenum;
page.odd : @outside-header $section; # set in data
page.even : @center-header $title;
endrule
# Data
data @main
transform @ingest;
include preface.txt;
include chapters/chapter*.txt;
enddata
data @index
include index.json;
enddata
# Transform
# Used for an initial transform if necessary
transform @ingest (text)
// JavaScript or other scripting language
// These aren't great examples, though
var response = text.replace(/CHAPTER/, "\n\nChapter");
response = response.replace(/\wteh\w/, "the");
return response;
endtransform
# Styles
# I don't have an example yet of how to use this, but imagine
# something ala InDesign or CSS
style @heading1
font Warnock Pro;
size 18/24pt;
onum;
-smcp; # turn off small caps
space-before 1.24em;
space-after 1.24em;
endstyle
Going forward
I’m open to feedback on all of this, of course, so feel free to comment or get in touch with me.
I’ve renamed inkpdf to Press (as in printing press).
I reached the point where creating the PDF manually is no longer feasible, so I’ve been working on getting Press to a point where I can implement the PDF generation. The basic structure is in place, sans the PDF part. (That’s next.)
Here’s what a Press script looks like right now:
from press import Press
p = Press('output.pdf', width=6*Press.INCH, height=11*Press.INCH,
margin=1*Press.INCH)
# Horizontal borders at top and bottom of page
p.stroke('#000')
p.pen(1.0)
p.line(p.page_min_x, p.page_min_y, p.page_max_x, p.page_min_y)
p.line(p.page_max_x, p.page_min_y, p.page_max_x, p.page_max_y)
# Page 2
p.page(2)
p.layer('base')
p.stroke(rgb=(1, 0, 0))
p.line(150, 150, 300, 300)
p.layer('fg')
p.stroke(hsl=(0, 0.5, 0.8))
p.line(300, 300, 450, 150)
# Go back and add another line to page 1
p.page(1)
p.stroke('#025')
p.line(p.page_min_x, p.page_min_y, p.page_min_x, p.page_max_y)
p.save() # this doesn't work yet
You can also do something like this:
with Press('output2.pdf', size=Press.LETTER,
margin=(0.5*Press.INCH, 1.0*Press.INCH),
inner_margin=0.5*Press.INCH,
outer_margin=1.25*Press.INCH,
bleed=.125*Press.INCH) as p:
p.line(50, 50, 250, 50)
# And so on
(Context manager, inner/outer margin, bleed, built-in paper sizes.)
Up next: adding more primitives, designing the font selection mechanism, getting it to generate an actual PDF, embedding fonts, using arbitrary Unicode code points, integrating HarfBuzz, etc.
I’ve been hand-coding PDFs in Vim, reading the PDF spec to learn how things work. It’s fascinating. My first, extremely simple PDF:
%PDF-1.4
1 0 obj << /Type /Catalog /Pages 2 0 R >>
endobj
2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 >>
endobj
3 0 obj << /Type /Page /Parent 2 0 R /Resources 4 0 R /MediaBox [0 0 500 800] /Contents 6 0 R >>
endobj
4 0 obj << /Font << /F1 5 0 R >> >>
endobj
5 0 obj << /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
endobj
6 0 obj
<< /Length 44 >>
stream
BT /F1 24 Tf 175 720 Td (Hello World!) Tj ET
endstream
endobj
xref
0 7
0000000000 65535 f
0000000010 00000 n
0000000059 00000 n
0000000116 00000 n
0000000220 00000 n
0000000263 00000 n
0000000333 00000 n
trailer << /Size 7 /Root 1 0 R >>
startxref
427
%%EOF
It’s not as bad as it looks, I promise. (I’m doing PDF 1.4 because CreateSpace doesn’t seem to support higher versions of the spec.)
Anyway, I’ve been reading through chapter 5 of the spec, learning how text works in PDF. I’ve learned how to modify character spacing with Tc, word spacing with Tw, leading with TL, and individual glyph positions with TJ (not sure yet if I can change vertical positioning or not). I’ve also learned how to change the text color. It’s all been fairly straightforward.
As part of this, I’ve used Hex Fiend (an OS X hex editor) to pry apart some simple PDFs I made with PlotDevice, to see how things were encoded. The streams themselves are generally compressed through Flate compression (opposite of deflate, har har), and I found this script to easily decode the streams:
#!/usr/bin/env python
import zlib
import sys
input = sys.argv[1]
output = sys.argv[2]
with open(input, 'rb') as f:
buffer = f.read()
decomp = zlib.decompress(buffer)
with open(output, 'w') as f:
f.write(decomp)
I copied each stream in hex from Hex Fiend, pasted it into a file, ran the Python script on it, and it would output decoded text to a new file.
Things I don’t know/understand yet, which are legion:
How to encode Unicode (I’m not to this point of the spec yet, but I believe it involves CID fonts and using cmaps to map glyph codes or something like that).
How to take a font name and, in a cross-platform way, get the path to the font file so I can embed it and also use it with HarfBuzz.
How to take the output of HarfBuzz (a list of glyphs with position coordinates for each) and use that in positioning the glyphs in the PDF. I believe HarfBuzz will handle parsing the OpenType features of the font, but I’m not positive on that. I did get HarfBuzz Python bindings working, though, and I plan to play around with it soon.
Whether I need to use FreeType at all. I might need it for font metrics, but HarfBuzz might give me everything I need there.
When typesetting multiple lines, I don’t know whether it’s best to use the PDF built-in support (T* and TL and such), or to set each line manually as its own text object. The built-in support seems better, though I don’t know if that limits what’s possible.
At some point soon — I think when I start embedding fonts — doing this by hand in Vim will stop being as feasible, and at that point I’ll start writing Python to manage the PDF creation process for me. For now, though, it’s easier to just edit the PDF manually.
As mentioned on Twitter, I’ve decided to write my own typesetting engine, called Ink. Apparently I’m crazy.
The details are still very much in the air, but here are some quick notes:
Written in Rust (for speed)
Programmatic (sort of like TeX)
Scripting language for extensibility (JavaScript or Lua or Python)
Intended for use in typesetting book interiors, covers, and charts
Possibly some kind of template/data division
Full OpenType feature support (shaping via HarfBuzz)
Custom PDF generation library (inkpdf)
Reasons for doing this insane thing:
PlotDevice only runs on OS X and I want the source of my language charts to be usable on other platforms.
I’d like to open source the books I typeset, so InDesign isn’t a great solution.
TeX is powerful and well-seasoned and all, but it’s not exactly pleasant or easy to work with, especially for the kind of stuff I do.
I’ll learn a lot and have fun while I’m at it.
The initial roadmap, not necessarily in order:
Write inkpdf in Python (which I think will be a better fit for the charts anyway)
Get familiar with HarfBuzz
Learn Rust
Port inkpdf to Rust
Plan out the Ink language (I’ve started on this and it’s looking promising)
Figure out how scripting is going to work and embed the interpreter
I’ll document the process on this blog, of course. First steps: reading the PDF spec and figuring out how to make PDFs by hand.
(For those who’ve been reading for a while, Ink was also the name of my static blog engine. That’s now ink-static, and at some point I’ll either retire it completely or change the name to something unrelated.)
I’ve missed learning new programming languages, so I’m now teaching myself Haskell for fun. I’ll post my progress as I go along.
Initial impressions based off what I know at this point: Haskell is very functional. The code looks different to my imperative-language eyes, though I’m intrigued by how short Haskell code often seems to be. Oh, and there are monads, though right now I have no idea what those are.
I’m excited to delve deeper into a pure functional language — I’ve done some limited functional programming in Python, and I have a cursory knowledge of Lisp, but I haven’t written anything real in a functional language before. (And I do plan to write a real project in Haskell. Not sure what it’ll be yet.)
Lifter is “a lightweight query engine for Python iterables.” Looks nice. One of their examples:
# vanilla Python
results = [
user for user in users
if user['age'] == 26 and user['is_active']
]
# lifter
results = manager.filter(User.age == 26, User.is_active == True)
