tarleb https://tarleb.com/ tarleb's blog quarto-1.4.554 Fri, 24 Mar 2023 00:00:00 GMT Typst Musings Albert Krewinkel https://tarleb.com/posts/typst-musings/ Typst, the new writing tool, was open sourced a couple of days ago. This is right up my alley of course, and I have a couple of thoughts on it, which I share here.

What is it?

Typst is a writing tool that’s described as a LaTeX alternative: it takes plain-text markup as input and can produce nice looking PDFs from that. The open-sourcing of the code created a lot of excitement and interest, with the GitHub repository getting starred over 9,000 times in just a few days.

The tool comes with fresh and interesting ideas, impressive technology, and a good bit of hype—so let’s take a closer look.

How does it work?

Typst is written in Rust, a comparatively low-level programming language that fixes many shortcomings of other languages operating in the same domain. One interesting feature of Rust that Typst makes use of is compilation to web assembly, which allows us to run Rust programs in the browser. This is how Typst can do the blazing fast PDF generation in the browser!

Of course this only works if it is Rust all the way down, which has important consequences. My first thought when I looked at the code was that it suffered from some serious nih syndrome: basically the whole stack is built on libraries written by the Typst team, even when high-qualitify open source libraries were available and could have been used with Rust.

But, while nih still seems relevant here and there, the majority of libraries make sense when viewed through the lens of wasm-compilation. My understanding is that it’s not always easy to compile C libraries to wasm. And with a complex system like Typst, it is often essential to have full control over the underlying libraries. So I think most choices are quite reasonable.

Personal opinions

This wouldn’t be a proper “old man shakes fist at the clouds” tech blog post if it didn’t come with a number of “I see room for improvements” comments. Here we go.

Scripting

One of my main concerns is the scripting language: It seems well designed, but I’m not convinced that it was necessary to create a completely new language. I actually believe that something like Lua, which is established1 but flexlible, would have worked well here. But what’s more is that, if we accept that it was necessary to have a custom language, I would have preferred one that isn’t Turing complete, i.e., one where we can be sure that document rendering will terminate. Something like Dhall.

As it currently stands we can neither reuse pre-existing code, nor do we have a guarantee that the document will stop evaluating at some point. It feels like a missed opportunity.

Notebooks

Notebooks are a great tool to improve reproducibility in science writing. The code for analyses is collocated with the descriptive text in a single place. The data is closely linked to the final tables and graphics. It is one of the great features of Quarto, Jupyter, Org, Stencila, Mathematica, and so on, which makes them well suited for scientific writing in fields like medicine, psychology, and the natural sciences.

This could be an issue for Typst, but I believe that it won’t be a problem when Typst is used in combination with other tools. John MacFarlane (of CommonMark and pandoc fame) has started work on a Typst pandoc writer, which will convert existing Markdown documents to Typst, and could enable the use of Typst as PDF generator while still writing Markdown and Python code in Quarto.

Accessibility and metadata

It’s good practice to make PDFs accessible, which requires adding semantic information to a PDF instead of “just” placing characters on a page. Accessibility is even a legal requirement for government-issued documents in many countries. There are standards like PDF/A-1a and PDF/UA that ensure people with disabilities can access the information contained in the PDF. This is currently not supperted in Typst. LaTeX is still trying to catch up there, too, while ConTeXt and speedata are doing well, for example.

Predictions

I believe Typst will succeed, but not as a full-fledged writing tool. The greatest value I see is in the responsive, interactive, and even collaborative styling of PDFs. That feature is truly unique and sets Typst appart from all other software out there. The tool obviously has the potential to reach the critical mass of contributors to become a sustainable open source project, and the tech choices help to attract more developers. Last but not least, the app is shiny, works well, and makes people want to use it.

At the same time, I believe that the Typst writing app won’t take hold in scientific writing. Most of the current enthusiasm is contained to technical circles, and scientists don’t have strong reasons to switch yet. To the contrary, journals expect paper submissions to be done in Word or LaTeX, not PDF. This won’t change anytime soon. Typst will have to insert itself into the current publishing landscape, and that’s not a trivial task.

For example, SciFlow and Overleaf already exist, cover most of the market needs, have a solid headstart, and – this is the important part – are well aligned with the needs of science publishers. Any new tool has to compete with them. Also, let’s not forget all the other tools that I mentioned above.

Regardless, I’m optimistic that Typst can carve out its own business niche to occupy. For example, hardly any of my points above matter in print publishing. An indie book publisher that allows authors to use Typst for layouting would be amazing.

I hope that the Typst team’s hard work will pay off, and I will continue to follow their progress with great interest.

Acknowledgements

Heart-felt thanks to @maegul@hachyderm.io for the insightful feedback on an earlier version of this post, and to Ilona Silverwood for her skillful editing. This post became much better thanks to their input.

Updates

An earlier version listed Authorea as an authoring tool, but the platform has shifted focus since I last looked at it, so I’ve removed it.

Footnotes

  1. I consider Lua to be the language of publishing tools. It fuels pandoc, Quarto, ConTeXt, LuaLaTeX, speedata, SILE, and probably a few more.↩︎

]]> PDF https://tarleb.com/posts/typst-musings/ Fri, 24 Mar 2023 00:00:00 GMT Auto-numbered list continuations Albert Krewinkel https://tarleb.com/posts/list-continuation/ Pandoc’s Markdown allows for “fancy lists”, i.e., lists with different styles used for the marker of ordered list items.

E.g., the list

(I)  primus
(#)  secundus
(#)  tertius

uses uppercase roman numerals and double parentheses for the markers. It gets rendered as

  1. primus
  2. secundus
  3. tertius

Continuations

The fancy lists feature also allows to continue lists after an intermediate paragraph:

i.   one
#.   another

Interruption; not part of any list.

iii. continue
#.   keep counting

This becomes

  1. one
  2. another

Interruption; not part of any list.

  1. continue
  2. keep counting

While very convenient, this requires us to carefully keep book of the number of items in previous parts, or risk the item numbering to become inconsistent. Imagine having to find and update all other list parts after adding a single item somewhere. Tedious work, that gets tiresome quickly.

Convention

Wanting none of this, we magicked a method that allows us to mark a list as a continuation. This way, we can search for those lists with a Lua filter and let pandoc do the counting and numbering for us.

Our convention is this: any list that starts with a number of 90 or above is treated as the continuation of a previous list. Why 90? Because it’s large, but still easy to express using roman numerals, should one be inclined to use those.

i.  primus
ii. secundus

Lorem ipsum.

xc. alius item
#.  ut custodians iens

Lua doing the counting

The next step is to let pandoc do the counting with the help of a Lua filter. In its simplest form, the filter will step through all ordered lists in the document, remember the number of items it has encountered, and renumber any list whose start is above our chosen threshold.

local next_start = 1

function OrderedList (ol)
  if ol.start >= 90 then
    ol.start = next_start
    next_start = next_start + #ol.content
  else
    next_start = #ol.content + 1
  end
  return ol
end

Applied to the example above, the filter produces the desired result:

  1. primus
  2. secundus

Lorem ipsum.

  1. alius item
  2. ut custodians iens

Refinements

This is fairly good already. But what happens if there are other lists appearing in the intermediate blocks?

1. First item

- Nested list:

    a. alfa
    b. bravo
    c. charlie

Which list will be continued?

99. ???

With the current state of our filter, the last item will be numbered d., which is most likely not what we want. So let’s make the filter sensitive to the list style; only a list with the matching style will be continued.

The style and delimiter of ordered list markers can be accessed via the element’s style and delimiter property, respectively. We use those values to construct a string key under which the start number of the next continuation is stored in table next_starts.

local next_starts = {}

function OrderedList (ol)
  local key = ol.style .. '|' .. ol.delimiter
  if ol.start >= 90 then
    ol.start = next_starts[key] or 1
    next_starts[key] = ol.start + #ol.content
  else
    next_starts[key] = #ol.content + 1
  end

  return ol
end

This way we can keep track of multiple lists, distinguishing between lists by using the style of their markers. The above now gets output as

  1. First item

  • Nested list:

    1. alfa
    2. bravo
    3. charlie

Which list will be continued?

  1. ???

Just what we want. Happy list writing!

]]> Markdown pandoc filter https://tarleb.com/posts/list-continuation/ Mon, 28 Nov 2022 00:00:00 GMT Semantic line breaks Albert Krewinkel https://tarleb.com/posts/semantic-line-breaks/ Line breaks usually have no semantic meaning within a Markdown paragraph. However, using line breaks to mark the end of a sentece can help with productivity for various reasons.1 Documents with one sentence per line are also called “ventilated prose”, and the Write the {Ascii}Docs website has a good article on that topic.

A question came up in a Slack channel, asking whether it was possible to convert existing Markdown docs to this style. Naturally, the answer is “pandoc can do that”, but that isn’t obvious here.

The solution that I came up with is centered around pandoc’s SoftBreak AST element. Pandoc uses SoftBreak elements internally to mark the place where line breaks occured in the input, but only if those breaks should be treated like spaces in most situations. The way to make those breaks visible in the output is to call pandoc with --wrap=preserve – only then will a line break in the input result in a break in the output in the same location.

$ printf 'Hello\npandoc' | pandoc --to=markdown
⇒ Hello pandoc

$ printf 'Hello\npandoc' | pandoc --to=markdown --wrap=preserve
⇒ Hello
⇒ pandoc

We are going to use SoftBreak for semantic line breaks, so the first step is to get rid of the SoftBreak elements created during parsing. A Lua filter can do so with

SoftBreak = function () return pandoc.Space() end

Then we check for strings that end with a period and are followed by a space. The space in such a combination is then turned into a SoftBreak.

local function semantic_line_feeds (el)
  local inlines = el.content
  for i = 2, #inlines do
    if inlines[i].t == 'Space' and
       inlines[i-1].t == 'Str' and
       inlines[i-1].text:match '%.$' then
      inlines[i] = pandoc.SoftBreak()
    end
  end
  return el
end

return {
  -- remove soft breaks inserted during parsing.
  {SoftBreak = function () return pandoc.Space() end},
  -- insert semantic soft breaks
  {Para = semantic_line_feeds, Plain = semantic_line_feeds},
}

This filter can then be combined with the --wrap=preserve option to get the desired semantic line breaks.

printf 'This. And that.' | pandoc -L semlf.lua --wrap=preserve
⇒ This.
⇒ And that.

There, mission accomplished.2

Footnotes

  1. It’s not my cup of tea, but I can see why people might like the concept.↩︎

  2. There are some cases in which the filter does not give the correct result, for example when using American-style punctuation after quotes, or when a full sentence is emphasized. It should be possible to fix the filter for these edge cases, but it doesn’t seem worth the effort: the presented solution should work fine for 95 % of all use cases.↩︎

]]> Markdown pandoc https://tarleb.com/posts/semantic-line-breaks/ Sun, 30 Oct 2022 00:00:00 GMT Install pandoc in custom Docker images Albert Krewinkel https://tarleb.com/posts/tip-install-in-docker/ When pandoc is used in larger pipelines, e.g. in combination with Sphinx, it can be useful to pack everything up into one neat container. One option is to base the container on a pandoc image like pandoc/core, but this may be difficult due to other dependencies. In this case, the easiest option is to do this:

COPY --from=pandoc/minimal:2.19.2 /pandoc /usr/bin/pandoc

The snippet, when added to a Dockerfile, copies a statically compiled executable from an official build to /usr/bin/pandoc. The static pandoc binary can run on all Linux distributions, so it does not matter what distro the custom build is based on.

The documentation for the https://hub.docker.com/r/pandoc/minimal has more info, including a list of supported versions (tags).

]]> pandoc tip Docker https://tarleb.com/posts/tip-install-in-docker/ Wed, 07 Sep 2022 00:00:00 GMT Haskell usedata objects Albert Krewinkel https://tarleb.com/posts/haskell-lua-objects/ When extending pandoc (or Quarto) with Lua filters, we interact with so-called Lua userdata objects. These objects are used to wrap document AST elements, making them accessible from Lua scripts. They mostly behave like normal Lua tables. This post is intended as a quick overview, listing interesting properties of userdata objects.

Userdata objects

Haskell-generated userdata objects have three main components: a type name, properties, and methods:

Type name

The name can be retrieved with pandoc.utils.type. It should be treated as a read-only constant. Internally, the name is used as a type tag, which is important when retrieving an object from a Lua script back into the main program. It is possible to access the name through the debug interface, e.g.,

function typename (x)
  return debug.getmetatable(x).__name
end

The above typename function is actually a bit faster than pandoc.utils.type. It can improve performance when accessing this info in a tight loop, but its use is not recommended.

Properties

Properties, e.g., the content fields of Para elements, are “lazy”: the property value is marshaled to Lua when the property is accessed for the first time. We are usually interested in no more than one or two element properties, so this is a big performance improvement for most scripts. Lazy properties are especially useful with large objects like Pandoc, which would otherwise take a long time to marshal and unmarshal with all their child elements.

However, this lazy marshaling is slower if all properties will be accessed anyway. If there are performance issues due to lazy properties, then please let me know, and I’ll try to find a fix.

Methods

Methods are wrapped Haskell functions: when calling a method, the arguments are unmarshaled back into Haskell objects, which are then passed to the wrapped function and processed in Haskell. The computation’s result is then pushed back to Lua.

This may sound weirdly complicated. While it is slow for very simple functions (like pandoc.utils.type), it’s very fast and convenient for complex methods like Pandoc:walk: objects with lazy properties are fast to unmarshal, the main Haskell code is fast, and it frees us from having to re-implement Haskell algorithms in Lua.

Iterating

The properties and methods are listed when iterating with pairs. The iteration order is defined at compile time, with properties listed first, followed by methods. We usually try to keep each of these lists sorted alphabetically, but there may be exceptions.

Calling pairs on a Haskell userdata object will always succeed, even if it has neither methods nor properties; the result will be the empty iterator in that case.

Aliases

Some objects also have property aliases: E.g., div.classes is really just an alias for div.attr.classes. Both entries point to the same list object. Aliases are not included in the iterator generated by pairs. See the internals on how to get a hold of them.

List behavior

Userdata objects can be made to behave like lists, but iterating over those “lists” is comparatively slow. That’s why the only object that uses this feature is PANDOC_VERSION: for example, we can write PANDOC_VERSION[2] >= 19 to check just the major version.1

Internals

Each type has a metatable, which defines its behavior. Most users should not need to access the metatable, so the getmetatable function returns true instead of the actual metatable when called on a Haskell userdata object. As we’ve seen with type names, It’s still possible to inspect the userdata metatable with the help of debug.getmetatable.

The metatable has four interesting fields: methods, aliases, getters, and setters. The fields contain just what you’d expect.

E.g., we can inspect the list of aliases defined for Inline objects with

local InlineMT = debug.getmetatable(pandoc.Str '')
for name, keys in pairs(InlineMT.aliases) do
  -- print the alias name and the alias value
  print(name, 'is an alias for', table.concat(keys, '.'))
end

Do not rely on the internal structure, and do not modify the metatable. Here be dragons.

Footnotes

  1. This feature exists only to ensure backwards compatibility. It is better to do comparisons in like PANDOC_VERSION >= '2.19' instead.↩︎

]]> Haskell Lua pandoc Quarto https://tarleb.com/posts/haskell-lua-objects/ Wed, 07 Sep 2022 00:00:00 GMT Document font Albert Krewinkel https://tarleb.com/posts/tip-document-font/ Setting the document font this way works for ConTeXt, LaTeX, and HTML output. The fonts used in docx or odt output must be controlled with the reference document instead.

The default LaTeX engine is pdflatex, which only supports TeX’s own font format and cannot use the TrueType or OpenType fonts installed on the system. However, XeLaTeX was written with that in mind; switching to that engine allows to specify any font available on the system.

A good source for free fonts is the Google Fonts repository.

]]> pandoc tip PDF HTML https://tarleb.com/posts/tip-document-font/ Wed, 07 Sep 2022 00:00:00 GMT Generating a sitemap with Quarto Albert Krewinkel https://tarleb.com/posts/quarto-sitemap/ Sitemaps are an easy way to list all pages that a search engine should crawl and index. Quarto supports the standard, XML based Sitemap Protocol, although that fact is a bit hidden in the docs.

Quarto will automatically produce a sitemap if the website’s URL is given as site-url property:

website:
  title: tarleb
  site-url: 'https://tarleb.com'
  site-path: '/'

The URL will contain a colon :, which is why the YAML value must be put in quotes.

You’ll find the sitemap.xml file in your _site folder after re-rendering your pages with quarto render.

]]> quarto https://tarleb.com/posts/quarto-sitemap/ Wed, 24 Aug 2022 00:00:00 GMT Pandoc resources Albert Krewinkel https://tarleb.com/posts/pandoc-resources/ These are the resources that I’d want to have if I was to learn about pandoc (the universal document converter) all over again:

Installing

latest release
The GitHub release page has installers for Windows, macOS, and Linux.
Docker images

There are three types of pandoc Docker images:

  • minimal – very small, just the bare pandoc binary;
  • core – includes pandoc-crossref and helpers programs, e.g. those used by pandoc for SVG image conversion;
  • latex – like core, plus TeXLive with all packages required by the default template.

These images can be used as an alternative to a system-wide installation. The image repository descriptions also explain how the images can be used.

Documentation

manual

The official pandoc manual. When in doubt, this should always be the first resource to check. It covers all command line options, defaults files settings, format extensions, etc.

Note that this always documents the latest pandoc version. If something isn’t working as documented, you may need to update.

FAQs
Frequently Asked Questions; click on the questions to see the answer.
Wiki
Pandoc’s GitHub wiki; contains many additional resources, links, and tips.
Lua filters
How to modify a document programmatically; also documents all constructors and utility functions exposed by pandoc.
Custom reader
Interface that allows to write parsers for otherwise unsupported formats.
Custom writers
Interface that allows to use pandoc to convert to a custom output format.
Format-specific documentation

The support for some formats is tune-able enough to warrant additional documentation.

  • EPUB – How to make an ebook
  • Org – Emacs Org-mode
  • JATS – The Journal Article Tag Set

Getting help

Please make sure to search the web before posting.

pandoc-discuss

Official pandoc mailing list; usually the best way to get help. The main developers and many seasoned users lurk here and answer questions.

The web interface makes it seem like a Google account is required to post, but this isn’t so: Write a mail to obfuscated mail address to subscribe to the list with any mail address.

StackOverflow
Post here to reach a more programming oriented group of people. Please do not cross-post, many people that follow the pandoc tag have also subscribed to the mailing list.
TeX/LaTeX Stack Exchange
This is a good place to ask for help when pandoc is used to generate PDFs via LaTeX. Be sure to include the (relevant parts) of the generated LaTeX output in your question, as most people there do not use pandoc. Some even seem to actively dislike it, so be prepared.

Misc

pandoc in the fediverse
I use this Mastodon account to post small tips and updates.
Quarto
Scientific and technical publishing system based on pandoc.
Zettlr
Markdown editor with “zettelkasten” functionality.

Edits

  • 2022-12-09: Added Mastodon account in favor of Twitter; the account on the latter platform is no longer updated.
]]> pandoc https://tarleb.com/posts/pandoc-resources/ Tue, 23 Aug 2022 00:00:00 GMT Quarto Website with GitHub Actions Albert Krewinkel https://tarleb.com/posts/quarto-with-gh-pages/ Quarto makes it very easy to publish a website via GitHub Pages: It is as simple as running quarto publish gh-pages. Here we explore a slightly different method that uses a GitHub Action to publish the website automatically every time it is updated.

Classic GitHub Pages

The classic way to publish a website via GitHub pages is to maintain a separate branch gh-pages. The branch is used to store the rendered HTML pages, and GitHub will publish the branch’s contents as website everytime that branch is updated.

Quarto uses this mechanism when called with

quarto publish gh-pages

We can combine this with GitHub Actions easily, ensuring that the site is updated every time new content is pushed to the main branch.

# file: .github/workflows/publish.yml
name: Publish Website

# Allow one concurrent deployment
concurrency:
  group: "pages"
  cancel-in-progress: true

on:
  push:
    branches: ['main']

jobs:
  quarto-publish:
    name: Publish with Quarto
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Install Quarto
        uses: quarto-dev/quarto-actions/setup@v2
      - name: Publish to GitHub Pages
        uses: quarto-dev/quarto-actions/publish@v2
        with:
          target: gh-pages

The quart-dev/quart-actions/publish action calls quarto publish internally. This is short, to the point, and won’t interfere with local calls to quarto publish gh-pages.

Actions-only Pages (Beta)

GitHub recently added support for GitHub Pages that do not require an extra gh-pages branch. Instead, the website is compiled and pushed directly from an action.

This takes slightly more code to set up, as the action must be granted the necessary permissions. However, it is still fairly short and quick to do.

# file: .github/workflows/publish.yml
name: Publish Website

# Allow one concurrent deployment
concurrency:
  group: "pages"
  cancel-in-progress: true

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: read
  pages: write
  id-token: write

on:
  push:
    branches: ['main']

jobs:
  quarto:
    runs-on: ubuntu-latest
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Install Quarto
        uses: quarto-dev/quarto-actions/setup@v2
      - name: Setup Pages
        uses: actions/configure-pages@v1
      - name: Render Website
        run: quarto render
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v1
        with:
          path: '_site'
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@main

Now we must change the setting to use the new publishing workflow:

Under Settings → Pages → Build and deployment the source must be switched to “GitHub Action”:

Required setting to enable the new method

The gh-pages branch is no longer needed and can be deleted.

Trade-offs

The branch-based method for GitHub pages always felt slightly inelegant. I prefer the new method, as it does not require an extra branch and feels much cleaner. We lose the ability to update the website via quarto publish, but as I usually rely on GitHub Actions to perform the updates, it doesn’t affect me much: git push has the same effect. However, it seems important to be aware of this trade-off.

]]> quarto https://tarleb.com/posts/quarto-with-gh-pages/ Mon, 08 Aug 2022 00:00:00 GMT Santa’s Little Lua Scripts Albert Krewinkel https://tarleb.com/posts/santas-little-lua-scripts/ Santa sighted deeply as worry and uncertainty gave way, leaving a feeling of relieve and accomplishment. The year was one of the worst he’d seen so far. Large numbers of his helpers were moving from the North Pole to Antarctica to satisfy their ambient temperature preferences. There would be many telecommuting Elves this year, and each helper enjoyed additional autonomy. Tying everything together was a challenge. But he had succeeded: the wishes processing program was finished, and the elves would be able to help Santa from the comfort of their new homes.

Wishes

The part of the wishes system that Santa had been working on was focused on classic toys: wooden bricks, dolls, and train sets.

data Toy = Bricks | TrainSet | Doll deriving Show

The system also kept track of basic data about the children:

data Behavior = Nice | Naughty deriving (Eq, Show)

data Child = Child
  { childName     :: Text
  , childBehavior :: Behavior
  } deriving (Show)

Children and toys were tied together in a wish.

data Wish = Wish
  { wishingChild :: Child
  , wishedToy    :: Toy
  } deriving (Show)

It was most elegant. The problem for Santa was that the Elves, being independent and autonomous workers, needed to access and process the data in very custom ways. Unfortunately for him, very few Elves had a Haskell build environment installed, so he had to distribute the binary. Writing a completely custom processing language seemed like an enormous rabbit hole.

Lua

Fortunately, Santa had a better idea: Lua, an embeddable scripting language. He had been using it for some projects1 and also made use of it in pandoc, which he used to answer his mails. Santa would just need to expose the relevant parts of the Haskell system, so the Elves could access and script it as their hearts desired. He looked for a library, found HsLua, and got to work.

Exposing data

Lua has a simple, yet powerful, stack-based API. The first step towards exposing Haskell data was to push them to the Lua stack. Keeping things simple, Santa chose strings to represent toys:

pushToy :: Toy -> Lua ()
pushToy = pushString . show

Lua offers only a single construct to structure data: tables. So that’s what Child and Wish were represented with.

pushChild :: Child -> Lua ()
pushChild (Child name behavior) = do
  -- create new Lua table on the stack
  newtable
  -- push string to stack
  pushText name
  -- table now in position 2; assign string to field in table
  setfield (nth 2) "name"

  -- push boolean to stack
  pushBool (behavior == Nice)
  setfield (nth 2) "nice"

pushWish :: Wish -> Lua ()
pushWish (Wish child toy) = do
  newtable
  pushChild child
  setfield (nth 2) "child"
  pushToy toy
  setfield (nth 2) "toy"

Running scripts

Santa’s goal for now was to allow his Elves to filter the list of wishes so each finds the ones relevant to them. For example, if an Elf only cares about wishes for train sets from children who were nice, then they should be able to use a script to filter those wishes out.

return function (wish)
  return wish.child.nice and
    wish.toy == 'TrainSet'
end

The script returns a (lambda) function that serves as a predicate for wishes. The function can be thought of having the type Wish -> IO Bool. Santa needed to turn the Lua lambda function into an actual Haskell function runPredicate :: Wish -> Lua Bool. If Santa assumed that the lambda function was at the top of the Lua stack, then he could push a Wish value to the Lua stack, call the function, and retrieve the result value from the stack.

runPredicate :: Wish -> Lua Bool
runPredicate wish = do
  -- Assume filter function is at the top of the stack;
  -- create a copy so we can re-use it.
  pushvalue top
  pushWish wish
  -- Call the function. There is one argument on the stack,
  -- and we expect one result to be returned.
  call (NumArgs 1) (NumResults 1)
  toboolean top <* pop 1

What remained was loading the Elves’ script files. Santa did this with dofile of type FilePath -> Lua Status. The predicate then ends up on the top of the Lua stack, and can be called through runPredicate, e.g. to select a subset of wishes via filterM.

main :: IO ()
main = do
  filterFile <- fmap (!! 0) getArgs -- get first argument
  result <- run $ do
    _status <- dofile filterFile
    filterM runPredicate wishes
  print result

Santa tested his creation on a short list of wishes

wishes :: [Wish]
wishes =
  [ Wish (Child "Theodor" Nice) Bricks
  , Wish (Child "Philine" Nice) TrainSet
  , Wish (Child "Steve" Naughty) Doll
  ]

by running runhaskell wish-filter predicate.lua. To his uttermost satisfaction, the terminal echoed the right information back to him.

[Wish {wishingChild = Child {childName = "Philine", childBehavior = Nice}, wishedToy = TrainSet}]

He reclined in his chair, shut down his device, and enjoyed a double chocolate chip cookie of which he felt very deserving now.

Santa’s full code, as presented here, is available as part of the examples at https://github.com/hslua/hslua.

Footnotes

  1. Santa learned about Lua from his game-devs. Now he uses it to keep his security teams on their toes with nmap and Wireshark; many of the North Pole’s servers contain custom Lua scripts, too (redis, nginx/OpenResty, HAProxy, PowerDNS).↩︎

]]> lua hslua example https://tarleb.com/posts/santas-little-lua-scripts/ Sun, 06 Dec 2020 00:00:00 GMT PDF Version of the Lua Manual Albert Krewinkel https://tarleb.com/posts/pdf-of-the-lua-manual/ A question came up on the Lua mailing list, asking whether there was a PDF version of the Lua manual. This is, of course, the home domain of pandoc, and I got nerd-sniped into producing a PDF (and ePUB) version of the manual.

This is a good opportunity to showcase some pandoc features. The post describes the process of going from an HTML web page to a PDF file via LaTeX and pandoc. We will see how to

  1. quickly convert documents with pandoc;
  2. use Lua filters to improve the result by modifying the document; and
  3. fine-tune the output by setting appropriate pandoc options.

Invoking pandoc

The first step is to call pandoc on the Lua manual website. Even when keeping everything bare-bones, the result is already decent:

pandoc --pdf-engine=xelatex --output=lua-manual.pdf \
    "https://lua.org/manual/5.4/manual.html"

Produces

First page of unoptimized PDF

This requires a somewhat recent version of pandoc as well as XeLaTeX to be installed. It is possible to forgo the trouble of installing the requirements by using the pandoc/latex Docker image:

docker run --rm -v "$PWD":/data -u $(id -u):$(id -g) pandoc/latex:2.9.2.1 \
    --pdf-engine=xelatex --output=lua-manual.pdf \
    "https://lua.org/manual/5.4/manual.html"

Replacing characters

The above commands will produce warnings about characters which are unavailable in the default fonts. We don’t want characters to go missing, of course, so let’s fix that first. The warnings are:

[WARNING] Missing character: There is no ≤ (U+2264) in font [lmmono10-regular]:!
[WARNING] Missing character: There is no ≤ (U+2264) in font [lmmono10-regular]:!
[WARNING] Missing character: There is no π (U+03C0) in font [lmroman10-italic]:mapping=tex-text;!

Searching the page for ≤ shows that it is used in inline code, while π occurs as emphasized character in the description of math.pi. We could, of course, search for a font which has the appropriate glyphs and instruct pandoc/LaTeX to use it. But we’ll go a different route.

A good way to improve the result of a converstion is to use a pandoc Lua filter. We create a file called beautify-manual.lua and pass it to pandoc via the --lua-filter=beautify-manual.lua command line option.

Handling ≤ is straight forward, we just replace the char with the slightly uglier looking ASCII sequence <= in all code elements.

function Code (c)
  c.text = c.text:gsub('≤', '<=')
  return c
end

While there is no italics version π in the default font, there is such a glyph in the default math font. Pandoc’s internal representation for π is Emph [Str "π"], which we replace with a math element holding the same content.

function Emph (e)
  local s = e.content[1]
  if #e.content == 1 and s.tag == 'Str' and s.text == 'π' then
    return pandoc.Math('InlineMath', 'π')
  end
end

The document now compiles without warnings, and all characters are properly included.

Add Table of Contents

The Lua manual is long, often used as a reference, and, in its HTML version, comes with a table of contents on a separate page. The PDF, for it to be useful as a reference, should have a table of contents as well. Pandoc can be told to generate a table of contents by adding the --toc command line flag. The toc depth is controlled via --toc-depth; 2 is a good setting here. However, in this case, the result is neither pleasing nor informative:

Bad looking table of contents

Something is terribly wrong. By inspecting the parsed document by running pandoc --to=native …, we see that all Headers contain a Span. That span holds the actual contents. Apparently LaTeX does not like this and omits the content of the span when generating the toc.

The span also has the id used by links to the header. Numbered sections start with the section number, which we’d rather produce via pandoc.

function Header (h)
  -- Unnumbered sections have the main contents as the first element.
  -- Numbered sections start with the number and an em-dash, so
  -- the Span is the fifth element (Lua multipass).
  local span
  if h.content[1].tag == 'Str' and h.content[1].text:match '[%d%.]+' then
    span = h.content[5]
  else
    span = h.content[1]
    h.classes:insert('unnumbered')
  end

  h.identifier = span.identifier
  h.content = span.content

  return h
end

The filter also removes the section numbering. We add it back by passing --number-sections to pandoc.

less-bad table of contents

Not bad.

Improve title and metadata

The PDF is already quite usable, let’s prettify it a bit more: It would be important to properly list the authors in the title and metadata, remove the unnecessary first header, and maybe add the Lua logo to the title. All this is easiest when acting on the full document.

function Pandoc (doc)
  -- comma separated authors
  local authors = doc.blocks[2]
  authors.content:remove(1)  -- remove 'by'
  doc.meta.author = pandoc.List()
  for author in pandoc.utils.stringify(authors):gmatch '[^,]+' do
    doc.meta.author:insert(author)
  end

  -- Remove unnecessary blocks
  doc.blocks:remove(4) -- menubar
  doc.blocks:remove(2) -- authors paragraph
  doc.blocks:remove(1) -- title header

  -- add subtitle image
  doc.meta.subtitle = pandoc.MetaInlines{
    pandoc.RawInline('latex', '\\vspace{1em}'),
    pandoc.Image("Lua logo", -- "https://www.lua.org/images/lua-logo.gif")
  }
  return doc
end

Final touch

Finally, we may want the PDF to add a little more visible structure, e.g., starting top-level sections on their own page.

The command used by pandoc to create the top level headings can be controlled with the --top-level-division option. Setting that option to chapter ensures that each major section starts on a new page. However, the default document class used by LaTeX doesn’t allow chapters, so a different class has to be set with --variable documentclass=report.

Summary

For completeness, here is the complete filter, and this is the full pandoc command used to generate the PDF:

pandoc \
  --toc \
  --toc-depth=2 \
  --metadata=documentclass=report \
  --pdf-engine=xelatex \
  --lua-filter=lua-manual-cleanup.lua \
  --number-sections \
  --top-level-division=chapter \
  --output=lua-5.4-manual.pdf \
  "https://lua.org/manual/5.4/manual.html"

One of the big advantages of pandoc is that it offers a lot of freedom. Since we already cleaned the content up, we can now also create other formats, like an ebook, just by changing the name of the output file. The final results are available below:

]]> pandoc lua pdf https://tarleb.com/posts/pdf-of-the-lua-manual/ Sat, 11 Jul 2020 00:00:00 GMT Extending pandoc with Lua Albert Krewinkel https://tarleb.com/posts/extending-pandoc-with-lua/ My first exposure to Lua has been as a pandoc user, and adding new Lua features to pandoc turned Lua into one of my favorite languages. In this post I will take a look at pandoc, the universal document converter, and explore how one can script and extend it with Lua. Pandoc includes a Lua interpreter since 2012, but the integration of Lua has been expanded significantly with the latest 2.0 release. My hope for this article is to highlight the beauty of these systems.

The universal document converter

Pandoc – written and maintained by John MacFarlane – is an relatively old project. It has grown considerably since the first version was published in 2006: at the time of writing, pandoc can read 27 different document formats and dialects, and can write 49 formats. Besides serving as a one-off document conversions tool, pandoc also frequently features as the central part of publishing pipelines. For example, Pandoc is used in static site generators and is frequently used by academic writers, due also to its excellent support for citations.

As a brief example, consider the following commands which transform Markdown input into docx, HTML, or PDF:

# command to convert a markdown file to docx
pandoc input-file.md --output=output-file.docx

# convert to HTML
pandoc input-file.md --standalone --output=output-file.html

# convert to PDF (via LaTeX)
pandoc input-file.md --output=output-file.pdf

Many conversion tasks need to alter the default behavior or require special conversion features. This highlights the importance of good customization support for a conversion tool, one of the areas in which Lua shines.

Pandoc is unusual for a Lua-extendable program, in that it is written in Haskell. Using Haskell is very productive, but is less suitable as an extension language: its concepts are often alien to users of other languages, and shipping a full Haskell interpreter with pandoc would result in considerable bloat. Lua is an excellent choice here, as it is lightweight, simple, and beautiful. It should be noted, however, that bridging Haskell and Lua is its own can of worms and worth a separate blog post.

Pandoc’s document AST

An important factor in pandoc’s immense transformation powers is its use of a unifying document representation: Every input is parsed into this document AST, which is then rendered in the desired output format. While a direct conversion between any of n input and m output formats would require n m* converters, using an intermediate representation reduces complexity to n + m.

There are additional advantages to this: as we’ll see, it becomes much simpler to work with a unified document representation than it would be to work with any of the input or output formats directly.

There are four main types in pandoc’s document model: inlines, blocks, document metadata, and the full document.

  • Inline elements represent text and text markup. Examples are Space for inter-word spaces, Str for (usually non-whitespace) text, and Emph for emphasized text.

  • Blocks are elements like paragraphs, lists, code listings, and headers. They are usually rendered in lines or blocks of their own; many block elements contain lists of inline elements.

  • Meta information is a simple mapping from string keys to meta values. Meta values can be thought of as a special JSON or YAML object.

  • Last but not least, the Pandoc type represents a full document. A Pandoc element consists of a lists of block elements, plus additional document metadata.

Pandoc’s Lua features revolve around modifying or converting these elements. The oldest use of Lua in pandoc enables the conversion of AST elements into strings as to output any document format.

Custom writers

Users can define custom writers in Lua to render any document format. Each of the aforementioned AST elements is transformed to a string by calling a Lua function of the same name as the element. E.g., this example demonstrates how emphasized text can be rendered as HTML:

function Emph(content_string)
  return '<em>' .. content_string .. '</em>'
end

A full custom writer is defined by specifying functions for all document AST elements. Example writers using this method include 2bbcode by @lilydjwg (依 云), as well as pandoc’s sample.lua. The latter is a well documented starting point for authors of new custom writers. The file can be produced by calling pandoc --print-default-data-file=sample.lua.

The pandoc-scholar project serves as an example for the power offered by custom writers. It is a publishing tool intended to help authors of scholarly articles and was created with custom Lua writers. The tool leans on the custom writers feature in ways that writers were not intended to be used, which resulted in the development of lua filters.

Filters

An additional benefit of a unified document type is that the document can be modified programmatically, regardless of which input and output format is chosen. Pandoc provides two interfaces for this.

JSON Filters

The first – very flexible – method is based on JSON. Pandoc can serialize the document to JSON; other programs can read and modify the document. The resulting document JSON is passed back to pandoc, thus allowing users to use any programming language capable of parsing JSON to alter the document. Many libraries for various languages have been implemented, including Haskell, Python, Ruby, and JavaScript.

The flexibility of JSON filters can also be a disadvantage, as it requires additional software and usually the full installation of a scripting language’s ecosystem. Pandoc is designed to work on all major platforms and without any dependencies on other libraries and binaries. Depending on additional software can be problematic, especially for non-technical users.

Lua filters

The Lua filter system added in pandoc 2.0 not only solves the portability issue of JSON filters, but also offers better performance and more functionality. Document elements can be selectively serialized to Lua tables, modified using the full power of Lua, and will then be transferred back, thus replacing the previous values.

Lua filters operate by calling filter functions on each element of the specified name. I.e., if a Lua filter contains a function with the same name as an AST element, then this function is called for all elements of the respective type. The serialized element is passed as input to the filter function, and the function’s return value is deserialized and used to replace the input element. This method is as simple as it is flexible, and fits well with the concept of immutability which is prevalent in Haskell programs: pandoc ignores modifications to the serialized object itself, it will just use the filter function’s return value.

The following example filter transforms all text set in small caps into emphasized text:

function SmallCaps (element)
  return pandoc.Emph(element.content)
end

The element constructor functions in module pandoc, like pandoc.Emph in the above example, are also the central step when transforming elements from their pandoc-internal representation to Lua values. This ensures consistency in the way element values are produced, whether during serialization or through a constructor call in the filter script. The current implementation uses only strings, tables, and some metatables when constructing element values, with the goal of marking these values easy and flexible to use.

Lua filter example: macro expander

Below is the code for a simple macro expander using pandoc’s Lua filter functionality. The expander replaces all macro occurrences in the given document. Macro definitions are hard-coded into the filter, but could as well be read from an external file.

-- file: macro-expander.lua

-- Macro substitutions: contains macro identifier as
-- keys and the expanded inlines as values.
local macro_substs = {
  ['{{hello}}'] = pandoc.Emph{pandoc.Str "Hello, World!"}
}

-- Replace string with macro expansion, if any.
function Str (s)
  return macro_substs[s.text] or s
end

The heart of the macro expander is the function Str. It is called on all simple strings in the document. The return value of this function is then read back into pandoc, replacing the original Str value.

Assume a Markdown file greeting.md:

Greeting: {{hello}}

We can apply the macro expander by calling

pandoc --lua-filter macro-expander.lua greeting.md

resulting in the expected expansion:

Greeting: Hello, World!

The function Str could be shortened further by dropping the trailing or s:

function Str (s) return macro_substs[s.text] end

This is a convenience feature of pandoc filters: if the function returns no value (or nil), the original value is kept unchanged. This makes filter functions easier to write and speeds up filtering, as unchanged elements don’t need to be deserialized again.

What’s good, and what’s next

Using pandoc with Lua is a fast, flexible, and platform independent way of augmenting pandoc with additional functionality. For me personally, having the full power of Lua at ones finger tips proved to be a lot of fun, while opening unexpected document processing possibilities.

Pandoc and its Lua subsystem are under constant development. E.g., the next versions will feature more utility functions exposed via Lua modules. There is constant work to make more and more internal functions available. The next big goal is to grant scripting access to all format-output functions. However, this requires some changes to pandoc’s internals. It remains a long way for pandoc to become a fully Lua-scriptable publishing platform.

If you want to learn more about Lua filters, the Lua filter docs is a good place to start. It includes up-to-date examples of Lua scripts, as well as a reference of all modules and functions accessible via Lua. Pandoc’s user manual is a good resource to learn about all of pandoc features and its command line options.

Feedback is always welcome!

Acknowledgements

A big thank you to Jennifer König, Birgit Pohl, and John MacFarlane for their feedback on an earlier version of this post, and to all pandoc contributors and users, who make working on this project incredibly fun.

]]> pandoc lua pandoc-filter https://tarleb.com/posts/extending-pandoc-with-lua/ Sat, 23 Dec 2017 00:00:00 GMT MetaNook2014 – Command Line Talk Albert Krewinkel https://tarleb.com/posts/nook-2014/ Last night was the night of this year’s MetaNook. It was the fourth time that the local hackers, most notably MetaMeute and Chaotikum, joined forces to organize a night full of beginner-friendly introductions, advanced tech talks, and project presentations.

My talk contribution this year was a brief introduction in the “magic” of the command-line. The talk slides are adapted to the topic in that the presentation must be given from the command line. Each slide is just a function, the whole presentation environment is held together by scripts and other shell-equivalences of duckt-tape. Feedback from the thirty or-so attendees was mostly positive.

It was a very fun experience over all, as usual.

]]> talk command-line https://tarleb.com/posts/nook-2014/ Sat, 22 Nov 2014 00:00:00 GMT rt6_redirect: source isn’t a valid nexthop for redirect target Albert Krewinkel https://tarleb.com/posts/rt6_redirect/ The zeitkraut server is configured to work with IPv6. For quite some time now, I’ve been seeing some strange errors in my log files. If you’ve been noticing something similar, here is what’s going on and how to prevent the messages from appearing.

The Problem

Everything works as expected, except for some weird messages in the logs:

rt6_redirect: source isn't a valid nexthop for redirect target

Not even startpage was of much help. Searching for the above line only lists only some unanswered forum questions and the kernel source code which is producing the message. Oh, and a somewhat unhelpful blog entry telling people to always use their routers link local address when routing. This is useless advice in my case, I don’t have a link-local address of the router, only it’s global address.

The Cause

I found a way to stop the message from appearing in my logs. On the way, I learned a bit more about IPv6 and improved server security on the way.

IPv6 contains functionality to tell a computer about better routes to the target destination. A router may send ICMPv6 redirect packages (type 137 to be specific), informing neighboring computers about more effective ways to reach their targets. This makes the most sense when applied within an environment heavily relying on auto-configuration – like a dynamic internal company or home network. It makes a lot less sense for servers very stable network topologies.

Attackers may try to exploit the redirect functionality by including themselves into the route to the target. The specification for those redirects includes some security-measures, requiring the attacker to correctly guess the server’s current next hop. If the attackers get it wrong, the Linux kernel refuses to use the new routing information. This is most-likely what happens when you see the above log messages.

The Solution

Long talk short, the solution is to disable IPv6 redirecting:

sudo sysctl net.ipv6.conf.all.accept_redirects=0

My server is not a router, so there is no need to accept any kind of route changing messages from external sources. We can simply disable redirects, using above command. The change can be made permanent by setting the value in /etc/sysctl.conf. In fact, we can disable routing for both IPv4 and IPv6. Be careful though, you might happen to be in a network environment requiring you to accept redirect commands for some reason.

If you are on Debian or similar distribution like Ubuntu, change the following lines in /etc/sysctl.conf from

# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0

to

# Do not accept ICMP redirects (prevent MITM attacks)
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
#
# Do not send ICMP redirects (we are not a router)
net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0

Running sudo sysctl -p loads the new settings.

Alternative Solution

Completely disabling redirects in the kernel should keep you reasonably secure. However, if you need redirects within your internal network, you could also block redirect packages reaching you through external interfaces. E.g., to block redirect packages coming in on eth1, one would issue

sudo ip6tables -A -i eth1 -p icmpv6 --icmpv6-type 137 -j DROP

However, firewall configuration is a complex topic, so I’m not going to go into details here.

If you have any questions, corrections or comments on the matter, please drop me a line.

]]> security sysadmin https://tarleb.com/posts/rt6_redirect/ Fri, 01 Aug 2014 00:00:00 GMT How to configure zsh with vi bindings and nice shortcuts Albert Krewinkel https://tarleb.com/posts/howto-zsh-vi-style/ Having a good working environment is vital for feeling comfortable being productive. This extends to computational tools and the command shell is an integral part of the daily work for many of us. It’s a good idea to configure the shell’s interface to be efficient and pleasant to use. Here we see how zsh, arguably the best shell around, can be configured to suit the needs of people used to vi key bindings (which are arguably arguably superior and more ergonomical than the default emacs-style bindings)1.

Using vi-bindings in the shell

The first step towards nice vi key bindings is almost too easy: The red pill takes the form of

bindkey -v

Type it into your prompt (and add it to your .zshrc file) and emacs bindings are going bye-bye. Escape will bring you to normal-mode, while i, a, o etc. will bring you back to insert-mode, just as with your favorite editor. Use j and k in normal-mode to go through your history and move around within the line with h, l, w, b and the like.

This is a good start, let’s see how we can bring it from “this is nice” to “that’s just awesome”.

First, we may want to keep some of the default key bindings in insert-mode since we’ve grown accustomed to them. No missing out, let’s put them back in:

# Kill input from the current point to the end of line with Ctrl-k
bindkey '^k' kill-line
# Search the history incremantally with Ctrl-r
bindkey '^r' history-incremental-search-backward
# Insert and go through the "last words" of previous commands with Meta-.
# (or Escape-. for that matter).
bindkey '^[.' insert-last-word
# Show the man-page or other helpful infos with Meta-h
bindkey '^[h' run-help

You can take a look at the key bindings defined for emacs-mode by typing bindkey -M emacs -L and reuse the bindings you like. See the zshzle manpage for more pre-defined widgets for which you could define bindings.

Configuring the prompt to show the current editing mode

So the key bindings are quite usable now, but it’s a bit unfortunate that it is impossible to see if the shell is in insert- or normal-mode. There should be a mode indicator right in the shell prompt!

# You may already have those in your .zshrc somewhere
autoload -U promptinit && promptinit
autoload -U colors     && colors

setopt prompt_subst

# Set the colors to your liking
local vi_normal_marker="[%{$fg[green]%}%BN%b%{$reset_color%}]"
local vi_insert_marker="[%{$fg[cyan]%}%BI%b%{$reset_color%}]"
local vi_unknown_marker="[%{$fg[red]%}%BU%b%{$reset_color%}]"
local vi_mode="$vi_insert_marker"
vi_mode_indicator () {
  case ${KEYMAP} in
    (vicmd)      echo $vi_normal_marker ;;
    (main|viins) echo $vi_insert_marker ;;
    (*)          echo $vi_unknown_marker ;;
  esac
}

# Reset mode-marker and prompt whenever the keymap changes
function zle-line-init zle-keymap-select {
  vi_mode="$(vi_mode_indicator)"
  zle reset-prompt
}
zle -N zle-line-init
zle -N zle-keymap-select

# Multiline-prompts don't quite work with reset-prompt; we work around this by
# printing the first line(s) via a precmd which is executed before the prompt
# is printed.  The following can be integrated into PROMPT for single-line
# prompts.
#
# Colorize freely
local user_host='%B%n%b@%m'
local current_dir='%~'
precmd () print -rP "${user_host} ${current_dir}"

local return_code="%(?..%{$fg[red]%}%? %{$reset_color%})"
PROMPT='${return_code}${vi_mode} %# '

This gives a prompt in the style of

user@host /current/working/path
[I] %

where [I] is the insert-mode indicator and is changed to [N] when normal-mode is activated. Neat, isn’t it?

Single- and multi-key shortcuts

This is all nice and dandy, but it’s not quite like vim yet. How about those sweet bindings where pressing jj in quick succession brings us to normal-mode without having to press Esc? Setting it up is easy as pie.

# Time in which two keys have to be pressed in order to be recognized as a
# single command (in centiseconds, set to 0.4 sec by default -- may be
# modified as needed).
export KEYTIMEOUT=40
bindkey 'jj' vi-cmd-mode

We can also add two-key bindings to jump to the start and end of the line:

# Bind to both possible orders in which the keys could be pressed.
# Move all the way to the left
bindkey ';l' end-of-line
bindkey 'l;' end-of-line
# Move all the way to the right
bindkey ';h' beginning-of-line
bindkey 'h;' beginning-of-line

Jumping to the beginning of the line is now as easy as pressing ; and h at the same time. No need to switch to normal-mode and your fingers don’t leave the your keyboard’s home-row. Try it, it’s great!

More

Customizations like this can make it much more pleasant to use the command line and boost your productivity. If the above is still not enough, here are some more ideas:

  • Define custom keymaps, e.g. to control other programs such as mpc or tmux.
  • Switch to said keymaps via some nice bindings.
  • Show the status of version control systems and build environments in the prompt.

If you don’t feel like doing all the work yourself, I can heartly reommend you take a look at oh-my-zsh. It offers a great collection of ideas to build on and some really cool ready-to-use plugins. Have fun!

Footnotes

  1. Emacs is a great program which I’ve been using for years and continue to use daily, but vi bindings just make good things better. Thanks to evil, that’s not a problem.↩︎

]]> command-line key bindings https://tarleb.com/posts/howto-zsh-vi-style/ Sun, 29 Jun 2014 00:00:00 GMT Exploring HTTP Headers with netcat Albert Krewinkel https://tarleb.com/posts/exploring-http-headers/ One of the many great things about free and open source software and the whole GNU/Linux ecosystem are the simple yet powerful tools available. The possibilities enabled by almost trivial programs are incredible. A very positive side effect this has on me is that I like to go and explore technologies with the tools at my disposal. My latest experiments revolved around the HTTP protocol, specifically HTTP headers, and very basic open source networking tools.

The HyperText Transfer Protocol

Webservers on the internet sending a website to a browser use the HyperText Transfer Protocol (HTTP) to do so. Along with the HTML data for the page itself, the server answer includes additional information: Response code, cookies, and how the browser or proxy server should handle the contents is transfered within the HTTP header. The ability of the headers to control state on the client side is what makes them so interesting and the reason why we are going to have a closer look at them.

Communicating with netcat

Instead of building our own HTTP client and server implementations – that would be total overkill – we restrict the goal to a simple networking tool that can be made to receive, send and alter basic HTTP commands: netcat, the self-described TCP/IP swiss army knife, combined with basic shell scripts.

We start by setting up a basic echoing server which sends everything back the same way it was received.

nc -l -p 8042 -e '/bin/cat'

Pointing the browser at http://localhost:8042, then killing the netcat process manually by hitting Control-C, we can see the headers we sent within our browser. Everything that is send to our simple server is put through the cat program, which just passes it on to STDOUT, which is then sent back to the connecting browser. The process has to be terminated manually, as it doesn’t know when to stop listening for more input. It’s crude, yet effective.

Shell Scripting for more advanced features

The above is neither comfortable to use nor very good to toy with, so we replace cat with a script of our own, we’ll call it exploring-http.sh:

#!/bin/sh

# Keep reading everything until we hit the first empty line
read_headers ()
{
    read i
    while [ -n "$i" ] &&
          [ "$(echo -n "\r\n")" != "$i" ] &&
          [ "$(echo -n "\n")" != "$i" ]
    do
        echo "$i"
        read i
    done
}
request_headers="$(read_headers)"

# Get some response headers ready
response_headers ()
{
    printf "HTTP/1.1 200 OK\r\n"
    printf "Content-Type: text/plain\r\n"
    printf "\r\n"
}

# Send the response
respond ()
{
    local response_headers="$(response_headers)"
    echo "${response_headers}"
    echo "Browser Request Headers"
    echo "======================="
    echo "$request_headers"
    echo "\r\n"
    echo "Server Response Headers"
    echo "======================="
    echo "${response_headers}"
}

respond

The request send by the browser is read till we reach the first black line, signaling the end of the request header. This time, we follow the protocol by prefixing the content with very simple response headers before sending it back to the browser. We also don’t have to manually terminate our netcat server, it terminates after answering to the request. Starting it again after each request is tedious, so we automate it and put it into a loop, restarting the server immediately once it terminates.

sh -c 'while true; do nc -l -p 8042 -e exploring-http.sh; done'

Now we are free to experiment with HTTP headers and the way browser and server interact. For example, we can let the server add a Last-Modified header, the content of which should be sent back by the browser in the next request:

response_headers ()
{
    printf "HTTP/1.1 200 OK\r\n"
    printf "Content-Type: text/plain\r\n"
    printf "Last-Modified: $(date --rfc-2822)\r\n"
    printf "\r\n"
}

Reloading twice, and the browser request will change to send an additional If-Modified-Since header.

ETags

The functionality of ETags, designed to communicate caching of old files, can be used follow users around without the need of cookies. Let’s see if we can do this with our little server.

The function generating the response headers is modified to extract any ETag supplied by the browser. If none exists, we generate a new one by hashing the number of nanoseconds passed since the beginning of the UNIX epoche. The parsed or newly generated etag is then sent back to the browser. We also add a few header to make sure the conents isn’t cached. As a result, we should be able to track a user through his or her browser cache.

response_headers ()
{
    local etag
    etag=$(echo "${request_headers}" | sed -ne 's/^\(If-None-Match: "\([a-f0-9]*\)".*\)/\2/gp')
    printf "HTTP/1.1 200 OK\r\n"
    printf "Content-Type: text/plain\r\n"
    printf "Last-Modified: $(date --rfc-2822)\r\n"
    printf "ETag: \"${etag:-$(date +%s%N | md5sum | cut -d' ' -f1)}\"\r\n"
    printf "Expires: Tue, 01 Jan 2013 00:00:01 GMT\r\n"
    printf "Cache-Control: max-age=0\r\n"
    printf "Connection: keep-alive\r\n"
    printf "\r\n"
}

We can test this by reloading our test page twice and… it works! We can reload as often as we want, the ETag header sent by the browser will not change unless we clear the browser’s cache. A stealthy kind of user tracking can be simulated with just a few lines of shell script.

Conclusion

Even though we used nothing but simple command line tools and shell scripting, we managed to build a simple server and to experiment with the ways in which stateless servers and stateful browseres can effect each other through HTTP headers. Standard UNIX tools are very powerful by themselves; together with a tool like netcat, power and fun extend even into experiments with networking and default protocols.

]]> command-line network https://tarleb.com/posts/exploring-http-headers/ Tue, 05 Nov 2013 00:00:00 GMT Russian Process Roulette Albert Krewinkel https://tarleb.com/posts/russian-process-roulette/ A couple of friends and I just discussed the idea of Russian Process Roulette: Everybody starts a little program which kills processes at random. The winner is the one whose computer remains usable the longest. Is this crazy? Yes. Immature? Sure. Fun? Definitely. Adviced against to actually play unless the consequences are fully understood? That too.

Playing the game is as simple as starting a command like the following:

while sleep 600; do kill -9 $(tr -cd "[:digit:]" < /dev/urandom | head -c5); done

This one-liner will wake up every 10 minutes, generate a random 5-digit number and then kill the process with the corresponding process ID. It will run forever, until it is either stopped, kills itself or has successfully crashed the user’s session.

Of course, there are some problems with this code. The time at which the next process will be killed is predictable, making it easier to prepare for the eventuality of a dying process. Worse, the range of random process ids (0 to 99999) is about three times as large as the actual range of process ids on a typical linux system (0 to 32768). This lowers the chance of hitting a valid process ID quite a bit. So let’s put our scripting-fu to some misguided use and “optimize” the code.

The first step is to write a function which gives uniformly distributed random numbers below a threshold.

random_number_below () {
  local upperbound=$1
  local candidate=$(( $1 + 1 ))
  local maxlen=$(printf $upperbound | wc -c)
  while [ $candidate -gt $upperbound ]
  do
      candidate=$(tr -cd '[:digit:]' < /dev/urandom | head -c "$maxlen" )
  done
  echo $candidate
}

Our random number generator is very very wasteful in terms of processor cycles. Even more, if the output of /dev/urandom is truly random, the function were not even guaranteed to terminate in any specified time frame. But given the nature of the application we have in mind, both caveats are completely acceptable here. Candidate numbers are generated by reading random characters from the systems urandom device, throwing away every character that isn’t a digit. After this, the candidate number is checked to make sure we haven’t generated a number greater than the specified upper bound. If that should be the case, we run the whole procedure again. Otherwise, the random number is returned. While wasteful, this function produces an unbiased uniform distribution on the interval [0, upperbound].

The second problem, regarding the range of possible process identifiers, is easier to fix. A little bit of searching reveals that the largest possible process identifier can be read from the file /proc/sys/kernel/pid_max. With this functionality in our hands, we can now write a “better” Russian Process Roulette script.

while sleep $(random_number_below 999)
do
    local maxpid=$(cat /proc/sys/kernel/pid_max)
    kill -9 $(random_number_below "$maxpid")
done

We should (probably not) run this as root, thereby making sure no process is safe from our “process gun”.

While the result of our efforts isn’t necessarily useful, it is still a nice exercises in shell scripting. Our game remains destructive and immature, but the execution is slightly more sophisticated. Hope you like it.

]]> command-line nonsense https://tarleb.com/posts/russian-process-roulette/ Fri, 27 Sep 2013 00:00:00 GMT Avoid Mail Harvesting through Address Munging Albert Krewinkel https://tarleb.com/posts/avoiding-spam/ Spam just doesn’t die. Every major mail provider has really good spam filters in place, as do the common mail clients. Nontheless, there seem to be enough people reading – and acting on – spam mail to make it a profitable business. No matter how much we hate it, we need to deal with spam.

Better than having a good spam filter is not to be spammed in the first place. Since nobody can sent me unsolicited bulk mail if they don’t know my mail address, keeping the mail address as private as possible is the way to go here. But what if we want our e-mail address to be public for other people to reach us? Adding minor obstacles to make it just a bit more difficult for spammers to get the address can be enough.

Address Munging

The usual way for spammers to collect email addresses is to crawl websites and to harvest everything that looks like a valid mail address. A common defense is address munging, i.e. resolving to bogus comments, additional markup, images or javascript to hide mail addresses from automatic harvesters. I find the use of images or javascript problematic for various reasons and just dislike unnecessary comments. Luckily, HTML5 and CSS3 make it really easy to mangle addresses without using either of these methods.

Obfuscating an Address with HTML5 & CSS3

Our approach here leverages the HTML5 data-* attributes as well as the ::before and ::after pseudo-elements. The mail address is divided and stored in custom data attributes and displayed as pseudo-elements. The address is kept in the markup, the presentation is handled in the style sheet:

HTML:

<span data-user="john.doe"
      data-domain="example.com"
      class="obfuscated-mail-address"><!--
-->@<span>obfuscated email address</span></span>

CSS:

.obfuscated-mail-address::before {
  content: attr(data-user);
}
.obfuscated-mail-address::after {
  content: attr(data-domain);
}
.obfuscated-email-address > * {
  display: none;
}

The result is readable, but can be neither selected nor copied:

obfuscated email address

The nested <span> and its styling could be omitted, without changing the presentation in most browsers. We use it to ensures that everything fails gracefully in browsers with limited CSS support. To see why this matters, have a look at this page in a text browser like w3m or links.

Drawbacks

The biggest advantage of this method is also it’s biggest disadvantage: The semantics are broken. The classic markup-code

<a href="mailto:me@example.com">John Doe</a>

is rendered as John Doe and makes it very clear, to machines and humans alike, that John Doe can be reached at me@example.com. Address munging, like the above, preserves this meaning for humans, but takes it away for machines. For better and for worse, browsers, search engines, and spam bots will not be aware of the meaning of the obfuscated address.

These effects can be alleviated by resolving to additional javascript:

// convert obfuscated mail addresses into clickable "mailto:" links
jQuery("document").ready(function($) {
  var mailAddressClass = "obfuscated-mail-address";
  $("."+mailAddressClass).each(function() {
    var address = $(this).attr("data-user") + "@" + $(this).attr("data-domain");
    $(this).html($("<a href=\"mailto:" + address + "\">" + address + "</a>", {}));
    $(this).removeClass(mailAddressClass);
  });
});

The webpage is still usable with JavaScript turned off, and even fails gracefully on text-only browsers. If JavaScript is available, the obfuscated links offer the same experience as properly coded <a href="mailto:...">...</a> links: @. I believe this to be a very good trade-off between usability and safety from harvesters.

]]> html css javascript https://tarleb.com/posts/avoiding-spam/ Sun, 25 Aug 2013 00:00:00 GMT