Pipelines for visual programming languages

November 1st, 2011 by balor

As a long time Unix user I’ve always found pilelines to be very useful. They allow you to build up extremely complex functions by sticking together much less complex parts. On Linux we put together commands and pipe the output into the next command. If you ask the question “How many mp3 files are there in this directory?” you can say “Ah, I know how to list all mp3 files, and I know how to count the number of lines in a list” then the answer becomes

$ ls -l *.mp3 | wc --lines

Pipelines can become arbitrarily complex. For example, the pipeline

$ cat /var/log/maillog | grep "status=sent" | grep -v phoric | wc --lines

counts the number of lines in my maillog that contain the text “status=sent” and omit the text “phoric” (my hostname). There is usually someone smart enough to write a shorter pipeline than what you’ve written, but that’s not the point. The point is that pipelines allow us to construct complex functionality by chaining together small programs.

If I were an academic (and I am), I’d probably generalise. I’d say somthing like, “pipelines have a source, they have many filter and translate actions and finally an action”. This, I believe, is a reasonable generalisation. In a pipeline, we often take some data, strip out the bits we don’t need, translate it into another format, maybe filter some more stuff and then finally do something. Such processes can be (relatively) easily turned into a visual programming language. The web application if this, then that is a simple visual programming language. Apple’s Automator is a more general visual programming language for such pipelines on OS X. And, going back to text-based pipelines, Window’s Powershell does away with much of the faffing about with text processing found in a Unix pipeline (by encapsulating the faffing in objects)

Get-ChildItem C:\Scripts | Where-Object {$_.Length -gt 200KB} | Sort-Object Length

The powershell example is stolen from technet.microsoft.com.

Let’s say I have an application area in mind. It’s a technical area with highly complex theory such that we can expect the individual components of our pipeline to contain complex functionality. Lets take the world of audio and video processing. The words are technical, we can expect to mux (i.e. multiplex) and demux audio. And we can also expect to transcode it. Decoding, say, a flac stream and re-encoding it as a vorbis stream requires complex algorithms. Furthermore, the language is generally well understood by geeks, like you. So, how do I do exactly that. Open a flac file, decode it, re-encode it as a vorbis stream and save it to a file. Fortunatly, a pipeline based toolset for audio (and video) already exists. We’re going to look at GStreamer. It’s effectively a domain-specific programming language that could be easily implemented as a visual programming language (it previously has been, but the visual side wasn’t of use to anyone who could actually understand the domain). We want to look at how GStreamer makes constructing complex pipelines easy.

The following pipeline is the start of an answer our problem (it intentionally doesn’t work):

$ gst-launch filesrc location=song.flac ! flacdec ! vorbisenc ! filesink location=song.ogg

In the above example we construct the pipeline, where ! is the pipe symbol and launch it with gst-launch. Each of the pipeline elements, on their own, are simple to understand. A filesrc element reads a file at the specified location. A flacdec element decodes a flac stream. A vorbisenc element encodes something in to vorbis and a filesink element stores a stream back into a file. The above pipeline produces the error

WARNING: erroneous pipeline: could not link flacdec0 to vorbisenc0

This is brilliant. A major difference between a GStreamer pipeline and a Unix pipeline is that I get static checking. It can tell me, from the structure of the pipeline, whether the type of data produced by each element, can be consumed by the following element. This is brilliant because it saves time.

Imagine that my pipeline was much more complicated. Imagine that the pipeline encoded a long, high-quality, video (taking several hours) and then tried to do something specific with the audio stream. As a Unix pipeline it would look like

$ cat video.raw | encodevid | modifyaudio > video.webm

We would have to run this pipeline before we saw passing of data from encodevid to decodevid failed. So having some static checking a-priori (I’m an acadmeic, I’m allowed to use Latin) could save us a lot of time. Static checking, like in a programming language, also ensures that we are passing the right data into an element that handles that data.

Ok, so what’s a working pipeline for the above question?

$ gst-launch filesrc location=song.flac ! flacdec ! audioconvert ! audioresample ! vorbisenc ! oggmux ! filesink location=song.ogg

Note: there’s a much shorter pipeline, but this one makes most of the stages explicit. I was missing the audioconvert, the audioresample and the filesink steps. Once again, this is a very complex domain. Each of the elements in this pipeline are more simple to understand than the entire pipeline.

Let’s do something magic. Take the original flac file, convert it to 8-bit audio and re-encode it as a vorbis stream in an ogg container. Why? Well this is the point of pipelines. Someone might want to do this sometime. The pipeline architecture allows you to combine elements in ways that other people may not have considered, or simply don’t need.

$ gst-launch filesrc location=song.flac ! flacdec ! audioconvert ! audioresample ! capsfilter caps=audio/x-raw-int,channels=2,width=8,depth=8 ! audioconvert ! vorbisenc ! oggmux ! filesink location=song.ogg

I think that GStreamer is a well desiged, well implemented pipeline architecture for a complex domain. So how does it achieve what it does?

Each of the elements in GStreamer advertises its capabilities. Each element advertises the capabilities of what it accepts (eg: raw audio with sample freq > 8hHz and < 44kHz) and the capabilities of what it produces (eg: audio bit stream conforming to the vorbis spec). You can then do static checking on pipelines by ensuring the capabilities of what is produced by one element are a subset of those consumed by the following element. It's very neat.

How might you retrofit this into an existing set of complex Unix applications? You could add a --caps flag (or -caps if you insist on BSD style) to each of your applications. If you have no source, wrap them in a shell script that takes --caps. Make the caps option outupt the input and output capabilities of your pipeline elements. Bonus hipster points if this is outputted in JSON. Then write a my_static_checker that takes your pipeline as a string, checks it statically a-la-GStreamer (I’m allowed to abuse French, I’m an academic) and then executes the pipeline.

Why! Now we’re in a position to write a very domain-specific visual programming language. This could be useful in a lot of cases. Particularly in the case of at least one person I’ll be emailing this to.

Recording presentations

September 26th, 2011 by balor

I have had a need to record lecture presentations. To this end I’ve hacked up some software which (a) takes a feed from a webcam, and (b) takes a PDF/ODF presentation and combines them into a WebM file.

For the code:

git clone git://gitorious.org/lecturec/lecturec.git

A simple overview:

Inkscape, textext and pixelated fonts

July 4th, 2011 by balor

I use Inkscape a lot. Particularly for drawing diagrams for my research. I use the TexText plugin for Inkscape to allow me to embed arbitrary LaTeX in documents. However, recently I’ve had an issue with it.

TexText uses pdf2svg to convert PDFs to SVGs to embed into an Inkscape project. Under the hood it runs pdflatex on the LaTeX fragment you provide, then it converts this to an SVG with a tight bounding box. This has been generating horribly pixelated fonts for me. The output of X+y=z is the following:
Garbled output from pdf2svg

Having looked into it there are two ways to solve the problem

  • The quick and simple way is to \usepackage{lmodern} at the top of your LaTeX preamble input to TexText – this forces latex to use type1 (scalable) fonts as opposed to type3 (bitmap) fonts.
  • The longer term way is to maintain textext for our research group to use. This will involve dropping all backends other than the pdf2svg one. Simply for the reason that I have neither the resources nor inclination to support multiple backends. Each supported backend is a deep mine of bugs and I’m comfortable with the pdf2svg/poppler/cairo mine.

I do wonder though, why LaTeX prefers type3 fonts over their type1 brethren.

FOlder – straightforward forms

June 28th, 2011 by balor

So I’ve got a problem. Every year I have to gather together a set of paper based forms and put them in a folder to hand to someone for QA processing. Most of these forms start off in some word-processing package, but by the time that they require review by a colleague and multiple signatures they end up on paper – it’s simply easier. But I’m not too good with paper and I’m even worse with uncodified processes that I follow once a year. Surely there’s a way to codify a process of handling documents? Yeah, there are plenty of tools that do it. And here’s another…

FOlder gives you a web based methods of creating, and writing documents. It’s based on the amazing Aloha-Editor and the fantastic Happstack. So the technical plumbing is thoroughbred, using a (soon-to-be) distributed (guaranteed) ACID data store. It’s missing three major features

  • Authentication (coming in version 0.1),
  • Collating the created files in a folder (simple fix in version 0.2),
  • Digital signatures (straightforward to implement, hard to devise a good UI).

and one minor feature

  • upload PDFs which are viewable using PDF.js.

You can find the source code here. I’m happy to accept patches!

Update:
A quick hacked video on YouTube (uploaded in WebM so it shouldn’t require flash)



Flattr this

Three Haskell web frameworks

June 4th, 2011 by balor

This model-view-controller pattern has been kicking around for quite some time now. It’s a nice way of separating concerns in many applications including web apps. And, a modern web-framework should both let you separate concerns and, ideally, bridge the gap between your programming model and data storage model. Some really nice people have done this in Ruby and Python and Python and …. well you get the picture. If you like a programming language there are probably several modern web-frameworks for it. So can we ask that question about Haskell? How would a modern web-framework work in a language that allows easy parallelisation, straightforward concurrency and even software transactional memory. Here’s my brief take on three Haskell web frameworks, with the caveat that I’ve only used one in anger (unsurprisingly my favourite one).

Snap Framework has been getting a lot of press lately. It’s fast. Faster than node.js but if you look at the documentation it appears that it uses a similar model to node.js i.e. a fast event loop. Snap provides a templating language called Heist. It has a nice method of handling RESTful queries (code stolen from here)

site :: Application ()
site = route [ ("/", index)
, ("/echo/:stuff", echo)
]
serveDirectory "resources/static"

Which just fires things off to specific handlers. Nice. And, just for good measure, I’ve stolen this heist snippet too

<html>
<head>
<title>Home Page</title>
</head>
<body>
<h1>Home Page</h1>
<apply template="nav"/>
<p>Welcome to our home page</p>
</body>
</html>

Snap leaves the choice of a storage backend up to the developer. They’ve got good reasons for this, however if doesn’t endear me. I’m happer with smart people doing all the integration work and giving me something to use.

Yesod gives you the choice of using either sqlite or postgres as a backing store. And it’ll do automagic migration of database schemas. Defining a datatype for storage is simple

mkPersist [$persist|
Entry
title String
day Day Desc
content Html'
deriving
|]

Again, this code is stolen from the very excellent tutorial. But this is looking straightforward, I get a Haskell datatype that’s persisted to a standard, trustworthy relational data store. Now, the people who are writing Snap seem to be scary smart and like to make fast things even faster, but this guy who develops Yesod seems to want to give me all the tools I need to just make a web app. It’s worth looking at his screencasts to see how easy OpenID and Facebook authentication is in a Yesod app. Furthermore, Yesod provides Hamlet, Cassius and Julius – respectively HTML, CSS and JavaScript templating languages. Julius, in particular, seems to have been designed with jQuery in mind. This, again, makes sense to me. Someone is giving me great tools to produce web apps under the standard use-case i.e. nice RESTful blingy things. You can even see that the Yesod query routing is simple

mkYesod "Blog" [$parseRoutes|
/ RootR GET
/entry/#EntryId EntryR GET
/admin AdminR EntryCrud defaultCrud
|]

And finally we get to Happstack. Happstack is the most mature of the three and features a wonderful data store called MACID. MACID has acid super powers but it distributes. I get to store any Haskell structure I want in a secure, distributed data-store! Representing reasonably complex information is easy

$(deriveAll [''Show, ''Eq, ''Ord, ''Default]
[d|
-- |ConferenceSubmission: a paper submitted to the conference
data ConferenceSubmission = ConferenceSubmission
{ author :: String
, title :: String
, date :: ClockTime
, email :: String
}

-- |Conference: a list of ConferenceSubmission
newtype Conference = Conference { conferenceSubmissions :: [ConferenceSubmission] }
|])

Routing is a little hairy, but you get a lot of power from it. And it’s nicely monadic.

appHandler =
do decodeBody (defaultBodyPolicy "/tmp/" 4096 4096 4096) -- decode the request body if present.
msum [ methodM GET >> renderIndex -- matches /
, renderSubmissions =<< conferenceHandler
]
conferenceHandler :: ServerPartT IO (HSP XML)
conferenceHandler =
dir "submissions" $ msum [postSubmission, getEntries] -- RESTful /submissions

Happstack uses Blaze or HSP or many other options for its templating language. Again, Blaze and HSP both play well with jQuery.

I have a fun project to play with over the next few months. I think Happstack is what I’ll be using.

Graphviz and the functional graph library

May 29th, 2011 by balor

A friend of mine asked how to display a Haskell Functional Graph Library (FGL) graph using graphviz. Turns out it’s really straightforward. Martin Erwig (the author of FGL) considered this when writing the library

$ ghci
> let g=Data.Graph.Inductive.Example.dag4
Use one of the example graphs so that we can print it out.  Then
> import Data.Graph.Inductive.Graphviz
> graphviz g "Foo" (100.0, 100.0) (100, 100) Portrait

where the documentation for the graphviz function is here

Of course, you can do much more complex stuff with the graphviz package…but that’s for another day.

Further homage to QuickCheck

May 20th, 2011 by balor

Lets suppose you’ve got two similar recursive types that differ only in their “payload”. For example, you want a binary tree of boolean operations to store a Char and another similar tree to store a set of Char. Let’s call them Tree and STree. Something like in the following, non-working, code

data Tree = And Tree Tree
  | Or Tree Tree
  | Leaf Char

data STree = And STree STree
  | Or STree STree
  | Leaf (Set Char)

We’re repeating a lot of boiler plate here. It’d be great if I could parameterise the tree, which I can.

data MTree t = And (MTree t) (MTree t)
  | Or (MTree t) (MTree t)
  | Leaf t

This is great, it removes boilerplate, and gives me less code to test.

Take our Tree as defined above. It’s easy to create an Arbitrary Tree. The following code creates an arbitrary tree with a maximum branch depth (generally for efficiency reasons you don’t want test trees that are either (arbitrary :: Int) or infinitely deep)

import Test.QuickCheck
import Control.Monad

data Tree = And Tree Tree
  | Or Tree Tree
  | Leaf Char
  deriving Show

arbitraryTree :: Int -> Gen Tree
arbitraryTree 0        =
  do
    c <- arbitrary
    return (Leaf c)
arbitraryTree maxDepth =
  do
    t <- oneof [arbitraryTree 0,
                liftM2 And (arbitraryTree depth') (arbitraryTree depth'),
                liftM2 Or (arbitraryTree depth') (arbitraryTree depth')]
    return t
  where
    depth' = maxDepth -1

instance Arbitrary Tree where
  arbitrary = do
    maxDepth <- choose (1, 10)
    t <- arbitraryTree maxDepth
    return t

So how do you create an arbitrary MTree? The following code makes it all work.

import Test.QuickCheck
import Control.Monad

data MTree t = And (MTree t) (MTree t)
  | Or (MTree t) (MTree t)
  | Leaf t
  deriving Show

arbitraryMTree :: Arbitrary t => Int -> Gen (MTree t)
arbitraryMTree 0        =
  do
    c <- arbitrary
    return (Leaf c)
arbitraryMTree maxDepth =
  do
    tr <- oneof [arbitraryMTree 0,
                liftM2 And (arbitraryMTree depth') (arbitraryMTree depth'),
                liftM2 Or (arbitraryMTree depth') (arbitraryMTree depth')]
    return tr
  where
    depth' = maxDepth -1

instance Arbitrary t => Arbitrary (MTree t) where
  arbitrary = do
    maxDepth <- choose (1, 10)
    tr <- arbitraryMTree maxDepth
    return tr

This is remarkably neat. QuickCheck, for me, is the killer feature of functional programming!

Returning to the Squad example from the other day. We can create Arbitrary instances of Squad.

import Data.Set
import Data.List
import Test.QuickCheck
import Control.Monad

data Squad a = Squad { players :: (Set a), team :: a}

instance (Ord a, Arbitrary a) => Arbitrary (Squad a) where
  arbitrary =
    do
    ps <- listOf1 arbitrary
    t <- elements ps
    return (Squad (fromList ps) t)

But now I have a complication (and this is where the Squad metaphor breaks down, but please bare with me). Suppose I want a Squad of players where I can in some instances choose a subset of them to play, and in other instances only choose one of them to play. So the Squad still constains a set of players, but the team can sometimes be a subset of the players, and sometimes is a single player. We’ve got to allow instances of TypeSynonyms:

{-# LANGUAGE TypeSynonymInstances #-}
import Data.Set
import Data.List
import Test.QuickCheck
import Control.Monad

data Squad a b = Squad { players :: (Set a), team :: b}
type SetSquad = Squad Char (Set Char)
type SingleSquad = Squad Char Char

instance Arbitrary SetSquad where
  arbitrary =
    do
    ps <- listOf1 arbitrary
    t <- elements (subsequences ps)
    return (Squad (fromList ps) (fromList t))

instance Arbitrary SingleSquad where
  arbitrary =
    do
    ps <- listOf1 arbitrary
    t <- elements ps
    return (Squad (fromList ps) t)

Which is awesome.

Mad props to my homeboy Eric (I’ve been told that’s how one addresses a USian) for listening to my overtired ramblings last night, and for pointing me at numerous ways to implement what I wanted to.

Recursive types in QuickCheck

May 18th, 2011 by balor

I want to build arbitrary sized trees of Compound

data Compound = And Compound Compound
  | Or Compound Compound
  | Product Compound Compound
  | Not Compound
  | Leaf { diagram :: Unitary} deriving (Show, Eq)

However, I want to limit to a max depth, and I want to have each branch to be of an arbitrary depth.

mkBinary :: (Compound -> Compound -
> Compound) -> Int -> Gen Compound
mkBinary f depth =
  liftM2 f (mkCompound depth') (mkCompound depth')
  where
    depth' = depth -1

mkNot :: Int -> Gen Compound
mkNot depth =
  liftM Not (mkCompound (depth-1))

mkLeaf :: Gen Compound
mkLeaf = liftM Leaf arbitrary

mkCompound :: Int -> Gen Compound
mkCompound 0 = mkLeaf
mkCompound depth = oneof [mkLeaf, mkBinary And depth, mkBinary Or depth, mkBinary Product depth, mkNot depth]

instance Arbitrary Compound where
  arbitrary =
    do depth <- choose (0, 4)
       d <- mkCompound depth
       return d

QuickCheck can do it. It’s nice. Use it.

Creating Arbitrary Foo for QuickCheck

May 17th, 2011 by balor

I always write semi-automated tests, whatever language I’m using. In Haskell my preferred framework is QuickCheck, which goes beyond what an XUnit framework is capable of. I come to QuickCheck as a convert from the old religion of Object Oriented programming. However much I’ve adopted functional practices I find myself, in the darkest of times, falling back on idioms from a past life. Recently I’ve been testing code that looks like the following:

data Squad = Squad { players :: Set Char,
team :: Set Char }

So what I’ve got is some type which is composed of a set and some other set which is a subset of the original. I hope the Squad metaphor is reasonable. In many sports you’ll have a panel of players, called the squad, and then for any given game you’ll have a team which is a subset of the players in your squad. To simplify this problem, each player is identified by a single character rather than a name. Now, we don’t have a dependent type system here. Just plain old Haskell. So the constraint that “team is a subset of players” is not enforced by the type system.

Now I want to tell Haskell to create Arbitrary instances of Squad.

randSubset :: Ord a => Set a -> Gen (Set a)
randSubset xs = do {
mask <- vectorOf (Set.size xs) arbitrary;
return (Set.fromList [ e | (e,True) <- zip (Set.toList xs) mask ]) }

instance Arbitrary Squad where
arbitrary = do {
players <- listOf1 (arbitrary :: Gen Char);
team <- randSubset (Set.fromList players);
return (Squad (Set.fromList players) team))}

The smarts here is in randSubset, but listOf1 ensures that there’s at least one player in our squad. I was struggling when playing around with a random number generator, to try and pull out random elements from a Set. Such thinking is a false idol. With some guidance from the only person on the planet to be 90% Real Jedi I realised you can just use the common generator combinators as provided by QuickCheck.

So the fragment

do {
mask <- vectorOf (Set.size xs) arbitrary;
return (Set.fromList [ e | (e,True) <- zip (Set.toList xs) mask ])}

asks for an arbitrary list of boolean values where the list is the same length as the size of the input Set. Then where the ith element of the list is true, the ith element of the set is considered to be in the subset, or omitted otherwise. It’s a nice idiom, useful anywhere you’re playing with sets that aren’t quite arbitrary themselves, but are constrained to be a subset of another.

An issue with QuickCheck 2

May 7th, 2011 by balor

I’ve been interested in functional programming for a while. I’ve hacked up a few things. Mira and Conferenceasaurus mainly. I’m not claiming that the code is great, I’d regard myself still as a n00b. I also have an interest in software testing, to this end the QuickCheck is both interesting and awesome. However, I’ve got one significant issue with my usage of it.

The types I’m playing with should probably be implemented using a dependent type system. For example, I’ve got a lot of

data Foo = Foo {bar::Set a, quux::Set a}

where quux should be a subset of bar. I’m not using a dependent type system, just regular Haskell, so the dependence is not enforced. However, I do want to generate instances of Arbitrary Foo where quux is a random subset of bar. Randomness is where my issues lie.

I could generate a random subset of a set

import Data.Set
import Data.Maybe
import Test.QuickCheck

randSubset :: Set a -> Gen (Set a)
randSubset xs = do {
mask <- vectorOf (size xs) (arbitrary::Gen Bool);
return (fromList (catMaybes [if b then (Just e) else Nothing | e <- l, b <- mask]))
}
where
l = toList xs

However, you’ll notice that the type signature is randSubset :: Set a -> Gen (Set a) and to create quux as a random subset of bar in an Arbitrary Foo I’d need this to return a Set a. I could unGen it, however this would require picking a seed which produces the same set of pseudo-random numbers each time, or generating a random seed, in which case I’d have to return an IO (Set a).

So, question time. Is it possible to generate arbitrary instances of Foo such that the property “quux is a subset of bar” is enforced in all arbitrary instances?

update:
As arbitrary :: Gen a then arbitrary = do x <- arbitrary; y <- randSubset x; return (Foo x y) is the solution.