Hoogle inside the sandbox


This is my first post from the (hopefuly fruitful!) series of blog posts as part of my Haskell SoC project. I will spend a great chunk of my summer hacking away on DarcsDen; in addition, I will document my hardships and successes here. You can follow my progress on my DarcsHub.

This particular post will be about my working environment.

The problem

Hoogle is an amazing tool that usually needs no introduction. Understandably, the online version at haskell.org indexes only so many packages. This means that if I want to use hoogle to search for functions and values in packages like darcs and darcsden, I will have to set up a local copy.

Cabal sandboxing is a relatively recent feature of the Cabal package manager, but I don’t think it is reasonable in this day to install from the source (let alone develop) a Haskell package without using sandboxing.

The problem seems to be that the mentioned tools do not play well together out of the box, and some amount of magic is required. In this note I sketch the solution, on which I’ve eventually arrived after a couple of tries.

Using hoogle inside a Cabal sandbox

The presumed setup: a user is working on a package X using the cabal sandboxes. The source code is located in the directory X and the path to the cabal sandbox is X/.cabal-sandbox.

Step 1: Install hoogle inside the sandbox. This is simply a matter of running cabal install hoogle inside X. If you want to have a standard database alongside the database for your packages in development, now is the time to do .cabal-sandbox/bin/hoogle data.

Step 2: Generate haddocks for the packages Y,Z you want to use with hoogle. In my case, I wanted to generate haddocks for darcs and darcsden. This is just a matter of running cabal haddock --hoogle in the correct directory.

Step 3: Convert haddocks to .hoo files. Run the following commands in X/:

.cabal-sandbox/bin/hoogle convert /path/to/packageY/dist/doc/html/*/*.txt

You should see something like

Converting /path/to/packageY/dist/doc/html/Y/Y.txt
Converting Y... done

after which the file Y.hoo appears in /path/to/packageY/dist/doc/html/Y/

Step 4: Moving and combining databases. The hoogle database should be stored in .cabal-sandbox/share/*/hoogle-*/databases. Create such directory, if it’s not present already. Then copy the ‘default’ database to that folder:

cp .cabal-sandbox/hoogle/databases/default.hoo .cabal-sandbox/share/*/hoogle-*/databases

Finally, you can combine your Y.hoo with the default database.

.cabal-sandbox/bin/hoogle combine /path/to/packageY/dist/doc/html/*/*.hoo .cabal-sandbox/share/*/hoogle-*/databases/default.hoo
mv default.hoo .cabal-sandbox/share/*/hoogle-*/databases/default.hoo

And you are done! You can test your installation

$ .cabal-sandbox/bin/hoogle rOwner
DarcsDen.State.Repo rOwner :: Simple Lens (Repository bp) String

For additional usability, consider adding .cabal-sandbox/bin to your $PATH.

ANN: Hastache version 0.6.1

Announcing: hastache 0.6.1

Happy holidays, everyone!

I would like to announce a new version of the Hastache library, version 0.6.1. Some interesting and useful changes, as well as improvements and bugfixes are included in the release. See below for an extended changelog.

Hastache is a Haskell implementation of the mustache templating system.

Quick start

cabal update
cabal install hastache

A simple example:

import Text.Hastache 
import Text.Hastache.Context 
import qualified Data.Text.Lazy.IO as TL

main = hastacheStr defaultConfig (encodeStr template) (mkStrContext context)
    >>= TL.putStrLn

template = "Hello, {{name}}!\n\nYou have {{unread}} unread messages." 

context "name" = MuVariable "Haskell"
context "unread" = MuVariable (100 :: Int)

Read Mustache documentation for template syntax; consult README for more details.

Whats’s new in 0.6.1?

Most of the new features in this release deal with generic contexts.

Context merging

composeCtx is a left-leaning composition of contexts. Given contexts c1 and c2, the behaviour of (c1 <> c2) x is following: if c1 x produces ‘MuNothing’, then the result is c2 x. Otherwise the result is c1 x. Even if c1 x is ‘MuNothing’, the monadic effects of c1 are still to take place.

Generic contexts for more datatypes

The mkGenericContext function now supports additional datatypes like Maybe (with Nothing being an empty/null value) and Either.

Context modifiers and renaming

The new mkGenericContext' is a generalized version of mkGenericContext and it takes two addition arguments. The first one, of type (String -> String) is simply a renaming function, similar to fieldLabelModifier of aeson. To see this feature in action, consider the following example:

{-# LANGUAGE DeriveDataTypeable #-}

import Text.Hastache 
import Text.Hastache.Context 

import qualified Data.Text.Lazy as TL 
import qualified Data.Text.Lazy.IO as TL 

import Data.Data (Data, Typeable)
import Data.Decimal
import Data.Generics.Aliases (extQ)

data Test = Test {f :: Int}
     deriving (Data, Typeable)

val1 :: Test
val1 = Test 1

val2 :: Test
val2 = Test 2

r "f" = "foo"
r x   = x

example :: Test -> IO TL.Text
example v = hastacheStr defaultConfig
                        (encodeStr template)
                        (mkGenericContext' r defaultExt v)

template = "An integer: {{foo}}"

main = do
  example val1 >>= TL.putStrLn
  example val2 >>= TL.putStrLn

In the example we use the renaming function r to rename a field “f” to “foo”.

The second additional argument is a query extension, of type Ext:

type Ext = forall b. (Data b, Typeable b) => b -> String

A query extension is a way of turning arbitrary datatypes into strings. This might come in very handy, if you want to generate mustache contexts from records/datatypes that contain non-primitive datatypes (from non-base modules) that you want to display. Before 0.6.1, if you had a record that contained, for example, a Decimal field, and you wanted to convert it to a context and access that field, you were simply out of luck. With this release you can basically extend the mkGenericContext' function to support any datatypes you want! Once again, I believe an example is worth a thousand words, so let us consider a slightly modified version of the example above:

{-# LANGUAGE DeriveDataTypeable #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE StandaloneDeriving #-}

-- Custom extension function for types that are not supported out of
-- the box in generic contexts
import Text.Hastache 
import Text.Hastache.Context 

import qualified Data.Text.Lazy as TL 
import qualified Data.Text.Lazy.IO as TL 

import Data.Data (Data, Typeable)
import Data.Decimal
import Data.Generics.Aliases (extQ)

data DecimalOrInf = Inf | Dec Decimal deriving (Data, Typeable)
deriving instance Data Decimal
data Test = Test {n::Int, m::DecimalOrInf} deriving (Data, Typeable)

val1 :: Test
val1 = Test 1 (Dec $ Decimal 3 1500)

val2 :: Test
val2 = Test 2 Inf

query :: Ext
query = defaultExt `extQ` f
  where f Inf = "+inf"
        f (Dec i) = show i

r "m" = "moo"
r x   = x

example :: Test -> IO TL.Text
example v = hastacheStr defaultConfig
                        (encodeStr template)
                        (mkGenericContext' r query v)

template = concat [ 
     "An int: {{n}}\n",
     "{{#moo.Dec}}A decimal number: {{moo.Dec}}{{/moo.Dec}}",
     "{{#moo.Inf}}An infinity: {{moo.Inf}}{{/moo.Inf}}"

main = do
  example val1 >>= TL.putStrLn
  example val2 >>= TL.putStrLn

As you can see, the query extensions are combined using the extQ function from Data.Generics, and the “unit” of this whole thing is defaultExt function.



This release would not have been possible without Tobias Florek, Edsko de Vries, Janne Hellsten, @clinty on Github, Stefan Kersten, Herbert Valerio Riedel, and other people who were submitting issues, patches, and requests.

ANN: hastache 0.6

Announcing: hastache 0.6

Hastache is a Haskell implementation of the mustache templating system.

Quick start

cabal update
cabal install hastache

A simple example:

import Text.Hastache 
import Text.Hastache.Context 
import qualified Data.Text.Lazy.IO as TL

main = hastacheStr defaultConfig (encodeStr template) (mkStrContext context)
    >>= TL.putStrLn

template = "Hello, {{name}}!\n\nYou have {{unread}} unread messages." 

context "name" = MuVariable "Haskell"
context "unread" = MuVariable (100 :: Int)

Read Mustache documentation for template syntax; consult README for more details.

Whats’s new in 0.6?

The interface of the library has been switched from ByteString to (lazy) Text. That means, for example, that the type of hastcheStr function is now the following:

:: MonadIO m => MuConfig m	-> Text	-> MuContext m -> m Text

The generic context generation (mkGenericContext) now supports functions of the types Text -> Text and Text -> m Text, as well as types with multiple constructors. That is, given a Haskell datastructure

data A = A { str :: String }
       | B { num :: Int }

it is possible to write a template like this:

A : {{str}}
B : {{num}}

Please take a look at the multiple constructors example if you are interested.

Additionally, a couple of new MuVar instances has been added.


Special thanks to Mark Lentczner, Alexander Voikov and others who reported issues, tested the library and provided feedback.

Representable functors in Haskell

I have finally managed to upload a (dry version) of a note about representable functors in Haskell. You can find it here: http://covariant.me/notes/rep-functors.html

Representable functors are functors that are isomorphic to $Hom(A, -)$ or $Hom(-, A)$ for some object $A$. In the note I give examples of some simple representable functors and prove that Maybe is not representable.

Cabal sandbox status in your ZSH prompt

Sometime ago I made a simple script for my zsh setup that allows me to see whether am I in a cabalized sandbox environment or not. I posted it to Haskell-cafe a month ago, but totally forgot to post it to this blog.

The result of checking for the sandbox is cached, which is probably unnecessary; it updates only when the user performs a cabal command or changes a directory.

After you place the contents of the script in your .zshrc file (or in a similar location), you should update your $PROMPT to use $(sandbox_prompt). The prompt I am using, by the way, is

local ret_status="%(?:%{$fg_bold[green]%}:%{$fg_bold[red]%})%?%{$reset_color%}"
PROMPT=$'\n$(ssh_connection)$HOST_COLOR%n@%m%{$reset_color%}$(my_git_prompt)$(sandbox_prompt) : %~\n[${ret_status}] %# '

and it is based on the oh-my-solarized theme.

ANN: scotty-hastache 0.2

The scotty-hastache library has been updating, following the introduction of the custom exception types in Scotty. Custom exception types (as opposed to plain-text exception mechanism that was previously employed in Scotty) is a more powerful way of handling exceptional situation; an example usage can be found here. Below are the appropriate release notes for scotty-hastache


Keeping up with the Scotty 0.6 updating. The introduction of the custom exception types is reflected in the scotty-hastache package as well. The main types are now polymorphic over an exception type

type ScottyH e = ScottyT e HState
type ActionH e = ActionT e HState

In addition, the specialized (to Text) types and runners are introduced

type ScottyH' = ScottyH Text
type ActionH' = ActionH Text
scottyH'     :: Port -> ScottyH' () -> IO ()
scottyHOpts' :: Options -> ScottyH' () -> IO ()

Thanks to Kirill Zaborsky for pointing out the incompatibility with 0.6.

As always, the scotty-hastache library is available on Hackage and on GitHub

True Concurrency and Net Unfoldings

This week I gave a talk at my university on true concurrency. My main focus, although, was partial-order semantics for Petri nets.

Slide screenshot

Slide screenshot


Computer scientists have been paying attention concurrency and parallelism since the dawn of the computing as science. In this talk I would like to discuss the topic called true concurrency – the concurrency in presence of parallelism, as opposed to concurrency defined by interleaving.

Carl Adam Petri played a crucial role in the area of true concurrency, he was interested in analyzing the relations and connections between the abstract machines (that of Turing, Church and others) and their physical counterparts. Petri observed that the assumption of the global state is unrealistic in big systems and he proposed to replace the “execution semantics’’ with the semantics of”nonsequential proceses’’, the notion which gave rise to the theory of occurrence nets and unfoldings. That corpus of work has be introduced originally to solve the very theoretical problem: to give precise semantics to concurrent systems. Goltz, Reisig, Nielsen, Plotkin, Winskel, Engelfreit, Montanari are among the people who have contributed to the topic.

Later on, with the rise of model checking techniques, it has been discovered that partial-order semantics (unfoldings) may be used in practice for formally verifying systems. McMillan, Esparza, Romer, Vogler, Khomenko and others have pioneered what is now can be called a “partial order approach to model checking”.

True concurrency is still a thriving field, and in my talk I hope to touch the foundations, as well as pragmatic applications of the approaches derived from the theoretical foundations of the subject.


.PDF link

Interactive-diagrams updates

It has been some time since I’ve written a blog post. Today I would like to present you the latest updates from the interactive-diagrams project.

  • diagrams-ghcjs finally got text support

Big thanks to Joel Burget for the implementation.

The testsuite demo for the package can be found at http://co-dan.github.io/ghcjs/diagrams-ghcjs-tests/Main.jsexe/ as usual.

  • New interactive widgets

We got rid of the old wizard-like widgets in favour of “all-in-one” style widgets. Thanks to Brent and Luite who came up with the type trickery to get this done.

  • New flat design

In order to match your slick & flat iOS7 style I’ve rolled out an update to Bootstrap 3.

  • Ability to paste code with errors. (It’ll ask you to make sure you didn’t make a mistake by accident)
  • Ability to quickly include a number of imports. By checking the “import standard modules” checkbox you’ll automatically bring several useful modules into the scope:
    • Diagrams.Prelude
    • Diagrams.Backend.SVG (or Diagrams.Backend.GHCJS if you are making an interactive widget)
    • Data.Maybe
    • Data.Tuple
    • Data.List
    • Data.Char
  • Other minor UI fixes such as documentation improvement

GSoC 2013, an afterword

The Summer of Code 2013 is over, and here is what I have to say about it.


The project is live at http://paste.hskll.org. The source code can be found at http://github.com/co-dan/interactive-diagrams.

I would like to say that I do plan to continue working on the project (and on adjacent projects as well if possible).

Interactive diagrams

Interactive diagrams is a pastebin and a set of libraries used for dynamically compiling, interpreting and rendering the results of user inputted code in a secure environment.

The user inputs some code and the app compiles and renders it. Graphical output alongside with code can be useful for sharing the experiments, teaching beginners an so on. If the users inputs a code that can not be rendered on the server (i.e.: a function), the app produces an HTML5/JS widget that runs the corresponding code.

The produced libraries can be used in 3rd party services/programs.



Technology used

The pastebin is powered by Scotty and scotty-hastache, the access to PosgreSQL db is done via the excellent persistent library. The compilation is done using GHC and GHCJS inside the workers processes powered by the restricted-workers library.

You can read some my previous report on this project which is still pretty relevant.

I plan on updating the documents on the wiki sometime soon.


The bad news is that I don’t think I was able to 100% complete what I originally envisioned. The good news is that I seem to know, almost exactly, what do I want to improve and how to do that. As I’ve mentioned I plan on continuing with the project and I hope that the project will grow and improve.

One thing that I felt was annoying is the (technical) requirement to use GHC HEAD. Because of that a lot of packages required updates and fixes. Due to changes in GHC and bugfixes in GHCJS I had to perform the tiring and not so productive procedure of updating all the necessary tools, rebuilding everything and so on. But I guess that’s just how computers work and I am sure that in the future (with the release of GHC 7.8 and a new Haskell Platform) the maintenance and installation will be much easier. Another thing that took a lot of my time was configuring the system and setting up the necessary security measures, which was also necessary.

Other stuff that kinda slowed thing down include: the lack of a good build system, in some cases non-American timezone (actually I think that the fact that my mentor, Luite Stegeman, was quite close to me in terms of timezones allowed us to communicated very frequently, as we did), the lack of knowledge of the tools I used (although you can think of it this way: I had an ability to learn exciting new things ;] ).

Among the grand things I plan to do: release a library for manipulating Haskell AST at the GHC level; make an IRC bot using the eval-api and restricted-workers; continue writing my notes/tutorials about GHC API (I have a few drafts laying around).

Some code refactoring should come along and a number of features for the pastebin should be implemented.


When the end of the program was approaching I predicted that I would have that sort of conflicted feelings that you usually get when you finish reading a good book – one part of you feels happy because you had an enjoyable experience, yet another part of you doesn’t feel so giddy, because the thing that you enjoyed is over. Well, I didn’t get this with GSoC. I did feel happy, but I didn’t get this touch of sadness. GSoC was a way for me to get into collaborating with people on real-world open source projects, and the end of GSoC for me is a beginning of something else. I can use my experience now to write better code, write more code and write useful code.


I had a very exciting summer and I would positively recommend anyone eligible to participate in the Google Summer of Code program. There is, however, a thing to remember. Programmers are known to be the kind of people who set ambitious goals. Reach for something inspiring, ambitious, yet realistic. Make sure to find something in between, that way you’ll have a concrete target that you know that you are able to achieve, but you also have a room for improvement.

PS. Acknowledgments

I would like to thank various people who helped me along the summer: Luite Stegeman, Brent Yorgey, Carter Schonwald, Daniel Bergey, Andrew Farmer; everyone in #diagrams, everyone in the #haskell channel who patiently answered my question; everyone on GitHub who responded to my comments, questions and pull requests. The Haskell community is lucky to have a huge amount of friendly and smart people.