Interactive-diagrams updates

It has been some time since I’ve written a blog post. Today I would like to present you the latest updates from the interactive-diagrams project.

• diagrams-ghcjs finally got text support

Big thanks to Joel Burget for the implementation.

The testsuite demo for the package can be found at http://co-dan.github.io/ghcjs/diagrams-ghcjs-tests/Main.jsexe/ as usual.

• New interactive widgets

We got rid of the old wizard-like widgets in favour of “all-in-one” style widgets. Thanks to Brent and Luite who came up with the type trickery to get this done.

• New flat design

In order to match your slick & flat iOS7 style I’ve rolled out an update to Bootstrap 3.

• Ability to paste code with errors. (It’ll ask you to make sure you didn’t make a mistake by accident)
• Ability to quickly include a number of imports. By checking the “import standard modules” checkbox you’ll automatically bring several useful modules into the scope:
  • Diagrams.Prelude
  • Diagrams.Backend.SVG (or Diagrams.Backend.GHCJS if you are making an interactive widget)
  • Data.Maybe
  • Data.Tuple
  • Data.List
  • Data.Char
• Other minor UI fixes such as documentation improvement

GSoC 2013, an afterword

The Summer of Code 2013 is over, and here is what I have to say about it.

Introduction

The project is live at http://paste.hskll.org. The source code can be found at http://github.com/co-dan/interactive-diagrams.

I would like to say that I do plan to continue working on the project (and on adjacent projects as well if possible).

Interactive diagrams

Interactive diagrams is a pastebin and a set of libraries used for dynamically compiling, interpreting and rendering the results of user inputted code in a secure environment.

The user inputs some code and the app compiles and renders it. Graphical output alongside with code can be useful for sharing the experiments, teaching beginners an so on. If the users inputs a code that can not be rendered on the server (i.e.: a function), the app produces an HTML5/JS widget that runs the corresponding code.

The produced libraries can be used in 3rd party services/programs.

Screenshot

Technology used

The pastebin is powered by Scotty and scotty-hastache, the access to PosgreSQL db is done via the excellent persistent library. The compilation is done using GHC and GHCJS inside the workers processes powered by the restricted-workers library.

You can read some my previous report on this project which is still pretty relevant.

I plan on updating the documents on the wiki sometime soon.

Progress

The bad news is that I don’t think I was able to 100% complete what I originally envisioned. The good news is that I seem to know, almost exactly, what do I want to improve and how to do that. As I’ve mentioned I plan on continuing with the project and I hope that the project will grow and improve.

One thing that I felt was annoying is the (technical) requirement to use GHC HEAD. Because of that a lot of packages required updates and fixes. Due to changes in GHC and bugfixes in GHCJS I had to perform the tiring and not so productive procedure of updating all the necessary tools, rebuilding everything and so on. But I guess that’s just how computers work and I am sure that in the future (with the release of GHC 7.8 and a new Haskell Platform) the maintenance and installation will be much easier. Another thing that took a lot of my time was configuring the system and setting up the necessary security measures, which was also necessary.

Other stuff that kinda slowed thing down include: the lack of a good build system, in some cases non-American timezone (actually I think that the fact that my mentor, Luite Stegeman, was quite close to me in terms of timezones allowed us to communicated very frequently, as we did), the lack of knowledge of the tools I used (although you can think of it this way: I had an ability to learn exciting new things ;] ).

Among the grand things I plan to do: release a library for manipulating Haskell AST at the GHC level; make an IRC bot using the eval-api and restricted-workers; continue writing my notes/tutorials about GHC API (I have a few drafts laying around).

Some code refactoring should come along and a number of features for the pastebin should be implemented.

Feelings

When the end of the program was approaching I predicted that I would have that sort of conflicted feelings that you usually get when you finish reading a good book – one part of you feels happy because you had an enjoyable experience, yet another part of you doesn’t feel so giddy, because the thing that you enjoyed is over. Well, I didn’t get this with GSoC. I did feel happy, but I didn’t get this touch of sadness. GSoC was a way for me to get into collaborating with people on real-world open source projects, and the end of GSoC for me is a beginning of something else. I can use my experience now to write better code, write more code and write useful code.

Concluding

I had a very exciting summer and I would positively recommend anyone eligible to participate in the Google Summer of Code program. There is, however, a thing to remember. Programmers are known to be the kind of people who set ambitious goals. Reach for something inspiring, ambitious, yet realistic. Make sure to find something in between, that way you’ll have a concrete target that you know that you are able to achieve, but you also have a room for improvement.

PS. Acknowledgments

I would like to thank various people who helped me along the summer: Luite Stegeman, Brent Yorgey, Carter Schonwald, Daniel Bergey, Andrew Farmer; everyone in #diagrams, everyone in the #haskell channel who patiently answered my question; everyone on GitHub who responded to my comments, questions and pull requests. The Haskell community is lucky to have a huge amount of friendly and smart people.

WIP: GHCJS backend for the diagrams library

About

I’ve picked up the development of the diagrams-ghcjs backend for the infamous diagrams library. This backend is what we use for the interactive-diagrams pastebin and it renders diagrams on an HTML5 Canvas using the canvas bindings for ghcjs. The diagrams-ghcjs is a fork of (unmaintained?) diagrams-canvas backend by Andy Gill and Ryan Yates.

The library is still WIP and it requires bleeding edge versions of ghcjs, ghcjs-canvas and ghcjs’ shims to function.

The library is scheduled to be released together with ghcjs.

Demo

The current demo can be found at http://co-dan.github.io/ghcjs/diagrams-ghcjs-tests/Main.jsexe/. It is supposed to work in both Firefox and Chrome.

Problems

• Text is not implemented. Some work is being done in the text-support branch. Generally, it has been proven hard to implement good text support, even diagrams-svg backend text support is limited;
• Firefox still does not support line dashing via ‘setLineDash’ and ‘lineDashOffset’. As a result we need to have additional shims.

Agile development and deployment in the cloud with Haskell and vado

In this post I would like to give you an update on vado – a piece of
software for running programs on vagrant VMs (or any other ssh server,
actually), projects I’ve contributed briefly to.

1 New build system

The old build system for ghcjs was a little bit messy. Basically, it was
just one Puppet configuration file that contained a hardcoded shell
script as a resource that is supposed to be written to the home
directory and executed. I decided to clean it up a notch and take more
of a Puppet approach to the whole thing.

You can find the new set of build script on the GitHub:
https://github.com/ghcjs/ghcjs-build

And since the errors are now printed to the screen it’s
easy to see which stage the build is going through and if anything
goes wrong you see an error trace for the current stage.

The prebuilt version has also been updated by
Luite Stegeman.

2 Vado

2.1 Vado intro

Hamish Mackenzie and I have been working on vado – a quick way to run
commands on a remote ssh server. Just mount the directory you want to
run the command in using sshfs, in that directory (or its
subdirectory) run vado like this:

vado ls -l

vado will run ‘mount’ to identify the user account, server name and
the remote directory to run the command in. It will then run ssh to
connect to the server and run the command.

You can also pass ssh options like this:

vado -t htop

This tells vado to pass -t to ssh (forces pseudo-tty allocation and
makes programs like vim and htop work nicely).

I will explain below how to set up vado for multiple remote
servers/sshfs mount points and how to develop Haskell projects on a
remote server/VM nicely using Emacs and ghc-mod.

2.2 .vadosettings

Vado is not tied to vagrant, but can be used with it and is faster
than vagrant ssh. If the user and host detected in mount are
specified in the ~/.vadosettings file, then the specified key and
port will be used.

The contents of the ~/.vadosettings file is basically a Haskell
list of MountSettings datastructures and we use standard Read and
Show type-classes for serialization.

The MountSettings data type is defined as follows:

-- | Mount point settings
data MountSettings = MountSettings {
    sshfsUser :: Text
  , sshfsHost :: Text
  , sshfsPort :: Int
  , idFile :: FilePath
  } deriving (Show, Read)

If the file is not present or incorrectly formatted
then the default settings for vagrant will be used:

• User: vagrant
• Host: 127.0.0.1
• Port: 2222
• Key file: ~/.vagrant.d/insecure_private_key

2.2.1 Example .vadosettings file

An example settings file might look like this:

[
  MountSettings {
    sshfsUser = "vagrant"
  , sshfsHost = "localhost"
  , sshfsPort = 2222
  , idFile = "/Users/dan/.vagrant.d/insecure_private_key"
  }, 
  MountSettings {
    sshfsUser = "admin"
  , sshfsHost = "server.local"
  , sshfsPort = 2233
  , idFile = "/Users/dan/keys/local_server_key"
  }
]

2.3 Vamount

Of course, using vado requires mounting the sshfs beforehand. But
it gets tedious typing out

sshfs vagrant@localhost:/home/vagrant ../vm/ -p2222
-reconnect,defer_permissions,negative_vncache,volname=ghcjs,IdentityFile=~/.vagrant.d/insecure_private_key

every time. A tool called vamount which is bundled together
with vado can be used for mounting remote filesystems based on
~/.vadosettings file.

You can use it like this:

vamount [ssh options] remote_path [profile #]

The remote_path from the remote server specified in the
~/.vadosettings file under number [profile #] will be mounted in the
current directory using sshfs.

The profile number count starts from 1. If the [profile #] is absent
or is 0 then the default (vagrant) configuration will be used.

2.4 Vado and ghc-mod

ghc-mod is a backend designed command to enrich Haskell programming on
editors like Emacs and Vim and it also features a front-end for Emacs
as a set of elisp scripts. It’s a really cool piece of software and if
you have not tried it yet I highly recommend you to invest into
installing and using it.

What we would like, however, is to edit files on the mounted
filesystem using Emacs on the host machine, but run ghc-mod inside the
VM. In order to do that we need to install ghc-mod both on our host
machine and on the VM.

While installing ghc-mod on the host machine running the latest
haskell-platform is pretty straightforward it is harder to do so on
the VM running GHC 7.7 due to the fact that many libraries are not
ready for GHC 7.7 and base 4.7 yet. We have to resort to installing
most of the things from source.

# run this on the guest machine
mkdir ghcmod && cd ghcmod

# patching installing convertible
cabal unpack convertible
cd convertible*
wget http://co-dan.github.io/patched/convertible.patch
patch -p1 Data/Convertible/Utils.hs convertible.patch
cabal install
cd ..

# installing ghc-syb-utils
git clone https://github.com/co-dan/ghc-syb.git
cd ghc-syb/utils/
cabal install
cd ../..

# finally getting and installing ghc-mod
git clone https://github.com/co-dan/ghc-mod.git
cd ghc-mod
cabal install

Ghc-mod itself uses the GHC API extensively so it’s no surprise we
have to change at least some code. Now that we have installed ghc-mod
on the guest VM we need to set up our host’s Emacs configuration to
communicate properly with the VM. First of all put this in your Emacs
config:

(setq load-path (cons "~/Library/Haskell/ghc-7.6.3/lib/ghc-mod-2.0.3/share" load-path))
(autoload 'ghc-init "ghc" nil t)
(add-hook 'haskell-mode-hook (lambda () (ghc-init)))
;; (setq ghc-module-command "ghc-mod")
(setq ghc-module-command "~/vado-ghc-mod.sh")

~/vado-ghc-mod.sh should contain the following:

#!/bin/bash
VADO=/Users/dan/Library/Haskell/bin/vado
LOCAL_PATH=/Users/dan/projects/ghcjs/mnt/
REMOTE_PATH=/home/vagrant/
$VADO -t ghc-mod ${@//$LOCAL_PATH/$REMOTE_PATH} | sed "s,$REMOTE_PATH,$LOCAL_PATH,g"

I know that it’s a hack, but it does work and I guess that’s what
shell scripts are for ;)

Now go to ~/.bashrc on the guest machine and make sure that the
PATH variable is set correctly:

PATH=/home/vagrant/ghcjs/bin:/home/vagrant/.cabal/bin:/home/vagrant/ghc/bin:/home/vagrant/jsshell:/home/vagrant/node-v0.10.10-linux-x86/bin:$PATH

# PATH is set *before* this line:
[ -z "$PS1" ] && return

# <snip>

And that’s it, you should be done!

Before (ghc-mod running on the host machine):

After (ghc-mod running inside ghcjs-build VM):

3 Conclusion and future work

We’ve seen how a small but useful tool vado can make our life easier if
we want to develop Haskell projects on a remote server or on a
virtual machine. You can get Vado from GitHub: https://github.com/hamishmack/vado

Next week we are planning on releasing our first version of
interactive-diagrams pastesite (not going to be very interactive
though) and writing out its security model.

Meanwhile check Luite’s post on using Sodium FRP library for creating
Functional Reactive Web interfaces. It’s astonishing how easily you
can just get a FRP library, compile to JavaScript and make nifty web
apps with it.

Building GHCJS

1 Intro

In this post I would like to talk about my experience with
bootstrapping GHCJS using the provided facilities ghcjs-build. I
never used tools like Vagrant or Puppet before so all of this was
kinda new to me.

2 Initial installation

GHCJS can’t actually work with vanilla GHC 7.* as it requires to
apply some patches (in order to get JS ffi to work, it adds
JavaScriptFFI language extension among other modifications).

ghcjs-build uses Vagrant (a tool for automatically building and
running work environments) to mange the work environment, so prior to
running GHCJS you need to install vagrant and VirtualBox. It’s actually
a sensible way to tackle a project like that: everyone has similar
work environments, you don’t have to mess with your local GHC
installation. It also make use of Puppet deployment system in
puppetlabs-vcsrepo module for cloning Git repositories.

Currently, there are two ways to start up GHCJS using ghcjs-build

2.1 Using the prebuilt version

git clone https://github.com/ghcjs/ghcjs-build.git
cd ghcjs-build
git checkout prebuilt
vagrant up

Using this configuration the following procedures are performed:

1. Vagrant sets up a 32-bit Ubuntu Precise system (/Note: if this is
   your first time running Vagrant it downloads the 280Mb
   precise32.box file from the Vagrant site/)
2. Vagrants does some provisioning using Puppet (downloads and
   installs necessary packages)
3. A 1.4GB archive with ghcjs and other prebuilt tools are downloaded
   and extracted.

2.2 Compiling from source

git clone https://github.com/ghcjs/ghcjs-build.git
cd ghcjs-build
vagrant up

Apart from setting up the box this will

1. Get the GHC sources from Git HEAD and applies the GHCJS patch.
2. Get all the necessary packages for ghcjs
3. Get the latest Cabal from Git HEAD, applies the GHCJS patch and
   build it.
4. Compile the necessary libraries using ghcjs
5. Compile ghcjs-examples and its dependencies (it appears that it
   can take a lot of time to compile gtk2hs and gtk2hs’s tools)

Please note, that depending on your computer, you might want to go for
a long walk, enjoy a small book or get a night sleep (assuming you are
not scared by the sound of computer fans).

Apart from being slow, the process of compiling everything from
source is error prone. To give you a taste, last night I was not able
to reproduce a working environment myself, because of some recent
changes in GHC HEAD. The prebuilt version on the other hand is
guaranteed to install correctly.

Hopefully, the GHCJS patches will be merged upstream before the GHC
7.8 is out. That way you won’t need to partake in building GHC from
the source in order to use GHCJS.

2.3 Communicating with the VM

After you’ve finished with the initial setup you should be able just
to

vagrant ssh

in your new vm and start messing around.

ghcjs command is available to you and Vagrant kindly forwards the
3000 port on the VM to the local 3030 port, allowing you to run web
servers like warp on the VM and accessing them locally.

You can access your local project directory under /vagrant in VM:

$ ls /vagrant
keys  manifests  modules  outputs  README.rst  Vagrantfile

However, copying file back-and-forth is not a perfect solution. I
recommend setting up a sshfs filesystem (Note: if you are on OSX,
don’t forget to install fuse4x kernel extension):

$ vagrant ssh-config
  Host default
    HostName 127.0.0.1
    User vagrant
    Port 2222
    UserKnownHostsFile /dev/null
    StrictHostKeyChecking no
    PasswordAuthentication no
    IdentityFile "/Users/dan/.vagrant.d/insecure_private_key"
    IdentitiesOnly yes
    LogLevel FATAL
$ sshfs vagrant@localhost:/home/vagrant ../vm -p2222 -oreconnect,defer_permissions,negative_vncache,volname=ghcjs,IdentityFile=~/.vagrant.d/insecure_private_key 
$ ls ../vm

When you are done you can just umount ../vm

3 Compiling other packages

Since the diagrams package on Hackage depends on the older version
of base we are going to use the latest version from Git:

mkdir dia; cd dia
git clone git://github.com/diagrams/diagrams-core.git
cd diagrams-core && cabal install && cd ..

cabal unpack active
cd active-0.1*
cat >version.patch <<EOF
--- active.cabal        2013-06-12 12:58:40.082914214 +0000
+++ active.cabal.new    2013-06-12 12:58:31.029465815 +0000
@@ -19,7 +19,7 @@

 library
   exposed-modules:     Data.Active
-  build-depends:       base >= 4.0 && < 4.7,
+  build-depends:       base >= 4.0 && < 4.8,
                        array >= 0.3 && < 0.5,
                        semigroups >= 0.1 && < 0.10,
                        semigroupoids >= 1.2 && < 3.1,
@@ -31,7 +31,7 @@
 test-suite active-tests
     type:              exitcode-stdio-1.0
     main-is:           active-tests.hs
-    build-depends:     base >= 4.0 && < 4.7,
+    build-depends:     base >= 4.0 && < 4.8,
                        array >= 0.3 && < 0.5,
                        semigroups >= 0.1 && < 0.10,
                        semigroupoids >= 1.2 && < 3.1,
EOF
patch active.cabal < version.patch
cabal install
cd ..

git clone git://github.com/diagrams/diagrams-lib.git
cd diagrams-lib && cabal install && cd ..

git clone git://github.com/diagrams/diagrams-svg.git
cd diagrams-svg && cabal install && cd ..

Other packages I had to install already had their Hackage versions
updated.

Now you can try to build a test diagram to see that everything works

module Main where

import Diagrams.Prelude
import Diagrams.Backend.SVG.CmdLine

d :: Diagram SVG R2
d = square 20 # lw 0.5
              # fc black
              # lc green
              # dashing [0.2,0.2] 0

main = defaultMain (pad 1.1 d)

then you can compile and run it

ghc --make Test.hs 
./Test -w 400 -o /vagrant/test.svg

And that’s it!

4 Outro

I would also like to note that we are currently polishing the GHCJS
build process. Luite, especially is working on making ghcjs work (and
run tests) with Travis CI (it take quite a bit of time to build ghcjs
and sometimes travis is timeouting) and I am working on tidying up
the build config.

Stay tuned for more updates.

Summer of Code

Hello, everyone!

I’ve decided to reinstate this blog since I’ve got accepted to this year’s Google Summer of Code program. I’ll blog about my updates, stuff that I’ve been working on and bottlenecks and problems I’ve encountered.

My project is a pastebin site using diagrams and GHCJS to generate embeddable interactive widgets and static images/text in case when the pasted code does not require additional interaction. My mentor is Luite Stegeman, and Brent Yorgey and other nice people from the diagrams community has agreed to help.

I am very excited about this and happy that I’ve got a whole bunch of smart people to help me with this.

Unfortunately, as we haven’t sorted out a completely safe way to evaluate code coming from 3rd parties, there is no public version hosted anywhere yet. Meanwhile, there is a project on GitHub.

Hopefully, soon I’ll be able to publish a post about my experience with bootstrapping GHCJS.
Until then, stay tuned!