Scala.js and Electron. The native parts.

20 Jul 2015

When we last left each other, we ended up with an Electron application, embedding a tiny bit of our own javascript, compiled from Scala.js. Any way you look at it, that was pretty useless. I mean, at best we were able to re-use some code and assets that we had already available from a website and distribute it to our users as a desktop app.

Stepping up

So, if you feel the need to write a desktop application, it is surely because you need to integrate with some native features such as accessing the filesystem or messing about with processes.

You may have read that java.io.File is not available from Scala.js. That makes sense, the java API around File will at some point interface with the OS and access the underlying filesystem, which in the context of a browser is not something you can do.

However in the context of Electron, access to native functionalities is provided through io.js. A quick look at its API documentation shows that there is a lot we can play with.

Dynamic all the things

Scala.js really shines when it comes to interoperability with javascript code (for more details, that talk is worth watching), and it’s no different for talking to io.js within Electron. All it takes is importing js.Dynamic and learning how to use it.

Node.js (which io.js inherits from) allows to load a module via its require function. So whenever we’ll need a module, we’ll have to call require with the module identifier as an argument (e.g. require("fs")). The io.js API is very helpful too, and reminds us of that.

Using the fs module

So, here is an example of how to list files from a directory (alternatively have a look at the skeleton’s code):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import scala.scalajs.js
import js.Dynamic.{global => g}
import org.scalajs.jquery.jQuery

object MyApp extends js.JSApp {

  val fs = g.require("fs")

  def main(): Unit = {
    display(listFiles("."))
  }

  def listFiles(path: String): Seq[String] = {
    fs.readdirSync(path).asInstanceOf[js.Array[String]]
  }

  def display(filenames: Seq[String]) = {
    jQuery("body").append("<ul>")
    filenames.foreach { filename =>
      jQuery("body").append(s"<li>$filename</li>")
    }
    jQuery("body").append("</ul></p>")
  }

}

Here we use the readdirSync() function from the filesystem module previously loaded via require. Because this is coming directly from javascript we get a native ‘javascript type’, which at compile time is no type at all. So we manually cast it to what we know it should be, i.e. a javascript array of String (and Scala.js provides the helpful conversion from js.Array to Seq). When we interoperate with javascript without using any kind of facetted library, we have to manually cast in a good number of places.

io.js often provides a synchronous and an asynchronous version of the same function, and following the node.js philosophy, we should really be using the asynchronous (read non-blocking) one here.

Callback -not quite yet- hell

Let’s try to rewrite our code in order to use the asynchronous readdir. Asynchronous in javascript means callback, so we will have to provide a function to handle the list of filenames once it is retrieved from the filesystem. In our case, we already have such function: display.

Here is what a first use of the asynchronous readdir could look like (don’t use this at home, we can do better -see below):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import scala.scalajs.js
import js.Dynamic.{global => g}
import org.scalajs.jquery.jQuery

object MyApp extends js.JSApp {

  val fs = g.require("fs")

  def main(): Unit = {
    listFiles(".", display)
  }

  def display(filenames: Seq[String]) = {
    jQuery("body").append("<p>Listing the files in the '.' using io.js API:")
    jQuery("body").append("<ul>")
    filenames.foreach { filename =>
      jQuery("body").append(s"<li>$filename</li>")
    }
    jQuery("body").append("</ul></p>")
  }

  def listFiles(path: String, callback: Seq[String] => Unit): Unit = {
    fs.readdir(path, { (err: js.Dynamic, files: js.Array[String]) =>
      if(err != null)
        println(err) // prints to console
      else
        callback(files)
    })
  }
}

What is nice about this, is that:

we are not blocking while waiting for the filesystem
the code is very close from the actual javascript that you would have written: the callback function that is passed to readdir takes two arguments (err, files) exactly as the io.js doc says.

What is not so nice, is that:

our code looks too much like javascript (wasn’t the whole point to write Scala instead?)
soon we’ll be trapped into multiple layers of callbacks in order to be non-blocking all the way. I know it may not look too bad for now, but if you’ve done a bit of javascript, you know callbacks can get messy very quickly.

Fortunately, there is a better way to do this in Scala.js. Believe or not, you can use scala.concurrent.Future in Scala.js the exact same way you do in Scala.

I’ll call you back

Let’s re-write our code using Future and see if things improve.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import scala.scalajs.js
import org.scalajs.jquery.jQuery
import js.Dynamic.{global => g}
import scala.concurrent.{Future, Promise}
// one of Scala.js ExecutionContext
import scala.scalajs.concurrent.JSExecutionContext.Implicits.runNow

object ElectronApp extends js.JSApp {

  val fs = g.require("fs")

  def main(): Unit = {
    listFiles(".").map(display)
  }

  def display(filenames: Seq[String]) = {
    jQuery("body").append("<p>Listing the files in the '.' using io.js API:")
    jQuery("body").append("<ul>")
    filenames.foreach { filename =>
      jQuery("body").append(s"<li>$filename</li>")
    }
    jQuery("body").append("</ul></p>")
  }

  def listFiles(path: String): Future[Seq[String]] = {
    val promise: Promise[Seq[String]] = Promise()
    fs.readdir(path, { (err: js.Dynamic, files: js.Array[String]) =>
      if(err != null)
        promise.failure(new RuntimeException("could not list files from fs"))
      else
        promise.success(files)
    })
    promise.future
  }
}

The implementation of listFiles still uses a callback, but outside of that function, we are using Future as we are used to. You just have to provide a Scala.js specific ExecutionContext at call site: here we do it by importing the implicit runNow but there are two you can choose from. To me that is a little miracle, but it turns out that Future maps perfectly the javascript asynchronous model.

Conclusion

There is more to learn about Electron and definitely a lot more to Scala.js but I hope that this gives you an idea of how to use some of Electron’s native functionalities from Scala.js.

Writing a native application with scala.js and Electron

19 Jul 2015

I have been looking at Scala.js lately, and like many other I am very impressed by it.

Scala.js

Scala.js lets you write Scala code that compiles to javascript, which you then use in your web application like any other .js file. That alone is a great achievement, but it goes far beyond thanks to all the work that has been put into providing interoperability with Javascript code. For example, manipulating the dom via jQuery in Scala is something I could not have even dreamt of.

The graal

To me, the killer feature of Scala.js is how cross compiling lets you share code, functions, algorithms and models between server and client logic. The uPickle library for example, makes it almost transparent to send and receive data between client and server.

But I don’t want to spend too much time on Scala.js, its features and its greatness (maybe in another post), and instead I want to focus on another neat use case.

Electron

From github with love

Electron was born in the process of building the Atom editor.

From Electron’s readme file:

The Electron framework lets you write cross-platform desktop applications using JavaScript, HTML and CSS. It is based on io.js and Chromium and is used in the Atom editor.

Now, with my own words: Electron enables you to build native applications using web technologies. If you use Slack, that’s how their native application is built. And that’s also why their web application looks so much like their native one: it’s made of the very same stuff.

At the heart of Electron is an instance of Chromium which is used to render the UI and whose regular HTML5 API is extended with node.js API to give access to native features.

Scala on the desktop

Another graal

Now, I am a Scala developer. I don’t do much front-end development, but I know more or less my way around html, css and ~~javascript~~ Scala.js. And many times I’ve felt the need to take a project beyond the command line interface, with the ambition to reach a different audience. What I’ve always wanted was to be able to quickly build native applications using web technologies but, at the same time, I never really had the motivation to dive deep into javascript. I just was too lazy.

The combination of Scala.js and Electron lets me write my code in Scala, compile it to javascript, bundle it with assets and obtain a native application.

I have put together a very basic skeleton, available on github, to help anyone get started. The result is a very basic hello world, electron-based application:

screenshot

From there onwards, the world is yours.

Bonus

One thing I have already mentioned about Scala.js is how it allows for code sharing between server and client, when used in the normal web application scenario. In fact, it is no different here: I can still write both my command-line application and my electron-based application using pretty much the same code (if that’s what you want, I recommend following the Scala.js cross-builing doc).

Conclusion

Scala.js is obviously the result of some hard work. And as often happens with such great projects, it enables use cases that were never anticipated by their creators. Building native applications might be one of those.

Next, we’ll see how to access the native parts (filesystem, process, etc) by using the node.js API from Scala.js, all within Electron of course.

hacking a sbt plugin

12 Aug 2014

A sbt is a powerful tool. And it turns out to be very easy to start hacking an existing sbt plugin. Let’s see how.

In the following case, we will be cloning the sbt-git plugin on our machine. It’s a small plugin (small enough to be able to look at the code and understand what is going on) that adds some git functionnality to your build. For example, you can set up your project version to include the sha of the HEAD commit.

Step 1: using the plugin

Let’s start with a project that uses the sbt-git plugin.

~ mkdir ~/simplistic-project
~ cd ~/simplistic-project

Our project/plugins.sbt should look like this:

1
2
3
resolvers += "jgit-repo" at "http://download.eclipse.org/jgit/maven"

addSbtPlugin("com.typesafe.sbt" % "sbt-git" % "0.6.4")

The first line is necessary because the plugin uses JGit internally so we need to add a resolver for sbt to find it. And the second line adds the sbt-git plugin to our project.

Our build.sbt should look like this:

1
2
3
4
5
6
import com.typesafe.sbt.SbtGit._

versionWithGit

// Optionally:
git.baseVersion := "0.1"

To demonstrate that the plugin indeed adds the sha of the HEAD commit to the baseVersion, we will initialize a git repository in that same folder and make a first commit:

~ git init
~ git add build.sbt project/plugins.sbt
~ git commit -m 'first commit'

And, in sbt, we should get a version that includes the HEAD commit’s sha:

~ sbt
> show version
[info] 0.1-b3942776b084fffd4b5291732f57d86171c6d9f5

Step 2: use our cloned plugin version

So far, we have just set up our project to use the sbt-git plugin. But we would like to hack locally and be able to link our project to our local version of the plugin.

Let’s clone the sbt-git repo (in our home folder):

~ git clone git@github.com:sbt/sbt-git.git ~/sbt-git

All we need to do to tell our current project to use our local version of the plugin is to modified our project/plugins.sbt to:

1
2
3
4
5
6
resolvers += "jgit-repo" at "http://download.eclipse.org/jgit/maven"

lazy val root = (project in file(".")).dependsOn(customSbtGit)

// replace the path with yours
lazy val customSbtGit = uri("file:///home/bchazalet/sbt-git")

That’s it. Sbt will build the plugin next time you build the project (our project now depends explicitly on the local sbt-git plugin).

Step 3: build the plugin from github

The problem with the local setup is that our build is not very portable: it will not work on another machine that does not have the version of the plugin locally at the exact same location. This is not good at all if we use tools like Jenkins, since they won’t be able to find the plugin and the build will fail.

Instead, we can push our version of the plugin to a new repository on github (or directly fork from sbt-git). If you fork the repository from github, your repository will be available at a new URL that will look similar to this https://github.com/bchazalet/sbt-git.git (I am using the https URI here, because the git one might fail depending on your firewall configuration; it does at my company).

So, let’s use that in our build instead of our local clone. We can modify our project/plugins.sbt to look like this:

1
2
3
4
5
resolvers += "jgit-repo" at "http://download.eclipse.org/jgit/maven"

lazy val root = (project in file(".")).dependsOn(customSbtGit)

lazy val customSbtGit = uri("https://github.com/bchazalet/sbt-git.git")

Note that you can even specify the branch you will like to build the plugin from by using the fragment part of the url. For instance, I have a branch called find-git-dir and I can use it directly in my build:

1
lazy val customSbtGit = uri("https://github.com/bchazalet/sbt-git.git#find-git-dir")

More information on sbt plugins, here.

basic init script for a play application running on centos

07 Aug 2014

When deploying a play app, you might need to make it a service so that it automatically starts after a reboot for example.

On a CentOS server, you can write a bash script in /etc/init.d and enable it with chkconfig.

This is my take on writing a -hopefully- generic script that should work for most play applications (tested with 2.3.2). It relies on the fact that plays automatically creates a file containing its pid within its own folder (you can always change the location of that pid file if you need to).

I am also assuming that the application has been package with the dist command. In my case, jenkins build it with clean universal:package-zip-tarball and copies over the tgz file to the server. The resulting folder (after extracting the files) is where the init script will point at. It looks like this:

To make it work for another play app, it should only be a matter of updating the parameters in the first 4 lines.

Note that I use DAEMON_OPTS to pass it production server specific options such as configuration files, e.g.:

DAEMON_OPTS="-Dconfig.resource=prod.conf -Dlogger.resource=prod-logger.xml"

running sbt in interactive mode with -feature

05 Aug 2014

This has annoyed me for a while, but it’s actually simple.

While running sbt in interactive mode, sometimes you might want to compile with the ‘-feature’ options, to see MatchError warnings for example.

Typing ‘compile -feature’ won’t work.

~ sbt
> compile -feature
[error] Expected ID character
[error] Not a valid command: compile (similar: completions)
[error] Expected project ID
[error] Expected configuration
[error] Expected ':' (if selecting a configuration)
[error] Expected key
[error] Expected '::'
[error] Expected end of input.
[error] compile -feature
[error]

All you need to do is to add your options to scalacOptions, exactly as you would in your build.sbt.

~ sbt
> set scalacOptions += "-feature"

You can also check or confirm its scalacOptions value.

~ sbt
> show scalacOptions

The option you just set is temporary (for the current session) and won’t affect your project build file. You can check your current session temporary settings.

~ sbt
> session list
  1. scalacOptions += "-feature"

Finally, let’s re-compile everything with the warnings showing.

> sbt
> set scalacOptions += "-feature"
> clean compile

More info on inspecting the build, from the sbt doc.

Edit: you can also save back your session’s options to your project build.sbt file:

> session save

Check this talk from Scala Days 2014 to learn about sbt and how to write an sbt plugin.

Well, that wasn’t so hard.

Older Newer

lazy {hacking}

Scala.js and Electron. The native parts.

Stepping up

Dynamic all the things

Using the fs module

Callback -not quite yet- hell

I’ll call you back

Conclusion

Writing a native application with scala.js and Electron

Scala.js

The graal

Electron

From github with love

Scala on the desktop

Another graal

Bonus

Conclusion

hacking a sbt plugin

Step 1: using the plugin

Step 2: use our cloned plugin version

Step 3: build the plugin from github

basic init script for a play application running on centos

running sbt in interactive mode with -feature