Home

Scala tooling in 2019

A lot of Scala tooling improvements got released in 2019 making it easier than ever before to build, document, deploy and maintain Scala applications. In this post, I want to highlight a few of the developments that I'm most excited about.

Scala Steward

Scala Steward is in a nutshell "dependabot for Scala". It's a bot that creates pull requests to keep your dependencies up-to-date.

Scalafix migrations: One cool feature of Scala Steward is that it can apply Scalafix migration rewrites on the codebase when upgrading dependencies. See avast/scala-server-toolkit#91 for an example pull request where Scala Steward automatically upgrades the build dependency on ScalaTest and renames usages of deprecated APIs.

--- a/http4s-server-micrometer/src/test/scala/com/avast/sst/http4s/server/micrometer/HttpStatusMetricsTest.scala
+++ b/http4s-server-micrometer/src/test/scala/com/avast/sst/http4s/server/micrometer/HttpStatusMetricsTest.scala
@@ -2,9 +2,9 @@ package com.avast.sst.http4s.server.micrometer

 import io.micrometer.core.instrument.simple.SimpleMeterRegistry
 import org.http4s.Status
-import org.scalatest.FunSuite
+import org.scalatest.funsuite.AnyFunSuite

-class HttpStatusMetricsTest extends FunSuite {
+class HttpStatusMetricsTest extends AnyFunSuite {

Self-host: It's possible to self-host Scala Steward to run it on proprietary codebases with internal Maven repositories, which is helpful if your company has multiple repositories with shared internal libraries.

To learn more about Scala Steward, check out the blog post on the Scala website.

Coursier

The Coursier dependency resolver got a lot of improvements in 2019 that may have gone by unnoticed.

Install command: there's a new experimental coursier install $APP command to install command-line applications on any operating system (macOS, Linux and Windows). To try it out, run:

export COURSIER_EXPERIMENTAL=1
curl -Lo coursier https://git.io/coursier-cli
chmod +x coursier
./coursier install coursier
export PATH="$PATH:$(./coursier install-path)"

coursier install scalafmt ammonite mdoc

The install command works for JVM applications, GraalVM native-image applications as well as Scala Native applications. Native binaries are linked on-demand during the install command.

# Requires a local GraalVM installation
coursier install --graalvm-home=$JAVA_HOME echo-graalvm
time echo-graalvm foobar
> echo-graalvm foobar  0.00s user 0.00s system 80% cpu 0.005 total

The cool thing about coursier install is that it allows tooling authors to release applications the same way you release libraries, avoiding the need to configure custom build pipelines to distribute platform-specific binaries.

Stable library API: the new io.get-coursier:coursier_2.12 module has a public API with high-level methods to programmatically resolve, download and cache library dependencies with only a few lines of code.

New Java-only API: the new io.get-coursier:interface module exposes the same high-level methods as the coursier_2.12 module except in a Java-only API without transitive dependencies (it doesn't even include scala-library.jar).

Mirrors: users behind corporate proxies can now globally configure that artifacts should be downloaded from an internal artifactory instead of Maven Central using mirrors.

Native command-line interface: recent releases of Coursier include GraalVM native-image binaries named cs for the Coursier command-line interface providing instant startup for resolving, fetching, launching and installing JVM applications.

❯ time cs fetch --intransitive org.scalatest:scalatest_2.12:3.0.8 | xargs du -h
8.1M	/Users/lgeirsson/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scalatest/scalatest_2.12/3.0.8/scalatest_2.12-3.0.8.jar
cs fetch --intransitive org.scalatest:scalatest_2.12:3.0.8  0.01s user 0.01s system 93% cpu 0.026 total
xargs du -h  0.00s user 0.00s system 19% cpu 0.029 total

Check out the talk from ScalaDays to learn more about recent improvements in Coursier.

All the fancy things flexible dependency management can do - Alexandre Archambault

All the fancy things flexible dependency management can do - Alexandre Archambault

Polynote

In October, Netflix open sourced a Scala notebook environment called Polynote. Polynote has several distinguishing features that makes it an appealing alternative to Jupyter.

Reproducible evaluation: Polynote enforces top-to-bottom evaluation of code cells avoiding the situation when a notebook works on your computer but not elsewhere.

Self-contained dependendency/configuration management: The environment where the code cells get evaluated is described and managed in the notebook itself, for example pip install requirements or Scala library dependencies.

Rich code editing: Polynote builds on top of the VS Code editor providing IDE-quality code completions, type-at-point, parameter hints and other desirable code editing features.

Scala/Python interop: it's possible to load up some data in Scala in one code cell and then plot the same in-memory data using Python in the next code cell.

To learn more about Polynote, check out the original announcement or watch the talk from the authors at Scale by the Bay (skip to 19min10sec to see demo of Python/Scala interop)

Solving the Scala Notebook Experience

Solving the Scala Notebook Experience

Almond

Almond is a Scala kernel for Jupyter with a built-in Spark integration, code completions, goto definition (including library definitions!), plotting integrations and more.

This year, Almond went from version v0.3 to v0.9 releasing a ton of improvements that I won't summarize here but if you use Jupyter notebooks you should check it out!

To try out Almond online, check out this post by Shadaj Laddad on doing machine learning with Scala in Google Colaboratory.

sbt

The sbt v1.3 release was a major upgrade. If you had a bad experience with sbt in the past, I encourage you to give sbt another try.

Faster edit/compile/test workflows: by using native file I/O APIs in combination with "turbo mode" (more aggressive caching of classloaders), the compile+test loop for incremental changes feels a lot faster compared to older sbt versions.

Super shell: the new "super shell" gives you information about what tasks are running in the background.

Faster dependency resolution: sbt now uses Coursier by default to resolve dependencies resulting significantly faster builds out of the box. I did an experiment to measure how long it takes to bootstrap sbt and download a sizeable number of library dependencies on a clean CI machine

sbt v1.3.5      54s  1x
sbt v1.0.0    1m26s  1.6x
sbt v0.13.17  1m54s  2.1x

sbt-sonatype

Releasing open-source libraries with sbt became faster and more reliable thanks to the v3.6 release of sbt-sonatype.

Faster releases: One open-source project reported that releasing a large build matrix takes 34 minutes with the latest sbt-sonatype compared to 3 hours with the previous version.

Reliable releases: sbt-sonatype now automatically opens and closes staging repositories for each individual release. Previously, a failed release would cause subsequent releases to fail unless you manually dropped the failed release.

I'm excited about the improvements to sbt-sonatype because they have dramatically improved the experience of publishing open-source libraries with sbt and everybody in the community benefit from a richer library ecosystem.

Use the new sbt-missinglink plugin to check if your project can crash at runtime with ClassNotFoundException or NoSuchMethodException. I haven't tried this plugin myself but it looks useful if you're building an application with transitive library dependencies that may be incompatible with each other. Read this blog post on "better management of transitive dependencies and conflicts" to learn more.

IntelliJ

IntelliJ is the most widely used IDE for Scala. This year the JetBrains Scala plugin team made a lot of nice improvements to the Scala code editing experience.

Error highlighting: diagnostics for type mismatches are now displayed with hints highlighting the difference between the expected type and the obtained type.

Compare that to how diagnostics are displayed in VS Code.

The blog post "functional highlighting for functional programmers" from the IntelliJ Scala plugin team goes into more depth about this new feature. I would love to see similar formatting of diagnostics to available in the Language Server Protocol.

Method chain hints Another new feature that looks useful is method chain hints.

I personally no longer use IntelliJ for day-to-day development so I have probably missed some other important changes.

Metals

Metals is a Scala language server that I personally use via VS Code. I worked on Metals so I'm biased here but I'm really proud of the progress that's been made.

In 2019, Metals went from being a bare-bones language server that only supported the most basic LSP features to supporting a ton of functionality including:

Some of those features are not yet available in a stable release so I won't go into depth about it here.

To learn more about Metals, check out the website or watch any of the talk below.

Rich code editing for Scala in VS Code, Vim, Emacs and beyond

Rich code editing for Scala in VS Code, Vim, Emacs and beyond

Casting Metals, a brief history of Metals - Gabriele Petronella

Casting Metals, a brief history of Metals - Gabriele Petronella

Building rich IDE features beyond the Language Server Protocol.

Building rich IDE features beyond the Language Server Protocol.

You can try out Metals online with Gitpod.

Bloop

Bloop is a Scala build server that has integrations with IDEs like IntelliJ and Metals as well as build tools like sbt.

New build integrations: you can now use Bloop to build Maven, Gradle and Mill projects. Below is a comment from a Java developer on Hacker News:

The last year I develop purely in Java and I use Scala bloop for day-to-day development, because I couldn't stand multi-minute Gradle compile times. Getting incremental compile times counted in milliseconds (!) or single seconds at worst is something very hard to give up on

— pkolaczk, Dec 23rd 2019

Compile de-duplication: Bloop avoids redundant compilations when handling concurrent compile/test requests from multiple clients. This means that when a user runs bloop test $PROJECT in the terminal it will re-use if possible any ongoing compilation that's already been triggered by another client like Metals or IntelliJ.

For more details check out the v1.3.0 release notes. I don't want to spill the beans but the upcoming v1.4.0 release will contain a lot of other exciting features.

Design Challenges of Bloop - Jorge Vicente Cantero

Design Challenges of Bloop - Jorge Vicente Cantero

Ammonite

The Ammonite REPL is a Scala REPL with tons of quality-of-life features such as built-in dependency resolution, Ctrl-c doesn't leave the terminal, multi-line editing, syntax highlighting and gorgeous pretty-printing (even for massive data structures!).

Tab completions when adding dependencies: one cool feature that got added to Ammonite in 2019 is tab completions when writing the Maven co-ordinates for library dependencies. I would love to have the same features in IntelliJ and Metals.

mdoc

mdoc is a tool that typechecks Scala code examples in markdown files. Quite a few Scala libraries started using mdoc in 2019 for writing typechecked documentation and the project released a few new features.

Interactive Scala.js documentation: the mdoc:js modifier makes it easy to write interactive documentation with Scala code examples that evaluate in the browser. For example, the markdown below renders into an interactive "Click me" button

```scala mdoc:js
var i = -1
val colors = Array("coral", "violet", "seagreen")
node.innerHTML = "<button>Click me</button>"
node.onclick = { _ =>
  i = (i + 1) % colors.length
  node.style.background = colors(i)
}
```
Loading mdoc:js demo...

HTML file support: the v2.0 release made it possible to process *.html files with mdoc the same way markdown files are processed. This makes it possible to for example write Reveal.js slides with typechecked Scala code examples.

Conclusion

In this post, we covered improvements that happened in 2019 to a range of Scala tooling projects including build tools, dependency resolvers, application installers, library releasers, dependency updater bots, IDEs, notebook environments, documentation tools and more. These tooling improvements are making easier than ever before to build, document, deploy and maintain Scala applications.

Discuss on Twitter

Home