Today, we're pleased to announce the open source availability of dbuild, a tool we use within Typesafe to test Scala's ecosystem. dbuild is a tool which can take a disparate set of projects and compile them against each other, ensuring that modifications to code in one project does not break code in another project. dbuild was instrumental in the Scala 2.11 release, where it was used to compile and validate more than 70 open source projects, 1.2 million lines of Scala code and 120 thousand lines of Java code. This helped ensure that our open source ecosystem would have a smooth transition from Scala 2.10 into the 2.11 series.
dbuild is not a build tool, it is a build aggregator. dbuild takes a set of distinct projects, like Scala, scalaz, Akka, spark, and attempts to rewire their builds such that all code is built together, as a cohesive unit. For example, we can build the latest in-development Scala, then use this recently built Scala to build and test a version of Akka. Finally, this newly built version of Akka is used to build and test the Play project.
Here's an example dbuild configuration which will build a few of the javascript plugins being actively developed for Play 2.3 and ensure they are compatible
build: {
projects: [{
name : sbt-web
uri : "https://github.com/sbt/sbt-web.git"
}, {
name : js-engine
uri : "https://github.com/typesafehub/js-engine.git"
}, {
name: sbt-less
uri: "https://github.com/sbt/sbt-less.git"
}]
}
Here, three projects: sbt-web, js-engine + sbt-less are compiled together to ensure their code and APIs are compatible.
The Scala ecosystem is composed of a rich and diverse set of projects. Each project has its own community and policies governing its development. It would be impossible to try to lock a community down into one build tool, and one standard of how to publish/consume artifacts.
At the same time, every Scala major release generally involved a bit of ceremony around ensuring the ecosystem projects were compatible with the latest version of Scala, and any changes/deprecations that may have occurred. It became apparent that we needed better tooling to simplify/ease the transition of the Scala ecosystem from one major release to the next.
dbuild does many things for the Scala ecosystem:
While, at Typesafe, we use dbuild to verify Scala’s standard library and compiler, it is applicable to any project which has an ecosystem of libraries/plugins which depend on it.
For example:
The Scala 2.12 roadmap includes switching over to a new compiler backend & optimizer. The new backend has been developed in a fork of Scala compiler created by Miguel Garcia from EPFL. Miguel’s fork replaces about 20% of the entire Scala compiler code base. Merging back such a large changeset and avoiding the risk of introducing regressions poses a significant challenge. We are using dbuild and community builds configuration to build 1.2 million lines of Scala code and verify that changes to the backend are not breaking existing Scala projects. This method of massive building and testing allowed us to uncover tricky bugs that were not caught by Scala’s standard test suite.
dbuild is offered for end users, with install instructions in the user manual. Additionally, OSS contributors interested in hacking on the core should check out the github repository: https://github.com/typesafehub/dbuild
Enjoy!
