Typesafe Activator

Play Reactive Mongo application with KnockoutJS frontend

Play Reactive Mongo application with KnockoutJS frontend

typesafehub
Source
January 31, 2014
playframework scala mongodb knockout bootstrap

Play Framework is the High Velocity Web Framework for Java and Scala. Play is based on a lightweight, stateless, web-friendly architecture. Built on Akka, Play provides predictable and minimal resource comsumption (CPU, memory, threads) for highly-scalable applications. This app will teach you how to start building Play 2.1 apps with Java and Scala.

How to get "Play Reactive Mongo application with KnockoutJS frontend" on your computer

There are several ways to get this template.

Option 1: Choose play-mongo-knockout in the Typesafe Activator UI.

Already have Typesafe Activator (get it here)? Launch the UI then search for play-mongo-knockout in the list of templates.

Option 2: Download the play-mongo-knockout project as a zip archive

If you haven't installed Activator, you can get the code by downloading the template bundle for play-mongo-knockout.

  1. Download the Template Bundle for "Play Reactive Mongo application with KnockoutJS frontend"
  2. Extract the downloaded zip file to your system
  3. The bundle includes a small bootstrap script that can start Activator. To start Typesafe Activator's UI:

    In your File Explorer, navigate into the directory that the template was extracted to, right-click on the file named "activator.bat", then select "Open", and if prompted with a warning, click to continue:

    Or from a command line:

     C:\Users\typesafe\play-mongo-knockout> activator ui 
    This will start Typesafe Activator and open this template in your browser.

Option 3: Create a play-mongo-knockout project from the command line

If you have Typesafe Activator, use its command line mode to create a new project from this template. Type activator new PROJECTNAME play-mongo-knockout on the command line.

Option 4: View the template source

The creator of this template maintains it at https://github.com/typesafehub/play-mongo-knockout#master.

Option 5: Preview the tutorial below

We've included the text of this template's tutorial below, but it may work better if you view it inside Activator on your computer. Activator tutorials are often designed to be interactive.

Preview the tutorial

The world is going reactive

Not long ago, response times in the seconds were considered appropriate. Browser refreshes were the norm in web applications. Systems would go down for hours of maintenance, or even be rebooted nightly, and this was ok because people only expected the systems to be up during business hours. Applications didn't have to scale because they didn't have big user bases. And the complexity requirements put on web applications meant that typical requests could easily be handled by a thread per request model.

Things are changing though. People expect web applications to react instantly. They expect them to be up all the time, while the applications are moving into the cloud, where failures are not exceptional, but rather are the norm, and so applications need to react to failure. Load on a web application can peak unpredictably, to be many orders of magnitude greater than normal, and so applications need to react to load and scale out. The complexity of business requirements means that in order to respond quickly to requests, things must be processed in parallel, reacting to events rather than waiting so as to utilise resources as efficiently as possible.

This application is an example of how to implement the tenants of the Reactive Manifesto.

It uses the latest in client side technologies to implement a reactive user interface. It uses a MongoDB database, using event driven IO via the ReactiveMongo database driver. And it maintains statelessness, ensuring that it can be easily scaled out to many nodes, for both fault tolerance and scalability.

Browse the app

In order to use this app, you'll need to install and run MongoDB on the default port. If you don't already have MongoDB installed, you can get it from here.

Once the application has been compiled and the server started, the application can be accessed at: http://localhost:9000. You can check in Run to see the server status.

Open the application by clicking the above link, and try adding some messages. The application is just a simple message board, you can post messages, the messages are paged.

Try opening the app in multiple browser tabs, or even different browsers. Notice that whenever any browser tab adds a message, the message instantly appears in every browser tab.

Code Overview - Templates

The HTML page you're seeing is implemented using Scala templates, so we'll start there. The main page is index.scala.html. It starts with a call to the main decorator template, main.scala.html. Inside it is first the modal dialog box for adding messages, then the text for the left hand side of the page, then the paging and rendering of messages.

You can see on many HTML elements there is a data-bind attribute. These are knockout.js binding attributes that allow binding data and behaviour into the HTML page. You can also see the use of many Bootstrap styles. Bootstrap allows you to create a very good looking site within minutes.

Moving into main.scala.html, this is the template that wraps all other templates (currently the only other one is the index). It is where the stylesheets and Javascript assets are brought in. Notice the @routes.Assets.at() and @routes.WebJarAssets.at() calls. These are reverse routes, they allow you to keep your routing path configuration separate from the templates that consume the routing information. Also notice that we are using RequireJS to handle JavaScript modularisation.

For more information on Scala templates, see here.

Code Overview - CoffeeScript

The client side code for this application is implemented using CoffeeScript. The main file for this code is app.coffee.

The first line of the file is the RequireJS declaration, if you've used RequireJS before, you may notice something different - some of the modules are prepended with webjars!. This tells RequireJS to use WebJars to locate these modules. Rather than being managed by downloading them onto the filesystem, WebJars modules are managed by the build system, you can see in the applications Build.scala file dependencies declared on various different WebJars. This has the advantage of upgrading being a simple configuration item, and also allows WebJars, in combination with RequireJS, to automatically pull in transitive dependencies.

You'll also notice the routes.js module, this is the Play JavaScript reverse router. Like the reverse router that we saw earlier, the JavaScript reverse router allows you to decouple your routing path configuration from your JavaScript code. The JavaScript reverse router is able to build dynamic URLs based on parameters that you pass in, and also knows the correct HTTP method for a given route.

The rest of the file is a knockout.js model, declaring a number of observable properties that are bound to in the templates we saw earlier, and declaring a number of behavioural methods that update the view and make ajax calls, using the JavaScript reverse router.

Code Overview - LESS

Most of the styles used by this application come from Bootstrap, and as we saw before, it was brought in via WebJars in main.scala.html.

We do need to define a very small amount of our own styling though, to do this we're using LESS, an extension to CSS that grants extra functionality so that you can write very concise, simple and easy to manage stylesheets. Play automatically compiles LESS stylesheets to CSS for you.

Our LESS style sheet can be found in main.less.

Code Overview - Routing and Actions

When Play receives a web request, it needs to know how to route that request to some code to handle the request. Play uses the routes file to declare this configuration, and then it compiles this into a statically typed router, and also compiles it into the reverse router and JavaScript router as we've already seen.

Each route declares a request method, a path, which may have some dynamic parts, and then an action to handle it. The action is a method on a controller, and the parameters to those methods are the dynamic parts of the path, as well as any query string parameters that you wish to pass. Since the routes file is compiled to Scala, an error, including type errors with parameters, will be found at compile time, you don't need to wait till runtime.

The first route in the routes file points to the index method on the MainController. This method simply renders the index.scala.html template that we saw earlier.

In the main controller you can also see the router method, this is what generates the JavaScript router. Each route that we want exposed in the JavaScript router is passed here.

You can read more about HTTP routing and actions in Play here.

Messages

This is an app that saves and retrieves messages, so we need to have some way to model these messages. This is done in the Message class.

You can see that a message as an id, the name given to it is _id because this is what MongoDB requires. The type of the id is a BSONObjectId, again, this is a MongoDB data type that allows generation of unique ids. And a message also has a message string.

In the Message companion object, we have an implicit JSON Format object. This will be used whenever a message is passed to a method that requires a Format type class. ReactiveMongo requires this when saving or loading objects from the database, and the Play JSON library requires this when serialising or deserialising objects to and from JSON.

The format object itself is created using the Json.format macro. This allows 100% type safe at compile time data binding of case classes to and from JSON, if Message was to have a field of a type that Play did not know how to handle, a compilation error would be thrown, you would not have to wait until runtime to find that error.

You can read more information about Play's JSON handling here.

Talking to MongoDB

We are using ReactiveMongo to interface to MongoDB. ReactiveMongo is a fully asynchronous database driver, no calls on it ever block a thread.

The code to save and load messages can be found in MessageDao. The collection method gets a reference to a MongoDB collection, if you haven't used MongoDB before, a collection is roughly equivalent to a table in a relational database.

Notice that every method in the DAO returns Future. A future is a value that may not yet be available, but will be available sometime in the future. You cannot interact with it directly, there is no way to get the value out of it now, in fact doing so would be very bad because it would require blocking if the value wasn't yet available. Rather, you can interact by passing callbacks to methods such as map and flatMap. map allows you to transform the value to another value, possibly of another type, while flatMap allows you to perform another asynchronous operation.

You can see in the save method, after calling collection.save(), we get back a future of the status of the save call. But we want to return a Future[Message], so we map the status future to a message, if the operation completed successfully. Since this code is non blocking, the save call will return almost immediately, before the database action has even started. But the returned Future[Message] allows the caller of the save method to perform its own operations once the future is redeemed.

You can read more about asynchronous programming in Play here.

Asynchronous actions

As we saw before, all actions that we do on MongoDB are asynchronous, they return futures. Navigate to the MessageController to see how we use these asynchronous calls on MessageDao.

As a simple example, take a look at the saveMessage method. You can see that after calling MessageDao.save, we map its result to a 201 Created response. At this point we have a Future[Result], we then wrap that in the Async call to return it to Play, and Play will handle that asynchronously, without blocking.

A more complex example is the getMessages method, which makes multiple calls on the MessageDao, one to get the count of all messages, another to get a page of messages. To make multiple asynchronous calls, we're using for comprehensions:

for {
  count <- MessageDao.count
  messages <- MessageDao.findAll(page, perPage)
} yield { ... }

Each line in the for block is equivalent to a call to flatMap on the result of the line before. This means each line is a callback that gets executed asynchronously - it looks like ordinary sequential code, but actually is non blocking and asynchronous. The yield block is equivalent to a map call on the eventual result.

Events

Up to now we've seen how the app implements the loading and the saving of messages, from the frontend through to the backend. There's one major feature left to show, and that's how the browser reacts to new messages being published.

Notifying browsers about messages requires publishing events. A naive approach to event publishing would be to have an in memory register of event listeners. This works fine if you have one node, but in a scalable, fault tolerant and reactive world, this will not be the case.

There are many technologies that we could use to publish events that all nodes can see, including Akka and RabbitMQ. Since we're already using MongoDB, we're going to take a very simple option, MongoDB tailable cursors on top of capped collections.

A MongoDB capped collection is a collection with a maximum size, when that size is reached, the oldest documents written to the database a overwritten with any new documents that are inserted. Tailable cursors allow you to reactively receive documents from the collection as they are inserted into the collection.

The EventDao

The EventDao allows publishing of events and also produces a stream of events to subscribe to.

We load the collection in a bit of a different way to in the MessageDao, we need to make sure that the collection is configured to be a capped collection, and this requires some asynchronous calls to the database that need to be made before the collection will be useable. For this reason, the collection is a future, and so you can see each time we use it we use a for comprehension to get the actual collection.

Additionally, we don't want to do the check every time we use the collection, only when Play starts up. So we've implemented a Play plugin, this allows us to hook into the Play lifecycle. The plugin is registered in play.plugins.

The stream method creates an enumerator based on a tailable cursor of the collection. An enumerator is an asynchronous stream producer, enumerators get used in combination with iteratees, which are asynchronous stream consumers. This enumerator is also managed by the plugin, we ensure that there is only one tailable cursor opened for the whole app, and then we wrap this enumerator in a broadcast enumerator to allow many clients to consume from it.

You can read more about iteratees and enumerators here.

Server-Sent Events

Our event stream is streamed to the browser using Server-Sent Events. The route for the stream can be found in the MainController. Our stream is fed into the event source enumeratee - an enumeratee is a transformer of asynchronous streams.

If you look in the Event model object, you can see that we've defined a comet message extractor for getting the data for the event, as well as an id extractor and a name extractor. The name extract in particular is important, it allows us to subscribe to events by name on the client side.

On the client side, at the bottom of app.coffee, you can see where we are creating the EventSource stream (using the JavaScript reverse router to get the URL), and then subscribing to message events. For each message event, we add the message (the data of the event) to the messages on the screen, if we are on the first page.

Configuration

Configuration for the application is defined in application.conf. This is for example where the mongodb.url configuration is defined.

Configuration for the build system is defined in Build.scala. This is an sbt build file. Here all the dependencies of the application are declared, along with any build configuration.

Testing

No application is complete without comprehensive tests. We are using specs2 to test the app. You can run the tests by going to the Test tab, and clicking start. By default, activator will run the tests automatically for you each time you change file.

We've started by declaring some helpers in MongoDBTestUtils, the primary thing being the withMongoDB method, which sets up and tears down both Play and MongoDB before and after the code block passed to it. It uses the Play test framework to help it, including the running method, and creates a FakeApplication.

Navigate to MessageControllerSpec to see an example set of specs. These invoke the methods on the message controller to test them. They also use a few methods provided by the Play test framework, such as header and contentAsString, which get headers and the body content out of possibly asynchronously produced results.

You can read more about testing your application here.

Adding an author field

Now that we've got an overview of the whole codebase and how it works, let's start to add new functionality. We want to add an author field to the messages, so we know who wrote them. Let's start with adding the author field to the add message popup. Open index.scala.html and add a new field below the exsting form-group div:

<div class="form-group">
    <label for="messageAuthorField">Author</label>
    <input type="text" class="form-control" id="messageAuthorField" placeholder="Enter author"
        data-bind="value: messageAuthorField"/>
</div>

You can see that we've bound the input element to the messageAuthorField property in the model. Save the changes and now let's add that to app.coffee, just below the declaration for messageField:

@messageAuthorField = ko.observable()

Save the file. Now if you refresh the app and click the Add message, you should see the new field, but it doesn't do anything yet. Let's modify the saveMessage method so that it adds the author field to the submitted JSON and then clear the messageAuthorField:

@ajax(routes.controllers.MessageController.saveMessage(), {
  data: JSON.stringify({
    message: @messageField()
    author: @messageAuthorField()
  })
  contentType: "application/json"
}).done(() ->
  $("#addMessageModal").modal("hide")
  self.messageField(null)
  self.messageAuthorField(null)
)

Make sure the indentation is correct and then save the file. Now to handle this newly submitted field on the server side, open the Message model, and add the authors field to it:

case class Message(_id: BSONObjectID, message: String, author: String)

Save your changes. Now update the MessageForm in MessageController to also accept this field, and pass it to the Message object when created:

case class MessageForm(message: String, author: String) {
  def toMessage: Message = Message(BSONObjectID.generate, message, author)
}

Save your changes. Before testing this, you may want to prep the database, since any existing data in there won't have the new field. Open a MongoDB command line shell and drop the messages collection:

$ mongo messages
MongoDB shell version: 2.4.6
connecting to: messages
> db.messages.drop()
true

Now we should be able to save authors, the only thing left to do is to render them. Back in index.scala.html, update the last <ul> at the bottom of the file to render the author field:

<ul class="list-unstyled messages" data-bind="foreach: messages">
    <li>
        <div class="messageAuthor" data-bind="text: $data.author"></div>
        <div class="message" data-bind="text: $data.message"></div>
    </li>
</ul>

Save the file. Refresh the app and add a new message. Now we can see the authors, but they don't look pretty. Add some styling to main.less:

.messageAuthor {
  font-style: italic;
  color: gray;
  float: right;
}

Save the file and refresh the app to see the styled author. And now you've finished adding your first feature to the app. The tests are broken though, let's fix them, and add a new test. Modify the createMessage method in the MessageControllerSpec to also handle the author:

def createMessage(msg: String, author: String = "None") = {
  Await.result(MessageDao.save(Message(BSONObjectID.generate, msg, author)), Duration.Inf)
}

And now modify the "save a message" spec to test saving authors:

"save a message" in withMongoDb { implicit app =>
  status(MessageController.saveMessage(FakeRequest().withBody(
    Json.obj("message" -> "Foo", "author" -> "Bar")
  ))) must_== CREATED
  val messages = Await.result(MessageDao.findAll(0, 10), Duration.Inf)
  messages must haveSize(1)
  messages.head.message must_== "Foo"
  messages.head.author must_== "Bar"
}

Save the file. Finally fix the EventsSpec to populate the author field:

val message = Message(BSONObjectID.generate, "Foo", "None")

Save the file and now the tests should all pass.

Implementing likes

Now we want to allow people to like messages. We'll start off with implementing the backend. You may want to first drop the messages collection again, since we'll be modifying it. Add a likes integer to the Message model:

case class Message(_id: BSONObjectID, message: String, author: String, likes: Int)

Before going on, fix the code that creates Message objects in MessageControllerSpec and EventsSpec.

Now we want to add a like method to the MessageDao:

def like(id: String): Future[Boolean] = {
  collection.update(
    Json.obj("_id" -> BSONObjectID(id)),
    Json.obj("$inc" -> Json.obj("likes" -> 1))
  ).map {
    case ok if ok.ok =>
      ok.n == 1
    case error => throw new RuntimeException(error.message)
  }
}

You can see here that we're using a MongoDB atomic increment operation to update the likes property. As with save, update returns a future, which we are mapping to be a Boolean, true if the message existed (ok.n returns the number of documents updated, so if it's 1, then the document must have existed), otherwise false.

Now let's go to the MessageController, and the first thing we want to do is pass 0 into the Message object created in MessageForm.toMessage, since a new message will start with 0 likes. Then we want to add a likeMessage action:

def likeMessage(id: String) = Action {
  Async {
    MessageDao.like(id).map {
      case true => NoContent
      case false => NotFound
    }
  }
}

We're using our newly implemented MessageDao.like method, which returns a Future[Boolean], and we're mapping that to 201 No Content if successful, or 404 Not Found if not found.

Now we need to define a route for our new action. In routes, somewhere above the last route (because it's a catch all route):

POST        /message/:id/like      controllers.MessageController.likeMessage(id)

And let's expose this route in our JavaScript router generated in MainController:

Routes.javascriptRouter("routes")(
  routes.javascript.MainController.events,
  routes.javascript.MessageController.likeMessage,
  routes.javascript.MessageController.getMessages,
  routes.javascript.MessageController.saveMessage
)

Finally, before we're done in the backend, we'll add a test in MessageControllerSpec:

"allow liking messages" in withMongoDb { implicit app =>
  val message = createMessage("Foo")
  status(MessageController.likeMessage(message._id.stringify)(FakeRequest())) must_== NO_CONTENT
  Await.result(MessageDao.findAll(0, 10), Duration.Inf).head.likes must_== 1
}

Run the tests to make sure they pass.

Likes on the frontend

Let's now implement support for likes on the front end. Let's start of with rendering them in messages list in index.scala.html:

<ul class="list-unstyled messages" data-bind="foreach: messages">
    <li>
        <div class="likes">
            <span class="glyphicon glyphicon-thumbs-up icon"
                data-bind="click: $root.likeMessage"></span>
            <span data-bind="text: $data.likes"></span>
        </div>
        <div class="messageAuthor" data-bind="text: $data.author"></div>
        <div class="message" data-bind="text: $data.message"></div>
    </li>
</ul>

We're using glyphicons from Bootstrap to render a thumbs up button. A little styling of our own main.less wouldn't go astray:

.likes {
  float: left;
  margin-right: 10px;
  .icon {
    cursor: pointer;
  }
}

Now you can see that we bound clicking on the thumbs up icon to a likeMessage callback, let's implement that in app.coffee:

@likeMessage = (message) ->
  self.ajax(routes.controllers.MessageController.likeMessage(message._id.$oid))

Note once again that we are using the JavaScript reverse router here. Now refresh your browser, and try liking a message. The likes won't appear immediately yet, but after a browser refresh, you should see them. Don't forget to drop your messages collection if you didn't already.

Reactive likes

The final thing we want to implement is reactive likes - when a user likes a message, every browser that is viewing the page should be updated with the new likes total for that message. On the backend, we'll utilise our existing events mechanism to publish a like event when a message is liked. Update the like method's success handler in the MessageDao file:

case ok if ok.ok =>
  EventDao.publish("like", id)
  ok.n == 1

Let's extend the existing spec for events in EventsSpec by adding the following to the "publish a new event when a message is saved" test:

val likeEvent = EventDao.stream |>>> Iteratee.head
MessageDao.like(message._id.stringify)
Await.result(likeEvent, Duration.Inf) must beSome.like {
  case event =>
    event.name must_== "like"
    Json.fromJson[String](event.data).asOpt must beSome(message._id.stringify)
}

In the front end, we want to start by making the likes property on each message observable. We'll write a small utility function to do this, and add it to the MessagesModel class in app.coffee:

bindMessage: (message) ->
  message.likes = ko.observable(message.likes)

Now when we load messages in the loadMessages function, we want to apply this function to each message:

loadMessages: (data, status, xhr) ->
  data.forEach(@bindMessage)
  @messages(data)

And we want to do the same for messages we've received via EventSource:

message = JSON.parse(e.data)
model.bindMessage(message)
model.messages.unshift(message)

Now we can write a handler for like events sent to us by the server. Each time we recieve a like event, we want to iterate through the messages that we have, and if the like event is for that message, we'll increment its likes:

events.addEventListener("like", (e) ->
  id = JSON.parse(e.data)
  ko.utils.arrayForEach(model.messages(), (message) ->
    if (message._id.$oid == id)
      message.likes(message.likes() + 1)
  )
)

Now check in your browser to see if it works. Congratulations, you've created a reactive app using Play, MongoDB and knockout.js!

comments powered by Disqus