Wednesday, April 18, 2012

One Web App = One Language

When I started the Opa project five years ago, my goal was to clean up the mess and build a clean technology for writing web applications.
At the time, Web 2.0 applications based on AJAX were increasingly popular and most technology stacks were growing in complexity, surpassing 5 technologies: A web server, a programming language for the server, another for the client, a database and at least one framework. Not counting an ORM layer, caching mechanism, etc., etc. There was room for improvement!
The design was based on two principles:
  1. There should be a single, unique language for writing web applications.
  2. There should be a single, clean, runtime on the servers.
This month, two new technologies launched that rely on this first principle. And both chose JavaScript as the single language:
  • Yahoo's Mojito framework features dynamic code migration, with the ability to shift computation at runtime from the client to the server for mobile devices, in order to reduce the amount of data transferred to the device.
  • Another framework, Meteor, automates the communication between the server and the client part of the web application.
Both use Node.js, the increasingly popular “application server” for JavaScript.
We are really happy to witness this move that makes web application development easier. Since 2008, we have learned a lot by using a single programming language for writing web applications. Especially when growing from prototypes to real-world applications. It turns out that one of the most needed features was the ability to automatically determine where the code should run, on the client or on the server. We named that feature the Opa “slicer”.

You can classically enforce a given side using “client” and “server” directives:

 // Client-side
client function client_function(value) { ... }
 // Server-side
server function server_function(value) { ... }

Most of the time, you want to omit the hint and let Opa guess. Imagine you have a DOM element of id "counter", and a database named "mydb":

// Will be on server side:
function db_update(value) {
    /mydb/counter <- value
}

// Will be on client-side:
function dom_update(value) {
    #counter = value
}

Even better! Both calls (and the automated communication) can be written with a single function:

function update(value) {
    /mydb/counter <- value;
    #counter = value
}

And even with a single LOC:

// Just write _what_ you want to do, without worrying about _how_ to do that
function init() {
    #counter = /mydb/counter
}
As you may imagine, automatic code repartition is a very complex problem that requires deep code analysis to make it work. In Opa, we tackle this problem by relying on an advanced static typing system. And let's be honest, the type system of Opa is the feature we are most proud of.
Every function in the Opa code gets a type (which is usually inferred). When the developer writes:
function foo(s) {
   String.length(s);
}
function bar(x, y) {
   foo(x) + y;
}
The Opa compiler infers:
int function foo(string s) {
   String.length(s);
}
int function bar(string x, int y) {
   foo(x) + y;
}
Similarly, the slicer infers wether a given function may run on the client, the server, both. Over the years, we developed and refined the approach to make it usable past the prototypes. We introduced hints to quickly precise if the client-server should be asynchronous, or synchronous. Wether, the function may, or may not be, callable. And use the rich typing information to make it work.
The compiler may warn you that:
File "errorslicer.opa", line 6, characters 17-19, (6:17-6:19 | 73-75)
A‚ 'malicious' is tagged as 'client' but it uses 'protected' values:
to prevent you from doing evil things. If you're curious, the sample code is here.
Opa has also built advanced support for MongoDb. And again, we make use of the rich typing information to make things simpler for the developer while automatically ensuring that the code is well written. You can read this post by Adam on this subject.
Opa has become a mature technology and now brings together incredibly clean and concise code, but also security and high-quality to web applications. The feedback we received hinted that Opa's biggest drawback is that it relies on its own application server.
Opa already and naturally uses JavaScript, as the Opa compiler translates the client-side Opa code to JavaScript. The popularity of the Node.js platform and the good set of tools and technologies now available, convinced us that we should also support Node.js as a new server backend. After all, we already know how to translate from Opa to JavaScript!
We have the feeling Opa could become the killer framework for writing Node.js applications. Do you also think that would be great?

Thursday, March 29, 2012

Programming tools UX: How we made MongoDB even easier to use with Opa

Most developers nowadays rave about “User Experience” or UX, i.e. designing software that pleases the users. Most of the time, UX involves GUI concepts and design, but not always. When we added support for the MongoDB database to the Opa programming language, we tried to make things as easy as possible for the developers. But a second look at the actual programming workflow showed room for improvement — and improvement we did.

Little background first. Opa is a web programming platform. It unifies the approach to web development by providing a single, uniform programming language for both front- and back-end coding. Under the hood the program gets automatically divided into the server and the client part, with the former compiled to native code and the latter to JavaScript and all the communication between the two taken care of by the compiler. Opa also nicely integrates database functionality, including extensive support for MongoDB, parts of which I’ll talk more about in this article. If interested in the language Opa's website is a great place to start.

A typical support for a database in any programming language consists of a driver for that particular database that provides connectivity with the DB and an API that allows the program to communicate with it. Opa goes few steps further. From the start the database was very tightly integrated into the language allowing one to very easily persist and query arbitrary data in one’s program. Opa was using its own DBMS to accomplish that.

That worked perfectly but our users wanted to have more freedom and to be able to use existing DBMSs. We quickly provided a driver — in the usual sense of this word — for MongoDB. But then we realized that a traditional driver comes with traditional problems: communication between the program and the database, going mostly via uninterpreted strings, is essentially untyped and unsafe. This flew in the face Opa’s principles, as it aims to provide a single, uniform language that is safe and secure to use and where type-safety is a given.

Our response to that was to provide an intelligent interface layer between Opa and MongoDB that ensures type-safety for all DB operations. On top of that we made the interface consistent with Opa’s internal database so that now one can very easily switch back and forth between those two different backends. The usage scenario for Mongo was to first start its instance (after installing it on the given machine) and then, once it was running, the Opa application using it, so it went something like this (output slightly simplified for brevity):

koper@thistle:~/test$ mkdir -p ~/mongodata/opa
koper@thistle:~/test$ ${INSTALL_DIR}/mongod --noprealloc --dbpath ~/mongodata/opa > ~/mongodata/opa/log.txt 2>&1
koper@thistle:~/test$ ./test.exe --db-local ~/mongodata/opa
Opening database at ~/mongodata/opa
Test (OPA/1488) serving on http://thistle:8080

this first creates directory ~/mongodata/opa for database data, then runs MongoDB server storing logs at \~/mongodata/opa/log.txt and finally launches the Opa application which will use that database server instance.

At the same time we also improved the query language in Opa. Just to give you an idea of how it works think how would you retrieve all American or French movies with rating between 7 and 10, released after year 2000, sorted by the number of ratings they received and limited to the first 50 results?. In Opa the query is actually shorter than this sentence:

movies = /movies/all[rating >= 7 and rating <= 10, country == "France" or country == "USA", release_year >= 2000; order +ratings_no; limit 50]

If you want to learn more about this cool, concise syntax check the relevant chapter in Opa manual.

That was cool. And we were quite pleased. Until one day we realized one fact: Opa’s approach tightly integrating persistence in the language, essentially meant that our users didn’t have to know first thing about MongoDB to be able to take advantage of this great, state-of-the-art database! They could just take a previous Opa program and now switch the database backend to Mongo. Except, for one thing. They of course had to be able to install and run it separately from the application itself, as illustrated above.

We could improve this by providing some installation hints… But why not going one step further and actually performing the installation and run an instance of MongoDB for the user? With this improvement in place the above scenario became this (again output slightly simplified):

koper@thistle:~/test$ ./test.exe
Opening database at ~/.opa/test.exe/db
MongoDB does not seem to be installed in ‘~/.opa/test.exe/db’; sit back, relax (or grab a cup of coffee) and I'll install it for you
Launching MongoDB...
Test (OPA/1488) serving on http://thistle:8080

The change just meant one day of work of one of the Opa developers (thanks Quentin) but the results is lots of love from our users. Thank you for your support, we love to make you happy!

Monday, March 12, 2012

OpaHOWTOs

Hey Opa coders! I do realize the blog was a bit low on tutorials lately. The problem is: writing them is very time-consuming and with so much stuff happening around Opa lately (just last week: very successful Opa DevCamp in Paris plus presence at GDC 2012, biggest gaming conference, in San Francisco, CA) it's sometimes difficult to keep up.

However, we get a continuous stream of interesting questions, via the mailing list, forum and otherwise. Many of those questions are of general interest and some of them are nicely self-contained. Such as today's question from Nathan Frund (private communication) who was wondering how best to map HTML's SELECT component to Opa. Perfect subject for a short HOWTO!… so that's how OpaHOWTOs were born :).

The idea is to take certain tasks/problems and to illustrate how to address them properly in Opa. Instead of writing a full-blown tutorial I just write a simple demo program: heavily commented, with HINTs and EXERCISEs. The nice thing about this is that even if you're not interested in the subject itself of a particular HOWTO, they should often contain techniques and solutions that are of general interest, so you can still learn!

Check out the freshly created repo. I may occasionally post some updates about it on the blog, but the best way to stay updated and hear about them is to follow us on Twitter:

Oh, one more thing: if you have good HOWTOs ideas don't hesitate to share them with me! (for instance in the comments to this post).

Friday, March 9, 2012

New release: Opa 0.9.1

(New to Opa? Read this to learn what it's all about)

Today we just published a new release: Opa 0.9.1. You're welcome to consult the CHANGELOG for a complete list of changes but the highlights include:

  • Improved opa-translate tool for conversion between old & new syntax.

  • Ability to mix db3 and mongo backends in a single Opa program.

  • Feature to automatically download, install and launch a MongoDB instance (--db-local).

  • Experimental feature: remote logging (logs send to a remote server).

  • Unification of runtime types (and fixed magic functions).

  • Better handling of Twitter's Bootstrap CSS (included in the app instead of referencing external resource) and updating it to 2.0.

  • Added hooks to register favicons (much the same as custom CSS and JS).

  • Numerous bug fixes.

As usual you can download it, discuss with us, report problems and provide contributions. Happy Opa hacking!

Tuesday, February 28, 2012

Monday, February 27, 2012

Opa 0.9.0: New syntax

When we released Opa 0.9.0 (codenamed S4) we promised to follow-up with posts presenting its two main new features: new syntax and MongoDB support. In this post we'll shortly introduce the former.

The main motivation behind introducing the new syntax was to make it more accessible and familiar to users of other programming languages. It is largely inspired by JavaScript and hence current web developers trying Opa should feel more at home with it. However, we still believe that our original syntax has a lot to offer (for one thing: it's very concise) and will continue supporting it. The choice between the two variants is done via a commend line option:

opa --parser js-like ...
opa --parser classic ...

choosing new/old syntax respectively. Starting with Opa 0.9.0 new syntax becomes the default (so to use it one does not need to include any of the above arguments). It will also be used exclusively in the manual and the API.

Instead of introducing the new syntax here, I'd like to refer you to a newly introduced section in our manual: Reference card, which succintly summarizes Opa's syntax, important libraries and everything to get you started with the language. It's a new material, so don't hesitate to give us your feedback on it.

I'll wrap up this post with our immortal Chat example; both in old & new syntax, stripped from comments for brevity (consult the original repo for a fully commented version).

import stdlib.themes.bootstrap

type message = { string author
               , string text
               }

exposed  room = Network.network(message) (Network.cloud("room"))

function user_update(message x) {
    line = <div class="row line">
              <div class="span1 columns userpic" />
              <div class="span2 columns user">{x.author}:</div>
              <div class="span13 columns message">{x.text}</div>
            </div>;
    #conversation =+ line;
    Dom.scroll_to_bottom(#conversation);
}

function broadcast(author) {
    text = Dom.get_value(#entry);
    message = ~{author, text};
    Network.broadcast(message, room);
    Dom.clear_value(#entry);
}

function start() {
    author = Random.string(8);
    <div class="topbar">
      <div class="fill">
        <div class="container">
          <div id=#logo />
        </>
      </>
    </>
    <div id=#conversation class="container"
      onready={function(_) { Network.add_callback(user_update, room) }}></>
    <div id=#footer>
      <div class="container">
        <input id=#entry class="xlarge" onnewline={function(_) { broadcast(author) }} />
        <div class="btn primary" onclick={function(_) { broadcast(author) }}>Post</>
      </>
    </>
}

Server.start(
    Server.http,
    [ {resources: @static_resource_directory("resources")}
      , {register: ["resources/css.css"]}
      , {title: "Chat", page:start }
    ]
);
import stdlib.themes.bootstrap

type message = { author: string
               ; text: string
               }

@publish room = Network.cloud("room"): Network.network(message)

user_update(x: message) =
  line = <div class="row line">
            <div class="span1 columns userpic" />
            <div class="span2 columns user">{x.author}:</div>
            <div class="span13 columns message">{x.text}
            </div>
         </div>
  do Dom.transform([#conversation +<- line ])
  Dom.scroll_to_bottom(#conversation)

broadcast(author) =
  text = Dom.get_value(#entry)
  message = {~author ~text}
  do Network.broadcast(message, room)
  Dom.clear_value(#entry)

start() =
  author = Random.string(8)
  <div class="topbar"><div class="fill"><div class="container"><div id=#logo /></div></div></div>
  <div id=#conversation class="container" onready={_ -> Network.add_callback(user_update, room)}></div>
  <div id=#footer><div class="container">
    <input id=#entry class="xlarge" onnewline={_ -> broadcast(author)}/>
    <div class="btn primary" onclick={_ -> broadcast(author)}>Post</div>
  </div></div>

server = Server.one_page_bundle("Chat",
        [@static_resource_directory("resources")],
        ["resources/css.css"], start)
Run

Monday, February 20, 2012

Spotlight on Opa app: OpaDo by Tristan Sloughter

We continue our presentation of the winning apps of the Opa Developer Challenge. Today is time for OpaDo, the 2nd place winning entry by Tristan Sloughter. OpaDo is a TODO list, that started as a clone of TodoMVC, but then Tristan added some cool features, such as user accounts and even Facebook authentication. We really liked this app as it does what it's supposed to do and does it well. We also appreciated the fact that Tristan wrote a series of blog posts on developing OpaDo - be sure to check it out. We're also, together with the author, working on some extensions — stay tuned for details!

Try OpaDo!

For your convenience we've embedded the OpaDo application in this post, but for best results we suggest you go to http://opado.org.

Interview with the author: Tristan Sloughter

me: Can you tell us a bit about yourself? What's your experience with web programming? Favorite languages? Web frameworks?
Tristan Sloughter: I'm Tristan Sloughter, and I am a professional working full-time for a startup in Chicago (eCDMarket) and on the side on a startup of my own (ClaimsTrade). Both are fully implemented in Erlang, my favorite language. I've been an Erlang programmer since college and have been obsessed with it ever since. For web frameworks there are a number in Erlang, but none have suited my needs or been based on OTP standards enough, so I started Maru, which is still very much a work in progress and is developed in parallel with my development of real world sites in Erlang.

me: Can you tell us a bit about your submission for the contest? How did you come up with the idea? What are your future plans with respect to the application?
TS: My submission started as a simple port of the TodoMVC project to Opa, called OpaDo. TodoMVC is an app that many have ported between different Javascript MVC frameworks to show their strengths and to learn. I thought it would be a good project to use to learn Opa and hopefully help others while I did so by blogging about my progress. After implementing the features of TodoMVC, I extended the project to include user accounts and personal todo lists. I intend to continue extending OpaDo. First, I'll be moving the database to CouchDB, not because the Opa database can not handle the app but more as a learning experience. A few other features I'd like to play with to learn Opa better is creating a RESTful interface for users to interact with their todo items and to add some sort of real time collaboration between users and their items, possibly through the PubNub Opa binding I've written.

me: How did you like programming in Opa? Was it different than other technologies you know? Anything that you particularly liked? Anything that could be improved?
TS: I really enjoyed working with Opa. While my language of choice has been Erlang and it is an amazing language, working with dynamically typed Erlang and Javascript has been prone to cause great frustration. Opa's type checking that covers both the client and server code is simply amazing and a great time saver. Like those who have worked with OCaml, Haskell or another strongly typed language know, if it compiles there is a good chance it works as well :). Continually I had the compiler catch mistakes that I commonly find and know that if I was using another backend language or Javascript on the frontend it would have taken till I was testing the page and had Firebug open to find the mistake.
So, I think my favorite features of Opa start small, before even thinking about its advanced architecture for easy network communication and scaling. The fact that it is a strong typed, functional language is its greatest feature. This combined with the ability to embed HTML (I don't know why this bothers others when it is so clean, I think it is perfect) and not having to use Javascript — and the more advanced features for creating large scale commercial apps — make Opa something that I think will greatly change the future of web development.
The main area for improvement, which the Opa team is already tackling head on, is database backends. Working with the embedded Opa database was very productive and much better than dealing with serializing/deserializing to an external relational database and writing complicated queries — similar to the ease and usefulness of Erlang's embedded database Mnesia. Having the same ease of use as the embedded database extended to CouchDB and MongoDB will be a huge feature.