mobile 2.0

Following up on my previous post on how we believe the best mobile apps are built , I’ve been asked what the somewhat nebulous term “mobile 2.0” means to me. From a technology perspective I would summarize it as the following things:

• locally executing applications that work to some level even when disconnected and thus have…
• locally cached data
• leveraging relevant open standards where possible. for example be opportunistic about the presence of Opera, Minimo, IE, Nokia on peoples devices
• controls that are optimized for the capabilities of a particular device and the unique functionality required by mobile apps. there is a natural tension between this goal and the previous point of just using a browser

From a user experience perspective I would suggest that it is the following three things:

• “context and user aware applications”: delivering the app to users with knowledge of their preferences, their previous uses, their defaults and their likely current actions, to accelerate their tasks and put them into the right form/page/screen/action without a lot of navigation
• “shared knowledge among applications” – in order to ease the previous goal of minimizing user data entry and navigation on mobile devices, applications should share information about user profiles, preferences, authentication, past usage and defaults. in this context its worth the extra effort to perform this integration
• user interaction and content contribution – this is probably the only point that truly echoes web 2.0. mobile 2.0 apps should incorporate user feedback and results in intelligently rapid ways

From a business perspective the following trends will drive massive user adoption and make mobile applications tools that everyone uses

• low to no cost through an advertising and transaction revenue model – users shouldn’t have to pay for services
• cheap to free data plans – this still has to happen and may be the biggest barrier to entry
• tools and platforms to make the nontrivial technology and user experiences described above happen for many applications and content in as cost effective and rapid way as it did for the web

WAP and mobile application development

Some of you may be using our Mobile Movie Times application or seen the demos of our various other mobile consumer apps. You may say “that sure looks a lot better than WAP”. Yes, all of our apps are locally executing rich clients optimized for their devices. Many of you may have noticed that most consumer-focused large web properties are pursuing (google, yahoo, ebay) are also not building WAP portals but instead focusing on locally executing optimized apps targeting a particular platform. So why is this? WAP’s been around for years. What’s wrong with it?The big problems with WAP that we see are no ability to execute functionality locally, lack of separation of presentation and data, a murky future standards picture, no ability to work with data offline, and no ability to take advantage of an increasingly diverse set of mobile device capabilities.For the record, there are several applications and use cases where none of these are problems, especially one way delivery of text only information. It may be that in an effort to allow an application to run on as many platforms as possible without rewriting them as native locally executing apps that use of WAP on some devices can be a way to multiply the payoff of investing in building the backend of a mobile application. That said, for most applications the following issues all inhibit WAP from delivering a truly optimal user experience on targeted devices<h3>Executing Logic Locally</h3>
The best applications (mobile, web-based and desktop) all execute some portion of their logic locally. This can be for:

• validation
• modifying forms, menus and screen display based on user behavior
• rapidly showing different data based on user actions and interests without roundtrips to the server

These ideas are behind much of the current trend of Ajax-oriented websites: highly responsive, interactive, adaptive web pages and user interfaces.

WAP sites don’t do this and they don’t have enough of a scripting model to enable these features. AJAX-capable browsers can try to do these things (the pros and cons of mobile Ajax browsers are the subject of another post). But most web and mobile app providers trying to create optimal user experiences are building “native applications” in the APIs of the mobile platform to achieve these benefits.

Separating Presentation and Data

Neither WAP nor traditional HTML pages separate presentation from data. Form controls and their embedded data are tied together in one stream of information. This is why most WAP sites that interact with users are painfully slow. By contrast, optimized native applications run locally on the device and send only the changed data back and forth. It is possible to approximate these benefits with Ajax and other use of Javascript on a mobile browser, but it is inherently more difficult and the data exchange less efficient (again, we’ll talk more about Ajax browsers later).

Future Standards

At one time it seemed like the mobile web industry might come together on some standards based on some derivative of both WAP and XHTML Mobile Profile. That effort ended with major vendors each taking their own tack on the mobile web using divergent subsets of a proposed WAP 2.0 standard. We’ll talk more about that later. But at this point, it is not a controversial statement that WAP 2.0 and XHTML MP are not being used consistently and robustly by major mobile software providers and device manufacturers.

Working with Data Offline

Mobile networks are still not always available with 100% coverage and this is likely to remain true for quite a while. The best mobile applications (such as Blackberry’s email) are optimized for syncing intermittently and transparently and allowing the user to use the application offline. The best usage experience even for “vertical” single purpose apps are written to the native device operating system, making optimal use of available storage, trickle syncing data to and from the server transparently to the user, and allowing the user to be as interactive as possible when not having coverage. WAP browsers (and Ajax browsers) are not capable of this.

Taking Advantage of Device Capabilities

Mobile phones are a diverse lot and, with the emergence of new “more than just voice call” features, becoming moreso by the day. There are several major device operating systems: BREW, Symbian, J2ME, Linux, Windows Mobile to name a few. More importantly phones have an increasingly differentiated set of capabilities: everything from taking pictures to recording voice to playing music to syncing contacts and PIM information to voice dialing to GPS. The best applications take full and optimal advantage each device’s capabilities in all of those areas and more. Browser based applications (of any flavor) just don’t do this. To do this requires writing to the APIs of the device, even when a device uses supposedly “open API” such as JSR extensions to J2ME for specific capabilities.

So How Do We Build Such Optimized Native Apps Efficiently?

All this said, its quite expensive and difficult to write application to these various devices in their native operating systems with their unique APIs in C, C++, or Java. And it becomes moreso once you want to target multiple devices. J2ME offers a variety of devices, but certainly not the majority of the market.

But this hasn’t daunted the the biggest web properties(such as Yahoo and Google) with extremely horizontal apps such as email and maps that can be targeted to many millions of users. The cost of hiring large teams of developers to spend months to years writing apps can pay off in these situations.

But is it possible to write apps at a higher level than the native procedural languages (maybe even higher level than WAP or HTML)? And can we do so and still allow single applications to be executed on a wide variety of other devices? And perhaps even better allow the apps to function at a gracefully degraded level by delivering content WAP and AJAX browsers on platforms and devices that its not economical to devote a lot of attention (while still just “writing them once”, as is referred to in another post here). All this is a topic for another post some time soon.

model oh model, where is my model?

One of the strengths of Rails and ActiveRecord is that it makes it easy to build your models. You Don’t Repeat Yourself, with your model attributes being derived directly from your database tables. Then all you have in your model definition itself is a discussion of its relationship to other models, relationship to tables and

There are problems introduced by this power however. When writing your application, where is your model definition? Your attributes are generally all in your table definitions. Your models relationship to other models is in your model definition file. Your validations are often in your model, unless your database is performing some form of validation (a practice that is generally encouraged outside the scope of Rails). So they are usually in both places! Plus there’s a good chance in your Rails application that you have introduced at least one or two views to handle things that Rails needs (such as the need for a single column primary key). So now there’s a third place for your model attributes definition.

Rails introduces another way of handling database definition and versioning: rake migration scripts. These are very powerful ways to version and maintain database schemas and data in a product neutral way. But they also spread the definition of “what is the model” out even further. I talk to Rails developers who say “I never look at my database administration tool – I just incrementally edit my Rake migrations”. The problem is that distributes the model definition out even further! It is spread amongst multiple rake migrate scripts.

Many developers used to programming with other tools and environments are also accustomed to some form of graphical frontend to define and manage their database tables. This can be at the “physical level” of tables or at a more conceptual level of entities and relationship (with tools such as ERWin or Visio). Such environments not only ease understanding and ease of manipulation and modify database (and model) relationships but they are also much better at concentrating the model definition into one place.

So here’s my suggestion: An incredibly useful improvement to the Rails app development toolset would be a graphical “model and rake migration designer” frontend. Probably an Eclipse plugin that plays nicely with RadRails (part of RadRails in the future?).

This frontend would allow you to define models, graphically draw relationships between models choosing from amongst the Rails palette of possible model relationships, and generate rake migration scripts from model attribute definitions. It would generate increment rake migrations when model changes were made. It would also allow you to set validations on attributes and models that would be reflected in the model definitions themselves. It would generate migration scripts for join tables when called for by as has_and_belongs_to_many relationship. There are a lot of other features I’d like to see in this tool, but I’ll leave off for now. No, I’m not considering building this. But I and many other Rails developers I know would be delighted by having this tool to use.

Rails, REST and SOAP web services

I’ve spent a lot of my career in the web services world (before it was even called web services). So a lot of my colleagues often ask me “you like Rails?! how is that possible, the Rails community hates SOAP-based web services?”. I’ve often wondered about this myself. More specifically I’ve wondered at the seemingly religious antipathy of the Rails community (including the illustrious DHH himself) to the “WS-Deathstar”.So (primarily for the people who are confused by a web services’ zealots embrace of Rails) here’s my take on the matter: REST is great for many problems and applications. The first class support that DHH is providing for REST in the Rails core is very useful. I love the scaffold_resource generator and what it provides (as I’ve mentioned earlier here). I also like the approach to providing it: the idea of single controllers that respond to different clients (HTML, REST, and others) with different content. I do have reservations about the need to put wordy respond_to clauses in each controller action (I’ve proposed what I think is a simpler way here). Anyway, automatic support for REST is a good thing. REST is a great way to build simple point to point distributed app to app connectivity. And I agree that, for someone who wants to be aware of the innards of the code they are working with that it is simpler. I also agree that there are scenarios where a REST interface just makes it easier for multiple arbitrary clients on more diverse platforms and languages (there is perhaps stil a small chance that it may be difficult to integrate with a client SOAP stack on a device, but there’s basically no chance that it will be hard to invoke a REST interface from another platform).

But does that mean that SOAP is bad? In the simple app to app connectivity scenario my opinion is that it basically doesn’t matter. In the vast majority of languages and platforms the work of consuming a SOAP-based web service is just as easy as consuming a REST service. But I agree that there’s little tangible advantage to using SOAP if all you ever need is a distributed method call.

The value of the SOAPbased WS-Deathstar is when more complex connectivity is required: multihop message routing, guaranteed delivery, publish-subscribe one-to-many integration, more advanced security in the plumbing than https (the latter probably only necessary when you’re routing messages). If this is never going to be necessary (probably true for the typical consumer-facing website), then the whole SOAP vs. REST debate is moot. And going with the default easier path of REST doesn’t hurt one whit. For more advanced distributed connectivity problems (endemic to many “enterprise applications” that are seemingly anathema to the Rails community), robust SOAP stack with extensive WS-Deathstar support would be useful for developers building complex connectivity. And yes I mean a whole bunch of those nasty evil WS-* specs, including those not yet widely implemented. My top requests include WS-Addressing, WS-Eventing, and full WS-Security. Doing these would also obviate the need to provide other APIs for such such services as publish/subscribe message routing. Due to Ruby’s stunning productivity advantages I think any and all of these would be trivial to implement. I remember implementing a very early WS-Eventing client prototype in C# in about a page of code. It would be much easier in Ruby (if there’s demand for it, I’d consider rewriting it, if only to demonstrate that a seemingly complex spec can in fact be made easy to use).

Are the WS-* overly complex? Yes, I think many of them are. There are plenty of “design by committee” artifacts in there. For the most part such unnecessary complexity can be hidden from the programmer using them. A good API with reasonable default behavior (convention over configuration) covers a thousand design sins. And the layered nature means that you can just pick and choose what facilities help you. Embracing the Deathstar specs would make Ruby a more powerful tool for a wider set of applications. The availability of Rails plugins (or just Ruby libraries) for these capabilities would make it more likely that Ruby gets used for a wider variety of programming tasks which can only help to grow and advance the Ruby and Rails community.

consuming web services from your Rails app

Despite the fact that SOAP-based web services seem to be considered politically incorrect by the Rails community, it is amazingly easy to expose web services from Rails. Ironically enough, it can be a little tricky to consume web services from Rails. Several of my colleagues have asked me to post this. This is mostly due to the strange web services client implementation in Rails (Action Web Services’ client). Essentially, you should just use the native Ruby web services client capability (SOAP::WSDLDriverFactory).Specifically, if you use the guidelines below it should be easier.

• Don’t try to use the Action Web Services client. It only works with AWS servers.
• Don’t use WSDL2Ruby.rb. Its buggy and not complete.
• DO use the native Ruby SOAP services such as in the following code (the example points to a free sample Trivia web service)require ‘soap/wsdlDriver’ triviaEndpoint=”http://mobquiz.com/question/api”; triviaWsdl=wsendpoint+”?wsdl”service=SOAP::WSDLDriverFactory.new(smswsdl).create_rpc_driver question=service.askQuestion(‘Entertainment’) puts question.question

Rails database guidelines

These are some guidelines on building databases that you will use with your Ruby on Rails applications. I have seen many of these elsewhere in scattered places, though I’m not sure that I agree with all the other takes on these issues. My intention is to both list good practices for building new schemata and how to handle inheriting legacy structure which diverge from these practices. Anyway here goes:

Create a Primary Key Column

It is a reasonable restriction to insist that tables have single columns for primary keys. If you’ve inherited a poorly designed, then just create a view onto it with such a column. Unlike the claims of many anti-Rails rants, the column does not need to be called id Using

set_primary_key your-other-primary-key-column

works just fine. But if you’re creating a new database, why not call the column id?

Foreign Key Columns

If you’re designing a new database you should follow the convention of naming the foreign_key column as othertablename_id.

If your existing table doesn’t have this please use a view.

Join Tables

Join tables should be named with the convention of firstModelnamePlural_secondModelnamePlural. The columns in the join table should use the convention just mentioned of joining back to the model name tables with columns named otherTableName_id.

Has Many Through Tables

If you are going to build a :has_many :through relationship, the :through model probably has its own attributes. Otherwise you would have done :has_and_belongs_to_many. Remember to give that model table its own :id column.

Parent Reference Columns

If you are using a model with :acts_as_tree the column which refers to the parent entities. You can override this with the :foreign_key option on :acts_as_tree

Pluralized Table Names

Its better practice if you’re creating a new database schema to use pluralized names. But if you’ve inherited a database that doesn’t follow these practices, you can enter the following in your config/environment.rb file.

ActiveRecord::Base.pluralize_table_names = false

Overriding Table Names

More generally, if you just don’t want to use the same name or plural of the same name as your model for your table name. Or if you’re inheriting a database schema, and you want to use a different name for your model than is reflected in your database, use the set_table_name option in your ActiveRecord model definition.

class YourModel < ActiveRecord::Base
  set_table_name "some_table_name_different_than_model_name"
end

Use Created_At and Updated_At Columns

If you create columns with these names, Rails will put the date and time of record creation in them for you. Be sure to leave the fields as allowing NULLs. For clarity you should avoid using other columns that have a similar purpose. For example, don’t use a column called “creation_time” that you put values in yourself.

There are also additional column names of created_on and updated_on that will be filled in with the dates only of record creation and modification times. I would tend to stick with created_at and modified_at flavors.

Avoid “Magic Columns” Unintentionally

Besides the columns mentioned there are several others that you could unwittingly use that will cause you problems. Don’t ever use these column names in your tables unless you know about and mean to use their special behavior.

created_at
created_on
updated_at
updated_on
lock_version
type
#{table_name}_count
lft
rgt
position
id
parent_id
method
quote

Rails 1.2 scaffold_resource generator suggestions

Rails 1.2 is out now of course. As most of you know the big feature in 1.2 is more support for “automatic CRUD style controllers”. The big feature here is the scaffold_resource generator. As documented elsewhere (but not that extensively) you invoke

ruby script/generate scaffold_resource <modelname> [attributename:attributetype]

where the optional attributename (or more than one of them) identifies an attribute of the model. The controller is generated for you with the same name.

The Good News

So, this is a great step forward. It promotes a REST style of design and the direction that Rails is moving. Its a very good thing that Rails is not agnostic about the right way to build apps. Just as Rails puts a stake in the ground and says that you essentially must build apps using MVC, using capabilities like this does the same to enforce and encourage REST.

More pragmatically, it saves you from writing those REST style controllers yourself, something I found myself doing in app after app. So I was very excited to see this feature and looking forward to saving some typing and being even DRYer.

Some Problems

But, alas, if you are starting from an existing table, you’ll immediately see the problem with this approach. If you don’t list your attributes explicitly (which those of you used to working with existing tables are not used to doing) you will get a surprise when you first invoke your controller actions(with URLs such as http://localhost:3000//view/1 or http://localhost:3000//new). When you view or add any particular record the forms will be empty. Of course, you can go look at your database table schema and just add several column name arguments to your call to the scaffold_resource generator. But that doesn’t seem very DRY, assuming that at some point you created that table (although its DRY enough if you don’t yet have the table).

Solutions

The best fix is probably that the forms introspect the model to find out its attributes and then display those attributes. This code can be placed once in the form.rhtml partial. Note that 1.2 doesn’t actually have that partial. But it appears to be in a fix put into Edge Rails, so it will probably be available in nearterm update to 1.2. Hopefully the form partial can include the dynamic model attribute display, at least as an option.

In the meantime, I may write a little scaffold_resource script generator “invocation generation” script that reads attributes to generate from the underlying table.

Handling Multiple Types of Clients in Your Rails Apps

If you are building a fullfledged application (versus just a web service), you may want to support both HTML and XForms clients (and possibly other clients such as WAP). This tip describes how to do this efficiently, in a way that is very much not the “standard Rails way” (but is much simpler).

Using A Common Client Checker Inside ApplicationController Class

Include the following code inside your parent application controller class (application.rb) which you use for all controllers in your application.

def checkClient
  if (@request.env['HTTP_ACCEPT']=="application/xml")
    @client="xf"
    if @request.env['REQUEST_METHOD'] == "POST"
      paramsFromXml(@request.env['RAW_POST_DATA'],'data')
    end
  else
    @client="html"
  end
end

def paramsFromXml(xml,parentname)
  doc=Document.new(xml)
  @params={}
  doc.root.elements.each do |x|
    @params[x.name]= x.text
  end
end

We will add this to the Rails application generator at some point. Note that the first method is the essential part: figuring out that the client is either “xf” (xforms) or “html”.

The second method handles the fact that Mobio’s XForms runner just populates XML in the POST data and doesn’t give the developer an @params hash to work with. paramsFromXml() populates the @params hash from the top level elements below the root XML element. This allows you to write a single controller action that handles data being available in an @params{} hash.

Note that if you wish to just manually parse the XML yourself then you may not need to use this second method. It is merely a “convenience” method for those of you that like to use a params hash in your controller actions

Hooking the Client Checker into Your Controller

Use the Rails filter capability in your controller to integrate the checkClient call to be called before each controller action. Specifically put the line “before filter :checkClient” at the top of each of your controllers. For example,

  class YourController < ApplicationController
   before_filter :checkClient
   yourMethod
      ...your method code....
   end
  end

We will be modifying our controller generator to include this call.

Render with the Appropriate Template

Put the following code at the end of each method.

   yourMethod
     ... your code
     render :action => "#{action_name}.r" + @client
   end

What this will do is either execute the ”.rhtml” template or the ”.rxf” template based on what client is accessing it. Note that you will need to follow the convention of calling your XForms templates ”.rxf” files. Again we will modify the controller generator to stub in these render calls

Note that this whole methodology is quite different from the “respond_to” mechanism proposed by Rails creator David Heinemeier Hanson. The method he proposes pollutes each controller action with several lines of code (versus adding ”@client” to the end of the render call as shown above).