DaveBouwman.com

Demystifying oAuth and ArcGIS

Sun, 27 Aug 2017 00:00:00 -0600

Over the years I’ve talked with a lot of developers who find oAuth to be mysterious, and really, when we watch what’s happening “on-the-wire”, it’s not too complicated. In this post we will review how it works, and we will implement it in ~120 lines of commented javascript. Let’s jump in!

What is oAuth?

Let’s start by recalling the web before oAuth - where you had a login at every different site… and of course you used good security practices and had a different password for every site… (me neither). So, while painful to manage, it was theoretically “ok”. But then we started building applications that integrated with other systems. Now we have a problem - how can application A take actions in system B on a user’s behalf without having the users credentials?

This is the problem that oAuth solves - it allows applications to use a third-party system for authentication, as well as managing access to services provided by that system.

Three Components

oAuth provides three main things:

Identity

We see “Sign In with Facebook” buttons on all manner of sites - from Strava to CodePen, millions of applications now leverage the fact that 3+ billion of us have Facebook accounts, and none of us want to create “yet-another-account”. Via oAuth, we can use our Facebook identity on other sites.

Privileges

When you use an oAuth provider to access a site, it will usually ask for some sort of privilege, allowing it to act on your behalf - i.e. Post to your Facebook timeline, or store items in your ArcGIS Online organization. For Facebook there are a wide range of grants that can be requests, but for ArcGIS Online it’s all or nothing, which makes things simpler for us.

Revokability

Should you decide you no longer want the application to have access to your account, you can revoke access at the origin - i.e. at Facebook - instead of having to find the site, login, and flail around trying to delete your account. In my mind, this is a huge feature.

ArcGIS oAuth

Similar to Google/Facebook/GitHub and others, ArcGIS Online / Portal / Enterprise support authentication via oAuth - specifically oAuth 2. This article will focus on the “web flow” (also known as the ‘implicit grant’), which is specific to web applications - if you are working with a desktop or native application you should use the “authorization code grant” flow. Documentation here

The Steps

Before talking about code, let’s review the basic steps.

Application Boots

Your application loads in a browser, and shows the user a screen saying they need to sign-in in order to do amazing things.

The application constructs a url to the ArcGIS Online Authorize login page, and it either opens a pop-up window (what we will do), or redirects the entire browser to this url.

This url contains two key pieces of information - a client_id and a redirect_uri. More on these later, but they basically tell ArcGIS Online what application the user is authenticating with.

This is the key part - the user provides their credentials to a page hosted by ArcGIS Online. Your super-cool application never has access to their credentials.

User Submits Credentials, and ArcGIS Online Approves access

A number of things happen at this point - and I think this is where it seems somewhat magical… but bear with me - it’s not that complex:

the login form is POSTed to ArcGIS Online
assuming the username & password checkout, it returns a page asking you to allow the app access to your account
clicking that button then POSTs another form back to ArcGIS Online, which then responds with a page that has a meta tag that redirects the browse… to the redirect_uri we passed in.
The browser then redirects to that page, which loads up, and passes the url to your application

See that’s not too bad?

Secret Sauce…

So the really crafty bit here is that the redirect_uri that ArcGIS Online returns to the browser is a hashed-url. This means that the url has a # in it - followed by the access token. Now, the cool thing about browsers is that they don’t send the hash or anything following that to the server.

The browser is told to load http://yourcoolapp.com/redirect.html#token=SEEKRET&username=joeuser, but the only part that goes over the wire is http://yourcoolapp.com/redirect.html. Once that page is loaded, we can access the entire url (including the hash) in javascript… and thus get the token. Since the authorize page @ ArcGIS is loaded over https, and the hash does not go over the wire, we can have a secured authentication system even when our app runs on http. That said - be sure that all requests sent with a token use https!

The Code

This demo app is about as stripped down as it can get and still be readable and have all the features. Please note - this is not production code! My point here is to strip back the mysteries so we can see what’s happening. The general process will work for any application, but when building a real application, you’d want to work with an established session management system for whatever framework you are using - i.e. torii for EmberJs etc.

Features

allow a user to sign-in
use localStorage to persist authentication info so they can be automatically logged in if they return to the app before their token expires
allow a user to sign out, which clears their current session and removes entry in localStorage
when a user is authenticated, show some information about them from their AGO user.

Step 1: Registering an Application with ArcGIS Online

Sign in at the ArcGIS for Developers site, and hit the plus icon at the top, and create a “New Application”. Fill in the basic information, and then go to the “Authentication” tab and leave it open. You will need the client_id and once you host your app somewhere, you will need to enter some valid redirect_uris. You can start with http://localhost:8080/redirect.html if you cloned and are running locally with http-server (see below)

Step 2: Clone the Repo and Install Dependencies or View the Demo App

The repo for this example is at https://github.com/dbouwman/vanilla-arcgis-oauth - you can either view it on github or clone it and follow along like that. The demo site is at http://vanilla-arcgis-oauth.surge.sh/ - all the code is inline in the page, so open the console and drop in breakpoints.

Step 3: Page Scaffolding

I simply created a basic page that loads the Bootstrap 3.x css from a CDN. Almost all our javascript is in-line in the page. If you are looking to optimize pageload, you could strip this down to ~30 lines of code, and stuff it in the <head> if you wanted to initiate fetching a webmap or some other resource as early as possible (read: before any frameworks have had time to load)

Step 4: Javascipt!

I put a lot of comments in the code, so here it is…

    // Setup some vars...
    // Client Id from your Application Registration at developers.arcgis.com
    let clientId = 'MReeG1Zt9JCulHLM';
    // change this if you are working with porta/enterprise
    let portalBaseUrl = 'https://www.arcgis.com';
    // does not matter what this is but it's used twice, so it's in a var
    let localStorageKey = 'agoauth';
    // simple "session" object
    let session = {
      isAuthenticated:false,
      portalInfo: null
    };
    let authObj = null;

    // With that setup, when the page loads, it will check for
    // a persisted session and it will automatically log the user in

    // check if the browser supports localStorage...
    if (window.localStorage) {
      let authObjItem = window.localStorage.getItem(localStorageKey);
      if (authObjItem) {
        // localStorage stores keys with STRING values!
        authObj = JSON.parse(authObjItem);
      }
    } else {
      // use cookies - you're on your own for this :)
      // authObj = someCookieParsetFunction()
    }
    // if we got an
    if (authObj)
      // check if the token has expired
      let nowTs = new Date().getTime();
      if (authObj.validUntil > nowTs) {
        // token should be valid...
        this.validateToken(authObj)
        .then((response) => {
          session.isAuthenticated = true;
          session.portalInfo = response;
          updateUI();
        })
        .catch((err) => {
          alert(`ZOMG! It did a bad! (an error occured)`);
        })
      }
    } else {
      // no worries, we assumed the user was not authenicated
    }


    // To valudate a token we make a call to the portals/self end-point.
    // If this is successful it will return a big fat object with information
    // about the org's portal and the current user.
    function validateToken (authObj) {
      // We have pulled in fetch via polyfill.io, so this should work in oldish browsers
      return this.fetch(`${portalBaseUrl}/sharing/rest/portals/self?f=json&token=${authObj.access_token}`)
      .then((response) => {
        // fetch does not auto-parse into json... so...
        return response.json();
      })
      .then((response) => {
        // Esri API's rarely use http error codes, but they may return an error payload
        if (!response.error) {
          // store the authObj in local storage
          window.localStorage.setItem(localStorageKey, JSON.stringify(authObj));
          return response;
        } else {
          throw new Error(`Error in response: ${JSON.stringify(response)}`);
        }
      })
    }

    // redirect.html simply calls this function, with the url
    // from there, we parse out the hash into a set of key/values
    // and stuff that into an object that our session then stores
    function checkOAuthResponse (href) {
      // url will look like: "http://localhost:8080/redirect.html#access_token=THE-TOKEN&expires_in=7200&username=dbouwman
      // parse the href - this could be tighter, esp if you have something like lodash loaded...
      let hash = href.split('#')[1];
      let parts = hash.split('&');
      let authObj = parts.reduce((acc, part) => {
        let k = part.split('=')[0];
        let v = part.split('=')[1];
        acc[k] = v;
        return acc;
      }, {})
      // the response has an expires_in value that is seconds-from-now...
      // lets turn that into a real date, so we can check if the token is valid later w/o doing an xhr
      authObj.validUntil = new Date().getTime() + authObj.expires_in;
      validateToken (authObj)
        .then((response) => {
          session.isAuthenticated = true;
          session.portalInfo = response;
          updateUI();
        })
        .catch((err) => {
          console.error(`Error: ${err}`);
          alert('error authenticating! Check the console!');
        });
    }

    // Start the sign-in process by creating the authorize url...
    // To make this code easier to deploy, we construct the currentBaseUrl
    // using the browser's current location, and use that to construct the redirect_uri
    // Remember - your application registration at ArcGIS Online MUST include the
    // redirect_uri that you specify here, or it will barf at you!
    function startSignIn () {
      let currentBaseUrl = [window.location.protocol, '//', window.location.host].join('');
      let authorizeUrl = `${portalBaseUrl}/sharing/oauth2/authorize?client_id=${clientId}&response_type=token&redirect_uri=${currentBaseUrl}/redirect.html`;
      window.open(authorizeUrl, 'authWindow', 'menubar=no,location=yes,resizable=no,scrollbars=no,status=no,width=500,height=550');
    }

    // Signing out is simple - just null out the session properties and kill the item
    // from localStorage
    function signOut () {
      window.localStorage.removeItem(localStorageKey);
      session.isAuthenticated = false;
      session.portalInfo = null;
      updateUI();
    }

    // Normally, you'd use a framework of some sort for UI updates etc...
    // But we're keep'in it real (simple) here today...
    function updateUI () {
      if (session.isAuthenticated) {
        document.getElementById('signInBlock').setAttribute('style','display:none');
        document.getElementById('signOutBlock').setAttribute('style','display:block');
        // ES6 templates ftw! But only on very modern browsers!
        let userInfo = `
          <h3>Hello ${session.portalInfo.user.fullName}</h3>
          <p>You are part of the ${session.portalInfo.name} organization</p>
        `;
        document.getElementById('user-info').innerHTML = userInfo;
      } else {
        document.getElementById('signInBlock').setAttribute('style','display:block');
        document.getElementById('signOutBlock').setAttribute('style','display:none');
        document.getElementById('user-info').innerHTML = '';
      }
    }

    // attach event handlers
    document.getElementById('signInBtn').addEventListener('click', startSignIn);
    document.getElementById('signOutBtn').addEventListener('click', signOut);

Redirect.html

This file is super simple - it literally get’s a reference to the window that opened it, and calls the checkOAuthResponse function, then closes itself.

<!DOCTYPE html>
<html>
  <head></head>
  <body>
    <script type="text/javascript">
      function closeWindow () {
        var win=window.open('', '_top', '', 'true');
        win.opener = true;
        win.close();
      }
      if (window.opener && window.opener.parent && window.opener.parent.checkOAuthResponse){
        window.opener.parent.checkOAuthResponse(window.location.href);
        closeWindow();
      } else if (window.parent && window.parent.checkOAuthResponse) {
        window.parent.checkOAuthResponse(window.location.href);
      }
    </script>
  </body>
</html>

And that is it!

If you cloned the repo, you can run npm install which will install two packages

http-server, which is a simple command-line http server - great for very simple projects like this
surge, which allows you to host static websites, for free, with a single command. Sweeet.

Ember Engines: Helper Not Found

Tue, 04 Oct 2016 00:00:00 -0600

Today I was doing some research using Ember 2.8 and Ember-Engines, and I ran into an issue that had me stumped for a few hours.

I had created a new ember engine addon as per the steps listed in the Ember-Engines README, and was consuming the engine in the Dummy app inside the addon. Pretty standard stuff.

Part of my research was to make sure that the internationalization system we use (ember-intl) worked with engines. So after getting some basic stuff working, I started to add some internationalization into a template served from my engine.

Ember intl uses a helper – `` – and as soon as I dropped this into the page, I started to get errors in the console.

Uncaught Error: Assertion Failed: A helper named "t" could not be found

So - the first thing I did was drop a `` into the index template in the Dummy app itself - and that worked, which told me ember-intl was installed, and working as far as the Dummy app was concerned.

After stepping through a bunch of Ember core code, I was able to resolve the issue by simply moving the ember-intl entry from devDependencies to dependencies in package.json.

Ripple Effects

This got me past my first problem… but then I ran into my next issue. The Ember Intl service gets initialized in the application route of the host app. But

DevSummit and DevMeetup Videos

Fri, 18 Apr 2014 10:44:48 -0600

Just a quick note that the video of the Javascript Unit Testing talk from the Esri 2014 Developer Summit is now up. We gave the talk twice, and while there are two recordings of the talk, the first one had some video and audio issues, so the one below is the one to watch (literally!)

Direct link to video

In this talk I start things off with the “zen of testing”, then David Spriggs talks about using the Intern, followed by Tom Wayson discussing Karma, and I close the talk with Grunt + Jasmine + automation.

Here is a PDF of the slide deck as well – may be useful in discussions with others.

All the tools we discuss in the session are listed in https://github.com/tomwayson/esri-js-testing-tools-and-patterns in Tom’s github account.

Speaker Info

Dave Bouwman – @dbouwman – http://blog.davebouwman.com/
David Spriggs – @davidspriggs – http://latitudeagile.com/
Tom Wayson – @tomwayson – http://tomwayson.com

DevMeetup Video

I also gave a quick presentation at the Fort Collins Dev Meetup, and recorded a screencast. In it, I talk about the soon-to-be-released-in-beta ArcGIS Open Data project at a high-level, and then demonstrate the front-end developer workflow related to automated linting and unit testing. I then showed some of our integration tests running (using selenium, driven by mocha + wd.js). Finally I talked about some work I’m doing taking “best practices” from the Open Data project and creating yeoman scaffolders to help people start off new projects with all the infrastructure in place. For this demo, I scaffolded and published a (really simple) web app up to github pages in ~2 minutes. As this moves forward I’ll be posting about the scaffolder’s themselves as well as how to create scaffolders.

Direct link to video

In the video I demonstrate a number of the tools and concepts that I’ve written about in these posts…

Chasing Numbers: Pragmatic Javascript Code Coverage

Tue, 25 Mar 2014 16:21:05 -0600

When writing unit tests, we want to spend our time wisely, and focus our efforts on areas of the application where test coverage provides the most benefit - typically this means business logic, or other complex “orchestration” code. We can gain insight into this by using “code coverage” tools, which report back information about the lines of code that area executed during your tests.

Coverage is typically reported as a percentages - percentage of statements, branches, functions and lines covered. Which is great… except what do these numbers really mean? What numbers should we shoot for, and does 100% statement code coverage mean that your code is unbreakable, or that you spent a lot of time writing “low-value” tests? Let’s take a deeper look.

Here is the output from our automated test system (grunt + jshint + jasmine) on our project. Included is a “coverage summary.

These numbers have been holding steady throughout the development cycle, but what do these numbers mean? Lets break them down a little

Coverage Measures

The first one is “statements”. In terms of code coverage, a “statement” is the executable code between control-flow statements. On it’s own, it’s not a great metric to focus on because it ignores the control-flow statements (if/else, do/while etc etc). For unit tests to be valuable, we want to execute as many code “paths” as possible, and the statements measure ignores this. But, it’s “free” and thrown up in our faces so it’s good to know what it measures.

Branches refer to the afore mentioned code-paths, and is a more useful metric. By instrumenting the code under test, the tools measure how many of the various logic “branches” have been executed during the test run.

Functions is pretty obvious - how many of the functions in the codebase have been executed during the test run.

Line is by far the easiest to understand, but similar to Statements, high line coverage does not equate to “good coverage”.

Summary Metrics

With that out of the way, let’s look at the numbers.

Since statements is not a very good indicator, let’s skip that. We notice it, but it’s not something we strive to push upwards.

On Branches we are at ~42%, which is lower than I’d like, but we’ll get into this in a moment.

Functions are ~45%, but this is a little skewed because in many controllers and models we add extra functions that abstract calls into the core event bus. We could inline these calls, but that makes it much more complex to create spies and mocks. In contrast, putting them into separate functions greatly simplifies the use of spies and mocks, which makes it much easier to write and maintain the tests. So, although creating these “extra” methods adversely impacts this metric, it’s a trade off we are happy with.

Where does this leave us? These numbers don’t look that great do they? Yet I’m blogging about it… there must be more.

Detailed Metrics

These summary numbers tell very little of the story. They are helpful in that they let us know at a glance if the coverage is heading in the right direction, but as “pragmatic programmers” our goal is to build great software, not maximize a particular set of metics. So, we dig into the detailed reports to check where we have acceptable coverage.

The report is organized around folders in the code tree, and summarizes the metrics at that level. I’ve sorted the report by “Branches”, and while we can see a fair bit of “red” (0-50% coverage) in that table, the important thing is that we know what has low coverage - as long as we are informed about the coverage, and we decide the numbers are acceptable, the coverage report has done it’s job.

Diving down to the file level, we can check if we have high-levels of coverage on the parts of the code that really matter. For us, the details controller and view are pretty critical, so we can see that they have high coverage. It should be noted that high coverage numbers don’t always tell the whole tale. For critical parts of our code base, we have many suites of tests that throw all manner of data at the code. We have data fixtures for simulating 404’s from APIs, mangled responses, as well as many “flavors” of good data. By throwing all this different data at our objects we have ‘driven’ much of the code that handles corner cases.

Here is a look at the Models in our application.

We can easily tell that our models have very good coverage - and this recently helped us a ton when we refactored our back-end API json structure. Since the models contain the logic to fetch and parse the json from the server, upgrading the application to use this new API was relatively simple: create new json fixtures from the new API, run the tests against the new fixtures, watch as most of the tests fail, update the parser code until the tests pass and shazam, the vast majority of the app “just worked”. Without these unit tests, it would have taken much longer to make this change.

The system we use allows us to dive down even further - to check the actual line-by-line coverage.

Adding Code Coverage Reports

There are a number of different tools that can generate code coverage reports. On our project we are using istanbul, integrated with jasmine using a template mix-in for grunt-contrib-jasmine. Istanbul can also be used with the intern, and karma test runners. If you are using Mocha, check out Blanket.js.

If you are just getting into unit testing your javascript code, this is kinda the deep-end of the pool - so I’d recommend checking out jasmine or mocha, and get the general flow of js unit testing going, look at automating things with a runner like karma or jasmine, and then look at adding coverage reporting.

Hopefully this helps show the benefit of having code coverage as part of your testing infrastructure, and helps you ship great code!

Transitioning To Javascript

Mon, 03 Mar 2014 09:08:41 -0700

Over the past few weeks I’ve been getting quite a few people asking questions about transitioning to javascript - perhaps this recent post about Esri’s Roadmap for Web Developers has spurred more people into action - whatever the cause, I thought I’d share a few thoughts and links.

First off, now is a great time to get into javascript! jQuery has leveled the playing field across browsers, and the truly horrible versions of Internet Explorer are nearly in behind us. The community is exploding, and it seems every day there is some new exciting project in javascript.

Javascript Application Architecture

I’ve got some great news: over the last few years javascript has matured as a language and as a community. No longer are javascript applications “spaghetti” code by default, and cross-browser issues are much less common and painful than in the past. Myriad Model-View-Something frameworks exist to provide structure for your code, and if you’re doing anything more complex than “Hello World” I’d strongly recommend investing in learning one (or more).

I was going to list out a bunch of frameworks along with pros and cons, but then I remembered this video by Rob Conery titled “Javascript Infero”. I really like this talk as it compares 4 javascript frameworks - Knockout, Backbone, Angular, and Ember. Go ahead and click through and watch it now… I’ll wait here…

Additional Framework Thoughts

BackboneJS + Marionette

Backbone was the first of the client-side MV* frameworks that really took off. It’s also barely a framework - very un-opinionated, thus allowing you do to virtually anything. Marionette is a backbone extension that helps developers implement additional patterns by adding in formal Modules, Controllers, Layouts, Regions, various types of views, as well as an Application. Leveraging these greatly streamlines development both by reducing repetitive code and enforcing a development pattern. Personally I liked this stack because it give you lots of freedom, while still providing pattern guidance. Coming from ASP.NET / C# on the backend, this resonated with me. Last spring I did a 6 part series on building a mapping app using Marionette, that would be a good intro if Backbone + Marionette sound appealing.

For what it’s worth, this is what we used to build the ArcGIS for Open Data application, as it gave us the benefit of solid structural and architectural patterns, while still leaving us lots of flexibility to implement the interface behavior we wanted.

Polymer (aka Web Components aka The Future)

Polymer is a Google project that lets you build applications based on Web Components. The tricky bit is that Web Components is an emerging W3C standard, and no browsers have support for them yet. Undaunted, Polymer provides a set of polyfills (stop-gap code) that let you build and use web components today (IE10+ and evergreen browsers). This project just hit “alpha” in mid-February 2014, so it’s great for experiments, but I’d recommend staying away from this for production. That said, Web Components will be the future of the web, so it is worth getting a general understanding of the concepts. Another side note - both Ember and Angular are aligning themselves to slip-steam their view infrastructure into web components.

General Stuff You Should Know / Use

Underscore / Lo-Dash

Underscore is a utility belt of awesome stuff that should be part of javascript but is not (yet). Lo-dash is the mo-better-faster implementation of the same library (yeah competition!). Get to know one/both of these, as they will save you a ton of time and effort. What is really great is that these libraries are smart enough to use native implementations of functions when they are available in the running browser, so you can use the same “code” in your app and in down-level browsers it will use a javascript implementation, but in newer browsers it will use the underlying C++ implementation.

Bootstrap

Bootstrap is a css framework that allows you to create a “reasonable” web app in minutes - no wonder it’s the most popular front-end framework! Sensible defaults based on a responsive base means that you can throw markup into a file, and after only a few minutes reading the documentation, have a site that looks good on a 27inch iMac and on your phone.

What’s more - Bootstrap is so popular that when you want to level up, there are tutorials on creating themes, or you can skip that and drop two lattes worth of cash on a pre-made theme. Boom. Beautiful, and you don’t have to fight the css. Bootstrap also comes with a bunch of optional javascript helpers. Start by using them, and then as you transition up to using a framework like Angular/Ember/Backbone, you can switch over.

Getting Started…

In the famous words of Nike: Just Do It. Start something - anything. Throw it up on github. A great starting point for working with maps is the bootstrap-map-js repo. Slap up something simple. Then build something else.

Will it break? Yes. Will you have problems with Internet Explorer? Yes. Will you scream at your monitor and rue the day you learned how to spell javascript? Likely.

But honestly, that’s learning. You did that when learning Silverlight or Flex. And really, if you are going to work on the web, javascript needs to be your new best friend. Think of it like this - you could switch to native app development, and then you’d have to know Objective C for iOS, Java for Android, and C# for Windows Phone/8.

In comparison, javascript is not so bad!

Resources:

Checkout my previous blog post on Leveling up your Javascript for lots of links.

YouTube has ton’s of resources for Angular and Ember

Html5Rocks & Polymer-Project - lots of info on Polymer and Web Components

Telemetry Part 2: The Code

Wed, 12 Feb 2014 13:00:56 -0700

In part 1 we covered the three most common types of telemetry data we want to collect. In this part we will review how to actually implement tracking within your code.

For this example we are going to use Google Analytics. Simple, Free, and virtually ubiquitous. Of course you could send this information to another service or a custom back-end, but that’s beyond the scope of this discussion.

Google Analytics API

The three types of telemetry we want to track - page views, user actions and in-browser performance - all map very nicely to three calls in the Google Analytics API:

Although analytics.js gives you a means to send this information to the backend service, it’s not the sort of thing that we want littered all over our code. Thus…

Separation of Concerns

Before we start into the details, let’s talk application design for a moment. While you could simply litter the code with calls to the analytics API, that would make a mess, and be a huge pain should you want to change to use some other tracking system.

Instead, we want to centralize the tracking and storing of telemetry in a central service. The specifics depend on your framework (Backbone/Angular/Ember/Dojo/other) but you will likely have some sort of “global event bus”, or a core “Application” object which all elements of your application can access.

In our application, we are using Backbone and Marionette, and so we have added methods to our “Application” object, as that is passed into all the modules.

Tracking Page Views

Again, your application architecture will dictate where to attach these events, but in frameworks that have the concept of a router, that’s a good place to start. For all “navigation” actions in our application, follow the same pattern and centralize things onto an application method, specifically Application.navigate(). Super handy, because we can just drop in the page view logging in this one function as shown below:

The actual logPageView function is just a call to the analytics function.

As we mentioned in the last post, this is simply a means to track what the user has interacted with. So, any place you have DOM event handlers, we want to assign that action a name, and make a call to Application.logUiEvent() method.

Depending on the amount of “magic” your application framework comes with (I’m looking at you Ember and Angular) this may be more or less difficult. Even with Marionette in the mix, our Backbone based app is pretty un-magical. All DOM interaction happens in handlers, defined in Views. So, all we do is drop in calls to logUiEvent in these handlers. While this could be made even more elegant by overriding the backbone and marionette view event binding infrastructure, in the interest of keeping the code easy to understand, we opted to just add these calls. Here is an example from one of our views.

Tracking In-Browser Performance

This is the trickiest of the bunch. As we mentioned, we need two calls - one to setup the timer, and a second to indicate the event we were tracking has completed. Or failed. We also need to handle the case where it does not complete.

For this we created a timer object - “Took.js” - which grabs a time stamp when it’s instantiated, and calculates the duration until the stop() method is called. We also have two additional methods - store() and log().

Here is a jsbin that you can play with as well (obviously it won’t report to Google Analytics)

We also expose this via the Application object as two simple methods startTimer(name, category, label, maxDuration) and stopTimer(name).

Through our code we wrap the various blocks we want timing data on in these two calls. Before we show an example, this brings up another area where we have centralized things - xhr calls. Although Backbone has a dependency on jQuery, and we could use $.ajax or $.getJson anywhere in the application, we have decided to route all requests through a central location - again on our Application object.

Anyhow - here is an example of a call that fetches rows from a remote feature service.

Using Telemetry

At this point, our project has not gone live, so we just have telemetry from dev and test environments in Google Analytics. That said, having this setup well before launch has already helped us re-arrange some of our naming conventions, and validated some ideas about the types of reports we can get out of the system.

Since we know that javascript performance varies greatly between browsers (orders of magnitude between recent browsers and older versions of Internet Explorer), we really wanted to make sure we could segment our performance data by browser and version.

Turns out that segmenting the data like this is not “built-in”, but it’s not hard to setup. Basically you create new “segments” and in the UI for that choose “Technology”, and then Browser & Browser Version.

With this in place, we can now easily compare performance of specific actions between different browsers. NOTE: data in this screen shot is from development - the actual performance is *much* better in production where all the code is combined and minified.

Summary

We hope this helps you get started with telemetry in your javascript applications. This is extremely useful information to have, and now more than ever, it’s very easy to get. Happy coding!

Truth in Numbers: Telemetry for Javascript Apps

Mon, 10 Feb 2014 13:00:49 -0700

Designing an application is really a set of educated guesses about what features users will actually use. We do interviews, conduct user testing sessions, and use our existing understanding of the “ideal user”, but without data coming in from real users, we just don’t know. In this two part series we will show you how to get this sort of data.

Telemetry: The Raw Numbers

The idea here is simple: track user actions in your application and send that information (“telemetry”) to a service that will aggregate it. Most analytics packages do this, but they are oriented towards so called ‘normal’ web pages, and the information they provide for javascript heavy pages (aka apps) is very limited without doing some additional work.

Types of Telemetry

We are interested in three classes of telemetry data - Page Views, User Actions, and in-browser performance. In this post we will introduce these classes, give some ideas of what to track, and how to organize things. Next post we will review how to actually implement the tracking.

Page Views

While there are many “page view” tracking techniques, they usually revolve around full-page refreshes. Of course, modern javascript applications eschew the full-page refreshes in favor of a more immersive applications that manipulate the url via html5 push-state.

Put another way, a user may spend 40 minutes using your web app, but traditional “page view” measurement may only register a single “hit” - when the page first loads.

Thus, when building rich javascript application, we need to help out and manually collect this information as the user changes pages, or “context” within your application. While most “single-page applications” don’t have clear boundaries between “pages”, we can usually break things down into reasonable units. Most applications will have some “home” or landing view, a search view, one or more types of search results views, item detail views etc etc.

The main thing is to come up with a naming convention for these “pages” that is consistent and makes logical sense.

We will look at the details of how to integrate this sort of tracking in the next post, but if your application has a “router”, that is likely a good place to attach page view logging.

Being able to track User Action is the core to being able to tell which features are actually being used. Since you are instrumenting the code yourself, you can track virtually anything that raises an event.

In our case, we want to measure the percentage of users change the base map, the size of the map, and virtually every other interface interaction.

The really boils down to adding code into every DOM event handler in the app, so how we structure our telemetry helpers will be really important - we don’t want to have brittle code littered all over the app. Think about having a central “telemetry service” that is available to all views or DOM events in the application.

Having a good naming convention is even more important for this type of tracking since these will likely be added by more than one developer and a mish-mash of naming will make the telemetry data a mess to work with. On the upside, it’s pretty easy to tweak

In-Browser Performance

The third type of information we want to track is related to performance. For our team, we develop on Chrome Canary, on maxed out Retina Macbook Pros while using high-speed internet. Unfortunately not all our users will have such an optimized environment. Add the fact that we are supporting IE8/9/10/11, Chrome, Firefox, Safari and Opera, virtually the only way to get realistic performance information for all those platforms is to harvest it from real users.

The end-goal of course is to help improve the real-world performance of the application. But, before we start wildly poking around the code base tweaking things we think may be performance bottle-necks, we want to have the system instrumented so we know where the real bottle necks are, and that when we deploy changes, we really do see improved performance.

So - what do we want to time? Initially, for our project, we want to track basic page load times, network calls (xhr’s), and computationally intensive code blocks (client-side filtering). Some specific timers:

how long did it take to load the page and initialize the app?
how long did it take to initialize the map?
how long did it take to execute a search?
how long did it take to display a layer?
how long did it take to sort a table?
how long did it take to filter a table?

While page view and events are essentially single calls, tracking timing requires two actions - one to start a timer, and a second to stop it and record the duration. Once again, having sensible, consistent naming is really helpful.

In the second part of this post, I will talk about how to integrate telemetry into an application.

Radio Telescope photo modified from Stephen Hanafin’s Flickr stream. cc by-sa.

Working Around Min/Max Scale

Mon, 03 Feb 2014 08:00:55 -0700

Another quick trick when using the Esri Javascript API. If you run into a scenario where the service you are accessing has min/max scales applied, but you need the data outside that scale range, here is a trick that can help out.

Before we get to the how, there are a few things you should know about this trick:

Data Holes: When Max Record Count Bites Back

Even with the gridded queries that dynamic mode feature layers use, at small scales (zoomed way out) it is common that you will be requesting more than the max record count number of features. When this happens, you get “holes” in the data returned, as shown below.

The default for max record count is usually 1000 features, but I think the latest release bumps this up to 2000. Regardless of the default, this value can be changed as part of the server configuration, so unless you control the server, it’s not something you can change.

Performance May Suffer

Scale ranges are commonly used to avoid sending very detailed or dense data over the wire. So, even if the layer you are working with only has a few dozen features, if they are really detailed geometries, things may get really slow, so perhaps reconsider.

How To

It’s actually really easy. Create a feature layer, then in it’s “load” event, reset the min/max scale properties. Then add it to the map.

Here is a link to a JSBIN you can play with.

The map below shows a feature layer from a demo server that has a minScale of 100,000 shown on a map that is at 1:36,978,595

[Another quick trick when using the Esri Javascript API. If you run into a scenario where the service you are accessing has min/max scales applied, but you need the data outside that scale range, here is a trick that can help out.

Before we get to the how, there are a few things you should know about this trick:

Data Holes: When Max Record Count Bites Back

Performance May Suffer

How To

It’s actually really easy. Create a feature layer, then in it’s “load” event, reset the min/max scale properties. Then add it to the map.

Here is a link to a JSBIN you can play with.

The map below shows a feature layer from a demo server that has a minScale of 100,000 shown on a map that is at 1:36,978,595

](http://jsbin.com/EXEwADO/6/embed?output){.jsbin-embed}

And of course you can use the same technique with MapService layers. Shown below is a layer that has a maxScale of 1,000,000 and we are zoomed in well past that.

MapService as FeatureService{.jsbin-embed}

Feature Layers from Map Services

Sun, 26 Jan 2014 22:04:34 -0700

Here is a lesser known fact: using the ArcGIS Javascript API, you can create a FeatureLayer from any vector layer in a MapService. Of course it will be read-only, but you still have all the usual control over it in terms of styling and interaction. If that works for you, it’s really simple: just drop the full url to the layer in the map, into the FeatureLayer constructor, and you’re up an running.

Of course you should be careful with this technique. Many times MapServices are used to display very dense data, so you may end up pulling a lot of data over the wire. But, if you happen to need more interactivity from a layer that’s already published as a MapService, this is a great way to avoid having to publish a FeatureService.

Here is a JSBin with the example running.

MapService Layer as FeatureLayer{.jsbin-embed}

Looking Ahead: Javascript in 2014

Thu, 09 Jan 2014 12:02:30 -0700

Ah 2013, what a great year you were – returning to full-time development has been a fantastic change of pace, and I’m super excited about what the next year will bring. In that spirit, here are a few things that the Esri DC team is looking forward to investigating.

Developer Tooling & Automation

This year our team is looking at adding Yeoman and Bower into the mix. We’ve been using grunt since we started our project, and I can’t imagine doing any front-end development with out it. Yeoman and Bower are companion tools that bring more awesome workflow options for front-end developers. Yeoman focuses on scaffolding out applications or components for you (think Rails or ASP.NET MVC scaffolding but for your javascript). Bower is a dependency management tool which makes it easy to pull down all the front end libraries you use, and easily update them over time. Grunt of course is a javascript task runner that can orchestrate things, and you should be using it. Or gulp.js, which is a “streaming build system”.

Web Frameworks

Although our team, and our primary project uses Ruby on Rails for a back-end web framework, as javascript developers, we can’t ignore all the activity around Express.js &Sails.js.

Personally, in switching over to OSX, I’ve been feeling a little off the back in terms of full-stack chops, so I’ve been hacking around with building simple web apps using Express, which is a back-end framework built on node.js. Express is the most mature of the node.js “frameworks” and thus the documentation is pretty good, and there are good intro and advanced books available. There are also a lot of modules that can be used to extend express.

Taking it up a level, Sails.js is a set of Express middleware that adds “stronger opinions” to Express. It is crazy quick to get started, and makes building out single-page style apps, and APIs a breeze. It’s similar to Rails in that it comes with scaffolding tools – so it’s a simple command to whip up a model and controller at the same time, and have that auto-magically become a REST and Socket.io based API. Again similar to Rails, it also has a plug-able data store model, allowing you to switch between database technologies as your project progresses. Sweet. That said, development is on-going and documentation is limited, so expect to fall back on some Express chops when you start digging into things. From a team perspective, Chris Helm (@chelm) has built the Koop project on top of Sails, so stay tuned for some interesting things coming on that front.

Front-End Javascript

If you are paying any amount of attention to the front-end web development space, you’ve likely heard people raving about Angular JS. While I am a big fan of Backbone & Marionette, the more I read about Angular, the more interested I am. Backbone is great when you want “extreme control” of the user experience (as is the case in our project), but in other scenarios, frameworks like Angular are a big win, as they can off-load a lot of work from the developer. On top of that, Angular seems to be able to play-nice with server-rendered markup, which is a big plus for building apps with really fast initial page-load times. Finally, being built by Google, using solid software engineering principles (dependency injection anyone?), with an eye towards our web-componety future (see below), I’d expect Angular to be relevant for quite some time.

I also found this very interesting post about Backbone and Angular . Not sure I completely agree with everything in there, but if you use backbone, it’s worth reading.

Speaking of looking forward over the next few years, the proposed web components standard is extremely interesting, and will be a huge shift in front-end development. At it’s core are 4 technologies (Templates, Shadow DOM, Custom Elements and Packaging) which combined will allow developers to create re-usable “components” for applications. What’s this boils down to is that we will be able to create new html tags and bake in behavior – just like built-in UI components like the

Thanks to Google, this future is available today. Chrome Canary has support for many of the underlying technologies, and the Polymer Project is a set of polyfills that allow web components to be used on all modern browsers (IE10+)

To learn more about Web Components…

If nothing else, this year will be a ton of fun as we all try to build the next awesome thing.

DaveBouwman.com

Demystifying oAuth and ArcGIS

What is oAuth?

Three Components

Identity

Privileges

Revokability

ArcGIS oAuth

The Steps

Application Boots

User Clicks Sign-In button

User Provides Credentials to ArcGIS Login Screen

User Submits Credentials, and ArcGIS Online Approves access

Secret Sauce…

The Code

Features

Step 1: Registering an Application with ArcGIS Online

Step 2: Clone the Repo and Install Dependencies or View the Demo App

Step 3: Page Scaffolding

Step 4: Javascipt!

Redirect.html

Ember Engines: Helper Not Found

Ripple Effects

DevSummit and DevMeetup Videos

Speaker Info

DevMeetup Video

Related Posts

Chasing Numbers: Pragmatic Javascript Code Coverage

Coverage Measures

Summary Metrics

Detailed Metrics

Adding Code Coverage Reports

Transitioning To Javascript

Javascript Application Architecture

Additional Framework Thoughts

BackboneJS + Marionette

Polymer (aka Web Components aka The Future)

General Stuff You Should Know / Use

Underscore / Lo-Dash

Bootstrap

Getting Started…

Resources:

Telemetry Part 2: The Code

Google Analytics API

Separation of Concerns

Tracking Page Views

Tracking User Actions

Tracking In-Browser Performance

Using Telemetry

Summary

Truth in Numbers: Telemetry for Javascript Apps

Telemetry: The Raw Numbers

Types of Telemetry

Page Views

User Actions

In-Browser Performance

Working Around Min/Max Scale

Data Holes: When Max Record Count Bites Back

Performance May Suffer

How To

Data Holes: When Max Record Count Bites Back

Performance May Suffer

How To

Feature Layers from Map Services

Looking Ahead: Javascript in 2014

Developer Tooling & Automation

Web Frameworks

Front-End Javascript