Starting in February we have been working with the Wildlife Conservation Society to create a map viewer for their Global Avian Influenza site (www.gains.org). This updated viewer launched a few weeks ago, and I thought I’d share a little bit about the project.

About the Project

The project is pretty straightforward, with only a few top level user stories:

- Map the scientific data that has been collected as part of the GAINS program.

- Allow users to filter the displayed this data by :

country,date range,species, and influenza subtype.

Also facilitate this functionality via Url parameters.

- Functionality to be delivered as a DotNetNuke module

- It has to be fast.

Technologies:

We used Virtual Earth for the mapping canvas since WCS needed a high-quality global data set, and this platform is free for non-profits.

For the UI, we dove into the Dojo Toolkit. This ended up working really well, with the exception of the tree view. It was just a huge pain to get the Dojo Tree to work the way we wanted it to, and dropping in the YahooUI TreeView solved these issues in minutes.

On the back end, it’s we used ASP.NET. Since this was going to be a heavy Ajax site, we also leveraged the ASP.NET web services which are marked up so that they “speak” json (anyone know if there is a more graceful / concise name for this??? ASP.NET Script Services?). Behind that there is a SQL 2005 tabular database, ArcGIS Server and ArcSDE.

The main UI for the map can be seen below. The data points are clustered, and symbolized based on the match to the filter criteria.

Data Filtering

The points displayed on the map meet a set of criteria, which can be chosen on this UI. Not that the design is overly beautiful, but it’s a mix of Dojo and Microsoft Ajax controls – using the best parts of both. Actually applying this criteria against the database and retrieving the points was non-trivial, so I’m going to write up another post about how we got all that working.

External Data: Dynamic WMS Tile Caching

The module can be configured to pull in tile services from different WMS servers. This shows my favorite – the global poultry density map. If you are on the site, you can turn these layers on / off this from Map Data –> Manage Atlas Layers. Administrators have a UI to setup the WMS Services. Since mass poultry migrations are rare, most of this data is pretty static. Thus we dynamically cache the tiles on the WCS server so we get improved performance. We do see some offsets in the tiles at the lower zoom levels due to the variations between WGS84 and the Web Mercator projection of VE.

External Data: GeoJSON Features from ArcGIS Server 9.2

We also have the ArcDeveloper 9.2 REST API for ArcGIS Server 9.2 in the mix, pulling in flyway geometries. For species that have flyways, an extra item shows up in the identify tree. Here’s a link to a map showing Calidris alpina, which has flyways. Mouse over a feature, click “View Details” in the pop-up and then expand the tree until you see “Show Flyway”. This makes a JSONP call to the REST service running on a different server, grabs the geometries and drops them into the map. Since this species as multiple flyways, multiple polygons are drawn with various colors.

It’s been a great project to work on, and we are continuing to work with the Wildlife Conservation Society to add additional functionality into this application.

In this go round, we’ll look at the questions related to Design Patterns…

Design Patterns

The question was "When you hear the term ‘Design Patterns’ what comes to mind?".

A majority of people did get that GOF = Gang of Four = Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides the authors of the book "Design Patterns: Elements of Reusable Object-Oriented Software", which describes recurring solutions to common problems in software design. Thick and meaty, it’s highly regarded and worth reading if you have the time and inclination. Martin Fowler’s site also has heaps of good info, and you’re not chopping down trees to read it. However you slice it, using patters will make you a better developer, so dig in.

Pattern Adoption

I guess this one has a call to action for someone to setup a community site to document and evolve some geospatial patterns.

If people are interested, I can stand up a wiki at ArcDeveloper.net, but someone else will need to push this forward as I’ve got a full plate.

Model View Pattern Use

"Rate your use of Model View * Patterns" was the question.

For those not familiar with MV* – this refers to a whole family of patterns, related to the Model-View-Controller pattern (Model View Presenter, Supervising Controller, Passive View), and the main intent of these is to separate business logic, user interface and data communications. Use of these patterns are helpful/ critical in enabling robust unit testing. I’ll be writing more about patterns and unit testing in the coming months, but Martin Fowler has a (burly) overview of these patterns. You can also check out the ASP.NET MVC stuff to get the Microsoft take on this, or check out MonoRail, which is part of the Castle Project. And for a change of pace, you could check out Ruby on Rails which is rooted in this pattern.

Inversion of Control & Dependency Injection

Ok, this is really maxing out the pattern / architecture geek factor, but it’s worth asking simply because this is another one of those patterns which enable unit testing.

So, with nearly 75% not knowing what this is… succinctly put, Inversion of Control and Dependency Injection patterns allow you to decouple classes from each other. This allows you to test things independently, which is critical if you want to have good test coverage. I had a series of posts partially written about this while at Sanborn, but I did not get them all polished off before leaving, and since I don’t have access to the source code anymore, I’ll need to look for opportunities to show some of this. For what it’s worth, the ArcDeveloper.net projects both use the Castle Windsor IOC container.

Ok, I’ll leave it here for now. Next time I’ll finish it up with a look at Build Processes, Unit Testing, Refactoring, API Documentation and some choice comments.

So we needed to do some testing of sites on Safari on a Mac, and I asked around the company, and low it turns out that there was an orphan MacBook Pro floating about. After a quick trip via FedEx, this lithe beast is now sitting on my desk. Turns out that the power supply has a short in it, so it will be a few days before I fire it up and take furtive steps into the the heart of darkness.

Actually I’m pretty excited to see what all the fuss is about. With a dead battery and no juice, my take thus far is purely physical, and as everyone knows, Apple groks design. Why can’t my Toshiba be as solid, and well designed at this Mac? Who knows, but it sure feels nice. Once I get it fired up, we’ll see how OS10 treats me. I’m got some rusty old-skool unix tricks up my sleeves, but it’s going to be a little bumpy at first. Clearly there is something good going in with them though, as MacBooks were the rule for anyone with any sort of Neogeo cred at Where 2.0 . We’ll see if 10 years of Windows development has ruined me, or if I can get my Mac on. Stay tuned!

Newly refreshed after a week of vacation, I thought I’d start things up with the results of the informal "2008 Geospatial Developer Survey". I’ll start by thanking the 320 people who took the time to complete it. Admittedly it was a little rough in spots – turns out that writing good survey questions is tougher than you’d think!

Since there were 30 questions in the survey, I’m breaking this into a few posts. As promised, I will post the results as an excel file with the last post.

Who are you?

I thought I should start off the survey with an easy one – what’s your role.

The split across titles was somewhat interesting – almost even between "Analysts who write code", and those who consider themselves "GIS Software Developers", with "Software Engineer" trailing slightly. I see this as a progression from "getting things done" coding towards more "built to last" coding.

Primary Development Language

There were a lot of "other" responses on this one, mainly along the lines of "Mix of languages – VB.NET, VBScript, JavaScript, Python, and others".

The idea of this question was to get a feel for what you most commonly work with. I went through the "others" and tabulated them into the mix. Since there were so many variations in the "other" categories, I collapsed them into one group for the pie chart. Perhaps not surprisingly, .NET was the prominent platform with a 61% share.

Here is how the "other" broke out by count of times they were mentioned. I’m pretty surprised that there are still people using VB6 and VBA. While these tools get the job done, the lifecycle for VB6 apps must be coming to a close.

Who do you Develop For?

Almost a 50-50 split on internal vs external clients. Some people were asking why I asked this. I was interested in seeing any relationships between engineering practices and the client base – would there be more advanced software techniques used for the external clients vs internal? With a 50-50 split, it’s not entirely obvious, but I’ve since downloaded the data into a database and I’ll try to split out the data a little more to see how this breaks things down.

Use of ESRI GIS Products

This is another question that I got some flak for – but I think it’s relevant in that I’m interested in knowing what my readers are using. 40% of the respondents are only using ESRI. Actually I had thought this would be higher – the idea that 60% of respondents are using some mix of non-ESRI GIS tools is very interesting. I’m sure a lot of that is Autodesk stuff, but there’s got to be some MapServer, GeoServer, PostGIS and other open source stuff making an appearance in here. Next year I’ll change this one up so we can see the mix of packages being used.

Coding Standards

It’s nice to see that most respondents have some type of coding standards.

Mostly informal, but that’s a start, and if you have a solid team, it’s likely all you need. Those folks with out any type of coding standards should really consider setting up something – most languages have some sort of recommendations or conventions. The goal of coding standards is simply to ensure that everyone on your team is writing code in such a way that anyone else on the team can easily tell what’s going on. If nothing else – simply using clear names for your variables is a good starting point.

Development Life-Cycles

This was interesting, and will require more analysis, but here’s how I look at this. 50% of respondents classified themselves as Developers or Software engineers, and just about 50% also fell into Agile or Waterfall. 30% classified themselves as GIS Analysts who Code, and we see about 35% of cowboy coders.

I’ll cut things off here for now – next time I’m going to cover some of the more technical questions – design patterns, unit testing etc. I also need to get the data out of survey monkey in a database format so I can use SQL to tease more information out of the results.

My thoughts after Where 2008…

In my view, the "Where 2.0" movement/phenomena is about the democratization of mapping – remix this, mash in that, geocode it and get it on a map. My Map. With My Stuff and My Friends. Social graphs and all that. For many things, "close" really is good enough, and since it’s "your" data, validation is less of an issue ( did you "really" like that pub? ) . Usability and performance are huge driving factors because many of these sites rely on users "wanting" to participate. This is driving the rapid evolution in terms of user experience, and poorly designed or slow sites fade quickly.

Meanwhile back in "GIScience"-land, the hard-core old guard have railed against pushpins as too simple, in-accurate or just lame. Not sure why there is so much insecurity, but maybe it’s just a little envy that a bunch of javascript hackers made maps "cool". Many GIS applications have a "captive" audience (i.e. staff who have to use the site to do their job), usability has been low on the list, and performance an after though. The emphasis was placed on trying to cram desktop type work-flows and interaction models into a browser. Needless to say this has its drawbacks.

Not to use the John Hanke and Jack Dangermond session as too much of a metaphor, but these two camps seem to be coming together, or at least recognizing they both bring something valuable to the table. Sure there is still lots of misunderstanding, but things are getting better. Many of the "mash up" people I’ve talked to are realizing that the next step is to add spatial analysis. Not buttons & dials type analysis for the sake of it, but leveraged in context of answering a question. Sure the social graph is cool, but there are all sorts of new and interesting ways to leverage that when used in conjunction with spatial analytics. It opens lots of new scenarios. There is a realization that the scientifically backed, validated data has serious value in addition to "my data" or crowd-sourced data.

Going the other way, the GIS crowd (sadly under-represented at Where) is coming around to the idea that even their "captive" user base appreciates (or demand) a good user experience – design & performance matter. A lot more than they thought. And maybe it does make sense to dial back some of the "science" options in order to optimize performance. Some times "close" is good enough, and the performance trade off to get to "really really really close" is not always worth it.

This is a good thing and the whole industry / community will be better for it. GIScience sites/services that embrace the usability and performance of the neogeography sites will flourish if for no other reason than their users "like" using them. And if you’re a social site looking to stand out, adding a shot of analysis into the sauce may be the trick.

It’s been fun and I’m leaving with a head full of interesting ideas.

If you are presenting…

A parting word for anyone presenting at Where (or anything else) – Please please please read Presentation Zen (the book & site) before you show up.  I (and others I talked with) had really expected that these presentations would be on the level of TED – sadly this was the exception, not the norm. There were some big name companies up on stage with horrid slide decks – some were literally painful to view. Sure you have a cool 3 minute video that’s super slick, and you likely spend 500K on, but following that with a jumble of bullets on neon-green background with heinous clip art some what spoils the effect. If you have 15 minutes in front of a crowd of thought-leaders, maybe you should spend more than 15 minutes on your slides.

Since there many other people are blogging the details of Where, I’m just going to take notes about the ideas that resonate with me…

Make your maps permalink friendly: Everyblock.com has a Uri hierarchy (RESTy?) that makes sense, and allows deep-linking and deep-search.

Need to do more than points: deal with "areas" – neighborhoods, routes etc. Especially relevant for geo-locating events/stories, where push-pins over simplify. The situation.

Roll your own maps: make the map relevant for your users. Don’t show what you don’t need. Control the look & feel of the cartography and make it part of the site design.

Federated Geodata: (Sean Gorman) someone other than ESRI talking about breaking data out of the silos. Not much detail on the "how", or synchronizing updates, but maybe he’ll post more details.

GOOG & ESRI: Enabling data sharing / deep indexing. Good demos of ArcGIS Server analysis fronted by a consumer UI experience – geoprocessing in Google Earth. The "real" application were a change of pace – not that "my friend recommends the calamari at this place" isn’t useful, but real-time forest fire modeling / evacuation planning just has a little more "bite" to it. 

Overall, an interesting morning. The John Hanke and Jack Dangermond session certainly got people talking – and that’s always a good thing.  Should be interesting to see how many groups open up their ArcGIS Server 9.3 systems to Google’s indexing. Discovering data that’s been walled off for so long could be really interesting.

More later…

So the session was hampered by some corrupted data on the USB sticks, and was more a loosely guided tour of the stack than anything else. I had PostGIS up and running, but could not get my data (loaded into ArcSDE on PostgreSQL using the PostGIS datatype) to load into GeoServer. Tried some other data, and same problem. Unclear what the issue is.

Many of the sample SLD’s would not validate (hence could not be loaded), and I could never get the WMS group layers to work. I did get some data up in Google Earth, and open layers, but overall this session confirmed for me that the open source stack learning curve is steep from the get go. I need to get into OpenLayers as the demo’s were pretty cool but until I can get the back end all working smoothly it’s somewhat moot.

I think that the docs for this will be up somewhere. The software installers are below.

Downloads: http://files.opengeo.org/where2/stick/software

Grass roots re-mapping efforts, focused on Europe. Crowd-sourcing. Pretty nice cartography. See the maps here: http://www.openstreetmap.org/

Note the pubs shown on the map via the Pint glass on the map. Nice.

Has features not in other maps – foot paths (dashed red lines), Pubs etc.

Starting in the US based off Tiger data, and updated.

The Back-End Story

Simple data model. Nodes, Ways (lines, or polys if node1 = nodeN), Relations

Runs on MySQL. 30,000,000 records. All changes stored. No "geo" columns, just points. Use scaled integers instead of doubles, and some saucy indexes. "SDE like" tiled indexes.

Planet.osm – rendering format in Xml. ~4Gb. Released weekly. From there you usually load it into PostGres with a Geocolumn, and then push this out via Mapserver, or GeoServer, or subset as you need.

Site runs in Ruby on Rails. OpenLayers for the slippy map. Off-line and on-line editors. All stored in Subversion

API was RESTful from day 1. http://www.openstreetmap.org/api/0.5/node/45

Tags – What is it?

Every object has keys and values associated with it. Community process for selecting keys and values. Keys and Tags must be approved, or features will not be on the map. There are 100’s of tags.

Current Rendering

Mapnik (apparently easier and simpler than MapServer) c-based tool that uses a huge Xml file for styling (looks like SLD?). Runs off the Postgres version of the OSM data. Tiles are created on request, and cached. There is a background tile cacher daemon. Tile cutter is on one box, cranking like 20 million tiles a day.

Osmarender – Xml –> Xslt –> SVG. Slow to convert. Rendering is done across a network of end-user PCs. 

Tiles match the Google scheme, which makes mashups easy.

Now moving onto mobile platforms

OSM is topological from the start, so it supports routing. Currently there are some simple routing options, but until the data density is higher, it’s not too useful. High-performance routing is CPU intensive, and no plans to add that as a service.

Lots of other projects going on, so check it out over at openstreetmap.org.

Ok, hopes for a Bruce Lee javascript smack down were thwarted when Andrew started off with "this is Where 2.0 101".

Started out with a quick run over the common formats – GeoRSS, KML and Microformats.

Microformats – additional HTML markup that are conventions for geolocating content. geo & adr for lat/long and address.

High-speed review of REST with "spiffy diagram"

Must do some reading on Atom – maybe have lunch with local Atom fan-boy Sean Gilles

What is Mapstraction?

Javascript mapping API wrapper covering least common denominator functionality across 9 or so map platforms. Essentially a layer of indirection that will allow you to change the back-end (i.e. Google Maps to Poly9) without breaking your application.

It can do image overlays – which is nice because you can then push very dynamic data (i.e. data that can’t be tiled) into the map.

Nice tile layer schema for roll-your-own tile services

GeoRSS and geojson direct consumption.

Support Filtering by a range of things – category, date etc.

It does not break terms of service – it still uses the underlying API’s, so the tile providers are cool with using this.

Still allows you to go below Mapstraction to the underlying provider. So, you’d be able to write most of the site in Mapstraction, then jump down into some VE/GM specific stuff if you need to (obviously this limits how you can change between the back-ends)

OpenSearch: Expose the search interface for a website directly into a browser search. Details at opensearch.org. OpenSearch-Geo – extensions to support location strings. Supports location name, lat/lon + radius, bbox.

This is very cool – need to push this into some of our sites.

Mapstraction is BSD license so do what you like.

Thanks to Ron Bruder who left a comment on the previous post with the link directly to the documentation for using the PostGIS geometries in the geodatabase. I did have some weirdness trying to access the link – of course you’ll need to login because it’s still beta documentation, but even after, that I’d see the page, and then it would jump me back to the login screen. I was able to access it by saving the link to Html and then viewing it. Moving on…

Using PostGIS Spatial Format

The ArcSDE doc says that to store data in PostGIS format, you need to create the "database" (presumably the "sde" database) using the postgis template, then create the sde user and sde schema before running the post install. Ok. Since I’d already done this, I dropped the sde database, then re-created it as per the instructions and… yeah.

So the post install failed. It did handily suggest looking at some logs buried away in the sde folder tree, and low the log did have a handy little gem:

db_table_grant_access: Grantee public does not have USAGE permission on schema sde.

Ok. Back into Navicat, grant USAGE to Public on the sde schema, and the post installer is completed. While my end goal is to use Direct Connect, to start, I’m just going to run the SDE service until I know everything else is working.

The next step is to get ArcGIS to use the PostGIS format. For this you need to edit the "dbtune file". From past experience with ArcSDE on SQL Server, I thought that "dbtune" was now a table, and it’s a whole lot easier to edit a table than fiddle with the dbtune export/import mess. So, I simply updated the GEOMETRY_STORAGE parameter in the DEFAULTS section of the SDE_DBTUNE table to be "PG_GEOMETRY". So now I should be able to use ArcGIS Desktop to edit / manage data in PostGIS, while still accessing the data as native PostGIS format for say GeoServer and uDig.

Adding and Editing Data

I used the standard Import feature in ArcCatalog, and this worked smoothly. I then fired up uDig to see if it would show up a native PostGIS layer… yep… along with all the other tables in the sde schema. I think that there are likely better ways to organize your data, so this is likely not a "real" issue.

I had some issues applying a thematic renderer to the layer – for some reason it would only apply the renderer to the vectors. When I looked at the SLD xml, everything was setup as a <sld:LineSymbolizer>… since it looks fine in ArcMap (see below), and the original shapefile loads into uDig as polygons, this may be an "beta" issue. Ordinarily I’d check the beta forums right now, but I’m writing this at 38575ft en route to Where 2.0, so that will have to wait.

Loading data via ArcSDE into ArcMap works very smoothly.

And after telling ArcMap to let me edit "unversioned" data, I could edit as usual. uDig says I can’t edit the layer. Being really new to this whole stack, I’m not sure if it’s permissions (although I connect as the same identity in ArcMap), or a limitation (heck – I don’t even know if uDig can edit vanilla PostGIS layers!)

I also wanted to try a direct connection – I stopped the ArcSDE service. Worked just fine – note the "service" name.

Summary

I’ve got a few other things I want to mess with, but I’ve got the basics working. It’s been a few miss-steps but not too bad once I read the documentation. Although I’ve been using ArcSDE on SQL Server for a long time, it’s definitely worth reading the docs when setting this up. Clearly I’d need to do more digging on optimizing the database structure, permissions and other sundry tuning things, but if you are looking to trade some time learning PostgreSQL for the $$$ involved buying Oracle or SQL Server, it’s pretty compelling. I can’t say much about performance as this is just one small layer (US Counties) running on my notebook. I do know people who are running some big databases on PostgreSQL + PostGIS (like millions of spatial features in a layer) and they’ve got it blazing.