Looking at the comments in James Fee’s post summarizing Doron Yaakobi’s feelings about the ArcGIS Server ADF, this seems to have struck a chord with many other people building solutions with ArcGIS Server.

The question I have is – will we do anything about it?

By that I don’t mean writing more angry blog postings, canceling our EDN subscriptions or other symbolic, by really meaningless actions. I mean lets write the APIs we want to have. This is akin to the Alt.Net movement (if you can call it a “movement”). We’re not saying we’re ditching ESRI for GeoServer and uDIG, but we are going to play the game by our rules. We will use ArcGIS Server for what it’s good for, and skip the rest. If we don’t like the ADF, lets cook up something better.

To show that this is not all talk, shortly I will be releasing a much more solid version of the ArcGIS Server Adaptive Virtual Earth Tile Cache I posted about last week. I’ll post more about it, but suffice to say it’s very flexible and extensible. It will be put up in a public Subversion repository on Assembla for all to have and play with.

They key thing is working together on this stuff – back when  ArcIMS 9.0 shipped and ESRI did a half baked job with the ActiveX/COM/not-dot-net connector, dozens of people went out and wrote their own .NET ArcIMS connectors. How many 1000’s of hours were wasted with people re-doing that same stuff? Wouldn’t hindsight suggest that we not all roll our own “Not-ADF” solutions?

So – if we want a REST API, lets built it. There’s not really a whole lot to it if it’s designed correctly. Heck if nothing else we can use what has been done on FeatureServer.org as a road map. Suppose we cook up a server object extension (SOE) that can do all the low-level stuff in raw ArcObjects. Then we write a httphandler that can take the requests, parse the Urls and make calls back to the SOE. If we but the JSON/GEORSS/KML/whatever conversion in the SOE, we’re very light-weight on the DCOM stuff, and can get away with little to no ADF. And if we run this all directly on the ArcGIS Server box, we’re good in terms of licensing.

Once we have that out of the way, then integrating with OpenLayers or Virtual Earth just got a lot easier. Sure it’s not full featured editing, but who cares? We can leave that to the ADF.

So how about it? What to see something change? Want a REST API before September? Want to be able to fix it if something does not work?

If so, drop me a message (contact link above) and for those attending the Developer Summit, lets plan on having a meeting (over beers) to get things rolling.

We just ran into a issue with the ADF – namely that the ADF Geometry model does not support curves (aka "True Arcs"). Instead, it just puked up an exception stating that "curves are not supported". After doing some searching we found this thread in the forums, and this little gem stood out…

I just want to add one more thing to what Kevin said: the SP3 will handle the exception gracefully when identifying on a true arc (i.e. a message stating "Curves are not supported") instead of throwing an unhandled exception. The goal is to get this functionality fully working at 9.3.

I take this to mean that we should be grateful we ran into this now and not pre-SP3 when we would have gotten a more generic exception. Nice.

Obviously we are not the first people to run across this. There is a write up about the issue in the Knowledge Base: Bug:  Web Application Developer Framework (ADF) does not support non-linear geometry (curves)

As I noted above, it seems that the issue is that the Web ADF geometry classes can’t handle curves. Fine – I can understand that it may be difficult to work with curves. Maybe implementing real curves will need to wait for v9.3 or v10 or whatever, but how about this – if a curve is sent into the Web ADF geometry how about you densify it for me? It may not be as "accurate", but "close" is a whole lot better than an exception – even if it has a nice message.

The thing is that this is the suggested solution. If you have curves, don’t use the IQueryFunctionality. Instead use the IMapServer.Identify and IMapServer.Query* methods. Loop over the returned features, check if the geometries are curves, if so then densify the geometries before stuffing them into a graphics layer.

Of course it’s nontrivial if you’re not into the ADF in a big way, and you are hosed if you are not using a local ArcGIS Server data source, but hey what do you expect.

ESRI has put some code up on ArcScripts that shows how to deal with this for the Identify tool, but you are on your own for anything else. Fun Fun Fun.

Today I needed to install SQL 2000 on one of our servers. MSDN subscriptions no longer ship out media for SQL 2000 so I grabbed the ISO image from MSDN. All well and good, but I needed to mount this as a drive on my server which is running Windows Server 2003 x64.

Windows Server cannot mount ISO’s on it’s own, and although there are lots of commercial products, this is not something I do a lot, so "free" was my price point.What I ended up using is called MagicISO, and it works really nicely.

Regardless of how you manage your software development, at the end of the day it’s really all about communication. Communications in a software development project are typically chain-like. The User tells an analyst what they want, analyst translates into analyst-speak and writes this down. This is passed to a system architect who re-interprets the document, and creates a design and a document in architect-speak. The requirements and design are then passed to the developers, who re-interpret both and then implement their understanding of what the user wants in code. This sounds well and good, but it only works if extreme care is taken at every link in the chain. Recall the game of “telephone”…

Although typically a game for grade school kids, “telephone” showed us early on that communication chains are fraught with peril. If the message started out “Today’s forecast is calls for snow”, at the end of the circle, it would be “Tookeys feet are slow”. This was quite funny back then, but we seem to lose this valuable lession as we age. We see this same thing can happen with software requirements – and then it’s not quite as funny. (Besides – how can I code slow feet?)

Agile seeks to reduce the bulk of the “traditional chain” by having the team (developers + designer + analyst) collaborate with the end users to create a workable shared vision of the product. The key thing here is ensuring that the collaboration actually occurs. Too often the users are busy doing their “real” jobs, and the team is “pretty sure” they know what to build. But if the collaboration is short changed, then one of the major benefits of agile falls out.

Recently our team finished just such a sprint. The client was very busy, and the team was pretty sure we knew what they needed. During the sprint review with the client, some of the functions we had coded missed the mark. When we went back and reviewed the requirements, what we had developed seemed a reasonable solution. The lack of collaboration had bit us. We did not have the shared vision – just a requirements document that was written by a different consultant.

Luckily we are doing two-week sprints, so we were only a little off course, and should have no trouble getting things back on track. To address this issue, we have decided to adapt our Sprint Planning meeting to include a more formal review of our understanding of what each user story entails. Prior to this meeting the team will review the top priority user stories, the associated requirements, and the existing application, and determine what they believe the “shared vision” should be. Then at the meeting we will review this with the users and they can correct or add anything we had missed.  I’ll let you know how this works out.

I keep getting more requests for the code, so I’m just going to put it up here. I sent the code out to a bunch of people over the weekend, and I thought I should put the same disclosure here…

As I said in the posting – this is really hacked up, pure "can it work" research code – it’s based on Rob Blackwell’s VE WMS Tile Server sample, with some “create a map” stuff from the AGS SOAP documentation added, and a few lines of projection stuff smashed in.

Known Issues:

Currently the projection is not exactly right, the caching stuff is just plain ugly, and requires that ASPNET (or whatever identity you run the handler as) have write access to the hard coded path. All the tiles are stored in the same folder, which clearly won’t scale. The Handler matches *.png, which means you need to map that extension to the aspnet_isapi.dll for the web application you drop this into. The SOAP end point is poorly named. And there are no comments of any use.

So – I think that about covers it – you have been warned! That said have fun with it, and let me know how it goes.

Download

So I’ve been playing more with the AGS SOAP API, and have found a few things…

Generating Proxy Classes

While you could pass Xml back and forth, most people will opt to use some kind of proxy classes. Visual Studio will generate these for you when you add a web reference, and this is a good way to get up and running quickly.

There is one downside however – it seems that the WSDL returned by ArcGIS Server for any particular service injects the service name into itself. So if you were adding a web reference to http://myserver/arcgis/services/mytestService/MapServer?wsdl then you’d get a class that was named mytestService_mapserver. While you can change the actual service Url to point to another Map Service, you’ll still have  “mytestService_mapserver” littered all over your code. To me this is ugly and a code smell if you are trying to write something which is not bound to a single service. I guess the idea is that it helps a developer figure out which web service they are talking to, and I suppose that if you needed to write an app that used multiple connections to multiple mapservices, then it may *help*, but I doubt it. Any insight on why this was done would be great.

Creating Manual Proxies: wsdl.exe

In addition to simply using Add Web Reference, you can take a more manual route and use wsdl.exe to create the proxies. The wsdl.exe tool that ships with Visual Studio 2005 and it offers a little more control. The syntax is pretty simple – here’s what I used:

wsdl.exe /nologo /l:CS /n:ArcDeveloper.AGS.SOAP.Proxies /o:c:\wsdl\proxies /urlkey:MapServerUrl MapServer.wsdl

So this creates a C# proxy class for the MapServer.wsdl file, in c:\wsdl\proxies, in the ArcDeveloper.AGS.SOAP.Proxies namespace.

However, if you run this against the WSDL you download from the MapServer WSDL page, you’ll get an error:

Warning: This web reference does not conform to WS-I Basic Profile v1.1.
R2028, R2029: A DESCRIPTION using the WSDL namespace and the WSDL SOAP binding namespace MUST be valid according to the XML Schemas found at http://schemas.xmlsoap.org/wsdl/2003-02-11.xsd and http://schemas.xmlsoap.org/wsdl/soap/2003-02-11.xsd.
  -  Warning: The ‘name’ attribute is invalid – The value ‘#NAME#’ is invalid according to its datatype ‘http://www.w3.org/2001/XMLSchema:NCName’ – The ‘#’ character, hexadecimal value 0×23, cannot be included in a name. Line 2465, position  11.
  -  Warning: The identity constraint ‘http://schemas.xmlsoap.org/wsdl/:service’  validation has failed. Either a key is missing or the existing key has an empty  node. Line 2465, position 3.

For more details on the WS-I Basic Profile v1.1, see the specification at http://www.ws-i.org/Profiles/BasicProfile-1.1.html.

Writing file ‘c:\wsdl\NAME.cs’.

The errors are not critical, since a file still gets written, but now the proxy class is named “NAME”, which is worse than the <mapservicename>_mapservice that we got using Add Web Reference. If we look at the wsdl file, the culprit is in the last few lines of the file – we see this:

<service name="#NAME#">
  <port name="MapServerPort" binding="e:MapServerBinding">
   <soap:address location="#URL#"/>
  </port>
 </service>

I’m betting that the #NAME# and #URL# are tokens which get replaced when you get the WSDL from a live service (i.e.  http://myserver/arcgis/services/mytestService/MapServer?wsdl ). So – the fix is pretty simple – just edit your copy of the wsdl file, and assign the name to whatever you want the proxy class to be called. In my case, I set the name to “MapServiceProxy” and the Url to http://localhost/use/app.settings to remind myself to always configure the url in the app settings (the “key” is part of the wsdl command line –> /urlkey)

Note: I think you can use Add Web Reference and just point to your own local “fixed” copy of the WSDL file, and this will produce nicely named proxy classes – I did not test this since I’d already baked my own using wsdl.exe

Anyhow – I’ve created a set of proxy classes (VB.NET and C#), and a batch file for cooking them yourself. Grab it here: ags-proxy.zip  [UPDATE - link now works]

In early January Chris Spagnuolo and I recorded a podcast with Adena Schutzberg of Directions Magazine. The topic was "An Introduction to Agile".

One of the fun things about recording a podcast is that it’s very "off-the cuff". The title of this post comes from my comparison of "waterfall" with ordering of a hamburger at a fast food joint. Give it a listen and let me know if you think this analogy works. 

Around the same timeframe we did the podcast, Chris did an interview with Ron Exler at theGeoFactor.com – his article titled "CanGIS Be Agile?" went up on January 15th.

Just doing some research into pulling ArcGIS Server map services into Virtual Earth, and thought I’d share a some of what I’d found.

ArcGIS Server WMS Layer in Virtual Earth

Using the “Accessing WMS from Virtual Earth” article and code as a starting point, I was able to add an ArcGIS Server WMS layer pretty quickly. The only problem I had was divining the correct values for the WMS “LAYER” element – apparently the default AGS WMS service expects the LAYERS to be specified as numbers – 1, 2, 3 etc. Maybe this is standard, but I’m not a WMS guru by any means. Anyhow, the performance was good, but I will say that I’ve got some beefy servers – the AGS box is a Dual Quad Core box, with 8 GB RAM. The SDE box has one Quad Core. Neither have any appreciable load, so you’re mileage may vary.

ArcGIS Server SOAP API

WMS is nice, but I’m going to need to do some more crafty stuff – adding things into graphics layers, drawing selections etc. This would normally mean using the ADF, but I want to be able to spread this across a web farm with out spending a zillion bucks, so the  ArcGIS Server SOAP API seemed to be the ticket.

The API itself is pretty easy to use, and the example on the AGS SOAP API page was about all I needed to get started. Where things became problematic was with the projections. Morten has a great rundown of the VE/Google Earth projection stuff which was very helpful.

The WMS requests specify a SRS which tells ArcGIS Server how to project the map so that it will match up with the Virtual Earth tiles. Going at the SOAP API, you need to do this yourself.

Armed with the Well Known Text definition of the Geographic Lat/Long system used in the VE front end, I was able to set the SpatialReference for the zoom envelope as well as the map.

wsMapService.SpatialReference sr = new wsMapService.GeographicCoordinateSystem();
sr.WKT = “GEOGCS[see Morten's post...]“;
env.SpatialReference = sr;

I also found that I needed to tell the map to be projected into the same space…

mapdesc.SpatialReference = sr;

Update: The tiles are still a little off. Need to do more messing with projections…

Overall, I like the SOAP API – and until the REST API shows up at 9.3 whenever that ships, this is a pretty cool option.

Missing Tiles

Ok – with that out of the way I started to get some tiles showing up, but I also got some run-time exceptions. Running in debug mode, I found that sometimes the SOAP API would return a Url to a map image which did not exist. I added in some code which would catch this, and shoot back a “Missing Tile” image.

At this point the behavior is seems random – different tiles at different scales will draw. When I refreshed the page, different tiles would be drawn. My initial thought was that I was overwhelming the Server Object Containers with requests.  I had been running with just two SOC’s, so I cranked it up to 10. Same results. And nothing in the logs. Next I thought that maybe it’s an issue of the Url being sent back before the image is written out – so I added a delay before fetching a tile. Still the same thing. Any insights on this would be great.

Solution: esriImageReturnMimeData

Since this was bugging me, I went in an changed the return type to esriImageReturnMimeData – which would return me a byte array containing the image. A little bit more hacking and it’s working just fine. Now, every tile draws every time. So – the issue seems to be AGS not writing out the files, but still sending back a url.

Adaptive Caching

Since I’m already throwing images around in the httpHandler, it was nothing to write the final images out to files, and put a quick check in up front to see if it exists before going out to AGS to create it.  Works great, but I need to add a little more logic so I don’t end up with one folder containing 50,000 files.

The Code

This is horrid evil messy hacked up “research” code that likely has substantial performance issues, high-cholesterol, security holes, and lice but if you really want it, shoot me an email (Contact link above) and I’ll zip it.

Just got the notice in via email – wow.

I may have to polish up that adaptive cache code!

Since leaving my former employer, I’ve gotten quite a few emails along the lines of "congratulations, I wish I could…"

The thing is anyone can make a major life change – all it takes is a few things:

1) Recognizing the status quo is not working for you, and

2) Get inspired and create a plan of action

I can’t help you much on the first one, but if you make it to #2, here are some things that can help.

Inspiration

Four Hour Work Week – Timothy Ferriss 

Getting down to four hours may be pushing it, but there are lots of great ideas in this book. It’s pretty focused on setting up a "hands off" business, but the other "zen" ideas are really good – mini-retirements, long international trips, becoming an expert etc.

TED Presentations

TED finds great speakers who talk about really cool stuff. These are inspiring – start with Hans Rosling – who would not want to work with/for him? What I think it so great about these presentations is that you see people who are genuinely excited about what they are doing. Given the general level of job dis-satisfaction today, it’s good to be able to identify real enthusiasm when you see it.

Free Prize Inside and The Big Moo (Seth Godin)

You could also add in all his other books, but these are two that I have read. They are both packed with good ideas – from how to stand out in an organization, selling your ideas, and marketing 2.0. It’s worth noting that if you are planning a change, reading about marketing is good – even if the only thing you market is yourself.

Presentation Zen Blog

Like it or not, Power Point is in your future. The difference between an average (i.e. painful) presentation and great presentation is easily seen, yet difficult to produce. This blog covers all things related to giving presentations from slide design to  speaking tips. It also has lots of links to great presentations.

Finally, when you need to vent, but can’t actually yell at your boss, head over to True Office Confessions and let it all out (anonymously). It’s also comforting to know that you are not alone.