ESRI Canada Tech Trek Talk: Tools for the Agile Developer

Posted by Dave Bouwman | Posted in Uncategorized | Posted on 17-08-2008

4

Just a quick note to any ESRI Canada readers heading to the Tech Trek next week – Thursday afternoon I’ll be giving a workshop about Tools and Techniques for the Agile Developer – I’ll be talking about a wide range of things – from source control to  unit testing to refactoring to design patterns – basically a whole wack of good (free/open-source) stuff that you can use to make your software development projects more sane. This is a zero-to-sixty talk – it assumes very little coming in, and you’ll walk out with a whole lot of stuff you can use to get fully up to speed!

Hope to see you there!

Trouble Running MbUnit tests with TestDriven.net on Vista 64

Posted by Dave Bouwman | Posted in .NET, Devt Tools, Unit Testing | Posted on 05-08-2008

3

Ran into some odd behavior today – I had installed MbUnit and TestDriven.net a while back after re-paving my workstaion with Vista64, but had not used either since then (writing more proposals than code lately). I went to add some new tests to the ArcDeveloper Tile Server code-base and found that when I ran the test from the Solution Explorer context menu, they would not run.

sol-context

In fact I got this message in the output window:

—— Test started: Assembly: TileServer.Core.Tests.dll ——

The target type doesn’t contain tests from a known test framework or a ‘Main’ method.

Nice. So something it likely messed up in the install. But for grins I tried running a test from the code window context menu – and it worked.

code-context

So clearly something is way jacked up. I un-installed TestDriven.net and MbUnit, and re-installed. No dice. Un-installed ReSharper, re-installed TestDriven.net and MbUnit. Nada. Repeat with various other options. Zilch. Googling turned up nothing. Yeech – not good.

Then I recalled getting an email from Greg Littlehales when I posted about switching to Vista 64… something about TestDriven.net. A quick search in Gmail pulled up the answer – there is a bug in the TestDriven.Net installer which causes it to put some registry entries in the wrong spot. Here’s the url of the issue over in Google Code: http://code.google.com/p/mb-unit/issues/detail?id=270 

The response from Jeff Brown is…

The installer doesn't know how to create the right entries on x64 machines.
We'll be fixing this in the new WiX-based installer as part of an upcoming release.

In the interim, you can export the
HKLM/Software/Wow6432Node/MutantDesign/TestDriven.Net/TestRunners key and import its
contents back into HKLM/Software/MutantDesign/TestDriven.Net/TestRunners.

I fired up RegEdit, did the export, deleted all occurances of “Wow6432Node/” in the file and imported it back in. Once I did this, all was well in the world and I could go home (and hit Publish on this post!) So far this is the only issue I’ve run into with Vista (other than a client who’s VPN does not support Vista, so I run an XP VM to connect)

For others who may run into this, here’s the what I used – this assumes that everything was installed in default locations.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners]

[HKEY_LOCAL_MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\AdHoc]
@="40"
"AssemblyPath"="C:\\Program Files (x86)\\TestDriven.NET 2.0\\AdHoc\\TestDriven.AdHoc.dll"
"TypeName"="TestDriven.AdHoc.TestRunner.AdHocTestRunner"

[HKEY_LOCAL_MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\Custom]
@="0"
"AssemblyPath"="C:\\Program Files (x86)\\TestDriven.NET 2.0\\AdHoc\\TestDriven.AdHoc.dll"
"TypeName"="TestDriven.AdHoc.TestRunner.CustomTestRunner"
"TargetFrameworkAssemblyName"="TestDriven.Framework, Version=2.0.0.0"

[HKEY_LOCAL_MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\MbUnit]
@="10"
"AssemblyPath"="C:\\Program Files (x86)\\MbUnit\\bin\\MbUnit.AddIn.dll"
"TypeName"="MbUnit.AddIn.MbUnitTestRunner"
"TargetFrameworkAssemblyName"="MbUnit.Framework"
"Application"="C:\\Program Files (x86)\\MbUnit\\bin\\MbUnit.GUI.exe"

[HKEY_LOCAL_MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\NUnit 2.2]
@="10"
"AssemblyPath"="C:\\Program Files (x86)\\TestDriven.NET 2.0\\NUnit\\2.2\\nunit.addin.dll"
"TypeName"="NUnit.AddInRunner.NUnitTestRunner"
"TargetFrameworkAssemblyName"="nunit.framework, Version=0.0.0.0-2.2.65535.65535"
"Application"="C:\\Program Files (x86)\\TestDriven.NET 2.0\\NUnit\\2.2\\nunit-gui.exe"

[HKEY_LOCAL_MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\NUnit 2.4]
@="5"
"AssemblyPath"="C:\\Program Files (x86)\\TestDriven.NET 2.0\\NUnit\\2.4\\nunit.addin.dll"
"TypeName"="NUnit.AddInRunner.NUnitTestRunner"
"TargetFrameworkAssemblyName"="nunit.framework, Version=2.3.0.0-2.4.65535.65535"
"Application"="C:\\Program Files (x86)\\TestDriven.NET 2.0\\NUnit\\2.4\\nunit.exe"

[HKEY_LOCAL_MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\NUnit_VSTS]
@="20"
"AssemblyPath"="C:\\Program Files (x86)\\TestDriven.NET 2.0\\NUnit\\2.2\\nunit.addin.dll"
"TypeName"="NUnit.AddInRunner.NUnitTestRunner"
"TargetFrameworkAssemblyName"="Microsoft.VisualStudio.QualityTools.UnitTestFramework"

[HKEY_LOCAL_MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\VisualStudioTestTools]
"AssemblyPath"="C:\\Program Files (x86)\\TestDriven.NET 2.0\\VisualStudioTestTools\\TestDriven.VisualStudioTestTools.dll"
"TypeName"="TestDriven.VisualStudioTestTools.VsttTestRunner"
"TargetFrameworkAssemblyName"="Microsoft.VisualStudio.QualityTools.UnitTestFramework"

[HKEY_LOCAL_MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\xunit]
@="4"
"AssemblyPath"="F:\\ArcDeveloper\\tileserver\\trunk\\Third Party\\XUnit\\xunitext.runner.tdnet.dll"
"TypeName"="XunitExt.Runner.TdNet.TdNetRunner"

Usability & Installing Pre-9.3 Direct Connect

Posted by Dave Bouwman | Posted in ArcMap, ArcSDE, SQL Server | Posted on 04-08-2008

4

First, a little background…

The current guidance from ESRI is to use “Direct Connect” to ArcSDE databases so that you don’t have to license all the cores on your DBMS box. This is great, and can be a cost savings if you’ve got a dual quad box, but only a single ArcGIS Server license (4 cores). If you are in a huge organization and run Oracle on some monster Sun box with 32 CPU’s you’re looking at a HUGE savings. So, a lot of people are happily running with Direct Connect. Great.

Now the Usability bit…

Last week I threw 9.3 ArcEditor on my workstation, and went to open some data…

Desktop-Fail

Based on the error message, and a little digging around I concluded that I needed to setup ArcSDE Services in order to connect. This is a non-trivial pain because I have a lot of spatially enabled databases. I ended up writing a bunch of SQL code that writes out the SDESERVICE command-line code to set this up (lower down in this post for those interested).

Then, on a flight over to Florida last Wednesday I was installing Desktop 9.3 on my notebook, and what did I see…

pre-93

Yep – the Pre-9.3 Geodatabase Direct Connect is an option on the installer splash screen. When I installed desktop on my Workstation I browsed directly to the Desktop installer and launched that (we keep the installers on a network drive). Thus I never saw this screen, and wasted whole wack of time writing the SQL Script etc. So this is great news, but why is this stuck 5 items below the ArcEditor install link? And ever better, why is it not an option INSIDE the Desktop installer package? It’s great that this is available, but burying it in a long list of stuff I typically ignore is pretty “meh” in my book. When I ran the installer it told me that the drivers for 9.0, 9.1 and 9.2 would take up a whole 91Mb – when you are looking at close to 2 GB for ArcGIS Desktop, WHY NOT JUST INSTALL THIS?

And the error message shown above fully deserves a “fail” from a usability standpoint. How does that message help the average user? And repeating it 4 times does not help. Would it be too difficult to change this to something useful like:

You are trying to create a “direct connection” to a previous version of ArcSDE. At this time, the pre-9.3 Geodatabase Direct Connect option has not been installed on this computer. Use the ArcGIS Desktop installer to add this component.

Anyhow – once installed, it works like a charm. Thanks.


For those interested in the more brute force option, and a little run down on using SQL to write SQL or command line scripts, read on…

We’ll use SQL to write out a bunch of scripts. Specifically:

1) Add SDE user to each database that contains the SDE metadata tables

2) Grant the SDE user correct permissions in the database

3) Create sdeservice command line(s)

4) Create command line scripts that will stuff service/port mappings into the services.sde and windows services files

 

Note: Before running this you will need to setup an SDE login to your SQL 2005 instance.

Here’s the SQL:

-- Make sure we are running in master...
use master
go

---make sure temp table is not present
IF object_id('tempdb..#SDEDatabases') IS NOT NULL
BEGIN
    DROP TABLE #SDEDatabases;
END

-- Create the tempory table
CREATE TABLE #SDEDatabases
(
    DatabaseName VARCHAR(255) NULL,
    Port int
)
GO
set nocount on

DECLARE @DBNAME varchar(50)
DECLARE @CMD AS VARCHAR(500)
DECLARE @PORT as Int
--Create a cursor on the sys.databases table
DECLARE dbCur CURSOR FOR
select [name] from sys.databases

-- open the cursor
OPEN dbCur
set @PORT = 5151
FETCH NEXT FROM dbCur INTO @DBNAME

WHILE @@FETCH_STATUS = 0
BEGIN
    Set @CMD = 'select ''' + @DBNAME + ''', ' + CAST(@PORT as varchar(4)) + ' from ' + @DBNAME +'.dbo.sysobjects where type=''U'' and name = ''SDE_Version'''
    set @PORT = @PORT + 1
    INSERT INTO #SDEDatabases EXEC(@CMD)
    FETCH NEXT FROM dbCur INTO @DBNAME
END

CLOSE dbCur;
DEALLOCATE dbCur;

--Create the SQL & commandline scripts
select 'use ' + DatabaseName + '  CREATE USER [sde] FOR LOGIN [sde] WITH DEFAULT_SCHEMA=[dbo]' from #SDEDatabases
select 'use ' + DatabaseName + '  grant CREATE PROCEDURE, create table, create view, create function, delete to [sde]' from #SDEDatabases
select 'sdeservice -o create -d SQLSERVER -p YOURPWD -i sde' + DatabaseName + '   -H C:\arcgis\ArcSDE\sqlexe ' from #SDEDatabases
select 'sdeservice -o register -d SQLSERVER -r ADMIN_DATABASE -v ' + DatabaseName + ' -i sde' + DatabaseName + ' -p dTs1dEv' from #SDEDatabases
select 'sdeservice -o delete -i sde' + DatabaseName + ' -d SQLSERVER'  from #SDEDatabases
select 'echo sde' + DatabaseName + '          ' +CAST(PORT as varchar(4)) + '/tcp  #ArcSDE for SQL Server >> C:\arcgis\ArcSDE\sqlexe\etc\services.sde' from #SDEDatabases
select 'echo sde' + DatabaseName + '          ' +CAST(PORT as varchar(4)) + '/tcp  #ArcSDE for SQL Server >> %windir%\system32\drivers\etc\services' from #SDEDatabases

-- clean up the temp table
drop table #SDEDatabases

Run this and specify results to text in your SQL tool. This will output something like this:

 

use CDOTSDE  CREATE USER [sde] FOR LOGIN [sde] WITH DEFAULT_SCHEMA=[dbo]
use TIMSSDE  CREATE USER [sde] FOR LOGIN [sde] WITH DEFAULT_SCHEMA=[dbo]
use ArcData  CREATE USER [sde] FOR LOGIN [sde] WITH DEFAULT_SCHEMA=[dbo]
use Gains_Spatial  CREATE USER [sde] FOR LOGIN [sde] WITH DEFAULT_SCHEMA=[dbo]
use sdeblank  CREATE USER [sde] FOR LOGIN [sde] WITH DEFAULT_SCHEMA=[dbo]
use TexasFire  CREATE USER [sde] FOR LOGIN [sde] WITH DEFAULT_SCHEMA=[dbo]
use WindSpatial  CREATE USER [sde] FOR LOGIN [sde] WITH DEFAULT_SCHEMA=[dbo]

use CDOTSDE  grant CREATE PROCEDURE, create table, create view, create function, delete to [sde]
use TIMSSDE  grant CREATE PROCEDURE, create table, create view, create function, delete to [sde]
use ArcData  grant CREATE PROCEDURE, create table, create view, create function, delete to [sde]
use Gains_Spatial  grant CREATE PROCEDURE, create table, create view, create function, delete to [sde]
use sdeblank  grant CREATE PROCEDURE, create table, create view, create function, delete to [sde]
use TexasFire  grant CREATE PROCEDURE, create table, create view, create function, delete to [sde]
use WindSpatial  grant CREATE PROCEDURE, create table, create view, create function, delete to [sde]

sdeservice -o
create -d SQLSERVER -p YOURPWD  -i sdeCDOTSDE   -H C:\arcgis\ArcSDE\sqlexe
sdes
ervice -o create -d SQLSERVER -p YOURPWD  -i sdeTIMSSDE   -H C:\arcgis\ArcSDE\sqlexe
sdeservice -o create -d SQLSERVER -p YOURPWD  -i sdeArcData   -H C:\arcgis\ArcSDE\sqlexe
sdeservice -o create -d SQLSERVER -p YOURPWD  -i sdeGains_Spatial   -H C:\arcgis\ArcSDE\sqlexe
sdeservice -o create -d SQLSERVER -p YOURPWD  -i sdesdeblank   -H C:\arcgis\ArcSDE\sqlexe
sdeservice -o create -d SQLSERVER -p YOURPWD  -i sdeTexasFire   -H C:\arcgis\ArcSDE\sqlexe
sdeservice -o create -d SQLSERVER -p YOURPWD  -i sdeWindSpatial   -H C:\arcgis\ArcSDE\sqlexe

sdeservice -o register -d SQLSERVER -r ADMIN_DATABASE -v CDOTSDE -i sdeCDOTSDE -p YOURPWD
sdeservice -o register -d SQLSERVER -r ADMIN_DATABASE -v TIMSSDE -i sdeTIMSSDE -p YOURPWD
sdeservice -o register -d SQLSERVER -r ADMIN_DATABASE -v ArcData -i sdeArcData -p YOURPWD
sdeservice -o register -d SQLSERVER -r ADMIN_DATABASE -v Gains_Spatial -i sdeGains_Spatial -p YOURPWD
sdeservice -o register -d SQLSERVER -r ADMIN_DATABASE -v sdeblank -i sdesdeblank -p YOURPWD
sdeservice -o register -d SQLSERVER -r ADMIN_DATABASE -v TexasFire -i sdeTexasFire -p YOURPWD
sdeservice -o register -d SQLSERVER -r ADMIN_DATABASE -v WindSpatial -i sdeWindSpatial -p YOURPWD

sdeservice -o delete -i sdeCDOTSDE -d SQLSERVER
sdeservice -o delete -i sdeTIMSSDE -d SQLSERVER
sdeservice -o delete -i sdeArcData -d SQLSERVER
sdeservice -o delete -i sdeGains_Spatial -d SQLSERVER
sdeservice -o delete -i sdesdeblank -d SQLSERVER
sdeservice -o delete -i sdeTexasFire -d SQLSERVER
sdeservice -o delete -i sdeWindSpatial -d SQLSERVER

echo sdeCDOTSDE          5155/tcp  #ArcSDE for SQL Server >> C:\arcgis\ArcSDE\sqlexe\etc\services.sde
echo sdeTIMSSDE          5156/tcp  #ArcSDE for SQL Server >> C:\arcgis\ArcSDE\sqlexe\etc\services.sde
echo sdeArcData          5159/tcp  #ArcSDE for SQL Server >> C:\arcgis\ArcSDE\sqlexe\etc\services.sde
echo sdeGains_Spatial          5163/tcp  #ArcSDE for SQL Server >> C:\arcgis\ArcSDE\sqlexe\etc\services.sde
echo sdesdeblank          5166/tcp  #ArcSDE for SQL Server >> C:\arcgis\ArcSDE\sqlexe\etc\services.sde
echo sdeTexasFire          5170/tcp  #ArcSDE for SQL Server >> C:\arcgis\ArcSDE\sqlexe\etc\services.sde
echo sdeWindSpatial          5171/tcp  #ArcSDE for SQL Server >> C:\arcgis\ArcSDE\sqlexe\etc\services.sde

echo sdeCDOTSDE          5155/tcp  #ArcSDE for SQL Server >> %windir%\system32\drivers\etc\services
echo sdeTIMSSDE          5156/tcp  #ArcSDE for SQL Server >> %windir%\system32\drivers\etc\services
echo sdeArcData          5159/tcp  #ArcSDE for SQL Server >> %windir%\system32\drivers\etc\services
echo sdeGains_Spatial          5163/tcp  #ArcSDE for SQL Server >> %windir%\system32\drivers\etc\services
echo sdesdeblank          5166/tcp  #ArcSDE for SQL Server >> %windir%\system32\drivers\etc\services
echo sdeTexasFire          5170/tcp  #ArcSDE for SQL Server >> %windir%\system32\drivers\etc\services
echo sdeWindSpatial          5171/tcp  #ArcSDE for SQL Server >> %windir%\system32\drivers\etc\services

From here, we need to do some copy pasting – the SQL code needs to go into another query window and be executed.

The sdeservice -o create, -o register and echo stuff needs to go into a batch file called “create.bat”. I included -o delete so you can easily remove all this crap when you get everything up to 9.3, so put those lines into a “drop.bat” file.

Then just run your create bat file, and start the services. You’ll need to add entries into everyone’s services file or just send out the port numbers, which is my preference… as shown below, the connection is good to go.

desktop-fix

as “fun” as this was, unfortunately it was a total waste of time.

Making Push-Pins Fly…

Posted by Dave Bouwman | Posted in ArcGIS Server, ArcSDE, SQL 2008, SQL Server | Posted on 04-08-2008

1

Note: I’ve been having some issues with LiveWriter posting things before they are fully baked. Despite deleting the pre-posted posts, they still seem to get stuck in dasBlog, and aggregated into my RSS feed. This is the “actual” post which seems to have snuck out on August 3rd. Hopefully this version make a little more sense.

A while back I wrote up a quick post about a site we built for the Wildlife Conservation Society that displays Avian Influenza data. This time I’m going to go into a few more of the details of how the system was built, specifically how we got the pins to be fast.

The Challenge

The main hurdle in this project was the database that contains the scientific observations. I should note that this is a database used by scientists to enter all sorts of avian influenza data – simple count surveys (number of dead birds and number of healthy birds) as well as detailed sampling. It’s designed to support revision management, and a complex set of peer review user stories. Thankfully we did not have to design this system, as the database has over 2000 tables, and >10000 stored procedures (mostly code-generated). There are more than 35,000 location records in the system, and many more samples or observations. While the locations (Lat / Lon) were in one table, we also needed to allow users to filter the data shown on the map by multiple criteria simultaneously.

wcs-criteria

Other than Country, all the other criteria (observation date, species, influenza sub-type) required at least one join, and most required multiple joins. Oh, and it has to be fast. Ideally we want the points drawn on the Virtual Earth Control at or near the same speed that the tiles load.

The ESRI Stack

The ESRI stack does alot of things well, but handling One-to-Many joins is not on that list. Despite that, I did some testing pulling the points out of ArcSDE via ArcGIS Server, converting the IGeometries into our custom point type, and ran them through a clustering algorithm, converted to GeoJSON and shipped to the client. While this worked, it was slow. And I could not see any way to handle the query criteria without doing the criteria filtering in code vs. in the database. Since we were already looking at several seconds to just get the points, this effectively took this stack out of the picture. Another approach was needed.

SQL 2008 Spatial:

Clearly if I was going to handle the criteria filtering, and hit the performance goals, I needed to do all the filtering (spatial & attribute) in the database itself. This sounded like a good opportunity to play with SQL 2008, thus the initial posting on the R & D back in February. That post goes into much of the details, but I found (and you can see on this demo page) that it was in the ballpark in terms of performance. On our LAN, zoomed in so there were not 1000’s of points returned, I was getting the points rendering in Virtual Earth about the same time the tiles would arrive. Good. But when we zoomed out (level 1 or 2) there were 10,000+ points being returned from the database (pre-clustering), and we still had 1-2 second lag times. As noted in this post, I also had some issues with the single hemisphere issue as well (fyi – this still an issue in the demo page).

SQL 2005 Non-Spatial:

While I was working on SQL 2008, more or less on a lark, I thought I would try working directly with the Lat / Long fields in SQL 2005. I cooked up a quick stored proc that took 2 Lat/Long pairs defining a box, and returned the points that fell inside. Simple, and for the purposes of this project accurate enough. Turns out that this is really crazy fast. Even at zoom level 1 (whole earth), I was still getting the points back rendered in Virtual Earth before the tiles. Checking performance in SQL Server, these queries were running around 100ms. 

So that pretty much decided how I would handle the locations. From there I took a set of criteria filtering stored procedures written by the Wildlife Conservation Society, added in the Lat/Long bounds code, and I had the criteria filtering working. This stored procedure than handles all the point and criteria queries is run every time you move or zoom the map, so it’s a good thing that we spent the time optimizing this part of the site.

Making Push-Pins Fly

Posted by Dave Bouwman | Posted in Uncategorized | Posted on 03-08-2008

0

 

The main complexity was the database that contains the scientific observations. I should note that this is a database used by scientists to enter all sorts of avian influenza data – simple count surveys as well as detailed blood sampling. As a result, the database has over 2000 tables, and 10000 stored procedures (mostly code-generated). There are more than 35,000 location records in the system, and many more samples or observations. Sure the locations (Lat / Lon) were in one table, but recall that we needed to allow users to filter the data shown on the map by multiple criteria simultaneously. Other than Country, all the other criteria required at least one join, and most required multiple joins. Oh, and it has to be fast. Ideally we want the points drawn on the VE Control at or near the same speed that the tiles load.

This effectively took ArcSDE and ArcGIS Server out of the picture since this stack does not effectively handle this sort of situation. We did some testing just pulling the points out of ArcSDE via ArcGIS Server, converting them into our custom point type, which then runs through a clustering algorithm, and is finally converted to GeoJSON and shipped to the client. While this worked, it was slow. And we could not see any way to handle the query criteria without doing the critera filtering in code vs in the database. Since we were already looking at several seconds to just get the points, I stepped back and too a different approach.

Clearly if I was going to handle the criteria filtering, and hit the performance goals, I needed to do all the filtering (spatial & attribute) in the database itself. This sounded like a good opportunity to play with SQL 2008, thus the initial posting on the R&D back in February. That post goes into much of the details, but what I found (and you can see on this demo page) that it was in the ballpark in terms of performance. On our LAN, zoomed in so there were not 1000’s of points returned, I was getting the points rendering in Virtual Earth about the same time the tiles would arrive. Good. But when we zoomed out (level 1 or 2) there were 10,000+ points being returned, and we still had 1-2 second lag times. As noted in this post, I also had some issues with the single hemisphere thing.

While I was working on that, more or less on a lark, I thought I would try working directly with the Lat / Long fields in SQL 2005. I cooked up a quick Stored proc that took 2 Lat/Long pairs defining a box, and returned points that inside. Simple, and for the purposes of this project accurate enough. Oh, and really really fast. Even at zoom level 1 (whole earth), I was still getting the points back rendered in Virtual Earth before the tiles. Checking performance in SQL Server, this was running in less than 100ms.

So that pretty much decided how I would handle the locations. From there I took a set of Stored Procedures written by WCS, and added in the Lat/Long bounds, and I had the criteria working. The main stored procedure than handles all the point and criteria queries is run every time you move or zoom the map.