Lab 6- Hotspot Analysis

I think this is the first week in GIS Applications that I finished the lab and felt less confused than when I started. Normally I work through lab but get completely stumped and frustrated at some point but that did not happen this week. Maybe that's just because I found the topic interesting but I'll find out once my lab is graded. The topic for the week was crime hotspot analysis and we created three different types of maps to show crime hotspots using ArcMap, Spatial Analyst Tools, and Spatial Statistics Tools.

The first type of analysis was grid-based thematic mapping. To analyze the data a grid was laid over the crime data point file and areas with high occurrences of crime were highlighted. The majority of the work for the first section was completed through a spatial join and selections with SQL queries. The spatial join counted the number of crime occurrences for each grid square and the SQL queries enabled the quick selection of grid cells with high numbers of crimes to export as a file to display on a map. The final product from this section can be seen as the pink colored block shapes in the graphic below.

The second map was a kernel density map. After using an a spatial join between census block areas and point crime data I used the Kernel Denisty Tool in ArcMap. Using a cell size of 100ft and a radius of 2640ft the computer analyzed the crime data and came up with the dark blobs in the graphic below to denote areas that had crimes rates the were greater than three times the average for the area.

The third map uses Local Moran's I (Cluster and Outlier Analysis) to identify hotspots. Again, this process started with a spatial join and SQL selection query. Then I used the Cluster and Outlier Analysis (Anselin Local Morans I) Tool in ArcMap. After the tool was ran I used another SQL query to select the results with a high-high outcome. This produced a result similar in area size to the grid-based method but with an irregular shape the covered different regions.

My final graphic shows the all three hotspot results from the lab.

The maps were based off of crime data from 2007. The final section of our lab was to look the the hotspots from the 2007 data and see how they compared to the actual crime spots for 2008. The grid-based map from 2007 encompassed 40% of the crimes reported in 2008 and a crime density of 43. The kernel density map accounted for 1/3 of the area that the grid-based map did but encompassed 18% of the crimes reported in 2008 and had a crime density of 66. I think based of of these statistics the kernel density map is a good predictor of potential high crime areas.

Module 6 - Python Goeprocessing

This week in GISProgramming we were challenged to write a standalone Python script in PythonWin for our lab. The script needed to add XY coordinates to a point shapefile, create a 1000 meter buffer around each point, and then dissolve the buffer areas into a single feature. We were given directions to use the AddXY Tool, the Buffer Tool, and the Dissolve Tool even though the dissolve portion could be done with optional parameters in the Buffer Tool.

 After completing the weekly exercise this task was not as daunting as I first thought it would be. The biggest challenge for me was syntax. I used psuedocode to establish my game plan for how to write the script and then I looked up the Python script example associated with each tool in the ArcGIS help area. The example scripts helped me understand how to write out the parameters for each tool because while they may be similar, not everything is exactly the same. I wrote out the script but expected errors so I "commented out" everything but the first tool and tried to run it. There were errors. A spelling problem and then an issue with a filepath but they didn't stump me for long. Once the first tool ran properly I "uncommented out" the second tool and worked through it. I found I had the same type of filepath issues so I decided to use full filepaths where the errors arose instead of the shortened versions that should have linked to my declared workspace. After that fix the Buffer Tool worked without a problem and I moved on to the last tool. I fixed the filepath problem for the input values on the Dissolve Tool before I tried to run it so when I did finally run the whole script it had no issues. My script results are in the graphic below.

One of the most important parts of this script was setting the environments and geoprocessing options. Since I had issues with my workspace and some of the filepaths that followed I ran the script a bunch of times. This means the same file was created over and over again. Turning on the overwrite option in ArcGIS's geoprocessing options was essential to completing this script.

When the script ran successfully I was still a little doubtful that the desired shapefiles turned out right so I opened up a map and checked them before calling the script a success.

Lab 5 - Spatial Accessibility

This week in Applications in GIS was a bit of a chore because it included quite a bit of Excel and I'll just say, maybe it was a good thing I had this refresher now instead of later. For Lab 5 we did a few exercises in the ArcGIS Help Tutorial to learn how to build network datasets. That Help guide is a lifesaver each and every time I get into ArcMap now so the tutorial lessons were a nice change of pace for this class. I am challenged every week but this week I already knew where to look in the Help directory because the page was already open and I just had to re-read the section again.

After working through the tutorial exercises we did a little work with spatial access based on proximity and this is where Excel came out to play. The lab required us to identify which areas (counties and then smaller census tracts) had access to hospitals based on the data we were given. After doing some quick work with Spatial Joins to determine distances between points and applying a nice graduated color scheme we started with the Cumulative Distribution Functions. I think I spent half of my time on this lab figuring these things out (because they came up again at the end of the lab). By exporting opening the .dbf file for a layer in Excel you can create a normal .xlsx spreadsheet and manipulate the data without messing with the actual map layer. Eventually I figured out how to sort the data efficiently, add the formulas i wanted, and sort the data again without messing up the formulas I entered earlier. In the end I had a couple of nice looking scatter plots that showed exactly what I was trying to show. Don't let me fool you, I skipped ahead and came back o finish this at the end.

Then came the spatial accessibility analysis with a network dataset. We looked at the effects a potential community college campus closure would have on student commutes and access. The graphic below is the result of Service Area Analysis using the Network Analyst Tool in ArcGIS with limits for a 5, 10, and 15 minute drive.

Module 5 - Toolboxes and ModelBuilder

In GISProgramming this week we took a look ModelBulder in ArcGIS and touched on exporting Python scripts for editing in PythonWin. Every week I find this class more and more useful and I find myself eager to look at the topics for the next week. Next week is more of the Python scripting side of ArcGIS rather than ModelBuilder so it should be interesting. Now on to the highlights from this week.

Farmland within flood basin boundaries

The purple polygons in the graphic above are the result of the model and script I created this week. As a bunch of purple polygons it may not look like much but if you had seen it at the beginning it would have looked like a bunch of colors, squiggly lines, and overlapping polygons depicting flood basins and soil composition for an area. Now the graphic only shows prime farmland within a specific basin. The model I created clip the soil file to the much smaller size of the basin file. From the clipped soil layer the areas listed as "Not Prime Farmland" were selected. This selection was erased from the basin layer leaving only areas suitable for farming in the basin visible in the final layer. The model used three tools and two original variables to create the final layer and was much easier to work with than trying to run each tool individually.

After verifying the model ran correctly I exported the model from ModelBuilder to a Python Script. I opened the script in PythonWin and added some minor edits to make sure the script would run outside of the Python Window in ArcGIS. The two major obstacles here were making sure the filepaths for the original data were correct and allowing the script to overwrite existing files. The filepath changes were simple but in order to overwrite files I had to add a line of code to change the environment settings that would normally be done in the Geoprocessing Menu in ArcGIS. Once the script was corrected zipped it with the Toolbox I created for the model I built and it was ready to share. (Really it was ready to submit for a grade - but that is still shareable.)

Lab 4 - Viewsheds

This week in Applications in GIS we took a look at viewshed and line of sight analysis. Most of the lab was completed in a 2D perspective but there were a couple of 3D components. All of the work was done using ArcMap and either the Visibility Tools in the Spatial Analyst Toolbox or the 3D Analyst Toolbox. Today my blog will focus on one of the 3D aspects of the lab.

We were provided with data for the Boston Marathon Finish Line to include the placement of one camera overlooking the finish line. We were asked to identify two other locations to place cameras to capture the happenings at the finish line based off of our viewshed analysis. The original camera was located to the west of the finish line at approximately 100 feet above the street next to a building with a 0-90 degree viewing angle. The two cameras I added are both located to the east of the finish line. I chose these locations based on the height of the building they could be attached because the viewshed was much more limited at ground level and for the different perspectives they would provide. The second camera (closest to the finish line) is located at a height of 194 feet and has a viewing angle of 150-260 degrees. I tried several different viewing angles but this angle created the largest area captured by all three cameras. This location also provides a decent birds eye view of the finish line. Third camera is located on a building one block to the east. It is positioned at a height of 117 feet with a viewing angle of 180-270. The locations of the three cameras capture the full finish line area from three different perspectives as seen in the graphic below.

Viewshed for Camera Placement Near Finish Line

Module 4 - Debugging

Bugs! This week in GISProgramming we worked through scripting errors and debugging in Python. After this introduction I think decided I prefer syntax errors to exceptions and logic errors because they seem easier to fix, or at least the computer is more likely to catch them as an error.

For the lab I tackled errors in three scripts supplied by our professor. In the first script the fixes were simple, just a couple of changes due to spelling and capitalization. The Check Tool in PythonWin did a lot of the work here. The output of the script is written in blue in the graphic below and the error that prevented the script from running is in red at the top.

Script 1 Output

There were more errors, 8 total, in the second script and they were a little more difficult to identify. The errors this time around included spelling issues, incorrect symbols, incorrect file paths, and missing/incorrect arguments for methods, I used the Check Tool again but it did not identify all of the errors so I also tried the Debugging Tool on this script. The debugging tool helped but one thing I learned this week is that it is hard to recognize an error for a method or function you know nothing about. How do you know to correct an argument if you do not know the requirements for a particular method or function? That is why I am a big fan of the Help Tool and the Python Manuals, they make weird codes make sense. Once everything was corrected and I ran the script, I got the results in the graphic below. Also make note of the last error I received- if the script is trying to save something to an open map document you may have to close the .mxd file first.

Script 2 Output

The final script seemed like it would be the most difficult to get through but it was actually much easier than the second. I guess the reason it was easier is because I used a try-exception statement to trap (and skip) and error instead of having to fix it. There was an error with one line of code that affected several others and caused the whole script to stop running. By adding a try-except statement around the whole section of affected code the rest of the script was able to run. You can see in the graphic of the script output that there was an error in Part A but that Part B was still able to run. The nice thing about a try-except statement is that you can add a print statement the describes the error that was trapped.

Script 3 Output

**Since I mentioned it in my post last week I feel obligated to follow up with my progress on my final project idea. So far I am still working on a good idea but this week I am thinking I may be able to write a code for this class that will help me with my final project in GIS Applications. I will ponder it this weekend and provide another update next week.

GIS Programming and Crime Mapping - Participation #1

For our first participation assignment in GISProgramming we had to find an article that applied to GIS and write a short summary about it. I tried to find an article that was both applicable to this class and the type of work I enjoy doing (intelligence analysis). The article “Fairfax County builds searchable crime-data mapping app” by Stephanie Kanowitz (Sep 15, 2014) filled both of these requirements.

Crime mapping has been used for years but Fairfax County’s crime mapping system began to have issues when the data from all of the years began to pile up. The original Oracle based relational database struggled to process the data, over 500,000 data points, in a timely manner and each day another 100-200 more data points are added from the county’s 911 computer system. Any time a search was performed in the old system every column of data in over 800 layers had to be searched which took a very long time to process. To fix this issue the county implemented a new system called Police Events.

In the new mapping system, specific programming codes sort the new data from the day into the appropriate databases every night. The coding also created a more robust indexing scheme for map data layers that reduced the total amount of layers needed and reduced data processing time. Now users searching through the crime data can conduct a more generic search but will receive quicker results. Users can search by specific terms, specific layers, and specific date ranges too. The creator of this new system made it developer friendly as well so new map data can easily be added. The new system cost $46,000 on top of cost to use ESRI’s JavaScript programming.

What I took away from this article is that the real time saving features in a GIS are all based on programming. With the right codes, gigantic databases can be searched and mined easily.


Article  - "Fairfax County builds searchable crime-data mapping app"

Police Events - New Crime Mapping System

Lab 3- Watershed Analysis

In GIS Applications we took a look at watershed analysis. I thought I had an understanding of this topic prior to beginning this weeks lab but I quickly realized I had only touched the surface. Almost all of the data we used this week stemmed from a DEM file available form the National Elevation Dataset (NHD). I am little amazed with all of the analysis that can be done with a single DEM and I am sure this is going to happen every time we use another tool in ArcGIS during this class.

One of the most important things I learned was also one of the first steps in the lab - how to fill in sinks (pixels that are anomalies or extremely lower than the cells surrounding them) to hydrologically correct the elevation data prior to conducing analysis. This was a really simple process using the Fill Tool in the Spatial Analyst Toolbox but without it the water flow data could have turned out much different. Then we went through the process of understanding water flow and identifying stream paths. To to this we used the Flow Direction Tool, Flow Accumulation Tool, Con Tool, Stream Link Tool, Stream Order Tool, Snap Pour Point Tool, Watershed Tool, and Basin Tool.

All of these tools have water related names except the Con Tool and that is because the Con Tool is not a part of the Hydrography Toolset even though it was important to use for watershed analysis this week. We used the Con Tool to establish a flow accumulation threshold in order to identify stream paths. This was the part that confused me for a while but eventually it began to make sense. To me this means we told the computer that in order for a pixel/cell to be a part of a stream, it had to have a flow connection to at least x other cells (for the lab this was 200).

After using all of these tools I eventually had a map that showed a computer model of all the streams and stream orders on Kuauai Island as well as the watersheds based off of these modeled streams. The task after creating the models was to compare the modeled data with existing data that was create from aerial images. I thought the amount of streams, stream length, and stream order were important in the stream comparison. I thought comparing the total area size and the amount of high order streams in each watershed was also significant. My final graphic is below.

Lab 3 - Watershed Analysis

x

Module 3 - Python Fundamentals Part 2

This week in GISProgramming we learned a few more fundamentals of scripting in Python using PythonWin. We were challenged to import a module, fix errors in an existing script, and write our own basic scripts using conditional statements and loops. In the weekly exercises we got practice doing all of these tasks and then for the lab we were given some general guidance to follow and several end-tasks to meet.

The first section of the lab required importing the random module in order to use some of the random functions later on in the lab. This step was pretty simple, just two words in one line of code. The next section required us to fix two errors in an existing script. During this section I used the help section of the Python window in ArcGIS to recognize discrepancies with some of the functions. I also remembered that Python is case sensitive (because I've been using all lowercase letters so far) which cleared up one of the errors. Once the errors were fixed and i ran the script the results were a simulated dice-like game using the player's names. The names and win/loss statements in the graphic below are the results of the game.

Module 3 Dice Game and Number List

The third and fourth parts of the lab went together. The third section required me to create a loop that would produce a random list of 20 numbers with values ranging 0-10. I used a list, the random range function, a while loop, the len function, and the count method to accomplish this. I learned that the range needed to be one number higher than 10 in order to actually include 10 in the random number selection. I ran the script a bunch of times waiting for a 10 before I figured this out. In the fourth section of the lab we had to remove an unlucky number from the list. In addition to number removal we also had to print a statement indicating if the list did or did not contain the unlucky number and if it did, the statement needed to indicate how many times was the unlucky number present. To complete this task I created a variable to represent the unlucky number. Then I used the variable in a conditional if/else branching structure and wrote out the two sentences that I wanted printed. To count the amount of unlucky numbers I used the count method and added it as a string function to one of my printed statements. To remove the unlucky numbers I used a while loop and the remove method. The results are pictured at the bottom of the graphic above. The initial random list of 20 numbers printed, a statement noting how many times the unlucky number occurred printed, and the new list with the unlucky number removed printed. The lab took me longer than it should have because I realized I was trying to make things a lot more complicated than they needed to be.

In addition to the normal coursework I am also t the point where I need to start thinking about what I want to do for my end of semester project and I have no idea. I will add a note next week about the progress I make with at least deciding on an idea. 

Lab 2 - Corridor Analysis

This week was a look at something totally new for me in the GIS applications realm, least-cost paths and corridor analysis. The lab consisted of two parts. The first section was creating a least-cost path based off of one raster input, then two then three. I used the Cost Distance Tool, Cost Path Tool, and Plus Tool in the ArcGIS Spatial Analyst Toolbox to create the paths. I assigned a cost to each cell in each raster dataset using the Reclassify Tool. I do not feel as confident about this section as I have about other applications so I think a little more practice will be in order.

The second part of the lab used the same tools to model a movement corridor for black bears based off of three input rasters. The input data were elevation, landcover, and roads. After reclassifying these layers to assign a cost value to each cell I used the Weighted Overlay Tool to combine the three with the landcover weighted higher than the other two. Then I had to invert the costs so I used the Minus Tool to accomplish the task. Once the cost surface was complete I ran the Cost Distance Tool for both of the National Forests pictured in the graphic below. I used the resulting cost rasters to run the Corridor Tool to identify the most suitable corridor for the black bear. My final results are pictured below.

Black Bear Corridor