GIS Programming and Crime Mapping - Participation #1

For our first participation assignment in GISProgramming we had to find an article that applied to GIS and write a short summary about it. I tried to find an article that was both applicable to this class and the type of work I enjoy doing (intelligence analysis). The article “Fairfax County builds searchable crime-data mapping app” by Stephanie Kanowitz (Sep 15, 2014) filled both of these requirements.

Crime mapping has been used for years but Fairfax County’s crime mapping system began to have issues when the data from all of the years began to pile up. The original Oracle based relational database struggled to process the data, over 500,000 data points, in a timely manner and each day another 100-200 more data points are added from the county’s 911 computer system. Any time a search was performed in the old system every column of data in over 800 layers had to be searched which took a very long time to process. To fix this issue the county implemented a new system called Police Events.

In the new mapping system, specific programming codes sort the new data from the day into the appropriate databases every night. The coding also created a more robust indexing scheme for map data layers that reduced the total amount of layers needed and reduced data processing time. Now users searching through the crime data can conduct a more generic search but will receive quicker results. Users can search by specific terms, specific layers, and specific date ranges too. The creator of this new system made it developer friendly as well so new map data can easily be added. The new system cost $46,000 on top of cost to use ESRI’s JavaScript programming.

What I took away from this article is that the real time saving features in a GIS are all based on programming. With the right codes, gigantic databases can be searched and mined easily.


Article  - "Fairfax County builds searchable crime-data mapping app"

Police Events - New Crime Mapping System

Lab 3- Watershed Analysis

In GIS Applications we took a look at watershed analysis. I thought I had an understanding of this topic prior to beginning this weeks lab but I quickly realized I had only touched the surface. Almost all of the data we used this week stemmed from a DEM file available form the National Elevation Dataset (NHD). I am little amazed with all of the analysis that can be done with a single DEM and I am sure this is going to happen every time we use another tool in ArcGIS during this class.

One of the most important things I learned was also one of the first steps in the lab - how to fill in sinks (pixels that are anomalies or extremely lower than the cells surrounding them) to hydrologically correct the elevation data prior to conducing analysis. This was a really simple process using the Fill Tool in the Spatial Analyst Toolbox but without it the water flow data could have turned out much different. Then we went through the process of understanding water flow and identifying stream paths. To to this we used the Flow Direction Tool, Flow Accumulation Tool, Con Tool, Stream Link Tool, Stream Order Tool, Snap Pour Point Tool, Watershed Tool, and Basin Tool.

All of these tools have water related names except the Con Tool and that is because the Con Tool is not a part of the Hydrography Toolset even though it was important to use for watershed analysis this week. We used the Con Tool to establish a flow accumulation threshold in order to identify stream paths. This was the part that confused me for a while but eventually it began to make sense. To me this means we told the computer that in order for a pixel/cell to be a part of a stream, it had to have a flow connection to at least x other cells (for the lab this was 200).

After using all of these tools I eventually had a map that showed a computer model of all the streams and stream orders on Kuauai Island as well as the watersheds based off of these modeled streams. The task after creating the models was to compare the modeled data with existing data that was create from aerial images. I thought the amount of streams, stream length, and stream order were important in the stream comparison. I thought comparing the total area size and the amount of high order streams in each watershed was also significant. My final graphic is below.

Lab 3 - Watershed Analysis

x

Module 3 - Python Fundamentals Part 2

This week in GISProgramming we learned a few more fundamentals of scripting in Python using PythonWin. We were challenged to import a module, fix errors in an existing script, and write our own basic scripts using conditional statements and loops. In the weekly exercises we got practice doing all of these tasks and then for the lab we were given some general guidance to follow and several end-tasks to meet.

The first section of the lab required importing the random module in order to use some of the random functions later on in the lab. This step was pretty simple, just two words in one line of code. The next section required us to fix two errors in an existing script. During this section I used the help section of the Python window in ArcGIS to recognize discrepancies with some of the functions. I also remembered that Python is case sensitive (because I've been using all lowercase letters so far) which cleared up one of the errors. Once the errors were fixed and i ran the script the results were a simulated dice-like game using the player's names. The names and win/loss statements in the graphic below are the results of the game.

Module 3 Dice Game and Number List

The third and fourth parts of the lab went together. The third section required me to create a loop that would produce a random list of 20 numbers with values ranging 0-10. I used a list, the random range function, a while loop, the len function, and the count method to accomplish this. I learned that the range needed to be one number higher than 10 in order to actually include 10 in the random number selection. I ran the script a bunch of times waiting for a 10 before I figured this out. In the fourth section of the lab we had to remove an unlucky number from the list. In addition to number removal we also had to print a statement indicating if the list did or did not contain the unlucky number and if it did, the statement needed to indicate how many times was the unlucky number present. To complete this task I created a variable to represent the unlucky number. Then I used the variable in a conditional if/else branching structure and wrote out the two sentences that I wanted printed. To count the amount of unlucky numbers I used the count method and added it as a string function to one of my printed statements. To remove the unlucky numbers I used a while loop and the remove method. The results are pictured at the bottom of the graphic above. The initial random list of 20 numbers printed, a statement noting how many times the unlucky number occurred printed, and the new list with the unlucky number removed printed. The lab took me longer than it should have because I realized I was trying to make things a lot more complicated than they needed to be.

In addition to the normal coursework I am also t the point where I need to start thinking about what I want to do for my end of semester project and I have no idea. I will add a note next week about the progress I make with at least deciding on an idea. 

Lab 2 - Corridor Analysis

This week was a look at something totally new for me in the GIS applications realm, least-cost paths and corridor analysis. The lab consisted of two parts. The first section was creating a least-cost path based off of one raster input, then two then three. I used the Cost Distance Tool, Cost Path Tool, and Plus Tool in the ArcGIS Spatial Analyst Toolbox to create the paths. I assigned a cost to each cell in each raster dataset using the Reclassify Tool. I do not feel as confident about this section as I have about other applications so I think a little more practice will be in order.

The second part of the lab used the same tools to model a movement corridor for black bears based off of three input rasters. The input data were elevation, landcover, and roads. After reclassifying these layers to assign a cost value to each cell I used the Weighted Overlay Tool to combine the three with the landcover weighted higher than the other two. Then I had to invert the costs so I used the Minus Tool to accomplish the task. Once the cost surface was complete I ran the Cost Distance Tool for both of the National Forests pictured in the graphic below. I used the resulting cost rasters to run the Corridor Tool to identify the most suitable corridor for the black bear. My final results are pictured below.

Black Bear Corridor

Module 2 - Python Fundamentals

This week in Programming we wrote our first Python script. It's a short script but without going through the 23 page practice exercise before trying to write it, I would have been completely lost. I learned a lot about syntax, assigning variables, and had a brief introduction to methods and functions. I know once we delve deeper into scripting for GIS I'm going to get a little confused with the names and uses of all of the tools and functions available. That being said, I am really glad that the Python Window in ArcGIS provides prompts when you start typing in the code and at the same time provides a help window with directions for how to fill in the required objects and arguments. I think those features will be lifesavers for me as this course continues on.

The script I wrote identifies my last name, counts the letters in my last name, and multiplies that number by three. The initial variable I started with was my full name and then I used functions and methods to create new variables and count the letters. The picture below shows the result of running the script.

Result of Last Name Script

One thing I learned this week that I am going to have to be extra careful about is using the right characters. Brackets, parentheses, apostrophes, and quotation marks all have specific purposes and are not always interchangeable.

Lab 1 - Suitability Analysis

This begins the blog section for Application in GIS (GIS5100). Last week was an introduction to the course and a review of everything I should have remembered from last semester. Let's just say GIS work is a lot more complicated with minimal directions but with ArcGIS Help, most things are possible.

The real work started this week with Suitability Analysis. The first challenge was to identify suitable areas for a mountain lion habitat using data provided by the instructor using a Boolean Suitability Model. The elements we used were a land cover raster file, an elevation data raster file, a road vector file, and a river vector file. I identified the suitable habitat areas using a vector method (converting all raster data to vector data) and a raster method (converting all vector data to raster). Both methods reduced all of the data to a basic yes or no option, does the area meet requirements or not? Using the Reclassify Tool in the Spatial Analyst Toolbox all of these yes and no answers were converted to 1's and 0's in the attribute tables for all of the files being analyzed. By the end of the process all I had to do was write one SQL query to sift through all of the data and find the areas that answered yes to every suitability requirement.

Then we took things a step further and rated locations by suitability, not only figuring out what areas were suitable but identifying how suitable they were. The process was very similar to the first part of the lab except in this section all of the data was analyzed in raster form. The difference came with the application of the Weighted Overlay Tool. Using this tool you can sort your data by how important it is, not just if it meets your needs. When assessing the slope of an area everything over 10 degrees slope may meet the suitability requirements but but a slope greater than 16 degrees would be better and a slope greater than 20 degrees would be ideal. You can assign values the data indicating these preferences. Then using the Weighted Overlay tool you can factor in these preferences associated with individual datasets as well the importance of one dataset over another. For example, soil composition may be more important than slope but you also want to consider the preference for different slope values and soil types. For the lab we evaluated five criteria; slope, soil, land cover, and proximity to rivers and roads. The graphic below shows the difference between an equally-weighted analysis and an alternate where slope was deemed the most important factor.

Suitability Analysis for Lab 1

Module 1- An Intro to Python

The summer semester has started and I just took my first look at Python! Python scripting as a concept makes complete sense to me but I have a feeling it is going to be a lot more tricky in practice. We used one script that the instructor provided in the Intro2GIS course last semester and it was easy to see the value in knowing how to use this language so I am excited to work through this course.

For the introduction to programming (for GIS more specifically) we used a Python script to set up the folder structure of our individual student drives on the remote GIS desktop that we use. I knew the tech guys at my last job knew how to do this stuff but I had no idea how to do it myself until now. Our lab walked us through how to use the basic functions of WinPython and then we were provided with a script to use to create our folder structure. Here is a look at the folder structure for everyone in the class --

The Standard Folder Structure for UWF's GISProgramming Course

We also had to write a pseudocode for a fairly simple math function. This step took me back a few years to basic algebra where my instructor regularly reinforced the idea of showing all of your work. I think that is going to be one of the most important things for me to remember as I continue through this programming endeavor, all of the work needs to be written out or the code will not work. I know sometimes I take shortcuts or overlook the simple steps that are always used to solve a problem but missing these little bits could cause big problems, like missing out on a piece of analysis that I was I expecting to be done in ArcGIS. I hope I didn't just jinx myself because I am eager to develop some Python scripting skills!