ArcGIS and Python Virtual Environments part 2

Making it work with Arcpy

In my last post I went over installing and setting up a Python virtual environment. Now I want to make it work with a specific library in my existing Python installation - Esri's Arcpy.

This is useful as it enables you to access other libraries without having to install them in the virtual environment. As I am learning OpenCV for example, I want to test if I can incorporate any spatial tools in the processing steps. Rather than installing arcpy in the virtual environment, I can simply set it up to be able to access my existing install, then I can just call import arcpy directly.
This could also be useful for GDAL or QGIS tools, or any other libraries you have already installed.

The USGS has a very good write up on calling arcpy from virtual environments.

To summarize:

To use arcpy in your virtual environment, you don't want to copy all the ESRI stuff in, you just want to tell it where to find the ArcGIS installed components. 

To do this, you'll need to place a *.pth file in the Lib\site-packages subdirectory of your virtual environment. If you used the WORKON_HOME variable from the last post, in Windows this defaults to %USERPROFILE%Env.
You can call this file whatever you want, something like "arcgis_arcpy.pth" works.

In the file, you'll need to put all the directories that ArcGIS uses to make arcpy work. The easiest way to get this list of pathnames is to start ArcMap, start the python window, type

 import sys 

then 

sys.path 

and copy the list into a text file. Save this as a '.pth' file. Note that the file should look something like the following (text after a # are comments and will not be read).

# Place in your virtual environment folder ...\\lib\site-packages\ 
C:\\Windows\\system32 
C:\\Windows\\system32\\python27.zip
# Esri specific for arcpy - Note 10.3 specific
C:\\Program Files (x86)\\ArcGIS\\Desktop10.3\\bin
C:\\Program Files (x86)\\ArcGIS\\Desktop10.3\\arcpy
C:\\Program Files (x86)\\ArcGIS\\Desktop10.3\\ArcToolbox\\Scripts
C:\\Python27\\ArcGIS10.3
C:\\Python27\\ArcGIS10.3\\Lib
C:\\Python27\\ArcGIS10.3\\DLLs
C:\\Python27\\ArcGIS10.3\\Lib\\lib-tk
C:\\Python27\\ArcGIS10.3\\lib\\site-packages

Put the file into the "\lib\site-packages" folder in your virtual environment folder (which was set up in the previous post). You should now be able to import arcpy and use it with other libraries. 

ArcGIS and Python Virtual Environments part 1

I recently learnt how to get Python virtual environments set up on Windows with ArcGIS installed, and thought it would be useful to write the notes up. This post covers installing a virtual environment.

Why would you want to do this (use virtual environments that is, not use arcpy)?

ArcGIS installs it's own version of python in the c:\python27\ArcGIS10.x folder. Upgrading ArcGIS will cause it's own particular version to be installed in a folder specific to that version. These should be left alone, as upgrading modules in them may cause all sorts of issues and potentially break ArcGIS stuff. So ideally we want to keep the ArcGIS python install as vanilla as possible. But what about when you want to experiment with other libraries, or need to test/develop against a different version of a library?

Virtual environments to the rescue! This post will explain the steps I took to getting Python virtual environments up and running for Windows, with particular notes for those whose Python install is the default ArcGIS installation (which is a little different than normal). However, most of the steps are applicable to any windows Python installation.

I found these sites very handy while reading up on installing virtual environments, and worth referring to if you have problems. In fact, you should open these and have a look through them now.

• https://virtualenv.readthedocs.org/en/latest/ 
• http://docs.python-guide.org/en/latest/dev/virtualenvs/
• http://python-packaging-user-guide.readthedocs.org/en/latest/installing/#optionally-create-a-virtual-environment

There are two main steps.

1. Installing and setting up a Python virtual environment so that we can install additional modules without affecting the ArcGIS python install, and 
2. Getting that virtual environment to interact with the ArcGIS python installation so that we can use arcpy.

To do this we will be working in cmd.exe (the command interpreter/shell/terminal) hereafter called 'cmd'.

This does not normally need to be an administrator level, so open up cmd.

In cmd, navigate to your installed python folder (example is ArcGIS 10.3) by typing:

cd c:\python27\ArcGIS10.3

Next we need to install pip.  Pip is a package management system that can install python packages. If you don't have pp installed, the easiest way to do this for windows is to download get-pip.py from here.

Move the downloaded get-pip.py file to the directory above and in cmd run:

python get-pip.py

Now you should have pip installed, which will make the rest easier.

Add the ArcGIS python folders to the environment path if they are not already. For ArcGIS installs (windows) also add the  \Scripts folder.

The path settings can be found in My Computer > Properties > Advanced System Settings > Environment Variables >
Add the following to the end of the environment path:

     ;C:\Python27\ArcGIS10.3;C:\Python27\ArcGIS10.3\Scripts
 

The next step is to install virtualenv. Virtualenv can be used to separate installs for development purposes. See the Python guide or Kenneth Reitz's virtualenvs github docs.

There is also virtualenvwrapper-win, a wrapper for virtualenv for windows that makes it easier to manage environments. You will need virtualenv installed first though.

To install virtualenv, you should only need to type

pip install virtualenv

In case of error:

Installing modules can be tricky on Windows, in many cases the files will need to be compiled and this might fail if you do not have Visual Studio 2008 installed. There are three potential ways to address this.

1. Install a compiler:
1. Install Visual Studio 2008 Express, or download and install just the compiler tools from https://www.microsoft.com/en-us/download/details.aspx?id=44266 .
2. upgrade setuptools by running pip install --upgrade setuptools
3. open a new cmd prompt and try the install again.
4. type pip install virtualenv
2. Use the  --use-wheel option:
1. Type pip install --use-wheel virtualenv
3. Download the appropriate precompiled wheel file and run directly
    

Install the virtualenv wrapper for windows:

 pip install virtualenvwrapper-win

In order to manage all our potential virtual environments, virtualenvwrapper has some handy features. You can set a default folder for your virtual envs to be installed to!

Type   

export WORKON_HOME=~/Env

to create an "Env" folder in your home directory (In Windows, the default path for WORKON_HOME is %USERPROFILE%Env). This will become the default location. If you don't want to do this, just make sure that you are in the appropriate folder before creating your virtual environment.

Now we are ready to create a virtual environment and switch to it. The following uses virtualenvwrapper. https://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html#managing-environments

In your command prompt type:

mkvirtualenv myenv

then (if needed):

workon myenv

The virtualenv name will appear to the left of the path, e.g.,

           C:\users\~

You can see what is installed by default using pip list to list installed modules. You can now install whatever you need, safe in the knowledge that it will not affect your other install of Python. 

To deactivate the virtual env  and return the to default installed Python, type 

deactivate

The cmd path will now not have the env name next to it, showing that we are out of the virtual environment.

If you installed virtualenvwrapper, you can also type 

rmvirtualenv myenv

to delete it. This will not delete the folder that was made though.
 

In the next post I will cover the setup needed to easily call arcpy from your new virtual environment.

Combining address fields into a Google Spreadsheet from a form

I was asked to help out with a project, involving a form-to-map website idea. This was to be based on a Google Form, feeding a Google Spreadsheet. This was then pushed to a Google Fusion table and eventually consumed in a WordPress app.

The form collected address data, and the first hurdle was that the data was in several fields (the fusion table expected one for geocoding). After they showed me what they needed I immediately started googling (as you do) to see what had been done before.

Fairly quickly I found that a) Google Scripts use javascript and b) you could connect a script to a spreadsheet and trigger it from a form submit. However much of the code I saw online used the e.values[i] approach, and I was using e.namedValues['name']. Once I tracked down a comment mentioning that you needed to either use e.namedValues['name'].toString() or e.namedValues['name'][0] it all came together quickly. Chris Keller has a particularly informative post on this sort of thing, so I included my version of his geocoding code below.

The input data is then concatenated to provide a geocodable string. The optional section (currently commented out) does the geocode and returns the lat long values. Either (or both) could be used as desired. This example assumes there is a "Location" column in the spreadsheet for the concatenated string, and a "LatLng" column if you are doing the geocode.

1. Open your spreadsheet, and click Tools, Script Manager. 
2. Make a new script and choose "spreadsheet". 
3. Add the code below, change as desired and save.
4. Click Resources and "Current Projects Triggers".
5. Add a new trigger as below, with 'concatAddress' under Run, 'From Spreadsheet' under Events and 'On form submit' at the end. You can edit the notification if you like.
6. Give it permission if requested.
7. Save that, and you should be done.

Trigger properties. When the form is submitted, concatAddress is run.

Code:

//concatenate address data and add it to a 'Location' column

function concatAddress(e) {

  var sheet = SpreadsheetApp.getActiveSheet(); //get the sheet

  var lastRow = sheet.getLastRow(); //get the last row (this is the newly added one)

  //fields to concatenate

  var address1 = e.namedValues["Address"].toString();

  var city = e.namedValues["City"].toString();

  var state = e.namedValues["State"].toString();



  //join the fields into single string, with delimiters

  var geocodeString = address1 + ", " + city + ", " + state;



  //update the value in the appropriate column

  sheet.getRange(lastRow, getColIndexByName("Location")).setValue(geocodeString);



  //we can also geocode the result directly into another column

  // uncomment this to do so

  /*

  var results = Maps.newGeocoder().geocode(geocodeString);

  if (results.status == 'OK') {

    var bestResult = results.results[0];

    var lat = bestResult.geometry.location.lat;

    var lng = bestResult.geometry.location.lng;

    var latLng = lat + ',' + lng;

    sheet.getRange(lastRow, getColIndexByName("LatLng")).setValue(latLng);

  }

  */



};

// look up the column index by supplied name

function getColIndexByName(colName) {

  var sheet = SpreadsheetApp.getActiveSheet();

  var numColumns = sheet.getLastColumn();

  var row = sheet.getRange(1, 1, 1, numColumns).getValues();

  for (i in row[0]) {

    var name = row[0][i];

    if (name == colName) {

      return parseInt(i) + 1;

    }

  }

  return -1;

};

2012 goals

So the new year has started, and even though my first day of it consisted mainly of sleeping (a result of partying until 5 am and half a bottle of 18 year old Johnny Walker Gold), a few thoughts began to surface around what I wanted to achieve this year.

The first thing that sprang to mind was improving my Spanish. Even a couple of weeks here in Arequipa being basically immersed in the language hasn't boosted me as much as I'd hoped, which I suspect is mainly due to a limited vocabulary (I am better than I was though). It's time to start getting serious about it and doing some structured learning and workbooks. Luckily I will get to practice it at home.

The second thing is improving my coding skills. I started with VBA, building MS Access front ends and small Office tools. MapBasic came next, learning by doing and by trying to fix a broken application at work. The next step was converting VBA code to run in ArcGIS10, deciding on VB.NET for ease and speed reasons. I also built a few add-ins for ESRI products. I also played around with Python, mainly in ESRI again (data driven pages etc). But with such an informal path I feel like I am missing some solid background, so will be addressing this first. I may also look at learning C#, which seems to mark a person as being a more serious GIS developer type than VB.

Both of these should have immediate benefits and feedback, which hopefully should go a long way towards keeping me motivated. Plus the coding aspect (esp Python) will be important for my thesis work which is an added incentive with specific timeframes included. And writing it here gives me that much more reason to stick with it!

print "Happy New Year!"
print "¡Feliz Años!"