Random Photos

My Flickr

Friday 22 January 2016

ArcGIS and Python Virtual Environments part 2

Making it work with Arcpy

In my last post I went over installing and setting up a Python virtual environment. Now I want to make it work with a specific library in my existing Python installation - Esri's Arcpy.

This is useful as it enables you to access other libraries without having to install them in the virtual environment. As I am learning OpenCV for example, I want to test if I can incorporate any spatial tools in the processing steps. Rather than installing arcpy in the virtual environment, I can simply set it up to be able to access my existing install, then I can just call import arcpy directly.
This could also be useful for GDAL or QGIS tools, or any other libraries you have already installed.

The USGS has a very good write up on calling arcpy from virtual environments.

To summarize:

To use arcpy in your virtual environment, you don't want to copy all the ESRI stuff in, you just want to tell it where to find the ArcGIS installed components. 
To do this, you'll need to place a *.pth file in the Lib\site-packages subdirectory of your virtual environment. If you used the WORKON_HOME variable from the last post, in Windows this defaults to %USERPROFILE%Env.
You can call this file whatever you want, something like "arcgis_arcpy.pth" works.
In the file, you'll need to put all the directories that ArcGIS uses to make arcpy work. The easiest way to get this list of pathnames is to start ArcMap, start the python window, type
 import sys 
then 
sys.path 
and copy the list into a text file. Save this as a '.pth' file. Note that the file should look something like the following (text after a # are comments and will not be read).


# Place in your virtual environment folder ...\\lib\site-packages\ 
C:\\Windows\\system32 
C:\\Windows\\system32\\python27.zip
# Esri specific for arcpy - Note 10.3 specific
C:\\Program Files (x86)\\ArcGIS\\Desktop10.3\\bin
C:\\Program Files (x86)\\ArcGIS\\Desktop10.3\\arcpy
C:\\Program Files (x86)\\ArcGIS\\Desktop10.3\\ArcToolbox\\Scripts
C:\\Python27\\ArcGIS10.3
C:\\Python27\\ArcGIS10.3\\Lib
C:\\Python27\\ArcGIS10.3\\DLLs
C:\\Python27\\ArcGIS10.3\\Lib\\lib-tk
C:\\Python27\\ArcGIS10.3\\lib\\site-packages

Put the file into the "\lib\site-packages" folder in your virtual environment folder (which was set up in the previous post). You should now be able to import arcpy and use it with other libraries. 

Tuesday 19 January 2016

ArcGIS and Python Virtual Environments part 1

I recently learnt how to get Python virtual environments set up on Windows with ArcGIS installed, and thought it would be useful to write the notes up. This post covers installing a virtual environment.

Why would you want to do this (use virtual environments that is, not use arcpy)?


ArcGIS installs it's own version of python in the c:\python27\ArcGIS10.x folder. Upgrading ArcGIS will cause it's own particular version to be installed in a folder specific to that version. These should be left alone, as upgrading modules in them may cause all sorts of issues and potentially break ArcGIS stuff. So ideally we want to keep the ArcGIS python install as vanilla as possible. But what about when you want to experiment with other libraries, or need to test/develop against a different version of a library?

Virtual environments to the rescue! This post will explain the steps I took to getting Python virtual environments up and running for Windows, with particular notes for those whose Python install is the default ArcGIS installation (which is a little different than normal). However, most of the steps are applicable to any windows Python installation.

I found these sites very handy while reading up on installing virtual environments, and worth referring to if you have problems. In fact, you should open these and have a look through them now.
There are two main steps.
  1. Installing and setting up a Python virtual environment so that we can install additional modules without affecting the ArcGIS python install, and 
  2. Getting that virtual environment to interact with the ArcGIS python installation so that we can use arcpy.
To do this we will be working in cmd.exe (the command interpreter/shell/terminal) hereafter called 'cmd'.
This does not normally need to be an administrator level, so open up cmd.



In cmd, navigate to your installed python folder (example is ArcGIS 10.3) by typing:
cd c:\python27\ArcGIS10.3

Next we need to install pip.  Pip is a package management system that can install python packages. If you don't have pp installed, the easiest way to do this for windows is to download get-pip.py from here.

Move the downloaded get-pip.py file to the directory above and in cmd run:

python get-pip.py

Now you should have pip installed, which will make the rest easier.

Add the ArcGIS python folders to the environment path if they are not already. For ArcGIS installs (windows) also add the  \Scripts folder.
The path settings can be found in My Computer > Properties > Advanced System Settings > Environment Variables >
Add the following to the end of the environment path:
     ;C:\Python27\ArcGIS10.3;C:\Python27\ArcGIS10.3\Scripts
 
The next step is to install virtualenv. Virtualenv can be used to separate installs for development purposes. See the Python guide or Kenneth Reitz's virtualenvs github docs.

There is also virtualenvwrapper-win, a wrapper for virtualenv for windows that makes it easier to manage environments. You will need virtualenv installed first though.
To install virtualenv, you should only need to type

pip install virtualenv

In case of error:
Installing modules can be tricky on Windows, in many cases the files will need to be compiled and this might fail if you do not have Visual Studio 2008 installed. There are three potential ways to address this.

  1. Install a compiler:
    1. Install Visual Studio 2008 Express, or download and install just the compiler tools from https://www.microsoft.com/en-us/download/details.aspx?id=44266 .
    2. upgrade setuptools by running pip install --upgrade setuptools
    3. open a new cmd prompt and try the install again.
    4. type pip install virtualenv
  2. Use the  --use-wheel option:
    1. Type pip install --use-wheel virtualenv
  3. Download the appropriate precompiled wheel file and run directly
     
Install the virtualenv wrapper for windows:
 pip install virtualenvwrapper-win

In order to manage all our potential virtual environments, virtualenvwrapper has some handy features. You can set a default folder for your virtual envs to be installed to!

Type   
export WORKON_HOME=~/Env
to create an "Env" folder in your home directory (In Windows, the default path for WORKON_HOME is %USERPROFILE%Env). This will become the default location. If you don't want to do this, just make sure that you are in the appropriate folder before creating your virtual environment.

Now we are ready to create a virtual environment and switch to it. The following uses virtualenvwrapper. https://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html#managing-environments

In your command prompt type:
mkvirtualenv myenv
then (if needed):
workon myenv

The virtualenv name will appear to the left of the path, e.g.,
           C:\users\~
mkvirtualenv result
You can see what is installed by default using pip list to list installed modules. You can now install whatever you need, safe in the knowledge that it will not affect your other install of Python. 

To deactivate the virtual env  and return the to default installed Python, type
 

deactivate
The cmd path will now not have the env name next to it, showing that we are out of the virtual environment.


If you installed virtualenvwrapper, you can also type 

rmvirtualenv myenv
to delete it. This will not delete the folder that was made though.
 
In the next post I will cover the setup needed to easily call arcpy from your new virtual environment.