Published on NGA Advanced Python Programming for GIS, GLGI 3001-1 (https://www.e-education.psu.edu/ngapython)

Home > Lessons > Lesson 3: Advanced Geoprocessing > Python Package Management

Python Package Management

There are many Python packages available for use, and there are a couple of different ways to effectively manage (install, uninstall, update) packages. The two package managers that are commonly used are pip and conda. In the following sections, we will discuss each of them in more detail. At the end of the section, we will discuss the merits of the two tools and make recommendations for their use.

We will be doing some more complicated technical "stuff" here so the steps might not work as planned because everyone’s PC is configured a little differently. For this section, it is probbaly best to just review it and know that if you needed to perform these steps, you can refer to the process to get you started. Anaconda environments can take days to install and get working. Installation of Anaconda is not required, but included for review and if you want to try it, I suggest referring to it at a later date.

pip

As already mentioned, pip is a Python package manager. It allows for an easier install, uninstall and update of packages. Pip comes installed with Python, and if you have multiple versions of Python you will have a different version of pip for each. To make sure we are using the version of pip that comes installed with ArcGIS Pro, we will go to the directory where pip is installed. Go to the Windows Start Menu and open the Python Command Prompt as before.

In the command window that now opens, you will again be located in the default Python environment folder of your ArcGIS Pro installation. For newer versions of Pro this will be C:\Users\<username>\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\. Pip is installed in the Scripts subfolder of that location, so type in:

cd Scripts

Now you can run a command to check that pip is in the directory – type in:

dir pip.*

The resulting output will show you all occurrences of files that start with pip. in the current folder, in this case, there is only one file found – pip.exe.

Figure 2.30 Files that Start with "pip"

Next, let’s run our first pip command, type in:

pip --version

The output shows you the current version of pip. Pip allows you to see what packages have been installed. To look at the list type in:

pip list

The output will show (Figure 31) the list of packages and their respective versions.

Figure 2.31 Package Versions

To install a package, you run the pip command with the install option and provide the name of the package, for example, try:

pip install numpy

Pip will run for a few seconds and show you a progress bar as it is searching for the numpy package online and installing it. When you run pip install, the packages are loaded from an online repository named PyPI, short for Python Package Index. You can browse available packages at Python's Package Index page(link is external). If the installation has been successful you will see a message stating the same, which you can confirm by running pip list again.

In order to find out if any packages are outdated you can run the pip list with the outdated option:

pip list –-outdated

If you find that there are packages you want to update, you run the install with the upgrade option, for example:

pip install numpy –-upgrade

This last command will either install a newer version of numpy or inform you that you already have the latest version installed.

If you wanted to uninstall a package you would run pip with the uninstall option, for example:

pip uninstall numpy

You will be asked to confirm that you want the package uninstalled, and, if you do (better not to do this or you will have to install the package again!), the package will be removed.

The packages installed with pip are placed in the Lib\site-packages folder of the Python environment you are using. You will recall that that was one of the search locations Python uses in order to find the packages you import.

Lesson content developed by Jan Wallgrun and James O’Brien

Conda and Anaconda

Another option for packaging and distributing your Python programs is to use conda. Just like pip, it is a package manager. In addition, it is also an environment manager. What that means is that you can use conda to create virtual environments for Python, while specifying the packages you want to have available in that environment. Conda comes installed with ArcGIS Pro. We can double check that it is by opening the Python Command Prompt and then typing in:

cd Scripts

followed by:

conda –-version

The output should show the conda version.

In order to find out what packages are installed type in:

conda list

Your output should look something like Figure 2.34:

Figure 2.34 Conda Package List

The first column shows the package name, the second the version of the package. The third column provides clues on how the package was installed. You will see that for some of the packages installed, Esri is listed, showing they are related to the Esri installation. The list option of conda is useful, not only to find out if the package you need is already installed but also to confirm that you have the appropriate version.

Conda has the functionality to create different environments. Think of an environment as a sandbox – you can set up the environment with a specific Python version and different packages. That allows you to work in environments with different packages and Python versions without affecting other applications. The default environment used by conda is called base environment. We do not need to create a new environment, but, should you need to, the process is simple – here is an example:

conda create -n gisenv python=3.6 arcpy numpy

the –n flag is followed by the name of the environment (in this case gisenv), then you would choose the Python version which matches the one you already have installed (3.5, 3.6 etc.) and follow that up with a list of packages you want to add to it. If you later find out you need other packages to be added, you could use the install option of conda, for example:

conda install –n gisenv matplotlib

To activate an environment, you would run:

activate gisenv

And to deactivate an environment, simply:

deactivate

Full reference to the command line arguments and flags [1] are helpful.

To see the commands usage, you access the documentation through the command: conda env [-h] command ... as

conda env –h

In the conda prompt.

Output:
  positional arguments:
{create,export,list,remove,update,config}
create Create an environment based on an environment definition file. If using an environment.yml file (the default), you can name the environment in the first line of the file with 'name: envname' or you can specify the environment name in the CLI command using the -n/--name argument. The name specified in the CLI will override the name specified in the environment.yml file. Unless you are in the directory containing the environment definition file, use -f to specify the file path of the environment definition file you want to use.
export Export a given environment list List the Conda environments remove Remove an environment update Update the current environment based on environment file config Configure a conda environment

optional arguments:
   -h, --help Show this help message and exit.

Once you know the command you want to use, you can add the parameter and view its help.
  conda env export –h
  
  Export a given environment

Options:
optional arguments:
 -h, --help Show this help message and exit.
 -c CHANNEL, --channel CHANNEL

Additional channel to include in the export
 --override-channels Do not include .condarc channels
 -f FILE, --file FILE
 --no-builds Remove build specification from dependencies
 --ignore-channels Do not include channel names with package names.
 --from-history Build environment spec from explicit specs in history

Target Environment Specification:
 -n ENVIRONMENT, --name ENVIRONMENT

Name of environment.
   -p PATH, --prefix PATH

Full path to environment location (i.e. prefix).

Output, Prompt, and Flow Control Options:
   --json Report all output as json. Suitable for using conda programmatically.
   -v, --verbose Use once for info, twice for debug, three times for trace. 
   -q, --quiet Do not display progress bar.

examples:
conda env export
conda env export --file SOME_FILE

Once we have made sure that everything is working correctly in this new environment and we built our export command, we can export a YAML file using the command:

conda env export > “C:\Users\...\Documents\ArcGIS\AC37.yml”

This creates a file that contains the name, channel and version of each package in the environment and can be used to create or restore an environment to this state. It also includes packages that were installed via pip, and will attempt to install them through conda first when the file is used to create the environment.

It is important to note that using pip can break a conda environment which results in conda not being able to resolve dependencies. There are times when packages are only found in pip and for those instances, you can create the conda environment first, and then use pip. This is not guaranteed and could take several tries to find the right package version numbers that work nicely together.

There are other options you can use with environments – a great resource for the different options is Conda's Managing Environments page

Lesson content developed by Jan Wallgrun and James O’Brien


Source URL:https://www.e-education.psu.edu/ngapython/node/585

Links
[1] https://conda.io/projects/conda/en/latest/commands/index.html