GEOG 585
Open Web Mapping

Walkthrough: Creating tiles with Mapnik using TileMill

Print

In this walkthrough, you will use Mapnik (wrapped by TileMill) to create a general-use basemap of Philadelphia. The idea is that you will be able to overlay thematic layers on top of this map in lessons to come. The data for this walkthrough is the base data for Philadelphia that you preprocessed in Lesson 3. If you followed all instructions, you should have this data in a folder called c:\data\Philadelphia or something similar. It should be using the mercator projection EPSG:3857.

Mapnik is a FOSS map-drawing engine that is often used for the purpose of making sets of map image tiles. Mapnik incorporates techniques like antialiasing that make the edges of lines and labels appear smooth, not pixelated. It is not tied to the overhead of any other GIS software framework, a fact that improves its performance. Various types of file-based data and spatial databases are supported as data sources for the maps.

To use Mapnik, you define a set of data sources and then connect them to a bunch of styling rules. You can set up the styling rules using an XML file, or you can write them programmatically using a language like Python. Once you've configured the data sources and style rules, you can instruct Mapnik to export pictures of the map.

This is easy in theory but is not the simplest process for a beginner, especially if your programming skills are thin. To work with Mapnik in this course, you'll use a program called TileMill that makes the process a lot easier. TileMill puts a GUI around Mapnik, allowing you to browse to the datasets you want to use and see the results of your work as you apply styling rules for each layer. Instead of using an XML file or code to define the rules, you'll use a fairly intuitive language called CartoCSS that borrows from the structure of web stylesheets.

TileMill is FOSS, but was developed by Mapbox to integrate with the company's for-profit tile hosting service. This is evident when you export tiles with TileMill and the resulting file has the extension .mbtiles. If you don't want to host the tiles on Mapbox's servers, you can "unpack" the tiles out of the .mbtiles file and host them as individual images on your own web server. In this walkthrough, you'll unpack the images and host them on your Penn State PASS web space. Then you'll test the tiles by supplying their URL structure to a web map.

As you complete this walkthrough, you may see occasional software messages that Tile Mill is not under active development and that Mapbox has shifted its focus to Mapbox Studio. Tile Mill continues to be available as a FOSS tool for building tile sets of rasterized map images. In contrast, Mapbox Studio is geared towards creating vector tiles which Mapbox will then host in exchange for a fee. Be aware that Mapbox Studio-generated .mbtiles files you encounter professionally may contain vector tiles instead of containing rasterized images like the Tile Mill-generated .mbtiles files we will use in this walkthrough.

Installing TileMill and designing the map

  1. Download and install TileMill from the following page: https://www.mapbox.com/tilemill/. Take all the defaults. When you install TileMill, Mapnik is installed for you as well.
  2. Launch TileMill from your Windows Start menu by clicking TileMill > Start TileMill.
  3. Click New Project and fill out the information as in the image below. Then click Add.
     New project in TileMill
    Figure 5.8 TileMill New Project GUI
    If you were making a cache of raster imagery, you might want to use the lossy jpeg image format to keep the cache size down. Since we are making a vector map, the png format is fine.

    Note here that having a map with vector data does not mean you will need to use vector tiles. The vectors in our map will be rasterized by the tile creation process, meaning that users will just be seeing a bunch of "pictures of the data".
  4. In your project list, click PhillyBasemap and take a look at your canvas. In TileMill, your map preview appears on the left and your CartoCSS code appears on the right. The map updates whenever you save your CartoCSS code.

    Let's see how this process works by changing the background color from blue to white.
  5. Set the background-color property to #FFFFFF (the HTML hex code for white) as shown below, and click Save.
    Map {
       background-color: #FFFFFF;
    }
    The background should update to white as soon as you save your code. Now, let's add some layers and follow this process to style them.
  6. Click the layers button, and click Add Layer.
     TileMill Add layer button
    Figure 5.9 TileMill Add Layer GUI
  7. Add the city_limits.shp shapefile as a layer using the exact properties as shown below (makes sure it is in the EPSG: 3857 projection).
     Add city limits layer in TileMill
    Figure 5.10 TileMill Add Layer GUI
    A couple of things here deserve mention. First, TileMill allows you to set varying levels of granularity on your styling rules, using the concept of classes and IDs. For example, if you wanted to set some general rules for all streets, you could set up a class for all streets, and if you wanted to set some more specific rules for a certain subgroup of streets, you could use an ID for that subgroup. You're only going to set one style on the city limits layer, so these instructions tell you to set the Class and ID to the same name. In fact, each layer in this tutorial will use an identical ID and Class name for simplicity.

    Also, projection code 900913 must be defined as the spatial reference when you are using files that you originally projected into EPSG:3857 using OGR. The two projections are equivalent, but if you leave TileMill to autodetect the spatial reference, you will see an offset from Google's tiles due to the way the projection parameters are interpreted.
  8. The starting view at level 2 is zoomed too far out for work with Philadelphia, so use the Zoom to extent button to get your map zoomed in to the city limits layer.
     Zoom to layer
    Figure 5.11 Tile Mill Layers Zoom to extent dialog
  9. The default #CityLimit styling makes the city look like a big park, so change the style to a simple boundary using the following code:
    #CityLimit {
      line-color:#88789e;
      line-width:3;
    }
    When you save, you should see the following:
     City limit styled in TileMill
    Figure 5.12 TIleMill sample output
    Now that you've got the process down for adding and styling a layer, the instructions are going to start moving faster. I will provide information about which layers to add, and some basic code to style them. Make sure the ID you add for your layer matches the ID that I use in the code (remember the ID is the name following the # sign, such as #CityLimit or #Waterways).

    If you don't like my styling, you can try tweaking it to your liking, but be careful to maintain legal syntax. The Carto button provides reference documentation about how to use the different properties.

    Don't spend too much time making major changes to this map's styling, because you'll get a chance to make your own map during this week's assignment. After you finish the walkthrough, you can use this Philadelphia map for further practice and experimentation.
     TileMill Carto button
    Figure 5.13 Carto GUI
  10. Add waterways.shp as a layer, and add the following code to style it. The position where you paste this block of code into your CSS doesn't matter.
    #Waterways {
      line-width:1;
      line-color:#89aceb;
    }
  11. Add natural.shp as a layer, and add the following code to style it.
    #NaturalFeatures{
      [type='park']{
      polygon-opacity:1;
      polygon-fill:#ae8;
      }
      [type='riverbank']{
      polygon-opacity:1;
      polygon-fill:#89aceb;
      }
      [type='water']{
      polygon-opacity:1;
      polygon-fill:#89aceb;
      }
    }
    Notice above that you can add query terms if you only want to display a subset of features in the dataset. We are displaying just the parks, riverbanks, and water features, even though there are a lot more types of features in this shapefile.
  12. Add Neighborhoods.shp as a layer, and add the following code to style it.
    #Neighborhoods[zoom>12] {
      text-name:[NAME];
      text-face-name:"Arial Black";
      text-fill:#88789e;
      text-size: 12;
      text-character-spacing: 2;
      text-transform: uppercase;
    }
    In this case, we only specify text properties. This has the effect of hiding the neighborhood boundaries so that just the label of the neighborhood name is shown. This is more fitting to neighborhoods anyway, since they don't tend to have rigid boundaries.

    When you label a layer in TileMill, you need to specify the name of the field in your dataset that contains the label text. In this case, you want to label based on the NAME field, therefore you put text-name:[NAME].

    Also, notice that you can set a condition on the layer so that it only appears in when you zoom in beyond a certain level. Zoom in past level 12 to make sure this is working correctly and that you see the neighborhood labels.
     Neighborhoods viewed in the TileMill map
    Figure 5.14 Sample output
  13. Add roads.shp as a layer, and give it the ID and class name of MajorRoads. Then add the following code to style it.
    #MajorRoads{
      [type='motorway']{
       line-width:3;
       line-color:#606060;
      }
      [type='trunk']{
       line-width:3;
       line-color:#606060;
      }
      [type='primary'] {
       line-width:2;
       line-color:#838383;
      }
    }
    This will symbolize just the major roads. OpenStreetMap has a lot of different road types, and there are different ways you could apply styling here. In our situation, we will display minor streets by adding all the roads again and putting them beneath the major roads. This introduces some redundancy but keeps the code simpler.
  14. Add roads.shp as a layer again, but, this time, give it the ID and class name of Roads. Then, add the following code to style it.
    #Roads[zoom>12]{
      line-width:1;
      line-color:#b6b6b6;
    }
    		
    #Roads[zoom>14]{
      line-width:1;
      line-color:#b6b6b6;
      text-name:[name];
      text-face-name:"Arial Regular";
      text-fill:#838383;
      text-size: 11;
      text-placement: line;
      text-min-path-length:100;
      text-avoid-edges:true;
      text-min-distance:50;
      text-dy: 6;
      text-max-char-angle-delta: 15;
    }
    This puts some labels on the roads when zoomed in.
  15. Add railways.shp as a layer, and add the following code to style it.
    #Railroads{
      line-width:1;
      line-color:#d2bcb0;
    }
    		
    #Railroads[zoom>15] {
      ::line, ::hatch { line-color: #d2bcb0; }
      ::line { line-width:1; }
      ::hatch {
        line-width: 4;
        line-dasharray: 1, 24;
      }
    }
  16. Put your layers in the following order (by clicking and dragging to the left of the layer name), and save the map.
     TileMill layer order
    Figure 5.15 Layer list GUI
    Zoomed out, your map should look like this:
     TileMill map at zoom level 11
    Figure 5.16 Sample zoomed out output
    Zoomed in, your map should look like this:
     TileMill map viewed at zoom level 15
    Figure 5.17 Sample zoomed in output
    Your final code should look something like this, although the layers may be listed in a different order.
    Map {
      background-color: #FFFFFF;
    }
    
    #CityLimit {
      line-color:#88789e;
      line-width:3;
    }
    
    #Waterways {
      line-width:1;
      line-color:#89aceb;
    }
    
    #NaturalFeatures{
      [type='park']{
      polygon-opacity:1;
      polygon-fill:#ae8;
      }
      [type='riverbank']{
      polygon-opacity:1;
      polygon-fill:#89aceb;
      }
      [type='water']{
      polygon-opacity:1;
      polygon-fill:#89aceb;
      }
    }
    		
    #Neighborhoods[zoom>12] {
      text-name:[NAME];
      text-face-name:"Arial Black";
      text-fill:#88789e;
      text-size: 12;
      text-character-spacing: 2;
      text-transform: uppercase;
    }
    		
    #MajorRoads{
      [type='motorway']{
      line-width:3;
      line-color:#606060;
     }
      [type='trunk']{
      line-width:3;
      line-color:#606060;
     }
      [type='primary'] {
      line-width:2;
      line-color:#838383;
     }
    }
    		
    #Roads[zoom>12]{
      line-width:1;
      line-color:#b6b6b6;
    }
    		
    #Roads[zoom>14]{
      line-width:1;
      line-color:#b6b6b6;
      text-name:[name];
      text-face-name:"Arial Regular";
      text-fill:#838383;
      text-size: 11;
      text-placement: line;
      text-min-path-length:100;
      text-avoid-edges:true;
      text-min-distance:50;
      text-dy: 6
      text-max-char-angle-delta: 15;
    }
    		
    #Railroads{
      line-width:1;
      line-color:#d2bcb0;
    }
    		
    #Railroads[zoom>15] {
      ::line, ::hatch { line-color: #d2bcb0; }
      ::line { line-width:1; }
      ::hatch {
       line-width: 4;
       line-dasharray: 1, 24;
     }
    }

Exporting and extracting the tiles

Once you've finished the map design phase and your map looks good at each scale, you can start thinking about generating the tiles. For large maps, plan for this to take some time, taking into account some of the factors discussed earlier in the lesson such as the shape of the map and the scale levels you choose to generate.

This section of the walkthrough explains how to generate tiles of your Philadelphia basemap, unpack them, and host them on your web space. You will then test them out in a web page.

  1. Make one minor change in your code to increase the default buffer size around the metatile. This helps avoid your very wide labels from being cut off when the tiles are drawn. You add this near the top of your code, in the Map section, like this:
    Map {
      background-color: #FFFFFF;
      buffer-size: 512;
    }
  2. In TileMill, click the Export button and choose MBTiles. This is MapBox's format for storing all the tiles in a SQLite database. We'll eventually unpack them as individual PNG images.
  3. Define the area and origin point for the cache. To do this, use the left-hand map view to zoom in to Philadelphia. Then hold down the Shift key and drag the left mouse button to draw a box around the Philadelphia city boundary. Make sure you do this at a fairly large scale (zoomed in) so that you don't create tiles for a lot of peripheral white space.

    When you have zoomed the map to the area that you want, right-click in the middle of your map to place the center point for the tiles. If it looks like your map has disappeared - it hasn't - be sure to iteratively zoom and resize (TileMill might reset you to Zoom 0 making your map look tiny) until you get back to a more reasonable zoom level).

    Your map should look like this:
     TileMill area of interest
    Figure 5.18 Sample map output
  4. In the right-hand menu, set your zoom slider to cover levels 0 through 17 as shown below. Notice how the number of potential tiles is affected by moving the slider, especially at the larger scale levels.
     Tile Mill zoom slider
    Figure 5.19 Zoom slider GUI
    Don't change any of the other default settings on this panel.
  5. Click Export, and note the progress bar (under Export - View Exports) giving you an idea of how much time is remaining to finish building your tiles. You can move on and do a few of the next steps while the tiles are being created, although, if you open Windows Task Manager, you'll see that a process called node.exe (or Evented I/O) is using up most of your CPU power while it draws the tiles.

    To unpack these tiles, we'll be using a utility called MBUtil. You need to go download this from its home on GitHub (an online repository where developers often like to share their code with the world).
  6. Go to the MBUtil page on GitHub and click Clone or download > Download ZIP. Save the zip file to your computer.
  7. Extract the contents of the zip file into a very easy path to remember, such as c:\mbutil.
  8. MBUtil requires Python, so determine where python.exe is installed on your computer. Look for a folder such as c:\Python27 or c:\Python27\ArcGIS10.4 (it may have been installed with ArcGIS). If you cannot find python.exe on your computer, visit python.org and download and install Python 2.7 in the default path.

    In this tutorial, I will assume that c:\Python27\python.exe is your Python path. If your Python is installed in a different location, just adapt the path in the examples to match the path of your own python.exe.
  9. In Windows Explorer, go to My Documents\Mapbox\export and find PhillyBasemap.mbtiles. These are all your tiles. Copy this file to an easy place to remember, such as c:\data\Philadelphia. For some reason, I have not been able to extract tiles in the past when they are sitting in the My Documents folder.
  10. Open a command prompt. (On Windows, click Start and then type cmd in the search box.)
  11. Extract all your tiles by running a command such as the following:
    c:\python27\python.exe c:\mbutil\mb-util c:\data\Philadelphia\PhillyBasemap.mbtiles c:\data\Philadelphia\PhillyBasemap
    Notice that the parameters are, in order, the path to Python, the path to the mb-util utility, the path of your compressed tiles, and the new folder where you want your tiles uncompressed. If any of your paths are different, you will need to adjust them when you type this command. After you run this command, you should see a set of uncompressed tiles in the folder c:\data\Philadelphia\PhillyBasemap.
     Extracted tiles
    Figure 5.20 File dialog showing folders of tiles at each zoom level

Hosting and testing the tiles

If you navigate around your folder of unpacked tiles, you'll notice that the images are extracted into a highly organized structure of level\column\row. This structure is understood by various mapping programs and APIs, so all you have to do at this point is put your tiles onto a web-facing server. A convenient place for you to experiment with this is the www folder in your PASS space, a public-facing directory that all Penn State students are given in their personal web space. We will place the tiles there and then test them in a web map.

  1. In Windows Explorer, right-click your folder of uncompressed tiles and click Send to > Compressed (zipped) folder. This should produce a folder titled PhillyBasemap.zip.

    Zipping the folder like this allows you to send all the files to PASS at once, rather than uploading them individually (which would be too tedious to be practical).
  2. Open a web browser to the PASS Explorer site at http://explorer.pass.psu.edu.
  3. Log in with your Penn State Access Account and navigate to the [www] folder.

    If you don't have this folder, go to https://www.work.psu.edu and click on a link on the left-hand side for requesting a personal web space. Complete the short required quiz, then wait a minute or two for your personal page to be created.

    Manually creating a folder named [www] will not work for this exercise; you must request this folder as a personal web space from Penn State.

    Regardless of whether your [www] folder has already been created or not, it will be helpful to request a bit more storage. By default, Penn State gives you 100 MB, but you can increase this limit to up to 10 GB. You can do this on the same site https://www.work.psu.edu by adjusting the Change quota dropdown to 10 GB.
     
  4. Within [www] create a new folder called [tiles] and navigate into it.
  5. Click the Upload button.
  6. Browse to your PhillyBasemap.zip folder and check the Auto Extract box.
     PASS upload
    Figure 5.21 Tile upload GUI
  7. Click Upload, and wait for a bit as your tiles are uploaded and extracted. When you are finished, you should be able to navigate to a structure like below:
     Extracted files in PASS
    Figure 5.22 PASS Explorer GUI showing tiles folders
  8. Test out a tile by hitting a URL of the structure http://personal.psu.edu/<your_PSU_ID>/tiles/PhillyBasemap/15/9555/12400.png. The <your_PSU_ID> part in the URL needs to be replaced by your own ID, the part in your PSU email address before the @psu.edu. You should get back an image like the one below.
     Test image from server
    Figure 5.23
    Now, let's see if we can add this to an online map to test the tile appearance and navigation experience. We will use the ArcGIS.com Map Viewer, which is browser-based software for making a web map. It is free to use for experimentation or for sharing maps with the public.

    We're using the ArcGIS.com map viewer here solely because it is a quick and handy way to test a tiled map layer. There is no dependency on ArcGIS software in this walkthrough.
  9. Open a browser to http://www.arcgis.com/home and click the Map link at the top of the screen. You'll see an Esri-provided map, which we will switch out for our Philadelphia basemap.
  10. Click Add > Add Layer from Web. (If you don't see an Add button, first click the Modify Map link.)
  11. Choose to add data from A Tile Layer, and then populate the dialog box as shown below, substituting again your own Penn State ID in the URL (replacing the sdq107), but leaving everything else the same. Note that to set the extent, you must click Set Tile Coverage and draw a box around Philadelphia. You don't need to get the exact same coordinates shown below.
     Add tile layer in ArcGIS.com viewer
    Figure 5.24 Add Layer from Web GUI in GeoServer
  12. Click Add Layer. If nothing happens for a long time, repeat and make sure there is no error in the URL field. Navigate around your Philadelphia map to test it out. Performance should be fast and crisp. If you turn on the network utility of your browser's developer tools, you should be able to see the tiles being brought into the map.
  13.  Map tiles in ArcGIS.com
    Figure 5.25 Sample map output
    Optionally, you could add other web layers on top, but, for the sake of time, this will not be described here. In future lessons, you'll learn how to build this type of web map programmatically without using ArcGIS.com.