There's so much to learn about ArcGIS Server and GIS server technology in general that it's impossible to cover it all in this course. Instead, we've chosen to focus on some of the issues most commonly faced by people setting up and running a GIS server. In Lesson 2, you learned how to set up a server and a web service, and you viewed that service on the web. In Lesson 3, you took that a step further and learned how to prepare data for editing over the web. You also made a fully-featured web application.

In Lesson 4, you will learn how to build rasterized tile caches to improve the speed of your map services. This is a practice used by major web mapping services such as Google Maps, Bing Maps, MapQuest, and the ArcGIS Online services that you have already used in this course.

Building and maintaining tile caches requires careful strategy and planning, far beyond just knowing how to push the buttons to make tiles. For this reason, map tiling can be a fun and intriguing subject to study.

Lesson Objectives

At the successful completion of this lesson you should be able to:

•     understand how to design a map for tiling;
•     create and maintain a rasterized map tile cache; and
•     create a substantive application using your knowledge of ArcGIS for Server.

Deliverables

• Complete: L04: Assignment
• (No discussion this week)

By this point in the course, you may have observed that there's more than one way to take raw GIS data from your server and put it into a map in someone's web browser. Recall some of the map services you used in the previous two lessons:

• The AppalachianTrail and BighornReferenceLayers services used the server to draw a new map every time the user panned or zoomed. The completed map image was then sent to the user's computer for immediate display in the web browser.
  
  In ArcGIS Server-speak, this is a dynamic map service, because it is drawn "dynamically", or on-demand, by the server. Dynamic map services are the default that you get when you publish a map service to ArcGIS Server, but it's not just Esri services that use this pattern. The Open Geospatial Consortium (OGC) Web Map Service (WMS) also uses this approach of sending back a dynamically-drawn image based on URL parameters sent from the requesting computer.
  
  Dynamic map services are easy to set up, but they are not the fastest or most scalable type of service (scalability refers to how many clients can be served at one time). Accessing data and drawing a map image require the server to do work, and if a large number of people are requesting maps at the same time, the server can get overwhelmed. Complex maps with many layers or fancy symbols may take the server an unreasonably long time to draw.
  
  Recognizing that dynamic drawing speed was suboptimal in early versions of ArcGIS Server, Esri wrote a streamlined drawing engine for the server and introduced it as an option at 9.3.1, using a special type of file you had to explicitly create called an MSD (map service definition). Starting from 10.1, the optimized drawing engine is used in all map services and the MSD takes a behind-the-scenes role. However, even a dynamic service, based on the optimized drawing engine, may not be appropriate for some heavily-used or complex maps.
• The BighornHabitat layer from the previous lesson containing sheep sightings and habitat was not drawn dynamically by the server. Instead the coordinates of all the vertices of the points and polygons were sent to the web browser as text, and the browser did the work of drawing the features. This approach can be thought of as using client-side graphics, because the client (web browser) uses its graphic-drawing technology to put the features on top of the map.
  
  ArcGIS Server map services can optionally be drawn as client-side graphics. OGC Web Feature Services (WFS) are also designed to be queried and drawn by the client machine.
  
  Client-side graphics are great for providing interactivity to your web applications. Once you get all the geometry and attributes of the features onto the client machine, you don't have to go back to the server any more. You can change the shape or color of a feature in real time, allowing for highlighting of features, reclassification, or making edit sketches.
  
  Client-side graphics are not appropriate in every case. Most browsers have a limit on how many points or vertices they can draw before they really start to slow down. Also, you saw in the previous lesson how symbology was limited using client-side graphics. The diagonal hash line symbolizing the habitats could only be drawn with a simple solid fill. This is because web browsers know how to draw a limited set of symbols; for example, they can't use the whole gamut of ArcMap polygon fills.
• The imagery service from ArcGIS Online that you used as a background for your bighorn sheep map was a tiled (or "cached") map service. In these types of services, the server just hands out little square images (tiles) of the map that it has stored in a cache on disk. These images are usually generated by the server administrator at a number of different scales before the service is made available to the public. Then, when web users request to see a map, the server does not have to do the work of drawing the map; it can just send back whichever tiles are needed to fill the map request.
  
  A server can send out tiles a lot faster than it can draw maps; therefore, tiled map services are quicker and can accommodate a lot more users than the other two types of services. Just take a look at how fast the imagery layer loads in your bighorn sheep app, or how quickly all the basemaps appear in the ArcGIS.com map viewer. If you have a complex map (especially a basemap) or a map that will be viewed by a lot of people, it's usually worth the effort to make a tile cache.

Like the two other choices above, tiled maps also have their unique drawbacks. The biggest one is the time investment and server power needed to generate the cache, along with the disk space necessary to store it. Also, because a cache represents a snapshot of your data at one point in time, it requires maintenance. If your source data or your map symbology is edited, you have to update the corresponding tiles in order for people to see the changes.

In this lesson, you'll learn about designing a map with the goal of building a tile cache. You'll get a chance to make some tiles and use them on the web. Since the number of tiles in a cache can multiply with each scale level added and become unmanageably large, you'll also learn about strategies for building and updating very big caches.

A word about vector tiles and rasterized tiles

A word about different tile types before we begin: There are two main types of tiles commonly used in web maps today. The kinds of tiles we've been talking about above can be thought of as rasterized tiles; in other words, they are images made up of grids of pixels. Rasterized tiles are easy for clients to draw because most apps and all web browsers know how to display an image like a JPG or a PNG; however, the server has to construct the image and, after that, you're stuck with the colors and symbols you chose.

To get around issues with rasterized tiles, another type of tiles called vector tiles have been increasing in popularity. Vector tiles are similar in concept to rasterized tiles in the sense that they are square packets of information structured in a pyramid motif and sent by the server; however, they contain vector coordinates instead of a picture of the data. This allows the styling to be easily changed. Vector tiles are displayed as client-side graphics, so the client software needs to understand what a vector tile is and how to deal with it. Older mapping software and APIs may not be able to consume vector tiles.

We will talk more about vector tiles in Lesson 5 when we work with Mapbox software, since Mapbox pioneered this format and based their company on it. Esri vector tile support [1] is growing, although it has lagged behind that of Mapbox. Esri vector tile functions are concentrated in their ArcGIS Pro product, which we are not using in this course.

Be aware that all the remaining content in Lesson 4 refers to rasterized tiles, and some of the design and performance considerations discussed may be very different when thinking about vector tiles.

Increasing your instance size for this lesson

Building rasterized cache tiles is CPU and memory-intensive. Your server is making thousands of repetitive map draws, sometimes with a very complex MXD in the background. You can build a cache a lot faster if you assign the tile creation to a powerful machine.

This short-term need for high computing power is a perfect use case for cloud computing. A lot of offices don't have a powerful machine to spare for building tiles (usually their beefiest machine is the server that's already hosting their live apps and web services). In this situation, a server administrator could launch a high memory and/or high CPU instance for just a few hours for the purpose of building tiles. The extra cost is often worth the time savings that it takes to build the cache. Once the tiles are created, the machine can be shut down or scaled down.

For this lesson only, you'll change your ArcGIS Server site to run on a High-Memory Extra Large instance. This costs significantly more than the Medium instance type that you've been using, but it will allow you to work with a complex map document and build cache tiles much faster.

1. Log in to the AWS Management Console and, if it is not already stopped, stop your ArcGIS Server machine instance.
2. Right-click your instance, and select Instance Settings > Change Instance Type.
3. Choose r4.xlarge, and click Apply.
   
   This Memory Optimized Extra Large instance type has over 30.5 GB of memory and 4 virtual CPU units. For reference, the t2 instance type that you've been using up to this point has 16 GB of memory and 4 vCPUs.
4. To see the specs for all instance types, check out the Amazon EC2 Instance Types [2] page and drool away. Then take yourself back to reality by viewing the Amazon EC2 Pricing [3] page.
5. Start your machine instance when you are ready to work with it.

Now that you are running on an instance that costs (at the time of this writing) 45 cents/hour as opposed to 6 cents/hour, it's more important than ever that you remember to stop your site when you are done working on your lesson materials for the day. Also, be sure to set your instance type back to t2.xlarge after building all your tiles in Lesson 4.

You'll find during this lesson that a rasterized tiled map service takes a lot of planning. Let's look at a few of the considerations needed to get a map ready for publishing as a tiled service. You'll download and examine a predesigned map and publish it as a service in preparation for making some tiles yourself.

To cache or not to cache?

The first question to settle is whether or not to make a tile cache at all. If the map is going to put strain on your server or take a noticeable amount of time to draw (these two often go together), then you need to consider making a tile cache. Most vector basemaps that give geographic context to your web map contain a lot of layers and fall into this category. This is one reason that splitting up your layers into basemap services and business layer services is a good idea; you can potentially cache the basemap while leaving the business layers uncached.

Is it necessary to cache the business layers, since that kind of data changes more frequently? Google used to do it with the Wikipedia layer in Google Maps [4].  With so many features (Wikipedia articles) to show, and with the amount of traffic Google Maps receives, it was burdensome on the servers to draw those points on the fly.  (Sadly, the Wikipedia layer is no longer offered.)

In addition to high traffic scenarios, you can also consider caching business layers when the map covers a relatively small extent, the data doesn't change very often, or the data is displayed at small scales only. Layers like weather radar need to be updated frequently, but are rarely viewed at large scales and require relatively few tiles in the cache, thus the update can be performed in a reasonable amount of time.

Choosing scales

There are a lot of decisions you need to make about how to set up your tile cache, but the first choice is the set of scales at which you are going to generate tiles. These scales represent the snapshots at which web users will see your map. They also determine how long it's going to take to create the cache, and which other web services the cache will be able to overlay. Ideally, you'll decide on your set of cache scales before you start designing your map.

Keep these things in mind when choosing a set of scales:

• If you already know that your map is going to overlay, or be interchangeable with, another tiled map service, then you should match the scales of that map service. Many server administrators set out to build caches that will overlay Google Maps, Bing Maps, or ArcGIS Online. In these cases, the choice of scales is easy. You have to match the Google/Bing/Esri scales, which are thankfully the same and are built into ArcGIS Server as an option.
• Larger scales require more tiles to cover the extent of the map. It takes four times as many tiles to cache a map at 1:1000 than at 1:2000. Thus, avoid building tiles at scales zoomed in farther than you need to see. It's worth noting that you can put scales in your "tiling scheme" (the Esri term for a set of scales and other cache properties), but you don't necessarily have to build tiles at all those scales. For example, in this lesson, you'll choose to use the Google/Bing/Esri tiling scheme which includes scales all the way down to approximately 1:1000, but you won't build tiles at the largest scales, because those aren't necessary for your map.
• Most tiled web maps halve the scale's denominator when zooming in (for example, the next scale beyond 1:48000 would be 1:24000, then 1:12000 and so on). The Google/Bing/Esri tiling scheme follows this pattern, and if you decide to enter your own scales, you might choose to follow it as well. Scale sets that increase slower than this rate tend to make the user feel some tedium when zooming in, and can cause you to create a lot more tiles than you really need.

Designing the map

Creating detailed vector basemaps of the type that are typically cached presents a grand cartographic challenge. In contrast to paper cartography, in which the map has to be designed at just one scale, the web basemap has to be designed to look good at every scale in your tiling scheme.

Designing this type of multilevel basemap can require you to include varying symbols at different levels of your map. For example, a road might be represented with a 3-point line width at a large scale, a 1-point width at a medium scale, and may not be visible at all at a small scale. Since ArcMap does not allow scale-dependent symbols, you'll sometimes need to add multiple copies of the same layer into your map, set different scale ranges on them, then assign appropriate symbols for each scale range.

It's also important to choose muted colors for the base map that look good, but do not overwhelm other layers placed on top. Go to Google Maps: Designing the Modern Atlas [5] to see some examples of how the Google Map design has toned itself down over time to be more accommodating to overlays.The Esri Light Gray Canvas basemap is another study of designing a basemap specifically as a backdrop for more important thematic or operational layers.

When web mapping exploded during the past two decades, some cartographers expressed their chagrin at the simple, uniform maps churned out by websites. Some may have thought their very jobs and livelihood were threatened. However, the years have shown that cartography holds a critical place in web mapping. Projects like the OpenStreetMap terrain layer [6] and the Esri World Topographic Map [7] incorporate very advanced cartographic techniques. In a sense, map tiling gave cartographers a ticket to ride in the web world, since these detailed maps would be too slow to serve dynamically.

No wonder some GIS professionals shrink at the thought of trying to design such a map on their own. Some organizations that lack an in-house cartographer have just limped along with the same symbols they used when more primitive map server technology was available. Others have imitated the colors and symbols of the ubiquitous Google Maps in their own basemaps (perhaps in response to a manager's demand, "Make our maps look like that!").

In response to queries about how the ArcGIS Online basemaps were constructed, Esri has released sample ArcMap documents using all the ArcGIS Online base map symbols. People can insert their own data into the map or simply copy the symbol settings into their own maps. Examining one of these maps provides a good lesson in multilayer basemap design.

Examining and publishing a street map

In this part of the lesson, you'll download and examine a map template that Esri has provided for the ArcGIS Online street map. This sample map covers the Little Rock, Arkansas region. You'll then publish the map as a service and get it ready for creating tiles in the next section of the lesson.

1. If necessary, start your ArcGIS Server site. You will need to use your instance a few steps down the road.
2. On your local machine, open a web browser to the ArcGIS.com map viewer [8] (which you used in Lessons 2 and 3) and then choose the Streets basemap.
   
   This is an approximation of the map you'll be working within this part of the lesson (I say approximation because Esri has updated some of the symbols slightly since they released the template you're going to download). Zoom in and take note of some of the layers that appear and disappear as you do so. Also, note how the symbols for features like rivers and cities change as you zoom in and out.
3. On your EC2 instance, extract the street map template files found in our course management system into a location under C:\data. The cleanest way to organize it may be to place the map and data directly under a folder named C:\data\LittleRock. Notice that the files include an MXD and a file geodatabase with a bunch of sample data.
4. On your EC2 instance, open StMap_Template_LittleRock_WebM.mxd in ArcMap.
   
   The first thing you should notice is how long it takes this map to draw and label. Take a look at the number of group layers and sublayers in the table of contents, and you'll see why. In addition to the sheer number of layers available, a lot of the layers are symbolized with complex symbols such as multilayer lines that take more time to draw. The performance of this map on the web will not be acceptable unless it is tiled.
   
   The data frame coordinate system is WGS 1984 Web Mercator (essentially the same as WGS 1984 Web Mercator (Auxiliary Sphere), which you used in previous lessons). This is the projection used by ArcGIS Online, Bing Maps, and Google Maps. It was not designed for geographic accuracy nor aesthetic purposes, rather for convenience. You could fit the whole world on a square tile. You'll go ahead and use this projection in this lesson, even though you can certainly make a tile cache in whatever projection you want.
   
   As you examine the table of contents, notice that this map is organized with group layers, each corresponding to a scale range. This particular map is designed for viewing at about 1:1,000,000 scale down to 1:4,500 scale. Any other scale is not going to result in a map being drawn. The layers were painstakingly copied, symbolized, and grouped so that the map would look good at each scale.
   
   To view the map at the ArcGIS Online, Bing, and Google scales, click the scale dropdown in ArcMap and click Customize this list.
    
   

   Figure 4.1 Customizing the scales list
   
   Then click Load, and click ArcGIS Online / Bing Maps / Google Maps. Click OK, and you will see the ArcGIS Online, Bing, and Google scales appear in the dropdown. Take some time to jump between them and examine the layers and symbols that will be in your tile cache.
   

   Figure 4.2 The ArcGIS Online, Bing, and Google scales appear in the dropdown menu
5. You'll publish this map as a service in the next part of the lesson. But before closing this map, you'll do one more thing to prepare for creating your cache. This map of the greater Little Rock region would take several hours to cache between 1:1,000,000 and 1:4,500. Since your resources are limited (in the sense that you are just caching with one server and are paying by the hour), you will just create tiles for the area around the city of Little Rock.
   
   To constrain tile creation to just the area around Little Rock, you'll need to have a feature class defining where you want tiles to be created. You could just use the Little Rock city limit boundary, but such a complex shape is not needed and can sometimes slow down the software. It's more desirable just to have a general shape of the area that you want cached.
6. Use your ArcGIS Desktop editing skills to create a new polygon feature class called CacheArea in the WGS 1984 Web Mercator coordinate system (be careful not to choose WGS 1984 World Mercator, it's Web Mercator.) You could potentially create this feature class right inside the file geodatabase holding all the other Little Rock data.
   
   If you cannot remember how to create a new polygon feature class, please refer to the ArcGIS Help and, if necessary, the course discussion forums for guidance.
7. Use the ArcMap editing tools to start editing your new feature class. Create one rectangular feature surrounding just Little Rock and North Little Rock (on the north side of the river), and save your edits. The screen capture below shows the general area your feature class should cover.
    
   

   Figure 4.3 The general area your feature class should cover
8. Save your edits and stop editing.
9. Remove the CacheArea feature class from your map. This is important! You don't want to cache this rectangle feature class on top of your map, you'll just use it to define your cache boundary.
10. Navigate your map so that it covers the full Little Rock region and then save it.
11. Leaving ArcMap open, move on to the next part of the lesson.

Now that you've finished designing your map and you've published it to the server, you're ready to start creating the cache of map tiles. As an advance notice, you should plan at least one continuous hour to work on this page of the lesson.

In this lesson, you'll learn how to create tiles using ArcGIS Server. However, tiles can be created using many other types of GIS and mapping utilities. Mapnik [9] is an example, which is used to create the tiles for OpenStreetMap.

Map tiling has become so popular that the Open Geospatial Consortium (OGC) has even released the Web Map Tiling Standard (WMTS) detailing an open specification on how mapping web services should expose their tile sets. ArcGIS Server services that have a tile cache can respond to WMTS-formatted requests.

Creating a tile cache with ArcGIS Server

When you publish a map service or image service in ArcGIS Server, you can define whether it will have a cache and what the cache properties will be. You can either build the tiles right at the time the service is published, or you can instigate the tile building later using geoprocessing tools like Manage Map Server Cache Tiles. Building the tiles at publish time is appropriate for smaller cache jobs, and that's what we'll do in this lesson.

1. Make sure you are logged in to your EC2 instance and displaying StMap_Template_LittleRock_WebM.mxd in ArcMap. In other words, you should be at the exact same point where you left off in the previous section of the lesson.
2. Click File > Share As > Service, and start the process of publishing a map service named LittleRock using the pattern from previous lessons. Stop when you reach the Service Editor window.
3. Click the Parameters tab of the Service Editor, and set the anti-aliasing level to Fast for this lesson.
   When making cache tiles, you should typically use anti-aliasing to make them appear as nice as possible. However, each level of anti-aliasing quality can increase the time required to build the cache, sometimes by orders of magnitude. The Fast level is a good compromise for the purpose of this lesson.
4. Click the Caching tab of the Service Editor, and choose to draw this map service Using tiles from a cache.
   This is where you'll define all the properties of your cache. You've already set up your map to use the Web Mercator projection, and you've ensured the map is designed for the scales of the ArcGIS Online / Bing Maps / Google Maps tiling scheme. Therefore, you can take the default value ArcGIS Online / Bing Maps / Google Maps in the Tiling Scheme drop-down. You can think of a tiling scheme as defining an empty, three dimensional "container" of tiles. The three dimensions are the scale levels, the width of the map, and the height of the map. At this point, you need to selectively fill up that container with tiles. You won't create tiles at all the scales, and you won't create them across the entire width and height of the map. Doing so wouldn't make sense, since your map does not have data or symbols for the extreme large and small scales in the ArcGIS Online / Bing Maps / Google Maps tiling scheme. Also, you are not interested in creating tiles outside the Little Rock urban area. If you wanted to use your own set of scales or a projection other than Web Mercator, you could use the Generate Map Server Cache Tiling Scheme tool to make your own tiling scheme file (which is just simple XML), and then you could browse to it by picking A tiling scheme file from this drop-down.
5. Examine the Levels of Detail slider, but leave the defaults going from 1,155,581 - 1:4,513. Although there are 20 scales in your tiling scheme, some scales may be way too detailed or too far zoomed out to be of use. As the service publisher, you can use these sliders to define the range of scales at which people will be allowed to create tiles for this service. ArcMap set these defaults by examining the scale ranges and the extent of data in your map document.
6. Click the Advanced Settings tab in the left menu. Some of these settings are not really advanced, they are fundamental.
7. In the Area of interest to cache drop-down, choose Import from a feature class and browse to the CacheArea feature class you digitized in the previous section of the lesson. This will cause tiles to only be created around the city of LittleRock. By default, ArcGIS Server caches the full extent rectangle of the map document, but this can be wildly inappropriate if your map has a lot of peripheral data in it or if you are caching a non-rectangular geography like Florida (you don't need a bunch of tiles created out in the Gulf of Mexico). Always consider masking your job to a feature class boundary using this option. Also, any time you run the Manage Map Server Cache Tiles tool manually, you should consider using a feature class to define the tile creation area. The feature class you define here in the Advanced Settings tab only gets used when you build the tiles automatically at publish time, which we will do.
8. Change the Tile format to MIXED and set the Compression level to 90. The image format you select for your tiles can affect your map appearance, the amount of space it takes to store your cache, and the amount of time it takes to download your tiles to the browser. The MIXED format uses the compressed JPEG format in the middle of the cache while using the larger but transparency-supporting PNG32 on the periphery where the map background is detected. This allows you to overlay your cache on top of other caches, while getting a relatively small size of cache on disk. The Compression setting refers to the amount of compression applied to the JPEGs, where 0 represents the most compression (but poorest quality) and 100 represents the least. In a vector map like this one, you need to keep the quality setting relatively high to reduce fuzziness around lines and labels.There are many guidelines about image format in the Esri help, including discussion about a new PNG format that can optimize the storage space for a cache (but takes longer to generate). See the "Tile format" section of Available map cache properties [10].
9. Switch back to the Caching tab. At this time, you're going to make a key decision about whether to create the cache tiles right away at the time you publish, or create them later in phases by running the Manage Map Server Cache Tiles tool. For smaller jobs that can be completed in less than a few hours, it may be appropriate to build the tiles right away. One way you can determine the size of your job is to run the Calculate Cache Size utility, which builds some sample tiles and attempts to extrapolate the size of your cache. It will use the rectangular extent of your CacheArea feature class to perform this estimation.
    
10. Click Calculate Cache Size, set the Estimate Quality to Best, and click Start. Notice that sample tiles are being built at each scale level, and you get an estimate range based on the size of the sample tiles and the area of your cache. The estimate you see will vary somewhat from the image below, based on the size of your CacheArea polygon and the sample tiles created.
     
    

    Figure 4.4 Example of an estimate
    My estimate above is about 90 MB. Your results will vary somewhat depending on how big you made your area of interest rectangle around Little Rock, but it should be in the ballpark. That's a relatively small cache, and in this lesson, we'll just create the cache immediately at publish time.
11. Close the Calculate Cache Size utility, then go ahead and click Publish. You see the familiar message about the service being packaged and published, but then you should eventually also see a message that the tiles are being created.
     
    

    Figure 4.5 Message to let you know the cache is being generated
    Now you wait for the tiles to be created. This is the price you pay to get a really fast map service. The cache you're building for this lesson should take under an hour to generate. Do not stop your instance while tile generation is in progress. As the message says, you can view the progress of the cache in ArcMap. Just remember that your machine is currently using a lot of CPU and memory building the tiles, so ArcMap will be slow. Patience is of the essence, and you may have to wait a while for windows to open, and so forth. However, try the following steps to get a view of your cache progress.
12. In the Catalog window of ArcMap, right-click your service and click View Cache Status. Be aware that it can take an especially long time for this context menu to appear after you right-click. After a while, a cache status report appears that you can expand by clicking Show Details.
     
    

    Figure 4.6 The Cache Status window shows that tile generation is in progress
    This reports the progress of your cache as a percentage of the full extent of your map (NOT your area of interest). Therefore, your cache for this lesson will probably wind up including about 15 - 25% of the total available tiles for this map document. The key piece of information for you to focus on is whether tile generation is in progress, or has stopped (see the red circled area above). This status does not update automatically. You need to click Refresh Status every now and then (not in rapid-fire succession!) to request an updated report from the server.
    
    I have sometimes run into repeated crashes of the tiling processes when trying to build tiles of this map. If the reported number of tiles is not increasing when you view the status, or you get an error message about a crash, just take note of the scale level at which the tiles stopped generating, and move ahead to the next steps. It's not critical that you finish creating all the tiles for this Little Rock map; it's just for practice.
13. While your tiles are being built (or afterward), take a moment to open Windows Explorer and navigate to the path: C:\arcgisserver\directories\arcgiscache\LittleRock\Little Rock, AR\_alllayers.
    
    Take a look around. You'll see folders for each scale level of your cache. You might also see large files with the extension .bundle. A bundle file is Esri's way of storing a large number of tiles in one file. You can optionally store each tile as an individual image file using what ArcGIS Server calls the Exploded storage format, but when your cache starts containing thousands or millions of images it can be tedious for Windows to copy it around, assign permissions, or frankly do anything with it.
     
14. When you are done creating tiles (or if you stopped because of a crash), open a web browser on your local computer and open the Services Directory. The URL will look like this: https://namegeog865.e-education.psu.edu/server/rest/services/ [11]
15. If you don't see your LittleRock service in your list, open the ArcGIS Server Manager (https://namegeog865.e-education.psu.edu/server/manager [12]) in your browser, and repeat the steps we performed in earlier lessons to make your service viewable by the public (Everyone).
16. Back on the rest services page, click your LittleRock service, and choose to View in ArcGIS JavaScript.
17. Zoom and pan around the map for a while, and note how quickly the tiles appear. Note that if your tile caching processes crashed, then you'll only be able to zoom in to the scale level at which the tiling stopped.
18. When you are done creating and experimenting with tiles, stop your AWS Instance.
19. Following the same pattern you used earlier in this lesson, open the AWS Management Console, and set your instance back to the t2.xlarge Instance Type.

The tile cache you just built was pretty straightforward. You just gave the tool an area of interest, it created tiles, and within an hour or two, you had your cache. In this case, you were fortunate that you just needed a cache of Little Rock, Arkansas. But what if you needed a cache of the entire United States, or world, down to a large scale like 1:4,500? This could take days or weeks to build, and could require terabytes of disk space. Even if you were successful at building such a cache, would you be able to do it again if the source data were updated?

This section of the lesson discusses strategic approaches for building large caches. These are presented in the order that they should be considered, meaning that if you skip down and implement one of the later strategies first, you still may end up doing things inefficiently.

Using an existing tile cache

If you need a tile cache that covers an enormous area at large scales, it would be worth your while to consider using one that someone else has built. Why go to the trouble if someone else has done it already? You've seen these types of worldwide tiled map services already throughout this course. They include ArcGIS Online, Bing Maps, and Google Maps. The companies who have built these caches have spent many thousands of dollars and hours collecting the data (often competing against each other for the best quality), building the tiles, and purchasing the hardware to serve them out in a rapid way. If you can get away with using them, you may save much time and resources.

The disadvantage of using someone else's tiles is that you cannot guarantee the accuracy or currency of the data. You don't get to choose the symbology or projection of the data either. Usually, you have to work in the Mercator projection.

Finally, if the tiled service goes offline for some reason or you lose your connection, you may have no control over when it will reappear. No server, whether it's maintained by Microsoft, Google, or Esri, can guarantee 100% uptime; however, this applies to your own servers as well. It's likely that these third-party services have better hardware infrastructure than your own when it comes to serving tiles; however, those tiles must still cross the Internet to get to your app, and that opens the door to potential connectivity problems.

Some organizations, especially those in the military and intelligence communities, have much of their network blocked from Internet access. Recognizing this, some tiled map service providers sell an appliance, basically a big server containing all the map tiles that can be plugged into your network. This eliminates the Internet access requirements, but still requires you to load periodic updates to the appliance. The Esri Data Appliance for ArcGIS [13] is an example of this type of appliance.

Creating tiles selectively

Some areas of a web map generate a lot more attention than other areas. Someone looking for directions to a particular house may zoom in down to the largest available scale in an urban area. However, in the middle of the desert where there are few geographic features to see, it's unlikely that someone would ever zoom to a very large scale such as 1:1100 (the largest scale offered by ArcGIS Online/Bing Maps/Google Maps).

Creating tiles at small scales isn't a problem since it takes relatively few tiles to cover the map, but if you are limited on time or disk space, it pays to be selective about which tiles you cache at the largest scales.

Some GIS professionals have a hard time accepting the fact that they don't need to create every tile at every scale. They feel that all places are created equal, and shudder at the idea that someone might zoom to an area of their map and see a "Data not available" image. In fact, such an experience is now commonplace among laypeople who use web maps, who tend to blame themselves when they see a "Data not available" tile ("Oh, I zoomed in too far") as opposed to blaming the server administrator ("Why isn't there a map here!?")

A useful website for countering the idea that "all places are created equal" was Microsoft Hotmap, an old project by Microsoft Researchers to visualize tile usage in Virtual Earth (now Bing Maps). This site is no longer functioning, but a screenshot below will give you an idea of its appearance. You could open Hotmap and zoom into your town, then use the Select Data Level dropdown to visualize tile usage at different levels. At the zoomed out data levels, most of the tiles are requested fairly often. But when you get down to the zoomed in data levels (17 - 19), some clear patterns begin to emerge regarding where people want to see tiles: urban areas, major roads, coastlines, and other areas of interest. There are also some places where people never or rarely view tiles: wilderness areas, bodies of water, and so on. These are the tiles you don't want to spend your resources creating and storing (for more images and analysis see Fisher D 2007 Hotmap: Looking at geographic attention. IEEE Transactions on Visualization and Computer Graphics 13: 1184-91 [14] and Fisher D 2009 The Impact of Hotmap. WWW Document [15].

Figure 4.7: Screenshot of Microsoft Hotmap covering Southern California at a mid-range scale level. Notice that the tile usage classes in the legend are based on a logarithmic scale, not a linear one, showing that tile usage jumps by powers of 10 across a short amount of space.

A few years ago, one of the authors of this course undertook a project to selectively cache the state of California using the observed usage patterns in Hotmap. He and his colleague combined urban areas, roads, coastlines, and places of interest into a single vector dataset that covered about 25% of the land area of California, but included about 97% of its population. The use of this dataset to define tile creation, as opposed to the entire state boundary, saved nearly 1 million tiles when caching down to the 1:4500 scale (see Quinn S and Gahegan M 2010 A predictive model for frequently viewed tiles in a web map. Transactions in GIS 14: 193-216).

When using ArcGIS Server to create tiles, there are a couple of settings on the Manage Map Server Cache Tiles tool that allow you to be strategic about which tiles you create. These are the ability to check on and off the scales you want to create, and the ability to pass in a feature class boundary that will define the area of tile creation. For a large caching job, you'll probably run the tool at least twice. The first time, you'll have only the small scales checked, and you won't pass in a feature class, you'll just create all the tiles. The second time, you'll have only the large scales checked, and you will pass in a feature class constraining the area where you want to create tiles, just like you did in the previous section of the lesson where you passed in the urban Little Rock feature class.

Optimizing the map drawing speed

The faster a map draws dynamically, the faster it will create cache tiles. All GIS software has its potential tweaks that can be made to increase performance, and ArcGIS is no exception. You've already learned, for example, that you can analyze your map using the Analyze button on the Service Editor window and see a list of potential performance issues.

Anything you can do to reduce computation will help your map draw faster. Matching the coordinate system of your source data, your data frame, and your web map will eliminate any costly projection on the fly. Saving out your labels to annotation (a way of storing labels in a database) will relieve the server from having to make label placement decisions while it is drawing your map. Spatial indexes [16] can help your map more quickly find the features that it needs to draw for each requested tile.

Increasing computing resources

The more computing power you can put behind creating tiles, the faster you can build your cache. CPU and memory restraints are often more of a problem than having enough disk space to store the tiles.

There are two ways you can increase your server computing power, scaling up or scaling out. Scaling up means you replace your existing machine with something more powerful, like we did in this lesson. Scaling out means that you add more servers to your architecture, with these servers possibly all having the same size and spec.

The concept of having more than one server working on one job is called distributed computing. Although distributed computing can allow you to do great things, it comes with some unique challenges. All machines have to be able to see the data and access it, which may require some adjustment of paths used in your maps. For example, in a distributed setup, you want to use network paths like \\server\data, instead of local paths like c:\data. Cloud Formation sets up your site so that if you put your data in C:\data on the site server instance (one named SITEHOST, for example), you can reference it through the path \\SITEHOST\data from any machine in your site.

Distributed computing may also require some adjustment of security settings so that the tile creation software has permissions to access the data from any machine. In ArcGIS Server, this is accomplished by giving the ArcGIS Server account permissions to your data folder (Cloud Builder does this for you), and registering the data folder with ArcGIS Server (you did this earlier in the course).

Building a cache in the cloud

Cloud computing can be an attractive environment for building caches, because you can access a higher level of computing power than you might typically have in your office. Usually, you only need it for a short period (a few hours or days to create all the tiles), so the prospect of renting a server by the hour becomes very attractive.

One challenge with building tiles in the cloud is moving them around. First, you have to get your data onto the cloud so that your caching software can quickly get it as the tiles are being drawn. Then you have to move the tiles back to their final home, which may be on premises. Both of these transactions involve moving data across the Internet and can be influenced by your organizations' bandwidth and security policies.

When creating tiles with ArcGIS Server on Amazon EC2, it's a lot easier to scale up than to scale out. As you have seen, Amazon offers the option to change the instance type (in other words, CPU, memory, etc.) without terminating the instance. This is very handy when you start doing something and realize you need a bigger machine, although you are required to stop the instance before you change size. Some of the largest instance types on Amazon EC2 have an enormous degree of CPU power and may negate the need to scale out.

Scaling out ArcGIS Server on Amazon EC2 is accomplished by adding more GIS server machines to your site. You must do this in Cloud Builder, either by setting a fixed number of machines or by configuring auto-scaling rules that increase the power of the site on an as-needed basis. You are allowed to edit the number of machines either before or after you have built the site.

Summary

Think back over the above strategies and consider why the techniques at the beginning should be employed before those at the end. It can be exciting to think about how many tiles you can build with distributed computing and all the computing horsepower that's available through the cloud. You may actually save the most time and resources by carefully planning which scales you want to create and selectively generating tiles at the largest scales. If the cache is still going to be overwhelmingly large, consider using an existing cache or a data appliance. By using a combination of the above strategies, you can usually find a way to build the cache you need, whatever the size.

A tile cache is just a picture of your data at one point in time. If that data ever changes, you need to update the cache. This final section of the lesson gives some practical considerations for updating and maintaining a cache over time.

Update strategy should affect the decision to make a tile cache

Your update strategy probably should have come into consideration before you even decided you were going to create a cache. If you need to see data in real time, or you have frequent changes occurring over broad extents of the map, then creating a tile cache may not be appropriate.

For each map, there's a threshold of acceptable data currency. For a neighborhood street map available in your handheld GPS, you may find it acceptable if the street data is updated once every three months. For a tax assessor looking at land parcels, it may be acceptable to have the data current to within the past day or two. For a 911 operator tracking a vehicle's progress, a delay of more than a few seconds may not be acceptable.

If the cache update can be performed within the threshold of acceptable data currency, then it may make sense to create a cache. If the cache cannot be updated that quickly, then caching should not be used.

Focusing cache updates

There are two approaches for cache updates; generate the entire cache, or focus the updates on places where the data has changed. If your entire cache can be rebuilt within the threshold of acceptable data currency, then it may be easier to do the first option, you can just kick off a rebuild of all the tiles and be done.

If your cache is very large and it is undesirable to rebuild the entire thing, then you need some way to track places that have been edited (for the sake of this discussion, we'll call these "dirty areas"). You can then pass the dirty area polygons into your caching tools to define where tile updates should occur.

So how do you find the dirty areas? One approach is to track them as edits are being made, each transaction can be logged to a database and, at the end of the edit session, the spatial extents of all the transactions can be exported to create a vector dataset of dirty areas.

If real-time tracking of the dataset editing is not an option, you can attempt to compare two datasets directly for attributes or spatial features that do not match. This type of strategy is required when you receive a dataset update without any record of how it was created (such as from a data vendor every six months). It requires that features have at least one key field in common between the two datasets. Comparing attributes is necessary if map symbolization or labeling could change based on a field value.

Accomplishing either of the above solutions in ArcGIS requires custom programming. Fortunately, this problem is common enough that people have posted some scripts and tools online that help address it. The Show Edits Since Reconcile [17] tool, written by Tom Brenneman, compares two versions of an ArcSDE geodatabase and outputs a feature class of spatial discrepancies. It can be installed into your list of toolboxes in ArcCatalog. A similar tool Compare two feature classes in a file geodatabase [18], written by Sterling Quinn, is designed for those who do not have their data in ArcSDE.

Basing an ArcGIS tile cache update on dirty areas requires some degree of caution. A feature class full of small, adjacent polygons can cause the Manage Map Server Cache Tiles tool to work slowly and inefficiently. If there are a lot of small dirty areas in close proximity, they should be merged before the dirty areas feature class is used to define a caching job.

Automating cache updates

It's common to perform tile cache updates on a regular basis, such as every three months, every week, or every evening. Because caching is so resource-intensive, many server administrators like to build the updated tiles on a staging server and then copy them to their production server. This avoids disruption to those who are viewing tiles on the live website.

Whether you use a staging server or not, it's wise to perform the update during times when the fewest possible individuals will be using your site. For most sites, this is during the early morning hours or the weekend. Since you probably do not want to log in at 2 AM Sunday morning to run your caching tools, it's worth exploring whether your tile caching software can be automated and scheduled to run at given times.

The ArcGIS tools, for example, can be automated using a Python script. Python is a relatively simple programming language to learn, and it can be used to run any ArcGIS tool, including Manage Map Server Cache Tiles. For a full update process, you might decide to chain several tools and functions together in one script, such as:

• Compare two feature classes in a file geodatabase (to find the dirty areas)
• Manage Map Server Cache Tiles (passing in the dirty areas feature class to define where tiles are created)
• A copy function to move the new tiles from the staging server onto the production server

Once you have a script that does everything you need, you can use your operating system to schedule it to run on a regular basis. Task Scheduler, included with Windows, is an example of a program that can run scripts on a repeated basis at any time you specify (such as nights or weekends).

Python scripting with ArcGIS is taught in Penn State's Geog 485: GIS Programming and Software Development [19]. If you're curious to see an example of a Python script that updates a cache, check out the ArcGIS help topic Automating cache creation and updates with geoprocessing [20].

In this assignment, you will put together all of the ArcGIS Server skills that you learned in Lessons 2 - 4. Starting with a folder of raw GIS datasets, you will compose maps, publish them as web services, and assemble those services into a web application. You will create a video tour of your web application so that you don't have to leave your server running as the project is graded.

The data for this assignment consists of vector feature classes covering an area around a town. I downloaded these from the State of California Geoportal [21] (formerly the California Spatial Information Library - CaSIL) and did some post-processing on them so that they cover the same extent. Don't worry too much about what town this really is; for this assignment, consider that it could be Anytown, USA.

Download the data for this assignment [22]

The scenario

Pretend you work for a town that up until now has only done GIS in the desktop realm (maybe there is no pretending needed). You are moving to ArcGIS Server for the first time. You want to take your GIS data and make it available in a series of highly-focused web applications.

Your first application will focus on your urban flooding dataset. This is a point feature class that shows areas in the city that tend to pool with water and flood during a storm event. Your web app will allow "non-GIS-trained" personnel in other city departments to add and remove points from this layer.

You've been asked to create a basemap web service that will be used as a backdrop in this web application and other apps your town will create in the future. You must design this basemap yourself and create a tile cache for it. An existing basemap from ArcGIS Online, Bing Maps, or Google Maps cannot be used because the map needs to show your town's own data. However, you can imitate design principles and techniques used in those maps.

You are also to create a separate web service containing only the urban flooding layer. This layer should be exposed as a feature service and should be editable. This involves loading the source data into SQL Server Express as shown in Lesson 3.

Once you have created these two web services, you must overlay them in a web application that allows the urban flooding service to be edited by the application user. Do this using the ArcGIS Web AppBuilder unless you already have extensive coding experience with another API such as the ArcGIS API for JavaScript.

Because this assignment takes a fair amount of time, there is no cloud computing discussion assignment this week.

Deliverables and grading

To minimize the amount of time your cloud-based server is left running, this project will be graded based on a short video tour of your app. You should record this using Zoom, Screencastomatic, or a comparable screen recording utility of your choice. Your video must demonstrate the following features in your ArcGIS Services Directory and your flooding application. Each item is worth 3 points, resulting in a total of 30 points available for this project (making it three times the value of a typical weekly assignment):

• The application contains a basemap service that has a tile cache built at appropriate scale levels (not too large, not too small). To verify this, show the Services Directory page for this service and scroll down to the section that displays all the cache scales.
• The application contains a second service that displays only the urban flooding layer. This service should have Feature Access enabled so that it can be edited. Show the Services Directory to prove that it indeed is a feature service.
• The basemap has been cartographically designed not to overwhelm any operational services that are placed on top of it.
• The basemap has been designed to be cartographically sound at all cached scales. The symbology and amount of detail shown adjust appropriately as you zoom in and out. Zoom in and out in your application, and point out how you have designed the basemap using appropriate scale ranges and symbol changes.
• The urban flooding service has been designed to easily stand out when placed on top of the base map. If necessary, transparency has been set at an appropriate level.
• The web application allows the user to edit the urban flooding layer. In your video, show that you can add a point and edit its attributes.
• The services, the layers within them, and the attributes within them are intuitively named so that an end user can understand them. You can edit the dataset schemas to accommodate this requirement if needed.
• The web application starts at an appropriate initial extent, contains an intuitive title, and includes other customizations as appropriate.
• The video includes a short oral summary of things you learned, things you enjoyed, and challenges you faced when completing the project. If you can't fit this all in the video, you're welcome to write it up and post it as a comment when you share the video URL.
• The video is no longer than 5 minutes.