Web services have the potential to expose your data to a much wider audience than may have previously seen it. But beyond allowing simple visualization of the data, web services can also permit editing and creation of data over the web. This type of "web editing" can allow field workers and people who typically don't use GIS to contribute valuable information to your database, information that you might not otherwise get.
For all its benefits, exposing a database on the web comes with some challenges. How do you protect your data from becoming corrupted? If you put a database on the cloud, how do you keep it in sync with the database in your office? And what happens if multiple users edit a feature at the same time?
This lesson explores some of the requirements and challenges related to making a GIS database available for editing on the web. You'll put some data in a SQL Server Express database on your EC2 instance, and you will use that data to design a map for web editing. You'll learn about how ArcGIS Server provides a special type of "feature service" that is engineered to allow editing through a web service. Finally, you'll make a web application that allows others to edit your data over the Internet.
Throughout this lesson, you'll be guided with step by step instructions. At the end of the lesson, you'll post a screenshot of your work. Pay close attention to what you are doing, because next week you will be assigned a project in which you will have to think through these processes on your own.
At the successful completion of this lesson you should be able to:
The ability to expose a GIS dataset to Internet users and allow them to modify it presents some enormous opportunities and challenges. A GIS professional needs to carefully understand and weigh these considerations before making decisions about how to make data available for web editing.
Web apps for editing GIS vector geometries were cumbersome and somewhat rare until about 2005. Attributes could be sent to a database through a web service fairly easily, but sketching geographic features on a screen posed some different problems. How could vertices be drawn in the web browser in real time as the user sketched them, without the entire page refreshing? Or how could a user view a snapping threshold on the screen while making a sketch? These problems were somewhat alleviated when AJAX came on the scene.
The bane of web developers up to this point had been the necessity of doing a full send and retrieval of information to the server in order to accomplish anything, with the ubiquitous "page blink" occurring in between. AJAX was not a particular product or feature, but rather a technique that web developers devised to work with existing technologies, with the goal of making their apps more interactive.
JavaScript is a language web developers use to program actions on web pages (in contrast to HTML, which is markup language used to lay out the static elements on the page). AJAX stands for Asynchronous JavaScript and XML. Web developers discovered that they could use JavaScript to send and retrieve XML packets of information from the server to create certain effects in their applications, without doing a full refresh of the page or requiring any type of browser plug-in. This revolutionized the interactivity of web applications.
Perhaps you remember the first time you saw Google Maps. This was actually one of the first programs, mapping-related or otherwise, to really give people an idea of the power of AJAX. Google Maps used AJAX requests to request pregenerated map images as the user panned and zoomed, creating a smoother web map navigation experience than most people had ever seen. Virtually all major commercial web mapping sites now use this approach.
AJAX techniques helped open the door for interactive editing of GIS geometries through web applications. Users could now sketch edits on their maps and see each vertex of the sketch drawn in real time without being interrupted by page blinks or waiting for the browser to respond. They could press a key and immediately see a snapping threshold that would be applied to a vertex. People began to think about the ways web browser-based editing could improve their GIS. In more recent years, people have also began to consider benefits of smartphone and tablet-based editing.
There are often many individuals within an organization who lack extensive GIS training, but could still contribute valuable information to the GIS. These include receptionists, field technicians, planners, and managers. Web editing comes with several advantages for these types of professionals.
First, virtually everyone has used a web browser, so an intuitively designed web application can be much less intimidating for them than a full-featured desktop GIS program like ArcMap.
Another advantage is that the web app can be specifically tailored to certain editing tasks that are within the audience's realm of expertise - no more, no less. If you need field technicians to sketch the locations of telephone poles, you can design a web app that allows sketching of telephone poles and nothing else. You can make the symbols big and round and even tappable on a smartphone by someone wearing large gloves. You can make the app as simple as needed, remembering that a simple app can still collect very valuable information.
Because everyone knows how to use a web browser, web editing gives you the potential to allow everyone to contribute to your GIS. When might this be a good thing? When everyone knows something that you don't! Or when the power of everyone can create something more complete, useful, or accurate than you can create on your own.
Enter two buzz terms that have crept into the GIS world in recent years with the advent of web editing, crowdsourcing, and volunteered geographic information (VGI). Crowdsourcing is the idea of allowing anyone to edit an information repository, with the faith that this will make the repository more complete and accurate over time. Wikipedia is an example of a crowdsourced online encyclopedia. In the GIS realm, OpenStreetMap is an example of a crowdsourced street map of the world. People hold mapping parties for OpenStreetMap where they ride and walk around a town collecting data and then enter it all into a database. Regardless of your feelings on using crowdsourced data, this type of activity undoubtedly increases the quantity and accuracy of the data already in the database (especially if the database previously contained nothing).
Similarly, VGI allows individuals to enhance a dataset with information that they alone may possess. Whereas the term crowdsourcing evokes images of mass participation in the creation of publicly-available dataset, VGI is more versatile in that it can contribute to private or temporary datasets. For example, a city might set up a VGI application to allow citizens to report broken streetlights, graffiti, overgrown trees, and so on. This information may remain proprietary to the city and it may go away over time, unlike a crowdsourced database that is expected to remain more or less available.
You've learned that the beauty of web services is that they communicate by common architectures and protocols like REST and SOAP, that can be invoked by any device with a connection to the Internet. These devices include the numerous tablets and smartphones that have hit the market in recent years. The ArcGIS Server feature service that you will create in this lesson could potentially be used in editing apps for the iOS, Android devices, or Windows Phone.
Smartphone and tablet-based editing greatly facilitates the field work and crowdsourcing scenarios that you learned about above. It's a lot easier to record something you see on a street, such as a pothole, if you can send it to the database right away from your mobile device. The editing app may even allow you to attach a picture to your feature that you captured just seconds before with your phone!
Along with the benefits of web editing comes a number of challenges. These need to be well-understood and dealt with appropriately by anyone planning a web editing implementation.
When thinking about web security, it helps to consider all the different tiers, or levels, at which someone can access your system. Then consider how each tier might be vulnerable and how it might be secured. With web editing security, you need to at least consider the application tier, the web service tier, and the data tier.
If this sounds confusing to you, let's talk through them one at a time. First, consider the application tier. You need to decide which people will have access to your web editing application. Is it open to everyone on the Internet? This is the easiest to set up, but it also results in the most vulnerability. Your organization's existing firewalls might also make it fairly easy to set up an application that's only visible to members of your internal network, in other words, people who work for your organization. A trickier kind of application security to set up is one that has a login page where only certain members of your organization are allowed to log in, but not others. Setting up this type of security is certainly doable, but is beyond the scope of this course.
The next level of security to think about is the web service tier. An organization can set up its server such that certain services require a login to access. If the application itself also requires a login, the application developer must figure out how to get that name and password to be applied to the services accessed therein.
Regardless of whether you decide to require a login for your services, you should consider which layers should have editing allowed. There are some datasets that you'll want to expose for web editing, and others that you will not. You need to design your web services such that they only allow editing of those particular datasets that you want to have modified. Several days before writing this lesson, I came across a web map showing election results in a certain country. The authors of this map had used a feature service with popup balloons to display the election results. This was not a bad plan; however, the web service authors had inadvertently left the editing capability exposed on the feature service. I discovered that I could potentially click a popup balloon and literally rewrite the election statistics for any particular province!
Your software will give you controls over which features may be edited, and you must carefully understand and use these controls. Sometimes you may be required to group editable layers in one web service and non-editable layers in a separate web service that has different security settings.
The final tier of security to consider is the data tier. By exposing your dataset for web editing, you are opening your database to many more people than would otherwise have access to it. You need to plan for the scenario where a malicious party could corrupt or delete your data. Keeping a backup or replica of your data is recommended in web editing scenarios.
In addition to the threat of a malicious party corrupting your data, it's possible that a well-intentioned user could make a mistake and negatively affect your database. Before you take all the changes submitted by your web editors and push them into your on-premises database, you might choose to have a GIS analyst examine the edits and perform a quality check. This type of scenario is possible if you maintain separate replicas or copies of the database for web editing and for on-premises work.
You can reduce the possibility of data corruption by carefully limiting the types of features and attributes that web editors can access and create. Later in this lesson, you'll use feature templates that ArcGIS provides for editing. The feature templates help you give the web editor a palette of approved features that can be created, while making it impossible or difficult to create other types of features. For example, if you want your web editors to add only 8", 12", or 16" pipes to the database (no 20" pipes, or fire hydrants), you can create a feature template with only those three types of pipes, with the size attribute preset for each one.
You've learned in this section that keeping a copy of your data for web editing and a separate copy for your on-premises work can be a good practice for maintaining security and data integrity. The tricky part is synchronizing the two copies at the appropriate times, with only the appropriate changes. ArcGIS contains a feature called geodatabase replication that can help with this.
The simplest option for replication is to make a one-way replica of the geodatabase, which is essentially a one-off copy made in a desired projection and format. This is useful for exposing a read-only database for web use. Some sites use one-way replication into the mercator projection for their web database, since they need to use mercator on the web but not in their office.
A more complex, but more useful action is to make a two-way replica of your database. This creates a copy of the database for web editing that can be synchronized with the original (or "production") database at intervals that you choose. A GIS analyst can potentially examine the web edits before synchronizing them with the production database. If the two databases are separated by firewalls or reside on different networks, an ArcGIS Server geodata service can be used to synchronize the two. This is beyond the scope of the course, but it's important for you to have a basic knowledge of these architectures in case you ever need to implement them.
If replication is a new concept to you, or you would like to learn more about it, you can read the first several topics in the ArcGIS help section Managing distributed data [1].
GIS vector datasets come in many formats. Some of these are better suited to web editing than others. Since we are working with ArcGIS Server in this exercise, we'll talk about some of the data formats that Esri offers and which ones are required for web editing. You'll then load some GIS data into a database on your EC2 instance.
Whether you're working with Esri software or not, one of the most ubiquitous formats for exchanging GIS datasets is the shapefile. This is a data format developed and openly documented by Esri, meaning that other software companies are allowed to use, create, and share shapefiles. A shapefile actually consists of multiple files with the same root name and different suffixes (.shp, .dbf, .prj, etc.) that store the data's geometry, attributes, projection information, and so on. You'll often see shapefiles available on GIS data warehouse sites that allow you to browse and download geographic datasets.
A shapefile is handy for exchanging data, but it's not very useful for web editing. Because the shapefile is an openly documented file format, it may be possible for a web developer to write an application that edits shapefiles. However, this would be a significant amount of work and ArcGIS does not supply out-of-the-box web editing functionality for shapefiles. Nor does ArcGIS support web editing with the shapefile's more advanced (but less openly documented) cousin, the file geodatabase.
In order to perform web editing with ArcGIS Server, your data must be stored in the ArcGIS Data Store or in a geodatabase hosted in a relational database management system (RDBMS). Here's what those terms mean:
Why are these things required for web editing with ArcGIS Server? One thing you have to consider is that when you configure editing on the web, you may not want to expose your main production database to everyone on the network. Your data is valuable. You may have spent thousands of dollars collecting it. It may be required to meet certain quality standards. To protect your data, you'll probably choose to expose a copy, or replica, of it for web editing. This replica goes on your EC2 instance. You'll keep a separate replica of the data in your on-premises environment. This on-premises replica can be protected by your firewall, data quality checks, and so on.
From time to time, you can synchronize the two replicas using ArcGIS software tools. This means that one replica gets sent the changes that were made to the other replica, and vice versa. ArcGIS Server even provides a special type of web service for synchronizing two replicas, called a geodata service.
Terms you may see during this lesson include geodatabase and feature class. Geodatabase is an Esri-coined term to describe a database containing related GIS datasets, tables, relationship classes, topologies, and so on. A feature class is a vector dataset within a geodatabase.
Let's load some data onto your EC2 instance and prepare it for web editing. Your whole goal is to make a map on your instance and expose it through an ArcGIS feature service, which is the type of service that you can edit over the web. The first step is to get the data onto your instance and load it into SQL Server Express.
The first part of this process for us is to install SQL Server Express and its required licensing and system components.
You learned in the previous lesson how publishing a web service requires some extra thought beyond just taking your existing map document and putting it on the server. It requires that you think about basemaps and business layers and separate those out into different services. It requires that you think about the coordinate systems of your data and the services you will overlay. Throughout this course, you'll learn of even more things to prepare for as you design a web service. In this section, we'll cover some considerations for web editing.
There are perhaps some layers in your map that you will want users to edit, and other layers that you will not want anyone to modify. For the most fine-grained control, the editable layers should be isolated into their own web service, with the non-editable layers being published in a separate service.
Once you isolate the editable layers into their own ArcGIS map document, you can set up feature templates that determine the types of items users will be allowed to create. You can predefine the symbology and some of the attributes of these items to make the job of your editors (and, as you will see, your web application developers) as simple as possible.
When the map is ready, you publish it to ArcGIS Server with the Feature Access capability enabled.
Let's take a look at how you can design a map document (.mxd file) for web editing. You will start with an MXD that was included in the BighornSheep lesson data that you downloaded previously.
In the next section, you'll publish both your maps to ArcGIS Server so that they can be used in a web editing app.
In order to get your maps into a web editing app, you need to publish them as services. Specifically, in ArcGIS, you publish your map as a map service, with the Feature Access capability enabled. This creates a feature service that can be used for web editing.
The terminology here can be confusing. Is it a map service or a feature service? The answer is...both. When you look at your GIS server as an administrator, you'll only see one service, a map service with the Feature Access capability enabled. However when you look at the GIS server as a consumer of the service, for example when you are developing a web app with the service, you will see two ways that you can access the service. You'll see the map service URL and the feature service URL. You need to use the feature service URL in order to access web editing functions. The feature service provides methods (or REST "operations") for editing. These operations include Add Features, Update Features, Delete Features, and Apply Edits. You have to enable the Feature Access capability and use the feature service URL (it ends with "FeatureServer") in order to get these methods. They don't come with a regular old map service.
In the previous section, you created two maps: BighornHabitat and BighornReferenceLayers. You'll publish the BighornHabitat map as a feature service. The reference layers map also needs to be published, but it doesn't need to have the Feature Access capability enabled.
Try these steps:
Taking into account the service you published in the previous lesson, your Services Directory should now contain the following services:
In the previous sections of this lesson, you have laid all the groundwork for allowing web-based editing of your GIS datasets. You've set up the database, prepared the maps, and published a web service that allows editing. Your final step is to make a web application that allows editing.
Web editing can be a successful or frustrating experience for users depending on how the web services and app are designed. You already did some work with your web services to make them easy to visualize and understand. For example, you made only a few layers editable and verified that a feature template was available so that users could create only certain types of features and have some of the attributes pre-set.
In the same way that you made simple, focused services for editing, you also need to make a simple, focused application. An application that has too many buttons, functions, or GIS lingo can seem over-complicated to field workers and other professionals in your organization who may need to perform web editing. Fortunately, it's a lot easier to make a simple web app than a complex web app.
In this lesson, you'll build a web editing applications using the ArcGIS [11]development tools, which will let you build in a what-you-see-is-what-you-get (WYSIWYG) environment so you can quickly and easily create an app. For those of you that are interested in going beyond the simple functionality, ArcGIS has an "extensible framework" which means you can build your own custom widgets and themes if you have some programming knowledge. We won't cover those extensions in this class but you should know that you're not limited to the tools and templates that Esri provides.
Before we start creating the app, let's assemble the web map that we want to display inside of it. You'll do this using the same ArcGIS.com map viewer that you used in the previous lesson.
Now that you've got a web map set up, you can get down to the business of creating your web app.
In your ArcGIS Online website, click the Content link in the main menu.
Click the Create App button, and select Web AppBuilder.
Choose 2D, enter a title (e.g., BighornSheepEditingApp), tags (again at least one is required), and a summary (e.g., Bighorn Sheep editing app using ArcGIS Online), click OK.
You will see the Web AppBuilder for ArcGIS screen, and then you'll be redirected to a webpage displaying a theme and other graphical styles.
Choose a style and color scheme that you like. I'll leave it up to you to be creative.
The important elements live under the Map and Widget tabs. We'll start with the Map tab.
Click the Map tab, and click Choose Web Map.
Select the BighornWebMap you created above, and click OK.
This will bring in all the layers that you configured in the ArcGIS.com map viewer. That's all you need to do for the map design. Now, let's add some widgets.
Click the Widget tab and click Set the widgets in this controller and then the +.
From the list of widgets that appears, add the Edit and Measurement widgets. Also feel free to choose a couple of others.
Click Save in the bottom of the pane and then Launch. The web app will open in a new browser tab (it might take a few seconds to start). Take a quick look around.
Now, go to your own local computer (not your EC2 instance), and log in to arcgis.com. Click Content, then click your BighornSheepEditingApp, and click View Application.
From a look at the URL, you will see that your web app is, in fact, running on ArcGIS online servers via the Penn State URL (pennstate.maps.arcgis.com); however, it is still depending on services that are running on your EC2 instance. When you stop your instance, this app will not work as expected.
If you wanted to download the source code for this app and host it on your own web server, you could easily do that using the Download link that appears by each app in your ArcGIS Online content.
Test out your Web App thoroughly by reading and following along below.
Now that you have your app created, you can use your widgets to edit some of the underlying data. If you click the Edit widget, a sidebar window will appear with the list of editable layers within your feature layer. If you select the Sightings, you can add in some new sightings and some attributes. Try it!
You can also add a habitat area by drawing a polygon on the map (follow the on-screen instructions). You can use the controls at the bottom of the edit window to modify those changes (just like you might in the Edit window of ArcMap).
When you're done editing, click the X in the upper right of the window. These changes will be saved back to the server version. You could open your database and look at it in ArcMap on the EC2 instance to prove that this is the case.
Please create a new document or PowerPoint slide show. Paste your screenshots that you took from your app. Label each with a description of what is happening in the screenshot.
Also, answer the following question in a thoughtful paragraph: What things would you change about this walkthrough or the app design if this were a real-world deployment looking at animal sightings? If you're having trouble coming up with ideas, think about this question across several layers, or tiers, of the architecture, starting at the database tier and working your way up to the GIS server tier and the web application tier.
Upload this document to Canvas in the lesson drop box.
Cloud computing advocates often cite cost savings as a reason to adopt cloud computing. But, there is no guarantee that any given project can be more cheaply executed using cloud computing than using traditional IT provisioning. This week, we will discuss how cloud computing economics might apply to your organization or situation.
First, please read this white paper from Amazon: The Well-Architected Framework - Cost Optimization Pillar [13]. This document is part of a series available here [14] on best practices for using AWS's infrastructure services. Also, if you have purchased the optional textbook The Cloud at Your Service, please read chapter 3, "The business case for cloud computing". Despite the title, it gives a reasonably objective view of how cloud computing costs break out for different kinds of users (start-ups, small and medium businesses, large businesses).
Second, please post your reaction in the lesson discussion in Canvas on the topic below
Links
[1] http://desktop.arcgis.com/en/arcmap/latest/manage-data/geodatabases/understanding-distributed-data.htm
[2] https://enterprise.arcgis.com/en/data-store/latest/install/windows/what-is-arcgis-data-store.htm
[3] http://desktop.arcgis.com/en/arcmap/latest/manage-data/geodatabases/an-overview-of-versioning.htm
[4] http://desktop.arcgis.com/en/arcmap/latest/manage-data/geodatabases/replicas-and-geodatabases.htm
[5] https://www.e-education.psu.edu/geog865/sites/www.e-education.psu.edu.geog865/files/data/BighornSheep.zip
[6] http://gis.utah.gov/
[7] https://www.e-education.psu.edu/spatialdb/l5_p3.html
[8] https://pro.arcgis.com/en/pro-app/latest/help/data/geodatabases/overview/versioning-types.htm
[9] https://www.e-education.psu.edu/spatialdb/node/2032
[10] https://baxtergeog865su22.e-education.psu.edu/server/manager
[11] http://www.arcgis.com/features/index.html
[12] http://pennstate.maps.arcgis.com
[13] https://d0.awsstatic.com/whitepapers/architecture/AWS-Cost-Optimization-Pillar.pdf
[14] http://aws.amazon.com/whitepapers/