This last lesson will cover a mix of advanced topics related to mapping application development, all of them having applicability to either of the mapping APIs we've explored in this course. We saw in the last lesson that Esri requires developers working with their JavaScript API to use the Dojo framework. Recall that by the end of the lesson we saw how to easily position our map and sidebar table, and that the table itself had advantages over a plain HTML table such as the fact that it was sortable. These things are possible with the Google API as well, and since not all of you need or are able to take advantage of the richer functionality in Esri's API, we'll spend part of this lesson covering how to develop professional looking layouts and interactive tables with the Google API and another popular JavaScript framework, jQuery.
We'll also see how HTML form elements (such as drop-down lists and radio buttons) can be used to provide end users with an interface for selecting subsets of larger datasets to visualize on the map.
Finally, we'll look at methods of dynamically acquiring the data for your map from some other site on the web. These methods include accessing data through a published web service and "scraping" data off of pages in which the desired information is not packaged in a convenient format.
The goal of the graded assignment in this lesson is for you to apply what you've learned in the course to produce a sophisticated mapping application of your choice. My expectation is that you'll work on something that goes a bit beyond the assignments you've already completed.
There are a number of directions you could take with this project. One of those directions would be to model your project after the examples in the lesson that demonstrate giving users the ability to map a part of a larger dataset. However, those of you who are looking for a bigger challenge might want to do something involving web scraping or database-driven mapping. (For the latter, I point you toward the optional database lesson and the database-related examples found here in this lesson.) I'm also open to you delving into some other advanced topic that we haven't touched on here. Examples include, but are certainly not limited to:
With the Google API:
With the ArcGIS API:
Please e-mail me your idea before you begin working to make sure that it is appropriate. I will provide a list of project ideas at the end of the lesson for those who can't come up with their own. You can choose to complete this project using either the Google Maps API or the ArcGIS API for JavaScript. If you have excelled with both these APIs and would like to try another similar API like Leaflet, you are welcome to do so with prior approval (with the understanding that the depth of help the instructor can provide may be more limited).
Lesson 7 is two weeks in length. (See the Calendar in Canvas for specific due dates.) To finish this lesson, you must complete the activities listed below. You may find it useful to print out this page so that you can follow along with the directions.
Step | Activity | Access/Directions |
---|---|---|
1 | Work through the Lesson 7 content on jQuery, HTML forms, web scraping and consuming web services. | You are in the Lesson 7 online content now. The Overview page is previous to this page, and you are on the Checklist page right now. |
2 | Complete the project described on the last page of the lesson.
|
Follow the directions throughout the lesson and on the last page. |
3 | Complete the Course Evaluation Survey. | A link to the Course Evaluation Survey (SEEQ) can also be found in the Modules folder in Canvas. |
In this part of the lesson, we're going to return to working with the Google Maps API and focus on improving the user interface. We'll see how to gain greater control over the page layout and how to turn the plain HTML table in our sidebar into one that provides additional functionality (e.g., the ability to sort). To accomplish this, we're going to use the popular jQuery JavaScript framework (actually an extension to jQuery called jQuery EasyUI).
Before incorporating jQuery into our pages, I recommend you return to the w3schools site to get a quick jQuery crash course [1]. The entire tutorial is worth checking out, but in the interest of time I recommend working through only the following pages:
One of the earlier pages in the w3schools tutorial discussed two different ways of incorporating jQuery into your web pages: downloading it and hosting it on your web server, or including it from a CDN (content delivery network) like Google. I'm going to suggest including it from a CDN, especially in this course, so that you have less files to worry about putting in place on your own machine for testing and on the PSU web server when you're ready to post your assignment. As we'll see, this simply involves adding the following script element to the page:
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js [2]"></script>
The jQuery EasyUI extension, on the other hand, is not hosted on a CDN to my knowledge, so we will download it.
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script> <script type="text/javascript" src="../jquery-easyui/jquery.easyui.min.js"></script>
<link rel="stylesheet" type="text/css" href="../jquery-easyui/themes/default/easyui.css">
<body class="easyui-layout">
Replace the existing map div with these two divs, one placing the map in the center region and another creating a sidebar in the east region:
<div id="map" data-options="region:'center',split:true"></div> <div data-options="region:'east',split:true" title="Cities" style="width:230px;"></div>
Save your document and test it out in a browser. (Consult this working example [6], if necessary.) Note that the sidebar (empty currently) can be resized or collapsed.
To insert a list of city features into the empty sidebar, we'll follow many of the same steps we took earlier in the course.
var markers;
markers = [];
function myclick(num) { google.maps.event.trigger(markers[num], "click"); }
markers.push(marker);
<table id="sidebar" class="easyui-datagrid" rownumbers="true" sortName="city" sortOrder="asc" remoteSort="false" striped="true" singleSelect="true"> <thead> <tr> <th field="city" sortable="true">City</th> <th field="pop" sortable="true">Population</th> </tr> </thead> <tbody> </tbody> </table>
function addToSidebar(name, pop, i) { var lastRow = $('<tr/>').appendTo($("#sidebar").find('tbody:last')); lastRow.append( $('<td><a href="javascript:myclick(' + i + ')">' + name + '</a></td>') ); lastRow.append($('<td/>').text(pop)); }
addToSidebar(name,pop,i);
$('#sidebar').datagrid();
Now that you've seen how a JavaScript framework like jQuery can be used to improve your Google API-based web maps, let's go further into the topic of user interface development by looking at HTML forms.
Form elements are often used in mapping mashups to:
Guess where I'm sending you next. That's right, w3schools. Work through w3schools' tutorial on HTML forms [9], and then come back to the lesson to see how form entries can be processed.
Form data are typically processed in one of two ways. The first method was briefly discussed on the w3schools page you just read and involves setting the form element's action attribute to the name of a server-side (e.g., PHP or ASP) script that will handle the user's entries. There are situations when this method is desirable (e.g., if the user's entries are to be simply added to a database on the web server).
However, I'd like to focus on using forms to give the user a set of mapping options. These options could be categories of features to display, ranges of dates, or regions to zoom to. In that context, where selections made from the form affect the appearance of a map on the same page as the form, sending the user's input to a server-side script is not actually necessary.
So, let's move to a different method that makes more sense for a mapping application. This method involves using some of the DOM and Dynamic HTML concepts you learned about earlier in the course. Specifically, a button can be added to the form and its onclick event handler can be set to the name of some JavaScript function stored in the document (or in an external file that is referenced by the document). That function can then obtain the user's input from the form and act accordingly, whether that means adding overlays from some data source, zooming into a region or something else.
Have a look at this example that demonstrates the usage of four commonly used form elements [10]: text boxes, drop-down lists, radio buttons, and checkboxes. First, be sure to note the following aspects of the HTML that differ from the examples in the w3schools tutorial or may not have caught your attention:
The process() function obtains the values entered in the text boxes and the selections made from the other controls. It begins by obtaining a reference to the form element by its ID ("myform").
The script then takes advantage of the fact that the elements of a form are available in JavaScript as properties of that form object. In other words, the element named "firstname" can be accessed using the syntax frm.firstname, the element named "country" can be accessed using frm.country, etc.
Obtaining the string entered into a text box (the two name fields in the example) or the option selected from a drop-down list (the country list) is as simple as reading the element's value property.
Determining which option the user selected in a set of radio buttons is a bit more complex. In the example, each radio button has its name attribute set to "six". Thus, the expression frm.six returns not just one element, but all the elements having that name (as an array). The general strategy then is as follows:
Checkboxes can be handled in much the same way as radio buttons with the exception that the user can select multiple options from a group of checkboxes. The approach shown in the example is to create a list of the selected boxes separated by commas. The loop contains no break statement since all the boxes must be examined. After the loop is complete, the last two characters in the list string (a comma and space that aren't necessary because they come after the last selected box) are removed.
Now that you've gone through some of the basics of using HTML forms to obtain user input, let's put this information into a web-mapping context.
The previous page demonstrated the use of form elements, but not in the context of a working web map. As mentioned earlier, one form element that's commonly used in a web-mapping context is a dropdown list that provides access to a subset of a larger dataset. So let's take a look at an example like that.
Open this rather boring County Viewer [11] app, try it out, and have a look at its source code. This app loads into memory data from a shapefile of US counties. When the user selects a state and clicks the Show button, the app loops through all of the county features and adds overlays for the ones that are in the selected state. This approach is a bit inelegant -- having the data in a database and retrieving just the needed features is probably better -- but it can be "good enough" in certain scenarios. Here are the important aspects of the code worth noting:
With that, you've seen an example of how to incorporate form elements into a Google API-based web map. Moving on to the next section, we'll shift gears to another advanced topic: "scraping" data off of other websites.
A programming technique that offers a great deal of potential for the creation of mapping mashups (and for controversy) is web scraping [13]. Web scraping is essentially reading and parsing data published on the web in formats that were originally intended only for human consumption, not machine consumption. As we saw earlier in the course, JavaScript can't read files hosted on another domain. However, a server-side language like PHP can be used to parse the text from the host site and package the desired data into a format that can be consumed by JavaScript.
PHP is discussed in a bit more detail in the optional database lesson.
Because of time constraints, I will just demonstrate this concept through an example rather than lead you through it yourself.
As a native Pennsylvanian and Penn State alum, I am a big Penn State football fan. I even follow the recruitment of high school players a bit, though I'm a bit embarrassed to admit that! Anyway, as a way of comparing the geography of football recruiting across different schools in Penn State's conference (the Big Ten), I developed a mashup several years ago that showed the hometowns of the players on Big Ten rosters [14].
So, where did I get these rosters and the latitude and longitude of each player's hometown? Well, I scraped the rosters off of ESPN's website [15], then geocoded the hometowns using Google's geocoder.
Let's begin by having a look at the HTML source of one of these rosters. If you do a search for the word "hometown," you'll be taken to two lines above where the player listing begins in an HTML <table>. Take note of the pattern that repeats itself for each row in the table.
Now, let's have a look at the scraping program [16] in which I use string manipulation functions in PHP to extract the desired pieces of data from the table:
The program begins by opening up a new XML file to hold the output from the program and an XML header and root element are written to the file.
The usage of XML as the source data format for a Google API-based map is also covered in the optional database lesson.
Next, PHP's file_get_contents() function is used to read the HTML of the specified roster into a variable ($file). The contents of this variable are then shortened to just the part of the page that comes after the first occurrence of the word "HOMETOWN" using the strstr()function. A while loop is then used to process each row in the table. The loop contains several uses of PHP's strpos() and substr() functions. strpos() returns the index position of a search string within a larger string. substr() is used to obtain a smaller portion of a larger string (i.e., all characters to the right of the character at position x). I'm not going to bore you with the details of this part of the script. If you have any questions, you're welcome to post them in the discussion forum.
Let's skip down to the statement where a long maps.google.com URL is constructed. This URL is an alternate means of geocoding an address that is available when working within a server-side script (see http://code.google.com/apis/maps/documentation/geocoding/#GeocodingRequests [17]). Here, the player's town and state are inserted into the URL and the desired output is specified as comma-separated values (csv). This URL is read once again using the file_get_contents() function. The comma-separated values are then converted to an array using the explode() function, which places the latitude and longitude values in positions 2 and 3 of the array, respectively. Once the lat/long values are obtained, all of the desired values are written to the XML output file. The last step in the loop is to whittle down the HTML string stored in the $file variable so that it no longer contains the just-processed row. The program then processes the next row in the table and continues looping until no more rows are left.
There are a number of code libraries in different languages that specialize in scraping data found in HTML tables. If you're interested in doing this sort of thing yourself, I recommend looking into whether one of these existing libraries will make your task easier.
The resulting XML file is then used as the data source for the cfb_rosters.html page, which operates in much the same way as examples from the optional database lesson.
You may be wondering why I didn't just set up these scripts such that the XML was passed directly from the PHP script to the JavaScript in the HTML file. That would be a truly nifty mashup! I actually did set it up that way at first just to confirm that I could make it work. However, there are two reasons why a "geocoding-on-the-fly" solution is not a good idea:
Finally, at the beginning of this section, I mentioned the word "controversy" with regard to web scraping. Hopefully, thoughts on the legality and ethics of using this programming technique ran through your mind while you were reading through this section. I constructed this football roster mashup for fun and to get practice with the technology. I haven't advertised it and certainly haven't tried to profit from it, but there's a chance that I'm violating ESPN's terms of use by not checking with them first before re-purposing their data. This kind of ethics question makes for interesting discussion, which is why I asked you to read and comment on the "Mashups: who’s really in control?" [18] blog post by Richard MacManus.
In the last section, we saw how to acquire data that is generally not meant to be "mashed up." Here we'll take a look at the easier case -- that probably would have made more sense to do first :-) -- in which the data publisher has made the data available in a form that allows it to be easily incorporated into other applications. XML is a popular format for this kind of data dissemination, which is often referred to as a web service or data feed.
Again, JavaScript can't be used to read remote XML files, but if you have access to a server with PHP, it is not terribly difficult to create a proxy script that simply repackages the XML for consumption by the JavaScript. For example, the (U.S.) National Weather Service provides data feeds of current weather conditions [19], forecasts and watches/warnings for hundreds of stations nationwide. For example, the data for State College, PA can be found at NOAA [20].
In the web scraping example on the previous page, the file_get_contents() function was used to read in the contents of the remote file. That function is actually typically disabled by most server administrators nowadays, so the example below uses a PHP library called cURL to handle the reading of the remote file.
https://php.scripts.psu.edu/jed124/read_xml.php [21]
<?php function curl_get_contents($url) { $ch = curl_init($url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/20100101 Firefox/19.0"); $data = curl_exec($ch); curl_close($ch); return $data; } header("Content-type: text/xml"); $root = 'http://w1.weather.gov/xml/current_obs/'; $data = curl_get_contents($root ."KUNV.xml","r"); $data = str_replace('<?xml-stylesheet href="latest_ob.xsl" type="text/xsl"?>','',$data); echo $data; ?>
Earlier in the course, you read about Ajax and saw an example map in which PHP was used to retrieve data from a database. The data outputted by that PHP script were consumed within the JavaScript code using a downloadUrl() function written in a special library intended to simplify the usage of the XmlHttpRequest object that makes Ajax possible. Just as in that example (and others found in the optional database lesson), the name of the proxy script above can be plugged into a downloadUrl() statement to supply the source data for a custom Google Map.
That brings us to the end of the topics I wanted you to be exposed to before beginning your final project. On the next page, I list a number of project ideas for those who can't come up with an idea of their own. Even if you have a good project idea, you may find it helpful to read through this page to get a better sense of my expectation level.
There is an enormous amount of data in the world that contain a spatial component and are just begging to be mapped. If you use your imagination, you should be able to come up with an interesting idea that is suited to your ability. But just in case you're not sure of what you'd like to do, I will list some ideas that I came up with. They are generally in order of increasing difficulty.
The National Geophysical Data Center (NGDC) of the National Oceanic and Atmospheric Administration (NOAA) maintains databases on the occurrences of earthquakes, volcanic eruptions and tsunamis:
Earthquake Database [22] (http://www.ngdc.noaa.gov/hazard/earthqk.shtml [22])
Volcanic Eruptions Database [23] (http://www.ngdc.noaa.gov/hazard/volcano.shtml [23])
Tsunami Database [24] (http://www.ngdc.noaa.gov/hazard/tsu.shtml [24])
Each database includes the latitude/longitude coordinates of the event along with other descriptive data. Importing the information into a MySQL database and providing a GUI for the selection of subsets of the full dataset should be relatively straightforward.
The U.S. Geological Survey (USGS) maintains a database of geographic features called the Geographic Names Information System [25] (http://geonames.usgs.gov/pls/gnispublic/ [25]) and provides public access to it through their website.
The database includes natural and man-made features ranging from airports to lakes to woods. While the site makes it possible to plot an individual feature on a Google Map, I don't see a mechanism for displaying all features of a certain type in a certain place. Thus, it seems to me that this database offers the potential for lots of interesting applications.
If this idea interests you, here is a tip on retrieving the data you want: Select a feature type, then a state and/or county and click Submit. For example, I selected cemeteries in Pennsylvania. The features and their coordinates will be displayed in a table. Some queries (such as mine) may take a minute or so if they return a large number of features. Beneath the table is a Save as pipe "|" delimited file link. If you click this link, you can import the data into Access or Excel and specify that the pipe character separates the values.
This database contains the locations of major and minor airports around the world [26] (http://www.partow.net/miscellaneous/airportdatabase/index.html [26]). It is maintained by an Australian software engineer. I agree that seems like a strange place to find such a thing and I don't know if there is a more authoritative source out there.
At the bottom of the page is a link to the zipped data file itself. The values in the file are separated by semi-colons.
A very simple application would be one that allows the user to select a country/state and see all of the airports in that location. To go a bit beyond that, it might be possible to construct links to sites that provide information about the airports. Or perhaps there are web services that provide interesting information. For example, here is a site that makes it possible to find airports served by a particular airline, airports currently experiencing delays and current weather conditions [27] (http://www.flightstats.com/developers/bin/view/Web+Services/web_services... [28])
These services are not free, but they can be accessed using an evaluation account that is probably long enough to complete the project.
Returning to NOAA, they also provide a downloadable shapefile of hurricane tracks [29] (http://maps.csc.noaa.gov/hurricanes/download.jsp [29]).
One application of this dataset might be to allow the user to select a year and/or basin and view the hurricanes associated with that selection. The track segments could be symbolized based on the hurricane's intensity.
NOAA is once again the source for this idea. Their Storm Prediction Center provides public access to reports of tornadoes, hail, and high winds. The reports are comma-delimited and are accessed through URLs like the following examples for May 1, 2009:
Tornado Report for May 1, 2009 [30] (http://www.spc.noaa.gov/climo/reports/090501_rpts_torn.csv [30])
Hail Report for May 1, 2009 [31] (http://www.spc.noaa.gov/climo/reports/090501_rpts_hail.csv [31])
Wind Report for May 1, 2009 [32] (http://www.spc.noaa.gov/climo/reports/090501_rpts_wind.csv [32])
My idea here is that the user could choose a date (or range of dates) and one or more weather type to map. The application would then read and plot the data from the appropriate files.
Each year, the National Center for Education Statistics (NCES) conducts a survey of public schools throughout the U.S. to compile what they call their Common Core of Data [33]:
This rich dataset includes the name, latitude/longitude, enrollment figures by grade, gender and ethnicity, and more. One idea for this dataset is to allow the user to select a state and county and see all of the schools in that locale. The schools could be symbolized by their type (elementary/middle/high).
And venturing into the supernatural, the National Unidentified Flying Object Reporting Center (NUFORC) provides lists of UFO sightings reported to their center [34] (http://www.nuforc.org/webreports.html [34])
The reports include the city where the sighting occurred along with the date/time on which it occurred and some descriptive information. Because they lack coordinate information, the reports would need to be geocoded before they could be mapped. An application based on these reports could allow the user to select a date (or range of dates), a state/country, and/or a shape (disc, triangle, cylinder, etc.).
This is certainly not an exhaustive list. If you have an idea for a good application, but aren't sure where to find the data, I encourage you to search the web. You may be able to find a site that provides easy access to the data you need by doing a search that combines the data's theme with terms like "web service," "data feed," or "XML."
Move on to the next page to find more details on the project requirements.
As laid out in the Overview of this project, the goal is for you to apply what you've learned in the course to create a sophisticated mapping application of your choice. This application should go a bit beyond the projects you've completed thus far. For some of you, this may mean constructing a MySQL database and providing the user with a GUI to map subsets of the larger database. For others, it may mean mashing up data scraped off of another site or published as an easy-to-consume service. Still others may choose to delve into some other advanced topic, like the ones I listed on the Overview page.
This project is two weeks in length. Please refer to the course Calendar, in Canvas, for the project deadlines.
Links
[1] http://www.w3schools.com/jquery/default.asp
[2] https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js
[3] http://www.jeasyui.com/download/index.php
[4] http://www.jeasyui.com/documentation/index.php
[5] http://www.personal.psu.edu/jed124/shapefileloader/shp_load_single_info_window_fix.html
[6] http://www.personal.psu.edu/jed124/shapefileloader/shp_load_single_jquery_layout.html
[7] http://www.jeasyui.com/documentation/
[8] http://www.personal.psu.edu/jed124/shapefileloader/shp_load_single_jquery_datagrid.html
[9] http://www.w3schools.com/html/html_forms.asp
[10] http://www.personal.psu.edu/jed124/form_processing.html
[11] http://www.personal.psu.edu/jed124/shapefileloader/shp_load_subset.html
[12] http://stackoverflow.com/questions/2177055/how-do-i-get-google-maps-to-show-a-whole-polygon
[13] http://en.wikipedia.org/wiki/Web_scraping
[14] http://www.personal.psu.edu/jed124/cfb/cfb_rosters.html
[15] http://espn.go.com/college-football/team/roster/_/id/213/penn-state-nittany-lions
[16] https://courseware.e-education.psu.edu/php/scrape_roster.html
[17] http://code.google.com/apis/maps/documentation/geocoding/#GeocodingRequests
[18] http://www.zdnet.com/article/mashups-whos-really-in-control/
[19] http://www.weather.gov/data/current_obs/
[20] http://w1.weather.gov/xml/current_obs/KUNV.xml
[21] https://phpdb.aset.psu.edu/phpjed124/read_xml.php
[22] http://www.ngdc.noaa.gov/hazard/earthqk.shtml
[23] http://www.ngdc.noaa.gov/hazard/volcano.shtml
[24] http://www.ngdc.noaa.gov/hazard/tsu.shtml
[25] http://geonames.usgs.gov/pls/gnispublic/
[26] http://www.partow.net/miscellaneous/airportdatabase/index.html
[27] https://www.flightstats.com/developers/bin/view/Web+Services/web_services_directory
[28] http://www.flightstats.com/developers/bin/view/Web+Services/web_services_directory
[29] http://maps.csc.noaa.gov/hurricanes/download.jsp
[30] http://www.spc.noaa.gov/climo/reports/090501_rpts_torn.csv
[31] http://www.spc.noaa.gov/climo/reports/090501_rpts_hail.csv
[32] http://www.spc.noaa.gov/climo/reports/090501_rpts_wind.csv
[33] http://nces.ed.gov/ccd/pubschuniv.asp
[34] http://www.nuforc.org/webreports.html