The links below provide an outline of the material for this lesson. Be sure to carefully read through the entire lesson before returning to Canvas to submit your assignments.
Note: You can print the entire lesson by clicking on the "Print" link above.
In this lesson, you'll learn some of the history of web mapping and why web services are so important. You'll also learn about free and open source software (FOSS) and the benefits and drawbacks of using it. Finally, you'll get the chance to install and use QGIS, a FOSS solution for desktop GIS work. QGIS will help you preview and manipulate your data as you prepare to create web maps later in the course.
For decades, most digital geographic information was confined for use on desktop-based personal computers (PCs) or in-house mainframes and could not be easily shared with other organizations. GIS analysts would access data from their own workplace computers that were often connected to a central file server somewhere in the office. Specialized software was required to view or manipulate the data, effectively narrowing the audience that could benefit from the data.
With mass uptake of the Internet in the mid-1990s, people began thinking about how maps and other geographic information could be shared across computers, both within the organization and with the general public. The first step was to post static images of maps on HTML pages; however, people soon realized the potential for interactive maps. The first of these, served out by newborn versions of software such as Map Server and Esri ArcIMS, were horrendously pixelated, slow, and clunky by today's standards. Limited by these tools, cartographers had not yet arrived en masse on the web mapping scene, and most of the maps looked hideous. However, these early interactive web maps were revolutionary at the time. The idea that you could use your humble web browser to request a map anywhere you wanted and see the resulting image was liberating and exciting. See Brandon Plewe's 1997 book GIS Online (findable in many university geography libraries) to get a feel for the web mapping landscape at this time period.
These early, dynamically drawn web maps ran into challenges with speed and scalability (the ability to handle many simultaneous users). The server could only accommodate a limited number of map requests at a time before slowing down (at best) and crashing (at worst). Web maps matured significantly in these two metrics when websites began to serve out tiled map images from pregenerated caches. Why ask the server to draw every single map dynamically when you could just put forward an initial investment to predraw all possible map extents at a reasonable set of scales? Once you had the map images drawn and cached, you could serve out the images as a tiled mosaic. Each tiled map request was satisfied exponentially faster than it would take to serve the map dynamically, allowing for a server to accommodate hundreds of simultaneous users.
Following the lead of Google Maps, many sites began serving out “pre-cooked” tiled map images using a creative technique known as Asynchronous JavaScript and XML (AJAX) that eliminated the ubiquitous and annoying blink that occurred after any navigation action in earlier web maps. Now you could pan the map forever without your server gasping for breath as it tried to catch up.
Cartographers, who had largely been resigned to trading aesthetics for speed in web maps, also realized the potential of the tiling techniques. No longer would the number of layers in a map slow down the server: once you had pregenerated the tiles, you could serve a beautiful map just as fast as an ugly one. Web maps became an opportunity to exercise cartographic techniques and make the most attractive map possible. Thus were born the beautiful, fast, and detailed “web 2.0” basemaps that are common today on Google, Microsoft Bing, OpenStreetMap, and other popular websites.
As web browsers increased in their ability to draw graphics using technologies such as SVG and later WebGL, the possibilities for interactivity arose. On-the-fly feature highlighting and HTML-enriched popup windows became common elements. For several years, developers experimented with plug-ins such as Adobe Flash and Microsoft Silverlight for smooth animation of map navigation and associated widgets. More recently, developers are abandoning these platforms in favor of new HTML5 standards recognized by the latest web browsers without the need for plug-ins.
Although maps had arrived on the browser by the mid-2000s, they were still largely accessed through desktop PCs. The widespread adoption of smartphones and tablets in subsequent years only increased the demand for web maps. Mobile devices could not natively hold large collections of GIS data, nor could they install advanced GIS software; they relied on web or cellular connections to get maps on demand. These connections were either initiated by browsers on the device, such as Safari, or native applications installed on the device and built for simple, focused purposes. In both cases, GIS data and maps needed to be pulled from the organization's traditional data silos and made available on the web.
All the above web mapping scenarios are possible because of web services. If you search the Internet, you'll find many definitions of web services and can easily get yourself confused. For the purposes of this course, just think of a web service as a focused task that a specialized computer (the server) knows how to do and allows other computers to invoke. You work with the web service like this:
Examine how the following simple diagram describes this process, returning a map of current precipitation:
Now, for an example. Let's suppose that you've identified the URL of a web service out there somewhere that draws maps. You make a request by constructing a URL (http://...) containing the address of the web service and various parameters for the map, such as the format of the image you want to receive (JPG, PNG, etc.), bounding box (coordinates defining the geographic area you want to see mapped), and map scale. You paste this into your browser's address bar and the server sends you a response containing the map image you requested.
Here's an example of just such a request, using a radar map of the United States. First, see if you can identify some of the parameters in the URL. Then take a guess at what the response will look like when you get it back. Then click the link to invoke the web service and get the response:
As you examined the URL of this request, you might have noticed parameters indicating the width and height of the image, the image format, the image background transparency, and the bounding coordinates of the map to be drawn. These parameters provide specific details about how the web service should run its map drawing code. You see these parameters reflected in the response image sent back to your browser when you clicked the link. In a future lesson, you'll learn about more of the parameters in the request above.
Not all web requests invoke web service code. Some web requests just return you a file. This is how tiled maps work, and this is why they are so fast. You'll learn more about tiled maps in a later lesson, but examine the following request for a specific zoom level, row, and column of a tile:
http://a.tile.openstreetmap.org/15/11068/19742.png [3]
The request drills down into the server's folder structure and returns you the requested PNG image as a response. No special code ran on the server other than the basic file retrieval, therefore you could argue that a web service was not invoked. However, this type of simple web request is also an important part of many web maps.
At this point, you might be thinking, “I've used web maps for years and I have never had to cobble together long clunky URLs like this. Have I been using web services and other web requests?” Absolutely, yes. As you navigate Google Maps, your company's online map site, and so on, your browser is sending hundreds of web requests similar to this one. You've just never needed to know the details until now. When you begin setting up your own GIS server or designing your own client web application, it becomes important to understand the theory and architecture behind web traffic.
Not all web services use the same format of URL and parameters. In this course, you'll learn about some of the most common formats for online web services, especially ones that have been openly developed and documented to work across software packages.
Here's a simple way you can view web requests made “behind the scenes” by your browser as you navigate a website. These instructions are for the tools provided by Mozilla Firefox. Chrome and other browsers have similar tools that typically go under the name "developer tools" or "web tools" and should not be difficult to locate.
This kind of developer tool will be valuable to you later in the course as you develop your own web maps, mainly for troubleshooting scenarios where you expect a map to show up and your browser does not send you the expected response.
Plewe, B. (1997). GIS online: Information retrieval, mapping, and the Internet. OnWord Press.
The term “free and open source software” (hereafter referred to as FOSS) includes a number of ideas that can invoke complex and even emotional discussions within the technical community. To begin to describe this term, it's important to understand that software development is more enigmatic and artistic than other crafts in the tech industry [5], such as computer chip design. Software cannot be seen, touched, tasted, or described in a physical sense, other than the number of megabytes it occupies on your computer. Software often begins as ideas on a whiteboard, which are then encapsulated into intangible “classes” and "objects" of code by a programmer. These are then assembled into invokable sets of concrete tasks and dressed with a user interface of buttons, menus, and icons that require a whole other skillset of aesthetic design.
As a result of all this work, software is an empowering technology that enables a person to make practical use of computer hardware. In fact, specialized software can often cost much more than the physical machine that it runs on. Software is your window into printing, painting, calculating, storing data, and, in our case, making maps.
Given the value of software, it's no surprise that commercial businesses invest millions in researching, developing, and selling proprietary software. They protect it through patent and copyright laws. They obfuscate (scramble) the code to make it harder to copy or reverse engineer. Commercial software development has become a successful enterprise whose resulting tools have shaped our work and home environments.
At the same time, some software users and developers have advocated that there are benefits from making software source code freely visible and able to be modified or shared without legal or technical restraint. Business benefits, technical benefits, and moral arguments have been invoked in support of this concept of “free and open source software.”
It's possible to get confused when learning about FOSS, because the word “free” carries several meanings in the English language. A common analogy describing the F in FOSS is “free as in free speech, not free beer.” In other words, FOSS is “free” in the sense that it is open and amenable to use and modification. (You may sometimes see the term Free/Libre [6] Open Source Software (FLOSS) used to communicate this idea.) A whole range of license types [7] are used with FOSS that specify the conditions surrounding the modification and reuse of the software, along with the attribution required on any derived products.
While the selling of FOSS is not necessarily restricted, FOSS is usually available free of charge. All of the software we'll use in this course can be downloaded directly from the Internet and requires no fee, although donating a few dollars to your favorite project is a good way to invest in its continued development.
If FOSS is usually available without cost, why is it such a successful concept? And how do people make a living from coding with it? First, it's important to mention that many developers contribute to FOSS solely for personal enjoyment and for the desire to donate their skills to a project offering wide societal benefits. They enjoy working on virtual teams and facing the challenges of rigorous software development that their ordinary “day jobs” might not offer.
At the same time, though, many individuals make a generous living by selling services and training related to FOSS, and some of these efforts may increase the quality and number of FOSS features available. For example, suppose your company invests in a FOSS package that does everything you want at $30,000 cheaper than the proprietary software alternative; however, it's missing “Feature X” that is critical for your workflows, and you don't have a programmer who can implement Feature X. Because the code for the software is all open to examination, modification, and extension, you can contract with Rock Star Programmer to implement Feature X for you for $10,000. If Feature X is widely applicable to the good of the product, and you and the contractor are okay with the idea, Feature X may eventually be rolled into the core source code so everyone can benefit from it.
Other people may not contribute frequently to FOSS projects, but can still make a profit selling support services for FOSS software. When you buy proprietary software, you often are given access to a technical support package that allows you to call and talk to an analyst if needed. Because FOSS does not offer these official support systems, enterprising individuals have stepped in to fill this void.
Finally, several new firms are building subscription-based online services that are created using open source building blocks and may even be released under open source licenses. The services are offered for a subscription cost that is lower than most organizations could achieve if they attempted to build their own comparable infrastructure or quality control procedures. Through these FOSS-based Software-as-a-Service (SaaS) offerings, the value of the free software is passed on to many buyers.
Although there are many FOSS purists out there, the decision to use FOSS does not have to result in a full migration away from proprietary software. Many businesses and governments use what might be termed a "hybrid approach," incorporating a mix of FOSS and proprietary software depending on their budget, staff skills, and technical needs.
Let's consider some of the benefits, challenges, and other considerations that the adoption of FOSS brings into your workflows.
As you begin the endeavor of learning to use FOSS, it's helpful to understand some of the benefits that you may see:
A danger of evaluating FOSS systems is to allow the potential exciting benefits to obscure the real challenges that can accompany a FOSS deployment. Some or all of the challenges below can be mitigated, but proprietary software may offer a smoother road in these areas, if you can bear the cost.
Some aspects of FOSS and proprietary software are not as clear when it comes to deciding which type of software owns the advantage.
Quality and technical superiority – The “bugginess” of FOSS compared to proprietary software probably depends on the products in question, how mature they are, and who's developing them. FOSS advocate Eric Raymond argued that “given enough eyeballs, all bugs are shallow,” contending that it would be difficult for any major issues with FOSS to go unfixed for long among a broad community of developers. It's less clear who fixes the obscure bugs that you hit and others don't. If you can't convince someone in the FOSS community to fix it, you could at least attempt it yourself or hire a consultant.
Proprietary software vendors obviously have business incentives to fix bugs; however, they also have business incentives to NOT fix certain obscure bugs if the time, effort, and risk to do so would not provide a good return on investment. If you're a small-potatoes customer with an obscure bug holding up your project, you'd better start looking for a workaround.
FOSS has been developed to support all tiers of system architecture. You've probably heard of (or used) some of this software before. For example, the term “LAMP stack” refers to a system that is running:
Other variations of this acronym exist. For example, PostgreSQL is another open source relational database that is commonly used with GIS because of the popular PostGIS extension. This results in a LAPP stack rather than a LAMP stack.
Other general-use FOSS includes the LibreOffice suite (similar to Microsoft Office), the Mozilla Firefox web browser, the Thunderbird e-mail client, the Python scripting language, and more.
Some governments have begun mandating or encouraging the use of FOSS for government offices and projects. This is especially popular in Latin America and Europe, with one of the more recent government decrees occurring in the United Kingdom [10]. Often the FOSS is implemented first on servers and back-end infrastructure, then rolled out to desktop workstations in later phases.
These policies favoring FOSS come about for different reasons. Obviously, the savings in software licenses and the flexible security models offered by FOSS are desirable, but sometimes there are political motivations to reject proprietary software companies and their countries of origin (particularly in places where the United States is perceived as imperialist). If you are interested in more reading on this topic, I recommend Aaron Shaw's study of the FOSS movement in the Brazilian government [11] and its tie to leftist politics [here is an alternative link [12] to the paper in case the other link isn't working].
FOSS has a strong and growing presence in the GIS industry. Some tools and utilities for processing geospatial data have been around for decades. For example, GRASS GIS [13] developed by the US Army Corps of Engineers recently turned 30 years old. To see what it looked like back then, see this old GRASS promotional video [14] narrated by none other than William Shatner. I still use this video in introductory classes to teach the benefits of GIS and the main components of a GIS system.
In this course, we'll use some of these desktop workstation GIS tools for previewing and manipulating our datasets before putting them on the web. For example, later in this lesson, you'll install and explore QGIS [15] (previously known as Quantum GIS), one of the most popular and user-friendly FOSS GIS programs.
There are also various FOSS options for exposing your GIS data on the web, either within your own office network or on the entire Internet. These include Map Server [16], QGIS Server [17], and GeoServer [18], the latter of which you will learn in this course. These software offerings take your GIS datasets and make them available as web services that speak in a variety of formats. They include a web server or are designed to integrate with an existing web server, so that your web services can reach computers outside your own office or network.
FOSS can also be used to generate sets of tiled images or vector data that you can use as layers in your web maps. In this course, you'll learn how to do this using a new tool recently integrated into QGIS.
Underlying both desktop and server GIS are the databases containing your GIS data. If you need something a little more complex than a folder of shapefiles or want to make use of spatial database types, then you can use a FOSS GIS database. These include PostGIS [19] (which is an extension to the PostgreSQL relational database) and SpatiaLite [20] (which uses the SQLite database). A lighter fare option for smaller datasets is to use standalone files of GeoJSON, KML, GeoRSS, or other well-documented text-based data formats.
To pull all your map layers together and display them on a web page, you'll use a programming framework, or API. Some of the most mature web mapping APIs are OpenLayers [21] and Leaflet [22], which you will use later in this course with the JavaScript programming language. Other popular FOSS web mapping APIs include ModestMaps [23], D3 [24], and Polymaps [25].
Not to be confused with open software, open specifications are documented and mutually agreed-upon patterns of how software and digital data should behave in order to be interoperable between systems. For example, HTTP (hypertext transfer protocol) is based on a specification defining how web servers and web browsers should communicate in order to exchange information. Without open specifications, you would not be reading this web page.
In this course, we'll learn about two types of open specifications:
In this course, we'll also be using open data, which is data that has been made available to the public free of charge and bereft of most copyright restrictions. These data may be shared by government entities, researchers, not-for-profit organizations, or ordinary citizens contributing to online projects. For example, Data.gov [26] is a popular website offering datasets collected by the US government. And, later in this course, we'll learn about one significant source of open GIS data called OpenStreetMap [27]. This is an online map consisting of voluntary contributions in the style of Wikipedia.
Open datasets are often rich and exciting to include in web maps, but there are some precautionary measures you should follow in order to be successful with using them. First, be aware that even though the data is free, you are often still required to provide attribution describing where you obtained the data. You may also be restricted from redistributing the data in any way that requires a fee. When you use an open dataset, you are responsible to carefully research and adhere to any attribution requirements. You should also make an effort to verify the data quality by examining any accompanying metadata, researching the sources and collection methods of the data, and scrutinizing the data itself.
In this course, you will be getting some experience installing and using FOSS and creating web maps with it. While doing this, you will use open specifications for GIS data and web services. You'll also learn how to use open data and contribute to OpenStreetMap.
Each week, you'll complete a walkthrough explaining some FOSS tool. Following the walkthrough, I will often ask you to apply what you've learned to some of your own datasets you've found or collected. This will allow you to build up a comprehensive project as the course progresses. The goal is to produce something you can host on your personal webspace and reference in a professional portfolio.
The functionality of our web maps will be relatively simple, limited to layer display and point-and-click queries. However, the frameworks that we'll use are broadly extensible depending on how much programming you're willing and able to do.
Along the way, you'll pick up some skills with manipulating data (projecting, clipping, and so forth) with FOSS, which will hopefully prove handy throughout your GIS career. Many of these tools can be scripted with Python and other languages that you may have learned already in other GIS coursework.
The first FOSS product you'll use is a GUI-based program designed for desktop workstations. It's called QGIS (kyoo-jis), although you should know that sometimes in the past it was referred to as Quantum GIS. QGIS is somewhat similar in appearance and function to Esri's ArcMap, which you've likely used in previous courses.
In this tutorial, you'll install QGIS and make a basic vector map with it. You'll use some shapefiles of downtown Ottawa, Ontario, Canada that I originally downloaded from the OpenStreetMap database.
Download the Lesson 1 walkthrough data [28] (It is a folder of shapefiles that you should extract into a folder such as c:\data).
In this week's assignment, you'll watch and read several opinions on the use of FOSS within the GIS community. I will then ask you to respond to these in a detailed discussion below.
Make a post on the "Lesson 1 assignment forum" on Canvas that addresses both of the following questions:
You can optionally make two posts addressing the questions separately. You will be graded on the depth of the arguments that you put forward in your responses and the evidence you produce to support these arguments, not on your opinions toward FOSS.
To receive full credit, please also write at least one response to another student's post building on whatever he or she said.
I encourage you to continue thinking about these questions throughout this course as you get more hands-on experience with FOSS.
Links
[1] https://www.e-education.psu.edu/geog585/sites/www.e-education.psu.edu.geog585/files/lesson1/Figure_1.1_LD.html
[2] http://mesonet.agron.iastate.edu/cgi-bin/wms/nexrad/n0r.cgi?SERVICE=WMS&REQUEST=GetMap&FORMAT=image/png&TRANSPARENT=TRUE&STYLES=&VERSION=1.3.0&LAYERS=nexrad-n0r&WIDTH=877&HEIGHT=276&CRS=EPSG:900913&BBOX=-15252263.28954773,2902486.4758432545,-6671748.242369267,5602853.811101243
[3] http://a.tile.openstreetmap.org/15/11068/19742.png
[4] http://ride.trimet.org/?tool=routes
[5] http://parrt.cs.usfca.edu/doc/software-not-engineering.html
[6] http://en.wikipedia.org/wiki/Libre
[7] http://choosealicense.com
[8] http://www.linuxinsider.com/story/67655.html
[9] http://www.courtnewsohio.gov/cases/2013/SCO/0307/121296.asp#.XNmp2ZNKiu7
[10] http://www.computerweekly.com/news/2240179643/Government-mandates-preference-for-open-source
[11] http://aaronshaw.org/papers/Shaw-2011-Insurgent_Expertise-JITP.pdf
[12] http://innovacionucb.pbworks.com/w/file/fetch/77872493/Aaron_2011_Insurgent%20Expertise-%20The%20Politics%20of%20Free%3ALivre%20and%20Open%20Source%20Software%20in%20Brazil.pdf
[13] http://grass.osgeo.org/
[14] https://www.youtube.com/watch?v=U3Hf0qI4JLc
[15] http://www.qgis.org/en/site/
[16] http://mapserver.org/
[17] https://docs.qgis.org/2.14/en/docs/user_manual/working_with_ogc/ogc_server_support.html
[18] http://geoserver.org
[19] http://postgis.net/
[20] https://www.gaia-gis.it/fossil/libspatialite/index
[21] http://openlayers.org/
[22] http://leafletjs.com/
[23] http://modestmaps.com/
[24] http://d3js.org/
[25] http://polymaps.org/
[26] http://www.data.gov
[27] http://www.openstreetmap.org
[28] https://www.e-education.psu.edu/geog585/sites/www.e-education.psu.edu.geog585/files/lesson1/Ottawa.zip
[29] http://www.qgis.org/
[30] https://gistbok.ucgis.org/bok-topics/web-gis
[31] https://www.youtube.com/channel/UCaM-49dLjHOiQ0k0K4w5uhw
[32] http://www.esri.com/news/arcnews/spring11articles/open-source-technology-and-esri.html
[33] http://www.esri.com/products/arcgis-capabilities/open-source