In our very first lesson, we introduced the concept that GIS System Analysis & Design was an iterative process that featured multiple stages and should include evaluation components at each step whenever possible. In this lesson, we will focus specific attention on methods and techniques for evaluating GIS systems. Such techniques include methods you might already be familiar with - surveys, interviews, and focus groups. But the range of available evaluation methodologies also includes a variety of technology-enabled methods to automatically capture usage statistics and to capture/visualize what users see on a screen, for example.
In this lesson, you will have the chance to learn about some common methods for evaluation, and you will also have the opportunity to do a bit of research on your own to discover what else is available. Every year brings substantial innovation in the human-computer interaction community - and it usually takes some time for such innovations to trickle down into what is used to evaluate GIS systems specifically.
At the successful completion of Lesson 13, students should be able to:
If you have any questions now or at any point during this lesson, please feel free to post them to the Threaded Discussion Forum. (That forum can be accessed at any time by clicking on the Communicate tab, above, and then scrolling down to the Discussion Forum section.)
To finish this lesson, you must complete the activities listed below. You may find it useful to print this page so that you can follow along with the directions.
Step | Activity | Access/Directions |
---|---|---|
1 | Read the lesson Overview and Checklist | You are in the Lesson 14 online content now. The Overview page is previous to this page, and you are on the Checklist page right now. |
2 | "Usability Engineering for GIS: Learning From a Screenshot [1]" by Muki Haklay and Antigoni Zafiri (Read) On line material (Read) Review PowerPoint [2] |
There are three different styles of reading that are referred to in the lessons:
|
3 | Geospatial Think-Piece (Template) [3] |
Using Word (or a word processing program compatible with Microsoft® Word), identify and briefly discuss (<200 words): What circumstances in which you might employ the methods tested by Haklay and Zafiri? Name your file Lsn14_YourName.doc, Please turn-in your document the Lesson 14 Dropbox in ANGEL. What is a “Think Piece”? A “think piece” is a form of writing that is less polished than a formal paper or presentation but more fully developed than an entry in a personal journal. Think pieces are written to discover what an individual is thinking about a particular topic. Within this course, the writing of think pieces is a way of helping learners connect with the subject matter. Within this context, think pieces reduce the grading risk associated with an “all or nothing” term paper and allow the instructor to communicate with learners throughout the semester, to see the evolution of thinking, and to suggest resources that can further the learners’ understanding. What does a Think Piece look like? The starting point for a think piece for this course lie in the author’s immediate past experience. Because think pieces are as much a reflection of one’s ideas, there is no standard or uniform format for a think piece. In other words, each of us is writing from personal experience. We are not claiming to be objective not are we offering prescriptive, how-to, formulas or guidelines. |
4 | Read lesson Summary | You are in the Lesson 14 online content now. Click on the "Next Page" link to access the Summary. |
We previously focused attention on a design process for GIS that involves needs assessment, concept development, prototyping, and implementation. Underlying each stage is a focus on evaluation - with the implication that evaluation is something that occurs during and between each stage of design.
Evaluation is needed to ensure that the progress made between design stages is based at least in part on input from end-users or customers. Rather than testing out the system with users at the very end of the process, you test in each of the stages along the way to add/subtract features and capabilities according to user needs. Failing to evaluate "along the way" can result in wasted effort if fully implemented systems must be fundamentally revised based on user feedback gathered at the end of design and development.
Evaluation is typically categorized into two broad areas: formative evaluation and summative evaluation. Formative evaluations focus on developing and refining designs. Summative evaluations compare an implemented system to an alternative system with the goal of measuring differences in performance or user satisfaction between the two systems. Quite often, formative evaluations happen in the early/middle stages of a design exercise and summative evaluations take place toward the end when a system has been implemented.
Common methods used in both types of evaluation include:
I have provided links to additional content explaining some of the evaluation methods that may not be familiar to you. Check them out!
Not everyone can afford to spend time & money on conducting in-depth evaluations at each stage of the design process while developing a new GIS system. Like everything else associated with GIS system design, trade-offs are involved and it is important for you, the designer, to figure out how to balance the need to make sure your progress is meaningful against the need to make progress toward the final system.
A distinction used quite often is to characterize evaluation efforts as formal or informal depending on the degree to which the evaluation activity makes use of rigorous methods to ensure unbiased participants, sound methodology, and careful analysis of results. An informal evaluation might make use of a few of your coworkers to look over a prototype design, while a formal evaluation could involve a dozen real end-users who complete a realistic exercise using the new GIS system and complete a post-activity interview and survey to gather structured and unstructured feedback.
The point here is that there are times when an informal evaluation will help you make progress on design and development goals, but there will come a time when you really want to conduct something formal to measure your success.
Eye tracking is a technology that is becoming more and more popular for use in software evaluations. Eye tracking makes use of infrared and other types of sensors to detect and track where a user looks while working. Modern eye tracking equipment can be mounted underneath the computer screen to face the user, or can come in head-mounted configurations. Eye tracking studies typically involve task analysis of one type or another, with the goal of capturing what users saw while they were completing their work. Analysis of eye tracking data can reveal which parts of an interface a person spent the most time using, which parts they missed entirely, and which areas of an analytical graphic were studied the most to inform analytical conclusions.
One challenge associated with eye tracking is that it generates a tremendous volume of data in a very short amount of time, so often there is substantial effort involved with analyzing the results of eye tracking studies. Using eye tracking along with other common usability methods (talk aloud protocols in particular) is quite popular, as eye tracking can provide insight into what someone was looking at, while verbal reports and other methods can reveal what the user was thinking at the time.
The first video (2:13) I'd like you to look at is a short demonstration on how eye tracking works and what some of the common outputs look like: Eye Tracking Demo [8]. Next, I'd like you to look at another short (1:56) video describing how eye tracking can be used specifically for usability studies. Both videos this week are from marketing materials, so keep that in mind as you watch: Tobii Usability Eye Tracking [9].
* Content derived from Geog 583, GEOG 583: Geospatial Systems Analysis and Design [10], which is part of Penn State's OER initiative. Author/Instructor: Anthony Robinson.
In this lesson, we have taken a look at the range of evaluation methods available for gauging the results of each step in GIS Design. As you have seen, there are lots of types of evaluation, and evaluations can be formal or informal depending on the time/capital resources you have at your disposal. There are simple low-cost methods you can use that require no technology at all, and there are more advanced methods that require substantial technology investments. In our reading assignment, you encountered a couple (quite different) academic studies on evaluating GIS systems that hopefully suggest some good ideas (and some limitations) you can take to heart with respect to evaluations you might conduct in your future work.
Links
[1] https://www.e-education.psu.edu/geog468/sites/www.e-education.psu.edu.geog468/files/Haklay_GIS%2BUsability.pdf
[2] https://www.e-education.psu.edu/geog468/sites/www.e-education.psu.edu.geog468/files/New_Lesson_14/Evaluation%202012_1.ppt
[3] https://www.e-education.psu.edu/geog468/sites/www.e-education.psu.edu.geog468/files/Misc/Think-Piece%20Template.docx
[4] http://en.wikipedia.org/wiki/Heuristic_evaluation
[5] http://www.useit.com/papers/focusgroups.html
[6] http://www.usability.gov/methods/design_site/cardsort.html
[7] http://www.nysgis.state.ny.us/coordinationprogram/reports/cost/index.cfm
[8] https://www.youtube.com/watch?v=lo_a2cfBUGc&feature=player_embedded#!
[9] https://www.youtube.com/watch?v=xgRIjrlK1mA&feature=player_embedded
[10] https://www.e-education.psu.edu/geog583/node/25