Bringing Data into a Unified Format

A. Rationale

My story:
If you recall, the layers that the State College Borough Water Authority identified on their GIS wish list came from many sources. We have added GPS, schools data, forest data, Centre County data and the CAD Drawing. These were imported from a number of sources and comprise a lot of different data types. Now, we wish to hand the project off, and to do this we will give them a unified data source.

B. Load the Data into FME Workbench

Workbench is the real workplace of the FME package. It has an industrial strength Database Management System that is optimized for spatial datasets. The same thing can be achieved by using ArcMap and ArcCatalogue. To it in these packages each piece of data will have to be treated separately and it is once only operation. In Workbench, the action can be saved and run again as things are needed. Workbench also keeps extensive log files so that work actions and flows can be recorded and documented.

1. Open FME Workbench (If you have a support problem the course instructor, not FME, is the person to contact).
2. Drag the Building Plots Shape File into the workspace or use the Add Reader button or dialog.
3. Click the Add Writer button or menu and navigate to a directory and set up Esri Geodatabase, file based personal geodatabase, name the geodatabase "SCBWAGIS"
4. Add a Feature and set feature. Name the variable to something descriptive, e.g. "Building Plot Polygons."
5. Set feature geometry to suit the incoming geometry; in this case "polygons."
6. Connect workflow arrow to it. (To set a destination there is not a button or dropdown menu on the application.  Look for the little yellow can and the green arrow pointing to the right. That is the button that will take you to the place where you go to set the destination for the buildings and streets layers to the FileBasedGeodatabase; just drag it across.)
7. Do the same thing for Streets. The work flow is drag in a data set, set a destination in the writer, and connect the two with an arrow.
8. Continue adding data sources and setting the connections to the geodatabase as per the first connection. .
9. Run the model to populate the geodatabase (when it has run it will have the number of features as a label on the arrow line (See Figure 8.1, above).
10. Save the FME workspace so that it can be run again if the data are updated.
11. Document your work, save the logfile from FME with the Geodatabase. Save any metadata you have received for the layers you included (in this case it will be none). You will need to create Metadata if there is none; do this for one of the layers only using the Esri metadata creator in ArcCatalogue.

Look around in FME and you will see lots and lots of features that we do not have time to look into, but you can select, subset, transform, generalize, simplify, and otherwise modify data as it is loaded. This can be particularly useful if you are transferring data from a different program. It is also the time to generalize data to a similar scale. The repeatability of these programs really is useful if you are receiving data from another source on a regular basis. For example, the works department has data on government owned and controlled buildings and the planning department has information on new buildings and you also have parcel data. These different departments might send you new data on a update basis or might send out a weekly data update. These you have to integrate each week to ensure that you are working on the same data. It would be nice if all the data were in the same format, but as it has different sources, it quite likely will not be standardized. If you have a lot of interactions with CAD based data that is not in geographic coordinates but is in arbitrary CAD space units, it is sometimes better to build a custom CAD projection so that ArcGIS knows how to reproject them, rather than transforming them. The custom projection takes a flat projection like UTM and then provides a custom offset (False Easting and False Northing ) to line it up. It might also take a custom scaling index to ensure a similar scale. This is really worth doing if you have lots of CAD drawings in the same source coordinates or if you have a frequent update series.

C. Use open interchange propriety or interchange formats?

In the example above, we have used the scenario of a consultant working with a variety of other authorities to create a GIS for the Water Board. When we hand it off, they want to have the data in their GIS with minimal work. Assuming that they had an ArcGIS system, the geodatabase is the best choice for delivery. It is richer than the open source standard, the shapefile. Shapefiles do not have anywhere near the amount of information about domains, topology and time (temporality) that a geodatabase can or an ArcInfo coverage can have. If you want exchange files on a regular basis, as when a corporation updates files and passes them on to another entity, then it pays to have inbuilt translation methodology like FME Workbench. If the data are to be put on the web for anybody, then a widespread propriety file like ArcGIS or even ArcInfo Coverage e00 export file are okay, but open source data would be better. Most GIS systems have routines to deal with common propriety file types, and Esri is now the de facto statndard. How about if the downloader runs a rarer GIS? Has anyone received GRASS or SmallWorld files? These can be difficult to work with and not everyone has the Interoperability Extension or FME. Open source files are fine for simple geometries without topology and are probably the best in many situations. However, they are not universally the best. Sometimes, a more complex spatial entity is better preserved in a more complex data type. Make a lfinal ist of file types you have incorporated into your project and add it to your document e-portfolio.

That's it for Part I!

You have just completed Part I of this module, Part II, you will load GPS data you've acquired.