EARTH 801
Computation and Visualization in the Earth Sciences

Lesson 7: File Editing with awk and vi

PrintPrint

Part 1: awk

The awk command is a powerful way to manipulate the contents of textfiles. We are only going to skim the surface of what awk can do right now. Let's start with a simple example. Say you have a text file that has some columns of numbers in it. With one awk command you can rearrange the columns in a different order, or perform some arithmetic on the columns.

Try this!

Download the "file1.txt" textfile from the example above. This is accomplished by clicking the link and then choosing Save As . . . from your browser's file menu. Save it somewhere on your computer, then in the terminal, navigate to that place. Use awk to change it into a file in which the first column is unchanged, the second column is composed of the number 5 and the third column is composed of the number 9. Try it first, and if you are stuck, watch this movie to see how I edited the file.

Part 2: vi

vi (also called vim; the two are basically the same) is a text editor that allows you to create and edit text inside a terminal window without popping up another window and without using the mouse. Truthfully, most people would never want to get rid of their mouse if they are used to using it all the time, but if you want to get into a file, do a simple thing to it, such as deleting the first 30 lines or adding one line to the bottom or something like that, then vi is handy. However, the functionality of vi does not lend itself well to making a screen capture "how-to" movie because most of the action takes place on the keyboard and you can't see my hands with a screen capture.

When you are using vi to edit a file, you will either be in "insert" mode or in "moving around" mode. While you are in "insert" mode, whatever you type becomes part of the file. (just like in word or whatever word processor/text editor you are used to). But when you are in "moving around" mode, you use keyboard keys to move the cursor around the file. To get started, type vi filename at the prompt (in which "filename" is your actual filename, not the word "filename" unless that's the name of your file wink.

Here's an incomplete command list (the man page for vi will do better) but it's a start:

INSERT MODE

i to insert before the cursor, I to insert at the beginning of the current line

a to insert after the cursor, A to insert at the end of the current line

o to make a new blank line below the cursor and put the cursor at the beginning of it, O to make a new blank line above the cursor and put the cursor at the beginning of it

MOVING AROUND MODE

h moves one space to the left

l moves one space to the right

j moves one line down

k moves one line up

dd deletes the current line

x deletes the current character

typing a number before a command repeats the command that many times, so 10 dd deletes 10 lines beginning with the current line.

Part 3: Do something useful

Let's put our skills to work. Download this file of a catalog of earthquakes from the USGS. Use vi and awk to make a new file that contains just one column -- the earthquake magnitudes. This is the kind of thing that will be super useful for making a frequency magnitude diagram! Try it on your own and if you get stuck, watch this movie to see how I edited the file with awk and vi to get just one column. Keep in mind that there is just about always more than one way to accomplish an editing task like this. The point is to get the end result without lots of work and cumbersome steps in a non-mathematical spreadsheet program that was not intended to handle a big dataset.