GEOG 897D
Spatial Database Management

Create a Cloud-Based Enterprise Geodatabase

PrintPrint

Using the Amazon service mentioned above Esri makes it possible to set up cloud-based installations of ArcGIS Server that include an enterprise geodatabase (SQL Server).  Because our Cloud and Server GIS course also guides students through putting together an instance of ArcGIS Server, we will at times refer you to that content.  While our focus in this course will be on the enterprise geodatabase that will be installed on the instance with ArcGIS Server, a side benefit is that you'll also be able to experiment with ArcGIS Server if you like.

A. Prepare to work in the Amazon cloud environment

I have already asked you to perform the first 3 steps below, so you do not need to go through those again.

  1. Establish an account with Amazon and sign up for Amazon Web Services (AWS) by following these instructions [https://www.e-education.psu.edu/geog865/orientation/AWS_signup]. 

    Whenever you wish to log on to your AWS account, go to the AWS Management Console (http://aws.amazon.com/console/) and follow the Sign in to the AWS Console link.
  2. E-mail your 12-digit Amazon account number to your instructor so that he can give you permission to work with the necessary Amazon Machine Image (AMI).  
  3. Next you need to go to the AWS Educate page and apply for a student credit. Past students have reported easily staying under the allotted credits by not spending a lot of time outside completing the course exercises and by stopping their instance when done with it for the day. 
    You will need to have handy your Amazon Web Service 12-digit account number.
    Go to https://aws.amazon.com/education/awseducate/
    • Scroll down and under Students click on Apply for AWS Educate for Students.
    • In the Step 1 window make sure the Student radio button is selected, and click the Next button.
    • In the Step 2 window supply the requested information.
      Put Penn State University - Erie as the Institution Name. (As you begin to type an auto-complete function will present you with the correct selection.  Someone at PSU-Erie campus apparently joined the AWS Educate program before we did and Amazon has told us to just use that same option for other PSU locations.)
      Be certain to supply your <Access Acct ID>@psu.edu email address.
      Select Graduate for the Grade Level.
      Do your best to supply the rest of the information. (At this point you do not have a Promo Code.)
      Click Next.
    • In the Step 3 window you will be instructed to supply a Verification Code that was sent to the email address you supplied in Step 2. And, I think it will also ask you to verify that you are not a robot....
      Click the Next button.
    • The Step 4 window will ask you to Accept the terms of usage, and to hit the Submit button.
      You will then be informed that your application has been received, etc.
      Unfortunately it looks like it may take two or three days before you get an email, with the subject line AWS Educate Application Approved, that contains your Credit Code.

      Once you have your Credit Code you next need to log on to your Amazon Web Service account in order to redeem it.
  4. To redeem your AWS Educate credits, follow this link to go to the AWS Console site, then:
    • Log in to your Amazon Web Services account, and then
    • expand the menu beneath your name, at the upper right, and follow the My Account link.
    • On the page that you land on follow the Credits link found in the list on the left side of the page, and proceed to redeem your credit code.
  5. Once you have redeemed your Credit Code, come back to section B, below, and continue with the lesson.

    In the meantime, learn about the Amazon Elastic Compute Cloud (EC2) service here [https://www.e-education.psu.edu/geog865/node/261].

B. Create your own cloud-based instance of ArcGIS Server

While it's possible to launch an instance of ArcGIS Server from the AWS website, Esri's Cloud Builder application greatly simplifies the process. Cloud Builder is a simple wizard-driven desktop application that Esri created specifically for creating ArcGIS Server sites on Amazon EC2. Cloud Builder abstracts or hides some of the advanced options that you see in the AWS Management Console. Cloud Builder also offers some options that are not available in the AWS Management Console at this time, such as the ability to set automatic scaling rules for your site based on CPU usage. Cloud Builder also does a lot of work behind the scenes to "wire" your GIS servers together, license your software, and create your site. It's possible to build simple one-machine ArcGIS Server sites with the AWS Management Console. You can even put several of these "siloed" sites under a load balancer to get more computing power. However, to get the full benefit of the ArcGIS Server 10.1 (and later) architecture, in which multiple GIS servers process and balance loads in a peer-to-peer fashion, you must use Cloud Builder.

Getting and Installing Cloud Builder

Cloud Builder is available as a free download on the Esri Customer Care website for anyone who has purchased ArcGIS Server. Since you cannot access the Customer Care site, I have hosted the Cloud Builder installer download in Canvas.

(If you recently took our Geog 865: Cloud and GIS Server GIS course, you may still have Cloud Builder installed on your computer. If you do, you can skip to the Open and Sign In to Cloud Builder section, below.)

  1. Go to the Lesson 6 folder/module on the Home page in Canvas and, via the link on the ArcGIS Server Cloud Builder page, download the installer .exe file for the ArcGIS Server Cloud Builder.
  2. Install Cloud Builder by double-clicking on the ArcGIS_Cloud_Builder_on_Amazon_1031_1458.exe file you just retrieved. (The link to the installer file is not here in the lesson text because the lesson text is part of our Open Educational Resources library.)
    • When the Application Install window comes up,
      • Un-check the box for Start application after installation.  Installation Preferences settings. 
      • (Adobe Air is required to run Cloud Builder and Air will be installed for you if you don't have it.) Note - this statement may be in error for some people; if the Cloud Builder installer fails to run for you, you may need to download and install Adobe Air yourself. Search the Web for the free installer.
      • You can choose to add a desktop shortcut.
      • It's safest to go with the default Installation Location.

    The rest of the installation process should go quickly. The application will install as ArcGIS Server Cloud Builder on Amazon Services.

Getting access keys for your Amazon account

In order for a third-party application like Cloud Builder to access your Amazon Web Services account, it needs to have two pieces of information known as an access key and secret access key. As the owner of the AWS account, you can create these keys (with varying privilege levels) and distribute them to whichever application needs them.

You manage your keys through an AWS service called Identity and Access Management (IAM), and you work with IAM through the AWS Management Console.

Let's create a key that you can use with Cloud Builder. This particular key requires Administrator access to your account because Cloud Builder has the potential to create a lot of resources and do things of a sensitive nature.

  1. Log in to the AWS Management Console if you are not logged in already.

    If you ever lose track of your AWS account number, follow the instructions here: Finding your Amazon account number.
     
  2. Click IAM. Find it under All services| Security, Identity & Compliance.
  3. Click Groups, then click the Create a New Group button.
  4. Specify a group name, such as 'Students', and click Next Step. You'll end up having just one member in this group, but that's okay.
  5. In the Attach Policy panel, click to check the box for Administrator Access. In this panel you could potentially assign lesser privileges if you were using an application that didn't require you to create new things in EC2.
    Click Next Step.
  6. If all is well with your Group Name and Policies setting (it should show AdministratorAccess), click the Create Group button.

    You now have an empty group of administrators, but no one to put in it yet. Let's create a user for this group.
  7. Click the Users link on the left and then click the Add users button.
  8. In the Details pane of the Add user panel, type a User name, such as your own name.

    And check the box for Programmatic access.
    Note that it enables an access key ID and a secret access key.

    Click the Next: Permissions button.
  9. In the Permissions pane, check the box for the new Group you created, then hit the Refresh button (though I’m not sure it is needed).

    Click the Next: Review button.
  10. Take a look at the details and summary of the new user in the Review pane.

    Click the Create user button.
  11. In the Complete pane you should see that an Access key ID and a Secret Access key have been created for the new user.
    Click Show to see the Secret key.

    The Access Key ID and Secret Access key will be required when you log in to Cloud Builder or link your AWS account to other third-party applications.

    In the Complete pane there is a Download .csv button.
    Click it.
    Save the CSV file to a safe place on disk. Do not lose this file. Once you download your credentials, there's no way to get them back without creating a new user. This is for security purposes.
  12. Click Close.
  13. Go ahead and Sign Out of the IAM Management Console.

In summary, you made an AWS administrator group, created a user, downloaded the user credentials. Now, whenever you supply the user credentials to a third party application (such as Cloud Builder), that application will have administrative access to your AWS account, allowing the application to do such things as creating and terminating instances, etc.

Open and Sign In to Cloud Builder

In step 8 below, you are going to need to have on hand a License File for ArcGIS Server. Retrieve it by going into the Lesson 6 module/folder in Canvas and following the link on the Authorization file for ArcGIS Server - New page. The file has a .prvc file extension.

Cloud Builder creates EC2 instances, Amazon Elastic Load Balancers, and other resources that are charged to your Amazon account. Before you can start using Cloud Builder, you must sign in using your Amazon Access key ID and Secret Access key that you retrieved in the previous section. There are a couple of reasons that Cloud Builder requires the sign in. First of all, as described above, you can't create any resource on Amazon EC2 without providing the information about an account that will be charged for the resources. Cloud Builder uses these credentials behind the scenes for every action that it performs. Also, in a large organization the sign-in screen allows the management of multiple accounts with the same Cloud Builder install. You just sign out and sign in to work with the different accounts.

Follow these steps to retrieve your access credentials and log in to Cloud Builder:

  1. Open ArcGIS Server Cloud Builder on Amazon Services (Cloud Builder). It ought to be under ArcGIS in Start > All Programs.

    (When I opened Cloud Builder I was prompted to update my instance of Adobe AIR, which required me to close Cloud Builder. So, don’t be thrown if that happens to you.)
  2. The first time you run Cloud Builder you will need to provide values for the Access key ID and Secret Access key input boxes using the credentials you saved in the previous section. If you're the only one using this computer, you can also check the box for Remember these credentials so that you don't have to sign in each time you use Cloud Builder.
  3. Click Sign In.
    Note: If you ever open Cloud Builder and get a message about a newer version of Cloud Builder being available, ignore it. Upgrading your Cloud Builder will force you to upgrade your ArcGIS Server version, and we need to all work with the same version in this course.
  4. Now, click Create Site.
  5. In the Name slot provide a name for your site. This name will be used to identify your site in the Sites screen of Cloud Builder and will be part of the URL used to connect to the site. If you use capital letters in the site name, those letters will be converted to lower-case in the URL. For this reason, you may want to stick with all lower-case.
  6. Type a more detailed Description for your site. Other administrators will be able to read this description when they expand your site in the Sites screen of Cloud Builder.
  7. In the Software drop-down list, choose ArcGIS 10.3.1 for Server (Windows).
    (The process for creating a Linux site is nearly identical in Cloud Builder, but logging in and working with Linux is out of the scope of this course.)
  8. Provide the License file by browsing to the .prvc file that you downloaded at the beginning of this section.
    Then click Next.

    It can take a minute for Cloud Builder to send your license file to Esri and receive the proper authorization to continue.

    The number of options you see in the subsequent panels is determined by your license level. In our course we are using an ArcGIS Server Enterprise license, which allows for multiple machines and the use of large relational databases. A full range of options will be visible to you in Cloud Builder, however to control costs and to provide a simple learning experience you'll create a one-machine site.

    You should now be looking at the Amazon Web Services panel of the Create Site wizard in Cloud Builder.
  9. You need to choose a Region and an Amazon VPC.
    For this course please choose US East (Virginia) Region, and accept the default Amazon VPC options.
  10. Choose the option to Create new key pair.

    Note that the key pair file is named arcgis-<your site name>.pem
    Remember this: this same name is used in the Amazon EC2 Console to refer to the security group associated with your instance.

    Make note of the following, too:
    Cloud Builder temporarily places your key pair file in:
    My Documents\ArcGISCloudBuilder\temp\sysgen\keycodes.
    Shortly thereafter it will show in My Documents\ArcGISCloudBuilder

    Click Next.
  11. You should now be in the ArcGIS Server panel. Choose an Instance type of m3.medium
    (You can try t2.medium after installation which is about half the price).
    Medium is the least expensive instance that can run ArcGIS Server at a reasonable speed for this course. At the time of this writing, it costs about 6-7 cents per hour to run an m3.medium instance on Windows in most regions (https://aws.amazon.com/ec2/pricing/).
  12. Now define the size of the EBS volume (an additional storage drive) that gets attached to your instance by changing the default of 100 GB to a lower number of 50 GB. You'll encounter this drive later in the lesson, it's where you'll put all your data for this course. Because we won't be using 100 GB of data, I have asked you to reduce the size of the created drive. You pay a small fee for each GB of storage required by your disk drives.

    Leave unchecked the Deletion option.

    Now you'll choose the number of instances that will be participating in your site. This is where you start to see the power of Cloud Builder to launch a potentially big and scalable site. You can also create a really expensive site by not setting these values judiciously, so follow the instructions carefully!
  13. Leave Number of instances as 1.
  14. Leave Enable auto-scaling unchecked, however take a moment to examine how the auto-scaling options could help you build an elastic site. You can set thresholds that raise or lower the number of machines participating in your site based on CPU usage over a period of time. For example, the default thresholds would add a new instance to your site if CPU usage exceeded 80% for more than 5 consecutive minutes, and it would remove an instance from your site if CPU usage remained below 20% for 5 consecutive minutes.

    These auto-scaling features were not invented by Esri, they are built into Amazon EC2. However, the AWS Management Console does not currently provide an easy interactive way to set up auto-scaling. This is another advantage of using Cloud Builder.

    When you're done viewing the auto-scaling options, click Next. You're now viewing the Geodatabase panel of Cloud Builder, which is actually the part of the process that we're most interested in for this course.
  15. Check Include enterprise geodatabase and choose Microsoft SQL Server Standard on an ArcGIS Server EC2 Instance (the second from the top of the list). Then click Next.

    You might have noticed an option to put the geodatabase on its own dedicated instance. This is recommended if you want to use a larger instance for the geodatabase, or if you don't want ArcGIS Server and the geodatabase to be competing for resources on the same instance. However, it is more expensive to run a separate instance for the geodatabase and unnecessary for this course.
  16. This brings you to the Security panel. In addition to launching instances, Cloud Builder also creates the ArcGIS Server site (something you have to do manually if you install the software yourself). This requires a "primary site administrator" name and password that you'll use in the event you perform administrative functions on your ArcGIS Server site. The primary site administrator is not an account that exists on your domain or operating system, it's used by ArcGIS Server only. Therefore, you can type any name and password you want.
  17. Enter a Username and Password for the ArcGIS Server primary site administrator account.
    The password needs to be a least 8 characters long.
    You'll need to repeat the password to confirm.

    Optionally, when building a site whose communication will be encrypted, you can allow Cloud Builder to apply a Secure Sockets Layer (SSL) certificate that you have previously uploaded to Amazon. This is beyond the scope of this course, so leave Install SSL certificate unchecked and click Next to move to the Summary panel.
  18. Examine the details about the site you are going to create, then click Finish.

It is apt to take up to 30 minutes to create your site. While that's happening, you can click the In Progress icon to see more details about what functions Cloud Builder is performing behind the scenes.

There were a lot of steps involved in launching this instance and some of it may have seemed confusing but the good news is you should not need to go through these steps again. Stopping and re-starting an instance once it's been created is a much simpler process.

C. Starting and Stopping your Instance via Cloud Builder

Once your site is created, you'll see icons for Backups, Stop, Update and Delete.

When you've finished your coursework for the day, you should always click Stop in order to keep costs down. This takes your site back down to one instance (if you had built a large site), then stops that instance. You can click Start when you're ready to work again. It can take a few minutes to start the site.

Your Enterprise Geodatabase instance is currently running and we're going to work with it in the next section. If you are going to continue on, you don't need to stop your instance now.  But be sure to click the Stop button next to your instance in Cloud Builder when you're ready to quit working.

D. Logging in to your Enterprise Geodatabase Instance

Now that your site has been created, you can get ready to log in to the instance and start working with your software.
Your instance needs to be running, so if you Stop-ed it at the end of the previous section, open Cloud Builder and Start it again.

  1. Log in to the AWS Management Console [http://aws.amazon.com/console/], which you saw briefly in section A.  After signing in to the Console, this time click on the EC2 link to get to the dashboard that shows a list of your Resources at the top of the page.
  2. Click the Security Groups link found under Resources.

    When Cloud Builder created your EC2 instance it also created a security group for that instance. We need to add a rule to this security group that allows Remote Desktop connections through port 3389.
  3. Click the name of the security group that Cloud Builder created. Recall that the name will have the format arcgis-.
  4. In the lower panel, click the Inbound tab and click the Edit button.
  5. Click Add Rule. Another entry will show at the bottom of the rules list.

    Set the Type to RDP.

    Choose My IP from the Source dropdown list and click Save.

    The IP address in the slot to the right will/may change to reflect the domain of the machine/location you are working from. With some internet service providers your IP address changes frequently, or, if you take your laptop to a different location the source IP address will change. So you may need to repeat this procedure if you ever find that remote connections are failing.

  6. Now, open a Windows Remote Desktop Connection dialog.
    On most versions of Windows, you can browse to this from Start > All Programs > Accessories > Remote Desktop Connection. In older versions of Windows, it may be in a folder called Communications.  Remote Desktop Connection is a program that you can use to log in to other computers from your own computer.
  7. In the Remote Desktop Connection dialog, expand the Show Options list > Local Resources tab > More button and ensure that the box for Drives is checked, then click OK. This will permit you to copy data from your machine on to the remote machine (in this case, your Amazon EC2 instance).

    Leave the Connection box open. You come back to in the steps below.
  8. Log in to the AWS Management Console, if you need to, and navigate to your list of EC2 Instances in the region where you instructed Cloud Builder to create your ArcGIS Server site (US East (N. Virginia). The instance containing your site should be easy to identify because it is tagged as - SITEHOST in the Name column.
  9. Check the box next to your SITEHOST instance and then examine the instance details in the bottom panel of the page. Find the value for Public IP and copy and paste this value into Notepad. It will look something like this: 112.345.67.123

    The Public IP is the name (URL) that you can use to make a remote desktop connection into your machine. Be aware that the Public IP address will change every time you restart your site. So you will need to go to the AWS EP2 Instances list to retrieve the new Public IP address when you go to make a subsequent Remote Desktop Connection. This is not ideal, but it is what it is for our purposes.

    (It used to be that we would use the Public DNS site instead of the Public IP address as the URL, and that it would remain constant, because Cloud Builder would place an Amazon Elastic Load Balancer (ELB) in your site. But something is amiss with that process, and we do not have time to troubleshoot it.)
  10. In the list of Instances right-click your instance name and click Get Windows Password, then follow the Browse link to get to and open the key pair file that you created in step 10 in the Open and Sign In to Cloud Builder section, above.
    (It’s the .pem file in My Documents\ArcGISCloudBuilder.)
    The text box will fill with the key pair information.

    Then hit the Decrypt Password button.
    The Password can be seen at the bottom of the window.

    Copy and paste the decrypted password into Notepad temporarily.

    Hit the Close button.
  11. Back in your open Remote Desktop Connection dialog, under the General tab, type or paste the Public IP of your instance into the Computer input box.

    In the User name input box, type Administrator, then click the Connect button.

    In the Do you trust this Remote connection? Window, click Connect again.
  12. In the Windows Security dialog, in the Password input box for Administrator, carefully type or paste the password you decrypted, then. Click OK.

    A Remote Desktop Connection desktop will open up.


    Amazon gave you a pretty strong password for this instance, but it's not one you're liable to remember easily. You should change the administrator password once you've logged in.

  13. On the Remote Desktop Connection desktop, use the slider bars to show the lower-left corner.

    Click on the 4-paned window icon. This will take you to the Start list of icons.

    Click on Administrative Tools. An Administrative Tools window will open.
    Then double-click Computer Management. A Computer Management window will open. (I had to close the Admin Tools window before this dialog appeared.)
  14. Expand Local Users and Groups and click Users.
  15. In the list of users, right-click Administrator and click Set Password > Proceed. The password rules are fairly stringent; please see them in the image in Figure 6.1, below.

    Type and confirm a new password that you can remember. In the future you can use this password when logging in to your instance.

    Close the Computer Management and Administrative Tools windows.

    Do NOT close your Remote Connection desktop.
     
    List of password restriction rules
    Figure 6.1: Password Security Setting Rules and Restrictions

Disabling IE ESC

As a security precaution, it's usually not a good idea to go around browsing the web from your production server machine. To do so is to invite malware intrusions onto one of your most sensitive computers. The operating system on your instance, Windows Server 2012, enforces this by blocking Internet Explorer from accessing most sites. This is called IE Enhanced Security Configuration (ESC). IE ESC gets burdensome when you're using the server solely for development or testing purposes like we are. To smooth out the workflows in this course, you'll disable IE ESC right now and leave it off for the duration of the course.

  1. Start the Windows Server Manager by going the Windows taskbar at the lower left of the remote connection desktop window and clicking the icon that looks like a server. (To me, it looks like a tall rectangle with a suitcase in front of it.)
  2. Click Local Server.
  3. Scroll over to the right and find IE Enhanced Security Configuration. Click the On link to access the options for turning it off.
  4. Select Off for both Users and Administrators and click OK.
  5. Close the Server Manager

Licensing ArcGIS for Desktop

Finally, you must license any Esri software that you cause to run on Amazon EC2, including any software that is installed with Cloud Builder. When you went through Cloud Builder you provided ArcGIS Server license information, therefore you don't have to go through the Software Authorization wizard for ArcGIS Server.

However, you still have to authorize ArcGIS for Desktop so that you can use ArcMap and ArcCatalog on the instance. You will use an evaluation code that your instructor has sent you.

Go to the Lesson 6 module in Canvas, to the ArcGIS for Desktop authorization codes page. There you will find an EVA… code associated with your name.


Do the following steps (which only have to be performed once):

  1. First, you need to activate your code.
    Visit the ArcGIS for Desktop Student Trial site and log in with an Esri Global Account (if you don't have a global account, please create one. You will need it later in this course.)
  2. Paste your ArcGIS for Desktop student authorization number into the page where prompted and click Activate ArcGIS.

    On the next screen you are asked to Select a version to download, BUT you need not download the software installer, because the ArcGIS for Desktop software is already installed on your instance.
  3. Now, on the desktop of your Remote Connection session, double-click the shortcut icon for ArcGIS Administration.
  4. Select Advanced (ArcInfo) Single Use and click Authorize Now.
  5. Choose I have installed my software and need to authorize it and click Next.
  6. Choose Authorize with Esri now using the Internet and click Next.
  7. Continue through the wizard and supply your contact information as requested.
    Do not choose to authorize any Extensions; they will be authorized, automatically.
    Eventually you will be prompted to enter your authorization code (EVA...) and finish the wizard. This should allow you to now open programs such as ArcMap on your instance.

With that we're ready to begin playing with our enterprise geodatabases. First you'll read a bit about ArcSDE, a technology that enables the ArcGIS products to work with data stored in an RDBMS.