CyberGIS-HydroShare


HydroShare is a collaborative environment for sharing hydrologic data and models.  It primarily serves the hydrologic sciences community. Both CyberGIS and HydroShare are sustainable software integration projects following open and adaptable software development practices. Both projects move computation off the desktop into advanced cyberinfrastructure based on service-oriented architecture enabling computation on big data, avoiding platform dependency and software installation requirements and serving as gateways to high performance computing. Interoperability between the two systems will enable the coupling of data and multi-scale and multidisciplinary modeling capabilities from both communities and empower scalable geospatial and hydrologic sciences. As both projects grow to integrate big data analytics and advanced modeling capabilities, we face two major challenges in interoperability that we believe XSEDE can help address with its richness in cyberinfrastructure resources and supercomputing architecture:

  1. How to establish a user environment that seamlessly integrates distributed data, software, and computation from both CyberGIS and HydroShare in a sandbox where users can focus on domain research?
  2. How to achieve interoperability features as extensible, reproducible, and reusable software solutions for scalable development, deployment, and operation in order to support broader collaboration of various multidisciplinary research in our communities?

Grant: NSF award 1047916

People: Anand Padmanabhan (apadmana@illinois.edu), Dandong Yin (dyin4@illinois.edu), Fangzheng Lyu(flu8@illinois.edu), Shaowen Wang (shaowen@illinois.edu)

 

External Link:  Hydroshare Project

Computational Movement Pattern Analysis


The analysis of geographic mobility data and associated spatial interactions is of great importance to understanding complex geographic phenomena and their space-time dynamics.

This research project aims to design and develop innovative data models and analytical methods for data-intensive movement analysis. Statistical and computational approaches are developed to study spatial patterns of origin-destination movement flows and more generally spatial interactions. Specifically, a multidimensional spatial point data model has been developed, which models each OD movement flow as a point with four spatial dimensions, two for its origin and two for its destination. It is then possible to detect spatial movement patterns by analyzing the 4D point patterns, using methods such as scan statistics. For instance, with a bivariate 4D point process model, we can evaluate the differences and similarities between the spatial distributions of a pair of OD movement datasets, and detect areas where the two spatial distributions differ the most. A multidimensional Bernoulli spatial scan statistics method is developed to detect OD region pairs with abnormally high concentrations of one movement dataset over the other. Experiments demonstrate that our method can effectively detect spatial patterns from large movement datasets, and is applicable to both individual-level and aggregated movement data.

People: Yizhao Gao (ygao29@illinois.edu), Ting Li (tingli3@illinois.edu), Shaowen Wang (shaowen@illinois.edu), Myeong-Hun Jeong (mhjeong@myeonghun.org), Kiumars Soltani (soltani2@illinois.edu)


Publications:

  • Yizhao Gao, Ting Li, Shaowen Wang, Myeong-Hun Jeong & Kiumars Soltani (2018) “A multidimensional spatial scan statistics approach to movement pattern comparison”. International Journal of Geographical Information Science. 32:7, 1304-1325, DOI: https://doi.org/10.1080/13658816.2018.1426859

How to Use CIGI Wiki?


Get a wiki account

CIGI members will get a wiki account when joining the group.

Create a wiki page

By default, wiki users are not authorized to create a wiki page. Please ask staff members to create the page for you.

Edit a wiki page

Please follow wiki syntax to edit a wiki page.

Upload files/images

If you need to upload files/images, please follow steps as below:

  • Use “https://cigi.uiuc.edu” in Firefox or “https://cigi.illinois.edu” in IE to open the CIGI homepage
  • At the time of creating/editing a wiki page, click the image icon
  • In the popup dialog, upload files and select one to be inserted into wiki page

Configure Active DIR/Kerberos on Ubuntu and Red Hat


Ubuntu 10.04
  • copy /etc/krb5.conf from everest to /etc/
  • sudo apt-get install libpam-krb5
  • sudo pam-auth-update
Ubuntu 8.04
  • Install necessary packages
  $ sudo apt-get-install heimdal-clients libpam-heimdal
  • Configure Kerberos with the details of the AD realm and IP addresses, /etc/krb5.conf (copy it from everest)
  • Update the PAM configuration to check for Kerberos accounts, /etc/pam.d/common-auth, choose whether you want a Kerberos login prompt or a regular prompt first.

#

  # /etc/pam.d/common-auth - authentication settings common to all services
  #
  # This file is included from other service-specific PAM config files,
  # and should contain a list of the authentication modules that define
  # the central authentication scheme for use on the system
  # (e.g., /etc/shadow, LDAP, Kerberos, etc.).  The default is to use the
  # traditional Unix authentication mechanisms.
  #
  auth    sufficient      pam_krb5.so ccache=/tmp/krb5cc_%u
  auth    sufficient      pam_unix.so likeauth nullok_secure use_first_pass
  auth    required        pam_deny.so
  • To manage the Kerberos tickets update /etc/pam.d/common-session
  #
  # /etc/pam.d/common-session - session-related modules common to all services
  #
  # This file is included from other service-specific PAM config files,
  # and should contain a list of modules that define tasks to be performed
  # at the start and end of sessions of *any* kind (both interactive and
  # non-interactive).  The default is pam_unix.
  #
  session required        pam_unix.so
  session optional        pam_foreground.so
  session optional        pam_krb5.so minimum_uid=1000

Red Hat Specific

  • rpm -qa | grep pam
  • Make sure you have: pam_krb5-2.2.14-10
  • copy over system-auth from everest (update if necessary)
  • Make sure that the clock is insync, otherwise auth will fail
  • Add users to wheel group for sudo privilege
  • use authconfig-tui to configure pan to use krb

Reference

Cloning VM in VMWare Server


  • cp -r oldDir NewDir
  • Replace HD filename
  [root@n2 SimpleGrid]# for file in MalariaMap.*; do ext=`echo $file | awk -F . '{print $2}'`; mv $file SimpleGrid.$ext; done
  [root@n2 SimpleGrid]# for file in *-*.vmdk ; do ext=`echo $file | awk -F - '{print $2}'`; mv MalariaMap-$ext SimpleGrid-$ext; done
  • Replace all occurences in .vmx , .vmdk (and maybe .vmxf)
  :%s/MalariaMap/SimpleGrid/g (vi shortcut to replace all)
  • Just say VM is copied and it will change uuids and mac address (use uuidgen to generate unique IDs if needed)
  • Update /etc/resolv.conf
  search cigi.uiuc.edu
  search atlas.uiuc.edu
  nameserver 130.126.2.131

Ubuntu Networking:

Change /etc/hosts, /etc/hostnane, /etc/network/interface as needed

RedHat Variants

  • You can also try to run command system-config-network-tui
  • Update /etc/sysconfig/network-scripts/ifcfg-eth0 , /etc/hosts
  [cigi@simplegrid ~]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0 (/etc/network/interface for Ubuntu)
    # Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]
    DEVICE=eth0
    BOOTPROTO=static
    ONBOOT=yes
    HWADDR=00:0c:29:a3:6a:e3
    TYPE=Ethernet
    NETMASK=255.255.255.192
    IPADDR=130.126.3.70
    GATEWAY=130.126.3.65
    DNS1=130.126.2.131
    SEARCH=cigi.illinois.edu
  • Restart Networking /etc/init.d/network restart

OpenLayers Tutorial


OpenLayers makes it straightforward to put a dynamic map from various sources onto any web page. OpenLayers, developed as open source software, is written as a JavaScript library for visualizing map data using most modern web browsers. Its code can be found at http://www.openlayers.org/api/OpenLayers.js. The associated documentation of the Javascript API can be foound at http://trac.openlayers.org/wiki/Documentation. A number of example maps created using Openlayers can be found at http://openlayers.org/dev/examples/

Visualizing Temperature data using Openlayers

In this tutorial we will see how to visualize data using OpenLayers. The resulting map can be found at http://n3.cigi.uiuc.edu:8080/temperature.html. For this map we take the interpolated temperature data from our postgis data store and visualize it as a layer on top of the default background map. The source code for displaying this map can be found on n3.cigi.uiuc.edu in /srv/cigi/httpd/www/ directory.
One directory per group has been created with the code placed there for the purpose of convenience. Please feel free to modify the code in your directories.

Openlayers installed on n3 can be reached at http://n3.cigi.uiuc.edu:8080/OpenLayers-2.7/lib/OpenLayers.js.
Below is the code of the temperature.html with comments, so that you can understand how to visualize map layers using openlayers.

Example Code

<html xmlns="http://www.w3.org/1999/xhtml">
    <head>
        <!--COMMENTS: Page Title-->
        <title>Temperature Data Vizualization</title>
        <!--COMMENTS: Location of Openlayer library: No Need to Change-->
        <script src="http://n3.cigi.uiuc.edu:8080/OpenLayers-2.7/lib/OpenLayers.js"></script>
        <script type="text/javascript">
       //<!--COMMENTS: AJAX Code: No need to change-->
       function GetXmlHttpObject(){
          var xmlHttp=null;
          try {
             // Firefox, Opera 8.0+, Safari
             xmlHttp=new XMLHttpRequest();
          } catch (e) {
             // Internet Explorer
             try {
                xmlHttp=new ActiveXObject("Msxml2.XMLHTTP");
             } catch (e) {
                xmlHttp=new ActiveXObject("Microsoft.XMLHTTP");
             }
         }
         return xmlHttp;
       }
   //<!--COMMENTS: Defining Variables -->
   var map, jpl_wms, ol_wms, out_wms;
        // <!-- COMMENTS: avoid pink tiles -->
        OpenLayers.IMAGE_RELOAD_ATTEMPTS = 3;
        OpenLayers.Util.onImageLoadErrorColor = "transparent";
            function init(){
                //<!--COMMENTS: Define some Bounding boxes-->
                var minExtent = new OpenLayers.Bounds(-1, -1, 1, 1);
                var maxExtent = new OpenLayers.Bounds(-150, -60, 150, 60);
                var usBounds = new OpenLayers.Bounds(-135.44, 10.18,-65.08, 50.55);
                var dmsBounds =  new OpenLayers.Bounds(-128.0, 24.052, -65.126, 50.0);
                 //<!--COMMENTS: Instantiating a new map with 10 zoom levels-->
                 map = new OpenLayers.Map('map',
                    {controls: [new OpenLayers.Control.Navigation(),
                             new OpenLayers.Control.PanZoomBar()],
                           numZoomLevels: 10});
                //<!--COMMENTS: Creating a new default map layer. This creates a base layer in the map called "Globe"-->
                ol_wms = new OpenLayers.Layer.WMS( "Globe",
                    "http://labs.metacarta.com/wms/vmap0?", {layers: 'basic'},{ 'wrapDateLine': false} );
                ////<!--COMMENTS: Creating a new base layer with NASA landsat image called "Global Mosaic"-->
                jpl_wms = new OpenLayers.Layer.WMS( "Global Mosaic",
                "http://t1.hypercube.telascience.org/cgi-bin/landsat7",
                {layers: "landsat7"}, {'wrapDateLine': false});
                jpl_wms.setVisibility(false);
                //<!--COMMENTS: Create our overlay layer (not a base layer) using temperature data. -->
                //<!--COMMENTS: As you may notice this layer is created by contacting the geoserver on n3.cigi.uiuc.edu -->
                //<!--COMMENTS: Also notice the layer name is same as the layer name in geoserver-->
                out_wms = new OpenLayers.Layer.WMS(
                "Temperature Data",
                "http://n3.cigi.uiuc.edu:8080/geoserver/wms?",
                {
                   layers:'cigi:gisolveoutstriped',
                   styles: '',
                   srs: 'EPSG:4326',
                   format: 'image/png',
                   tiled: 'true',
                   transparent: true
                },
                {
                    'opacity': 0.7, 'isBaseLayer': false,  'wrapDateLine': false
                }
            );
               //<!--COMMENTS: Setting some map options -->
                map.setOptions({minExtent:dmsBounds});
               //<!--COMMENTS: Add all the layers to the map -->
                map.addLayers([ol_wms, jpl_wms,  out_wms]);
                //<!--COMMENTS: adding the control that allows you to switch between layers-->
                map.addControl(new OpenLayers.Control.LayerSwitcher());
                //<!--COMMENTS: Zooming map to a particular region-->
                map.zoomToExtent(dmsBounds);
            }
        </script>
    </head>
    <!--COMMENTS: Calling the init function when body is loaded -->
    <body onload="init()">
        <!--COMMENTS: Setting the layout in html -->
        <div id="tags">
            <table width="100%"><tr>
            <td width="40%"><div id="image"></div></td>
            <td><h2>Temperature Data Vizualization - Striped</h2></td>
            </tr><table>
        </div>
        <!--COMMENTS: The div tag for the region where the map will be displayed -->
        <div id="map" style="width:1200px; height:400px"></div>
        <div id="data"></div>
    </body>
</html>

Reference

Geoserver Tutorial


Publishing your data through GeoServer consists of three steps

  1. Login to the Geoserver
    • An account with the same username as passwd as database has been created
  2. Create a DataStore for your feature data (you will create Coverage stores for raster datasets)
  3. Create a FeatureType for feature data (create Coverages for raster datasets)

A good tutorial illustrating how to add shapefile data to geoserver can be found here. A step-by-step screencast of this process can be found here

Adding PostGIS datasets

  1. Login in to the server with username and password. For our tutorial we will use geoserver located at http://n3.cigi.uiuc.edu:8080/geoserver/welcome.do
  2. From the welcome page, click on Config → Data → DataStore → New
  3. Now in the new DataStore interface. Select ‘postgis’ for ‘Feature Data Set Description’ give it a ID (e.g. groupx) and click new.
    • You should get a new interface that will have various text boxes. The four important options for you to specify are a) Namespace; b) database; c) user; d) password. Select ‘geod480_group1’ or ‘geod480_group2’ as appropriate, from the drop down list. The database name should be the name of that contains the postgis data (e.g.group1_postgis, group2_postgis). Then enter your database username and password.
    • Note: hostname should be ‘localhost’, port should be ‘5432’ and schema should be ‘public’. These values should be correctly populated by default and you do not need to change them.
    • Click on Submit
    • Now you have created a new datastore that will talk to the postgis database
    • Click Apply and Save button (in that order) on the left hand side of the page to store the new datastore configuration in geoserver.
  4. Create a new feature type.
    • Follow these links from the Welcome page. Config → Data → FeatureTypes → New
    • When you click on new, you should get a list of all feature data available to geoserver.
    • You will see the data from the previously added postgis datastore in the drop down menu. The feature data will be named as dataStoreID:::database_table_name.
    • After selecting the correct dataset from drop-down menu click new.
    • You will get a page named ‘FeatureType Editor’. On this page a) Select an appropriate style sheet or create a new one; b) Click on ‘Lookup SRS’ button or manually enter the appropriate SRID; and c) Click on ‘Generate’ button which is next to the Bounding Box. You should be able to leave other values as defaults. Finally Click ‘submit’ button.
    • Your feature data from the PostGIS database should now be available as a map, through a WMS interface.
    • Click Apply and Save button on the left hand side of the page to make available the feature dataset you just created.
  5. Go to the demo page on the GeoServer (http://n3.cigi.uiuc.edu:9009/geoserver/mapPreview.do) and click on the layer you just created.

Style Sheet

GeoExt styler extention can be found at http://n3.cigi.uiuc.edu:8080/geoserver/www/styler/index.html

References

PostGIS Tutorial


Creating a PostGIS database

CREATE DATABASE your_database WITH TEMPLATE = template_postgis;
  • This step is already done on n3.
  • We have 2 databases group1_postgis and group2_postgis, created with the right privileges

Additional information about the template

  • Creating a database the template creates two tables named spatial_ref_sys and geometry columns
  • The table definitions for these metadata tables are as follows (both these tables are already created)
CREATE TABLE spatial_ref_sys (
  srid       INTEGER NOT NULL PRIMARY KEY,
  auth_name  VARCHAR(256),
  auth_srid  INTEGER,
  srtext     VARCHAR(2048),
  proj4text  VARCHAR(2048)
)

and

CREATE TABLE geometry_columns (
  f_table_catalog    VARRCHAR(256) NOT NULL,
  f_table_schema     VARCHAR(256) NOT NULL,
  f_table_nam        VARCHAR(256) NOT NULL,
  f_geometry_column  VARCHAR(256) NOT NULL,
  coord_dimension    INTEGER NOT NULL,
  srid               INTEGER NOT NULL,
  type               VARCHAR(30) NOT NULL
)

Using PostGIS

Creating a table that GIS Objects

  • Create a regular table
CREATE TABLE test_tutorial_postgis ( ID int4);
  • Add a GIS column named geometry of type polygon
SELECT AddGeometryColumn(
  '',
  'test_tutorial_postgis',
  'geometry',
  4326,
  'POLYGON',
  2
);
  • Syntax of AddGeometryColumn: AddGeometryColumn(<schema_name>, <table_name>, <column_name>, <srid>, <type>, <dimension>)
  • Why use the AddGeometryColumn() function and all the other OpenGIS functions?
    • In addition to creating a new column, using AddGeometryColumn also adds metadata that a simple create statement wont add. E.g. the metadata in geometry_columns added by AddGeometryColumn is used by Mapservers to ensure correct map projections.

Delete a GIS object column

SELECT DropGeometryColumn('', 'test_tutorial_postgis','geometry');

Inserting data into that postGIS Table

insert into
  test_tutorial_postgis (ID,geometry)
values
  (
   1,
   GeomFromText('POLYGON( (-128 50, -128 49.948, -127.874 49.948, -127.874 50, -128 50))',
                  4326)
  );

Perform spatial query

  • E.g. Find all objects with 100 meters of POINT(-128 49.948 49.948)
  SELECT id
  FROM test_tutorial_postgis
  WHERE ST_DWithin(geometry, GeomFromText('POINT(-128 49.948 49.948)',4326), 100.0);

References

Database Tutorial


Connecting to Server

  • Use openssh client (Start → All Programs → OpenSSH)
  hostname = geog473.cigi.uiuc.edu
  username = group1/group2
  port = 22
  • Enter your group passwd when prompted

Connecting to Database

psql -U username -d database_name
  • username = group1/group2
  • database_name = group1/group2
  • Enter your group passwd when prompted

Postgres Commands

SQL Cheat Sheet

  • List all accessible databases
\l
  • List all the tables in current DB
\dt
  • Quit
\q

SQL Commands

  • Create Table
create table your_first_name (key int, attr varchar(20),value float);
  • Insert a row
 insert into your_first_name values(1, 'attr0', 100);
 insert into your_first_name values(2, 'attr2', 101);
  • Update table contents
update your_first_name set attr='attr1' where key=1;
  • List contents of table
select * from your_first_name;
select * from your_first_name where attr='attr1';
select * from your_first_name where attr='attr0';
  • Delete Rows
delete from your_first_name where key=1;

Courses Offered by Prof. Shaowen Wang


Principles of Geographic Information Systems (GEOG: 480; Informatics Minor)

Introduction to Geographic Information Systems (GEOG/ESES: 379; Informatics Minor)

Advanced Geographic Information Systems (GEOG: 479)

Geovisualization (GEOG: 473)

Topics of Geographic Information Science (GEOG: 595)

Independent Study

Advanced Spatial Analysis