How-To: OMG Basemaps

How-To: OMG Basemaps


Quick Start: To get a skeletal Base Map Project directory structure and scripts, download this file, and type:
	tar -zxvf OMG_basemap_template.tgz
Before you dive in, you should the read the Caveats & Limitations at the very least.

0. Table of Contents

1. Introduction
2. Naming Convention and Directory Structure
3. Adding a New Source
4. Updating Configuration Files for a New Source
5. Gridding and Mosaicing Data for a New Source
6. Making Index Maps
7. Making the Overview map
8. Making Base Map Webpage (and associated files)
9. Making 'Extra' Base Map Files
10. Supporting Scripts
11. Summary of Commands
12. Common Problems
13. Future Additions
14. Caveats & Known Limitations
15. Document History and Bug Fix log


1. Introduction

I've slowly developed a set of monstrous scripts that allow us to create a web portal to display and distribute mapping products. The system was initially designed to specifically handle datasets whose acquisition span over several years (and use, potentially, different mapping platforms each year). The system was initially developed for the ArcticNet 15'x30' Basemap Series, and has since been generalized and modified to support the Southwest New Brunswick 2'x2' Basemap Series (SWNB). The image below shows an example of the web interface from the SWNB Basemap Series.

The interface is designed to allow users to geographically browse datasets and to download mapping products for basemaps of interest (e.g. ESRI grid files, geotifs). The mapsheet on the right is a standard OMG product with colour-encoded bathymetry and sun-illuminated topography. This is referred to the Base Map. The image on the lower left is what is referred to as an Indexmap, whereas the image on the upper left is called an Overview Map. These names are used consistently throughout this documenation (and throughout the scripts) to avoid confusion. The web portal also allows users to view the coverage achieved by the survey projects that contribute to the basemap, as shown in the image below. Coverage images are referred to as Coverage Maps. Immediately below the coverage map is a set of lists of survey lines that contribute to the Base Map.


2. Naming Convention and Directory Structure

The scripts use a STRICT directory structure and naming convention. The base map naming convention is 44_54_N_66_56_W, the first two number represent the latitude in degrees and minutes of the upper boundary of the basemap, the second two numbers represent the longitude in degrees and minutes of the western boundary of the basemap. Thus, the base map name is corresponds to the UPPER LEFT CORNER of the mapsheet. The name of a basemap does not imply its size. The basemaps cannot currently be less than 1'x1'. The following listing is an example directory structure used in the SWNB series. A bit of explanation is required, of course, this is done below.
	drwxrwxr-x  3 jonnyb omg 20480 Jan 30 13:32 1992_Creed_Passam/
	drwxrwxr-x  4 jonnyb omg  4096 Jan 30 13:30 1995_Creed_Passam/
	drwxrwxr-x  3 jonnyb omg  4096 Jan 30 13:22 2000_Plover_Letang/
	drwxrwxr-x  2 jonnyb omg  4096 Jan 30 07:53 2002_Heron_GrManan/
	drwxrwxr-x  3 jonnyb omg  4096 Jan 30 13:26 2002_Heron_StAnd/
	drwxrwxr-x  2 jonnyb omg  4096 Jan 30 13:16 2002_Plover_Passam/
	drwxrwxr-x  2 jonnyb omg 12288 Jan 30 13:15 2004_Heron_Deer/
	drwxrwxr-x  3 jonnyb omg 32768 Jan 30 13:33 2005_Heron_Deer/
	drwxrwxr-x  4 jonnyb omg 12288 Jan 30 13:25 2006_Heron_Deer/
	drwxrwxr-x  3 jonnyb omg 28672 Jan 30 13:31 2006_Heron_OAP/
	drwxrwxr-x  2 jonnyb omg  4096 Jan 31 14:58 bin/
	drwxrwxr-x  2 jonnyb omg  4096 Jan 31 14:58 config/
	drwxrwxr-x  2 jonnyb omg  4096 Jan 31 14:58 docs/
	lrwxrwxrwx  1 jonnyb omg    40 Jan 31 11:12 html -> /homes/jonnyb/public_html/swnb_mapsheets/
	drwxrwxr-x  2 jonnyb omg  4096 Jan 30 07:52 logs/
	-rwxr-xr-x  1 jonnyb omg    28 Jan 31 14:55 makefile*
	drwxrwxr-x  3 jonnyb omg 36864 Jan 31 14:43 maps/
	drwxrwxr-x  3 jonnyb omg 36864 Jan 31 14:43 maps_index/
	drwxrwxr-x  6 jonnyb omg  4096 Jan 31 14:56 misc/
	drwxrwsr-x  3 jonnyb omg  4096 Jan 31 14:56 tmp/

1. Source Map Directories

        drwxrwxr-x  3 jonnyb omg 20480 Jan 30 13:32 1992_Creed_Passam/
        drwxrwxr-x  4 jonnyb omg  4096 Jan 30 13:30 1995_Creed_Passam/
        drwxrwxr-x  3 jonnyb omg  4096 Jan 30 13:22 2000_Plover_Letang/
        drwxrwxr-x  2 jonnyb omg  4096 Jan 30 07:53 2002_Heron_GrManan/
        drwxrwxr-x  3 jonnyb omg  4096 Jan 30 13:26 2002_Heron_StAnd/
        drwxrwxr-x  2 jonnyb omg  4096 Jan 30 13:16 2002_Plover_Passam/
        drwxrwxr-x  2 jonnyb omg 12288 Jan 30 13:15 2004_Heron_Deer/
        drwxrwxr-x  3 jonnyb omg 32768 Jan 30 13:33 2005_Heron_Deer/
        drwxrwxr-x  4 jonnyb omg 12288 Jan 30 13:25 2006_Heron_Deer/
        drwxrwxr-x  3 jonnyb omg 28672 Jan 30 13:31 2006_Heron_OAP/

The base map concept works around the idea that we have many sources of data (in the above case, from many years of acquisition). Each source of data has a directory associated with it, in which that source's bathymetry, backscatter and sidescan maps are stored in separate subdirectories (bathy, backscatter and sidescan, respectively). The source directories should follow the naming convention of YYYY_PROJECT_NAME/ for consistency between base map projects (though this is not strictly required by the scripts).

2. Output Directories

        lrwxrwxrwx  1 jonnyb omg    40 Jan 31 11:12 html -> /homes/jonnyb/public_html/swnb_mapsheets/
        drwxrwxr-x  2 jonnyb omg  4096 Jan 30 07:52 logs/
        drwxrwxr-x  3 jonnyb omg 36864 Jan 31 14:43 maps/
        drwxrwxr-x  3 jonnyb omg 36864 Jan 31 14:43 maps_index/
The output directories have output from the various scripts, be it webpages, logfiles or mapsheets. The html/ directory contains the set of webpages that constitute the web portal to the dataset, in this case, it is a softlink to a location directly accessible via the WWW. The logs/ directory contains the output of various logfiles generated during the base map generation process. The maps/ directory contains mapsheets that have been created by combining the various source mapsheets. The maps_index/ directory contains index mapsheets, which are low resolution mapsheets that show the coverage (in addition to having backdrop imagery). These mapsheets in maps/ and maps_index/ are in OMG format (.r4, .mos, etc).

3. Other Files and Directories

        drwxrwxr-x  2 jonnyb omg  4096 Jan 31 14:58 bin/
        drwxrwxr-x  2 jonnyb omg  4096 Jan 31 14:58 config/
        drwxrwxr-x  2 jonnyb omg  4096 Jan 31 14:58 docs/
        -rwxr-xr-x  1 jonnyb omg    28 Jan 31 14:55 makefile*
        drwxrwxr-x  6 jonnyb omg  4096 Jan 31 14:56 misc/
	drwxrwsr-x  3 jonnyb omg  4096 Jan 31 14:56 tmp/
The remaining directories and files play a supportive role in base map production. Firstly, the bin/ directory contains all the necessary scripts to create base maps. The config/ directory has many configuration files that modify exactly how the basemaps are created (more on this later). The docs/ directory contains this documentation. The makefile may be used in the future to help streamline the update process using time-stamps. The misc/ directory has misc other files and directories required for base map production (again, more on this later). The tmp/ directory is a temporary directory that stores files generated during a run of any of the basemap scripts.

3. Adding a New Source

There are two steps here, first you need to create a few files in the source data directory. Once that's done, you move over to the base maps directory.

1. Working with source data

We need to start off in the directory where you have source data, i.e. .merged files. You'll need to make 2 files that will be used during gridding and mosaicing. These files contain a listing of all the .merged and .ss files you want to include in the base map series.

	cd /drives/hecla/disk1/NB_HydroData/SouthWest_NB/2002_Heron_GrManan/EM3000/
	ls merged/JD*/*.merged > merged_input.list
	ls ss/JD*/*.ss_bp > ss_input.list
If you want to get fancy, you can use the find command, as such:
	cd /drives/hecla/disk1/NB_HydroData/SouthWest_NB/2002_Heron_GrManan/EM3000/
	find merged/ -name '*.merged' | sort > merged_input.list
	find ss/ -name '*.ss_bp' | sort > ss_input.list
It's a good idea to check that you've got the same number of .merged and .ss files. You can use the wc program to count the number of lines in each file, e.g.:
	wc -l merged_input.list ss_input.list
This should dump out something like this:
  436 merged_input.list
  443 ss_input.list
  879 total
A final step would have you peruse both files and comment out any lines you specifically want to exclude from the base map series, e.g. transit lines, patch test lines, etc. These can be commented out by placing a '#' sign before a survey line's entry in either file. You can place comments in the files as well, as long as they're preceded with a '#' sign. This is a good way to document why certain lines are excluded, e.g. bad navigation.

2. Working with base maps

Now you can wander over to the directory (or machine) where your basemaps project exists and create a source project directory for your new source.

	cd /drives/boothia/disk1/jonnyb/SWNB_basemaps/
You can then run a script which will generate base map header files for your new source. It will generate a new source project directory and data type subdirectory for you and will populate it with base map .header files that encompass the spatial area covered by the .merged files specified in the xyzlist (or the ss files specified in the xyzlist). The base map size (in minutes of latitude and minutes of longitude) are pulled out from the config/config.map file. If you run this script again at a later date, it will compare the time stamps of the input .merged files against the .header files and will only update the .header files that need to be (this will force subsequent gridding and mosaicing for those mapsheets). This process is quick for .merged files that have had 'getBounds' run on them. It can be slow otherwise since it needs to get the spatial extent of the .merged or .ss files each time it is run.
	make_basemap_headers -data_type bathy -project 2002_Heron_GrManan/ -xyzlist /drives/hecla/disk1/NB_HydroData/SouthWest_NB/2002_Heron_GrManan/EM3000/merged_input.list
The same procedure can be done with ss_input.list files, whether they are backscatter derived from multibeam data or from sidescan data, as is shown in the example below.
	make_basemap_headers -data_type backscatter -project 2002_Heron_GrManan/ -xyzlist /drives/hecla/disk1/NB_HydroData/SouthWest_NB/2002_Heron_GrManan/EM3000/ss_input.list
	make_basemap_headers -data_type sidescan -project 2007_Heron_SS/ -xyzlist /drives/foxe/disk2/data/2007_Heron_GrManan/KSS/ss_input.list

The next step is to create OMG mapsheet files for the .header files. Each basemap project will have a custom script to do this, depending primarily on the map projection chosen for the base map series. This script parses the config/config.map file to get the base map size (again, in minutes latitude and longitude) and resolution.

	make_swnb_mapsheet 2002_Heron_GrManan/{bathy,backscatter,sidescan}/*.header
This will create a .{r4,mos}.gz file for every .header file in the source/data_type directories. In addition, it will create a copy in the 'maps' directory and it will make an appropriate index mapsheet in the maps_index/ directory (see notes on 'make_index_blank' in the supporting scripts section below). The 'make_swnb_mapsheet' script is part of the collection of base map scripts; it is customized for a New Brunswick Double Stereographic projection. A similar script exists for the ArcticNet base map series (make_arcticnet_basemap) that is configured for a Lambert Conformal Conic projection. In this case, the SWNB script generates 2'x2' basemaps.

4. Updating Configuration Files for a New Source

Now that you've created .{r4,mos} files for the mapsheets in your source directory, the next step is to create grids and mosaics. Before diving in to gridding and mosaicing, you need to update some of the configuration files to reflect the fact that you've added a new source. At this stage, you'll need to update the following configuration files, their contents are explained at length below the list.

config/config.sources: This file simple contains a list of the source project directories. You simply add your directory name to the list, without a trailing slash, e.g.

1992_Creed_Passam
1995_Creed_Passam
2000_Plover_Letang
2002_Heron_GrManan
2002_Heron_StAnd
2002_Plover_Passam
2004_Heron_Deer
2005_Heron_Deer
2006_Heron_Deer
2006_Heron_OAP

config/config.weigh_grid: This file contains customized weigh_grid options for each source. The first entry in the list is the source directory name, matching the name in the config/config.sources file. The remaining entries on the line are command line options that are passed to weigh_grid, verbatim, i.e. as you would type them on the command line.

1992_Creed_Passam -xyzlist /drives/hecla/.../1992_Creed_Passam/EM1000/merged_input.list -custom_weight misc/weights/creed_em1000.weights -beamwidth 2.4
1995_Creed_Passam -xyzlist /drives/hecla/.../1995_Creed_Passam/EM1000/merged_input.list -custom_weight misc/weights/creed_em1000.weights -beamwidth 2.4
2000_Plover_Letang -xyzlist /drives/hecla/.../2000_Plover_Letang/EM3000/merged_input.list -custom_weight misc/weights/heron_em3000.weights -beamwidth 1.5
2002_Heron_GrManan -xyzlist /drives/hecla/.../2002_Heron_GrManan/
2002_Plover_Passam -xyzlist /drives/hecla/.../2002_Plover_Passam/EM3000/merged_input.list -custom_weight misc/weights/heron_em3000.weights
2002_Heron_StAnd -xyzlist /drives/hecla/.../2002_Heron_StAnd/EM3000/merged_input.list -custom_weight misc/weights/heron_em3000.weights
2004_Heron_Deer -xyzlist /drives/hecla/.../2004_Heron_Deer/EM3000/merged_input.list -custom_weight misc/weights/heron_em3000.weights -beamwidth 1.5
2005_Heron_Deer -xyzlist /drives/hecla/.../2005_Heron_Deer/EM3000/merged_input.list -custom_weight misc/weights/heron_em3000.weights
2006_Heron_Deer -depthshift -0.7 -xyzlist /drives/hecla/.../2006_Heron_Deer/EM3002/merged_input.list -custom_weight misc/weights/heron_em3002_low_density.weights -beamwidth 1.5
2006_Heron_OAP -xyzlist /drives/duckie/OAP_2006/merged_input.list -custom_weight misc/weights/heron_em3002_high_density.weights -beamwidth 1.5
The major points to notice are the -xyzlist and -custom_weight options. The -xyzlist is the path to the merged_input.list file you created earlier. The -custom_weight option points weigh_grid to a weights file, normally stored in the misc/weights directory. Other options of interest in the above example is the different beam widths associated with each system, also, the 2006_Heron_Deer dataset accidentally had a draft of 0.0 so it is corrected during gridding by applying a depth shift of -0.7 metres to each sounding. Don't forget to create a .weights file for your system if it doesn't exist already. The convention established with the ArcticNet and SWNB base maps is to store the weight files in the misc/weights/ directory.

config/config.addWG: This file contains a list of source directories followed by the weight assigned to them by the addWG program when combining multiple sources into a basemap grid. A value of zero indicates to use the .weights and .weights_depth file. A negative value indicates to weight the aforementioned files by the absolute value of the number, e.g. the 1992 and 1995 Creed data are given half the weight when average with overlapping data from later surveys since the data from both of these surveys is of poor quality. A standard entry of 0.0 is fine in most cases. If you want to globally exclude a source from the compilation maps, you can comment it out in the config.addWG file. An example is shown below, with comments.

# 1992 and 1995 Creed are not worth adding to the grid due to poor data quality
#1992_Creed_Passam 0.0
#1995_Creed_Passam 0.0
2000_Plover_Letang 0.0
2002_Heron_GrManan  0.0
2002_Plover_Passam 0.0
2002_Heron_StAnd 0.0
2004_Heron_Deer 0.0
2005_Heron_Deer 0.0
2006_Heron_Deer 0.0
2006_Heron_OAP 0.0

config/config.mos2_{backcatter,sidescan}: These files work in the exact same manner as the config.weigh_grid file. The first entry is the name of the project source directory. Following entries are command line options passed to mos2 that dictate exactly how to mosaic data from this particular project source. Sidescan data (i.e. imagery from a true sidescan, not from multibeam backscatter) is treated slightly differently in that you cannot use the "-swap_dirs ss/ merged/" option. Again, you'll definitely want to run getBounds on sidescan files, otherwise the mos2 program will take a horribly long time to run. This isn't necessary for multibeam backscatter files.

1992_Creed_Passam -xyzlist /drives/hecla/.../1992_Creed_Passam/EM1000/ss_input.list -swap_dirs ss/ merged/ -maxdist 20.0
1995_Creed_Passam -xyzlist /drives/hecla/.../1995_Creed_Passam/EM1000/ss_input.list -swap_dirs ss/ merged/ -maxdist 20.0
2000_Plover_Letang -xyzlist /drives/hecla/.../2000_Plover_Letang/EM3000/ss_input.list -swap_dirs ss/ merged/ -addDN 40
2002_Heron_GrManan
2002_Plover_Passam -xyzlist /drives/hecla/.../2002_Plover_Passam/EM3000/ss_input.list -swap_dirs ss/ merged/ -addDN 40
2002_Heron_StAnd -xyzlist /drives/hecla/.../2002_Heron_StAnd/EM3000/ss_input.list -swap_dirs ss/ merged/ -addDN 34
2004_Heron_Deer -xyzlist /drives/hecla/.../2004_Heron_Deer/EM3000/ss_input.list -swap_dirs ss/ merged/ -addDN 34
2005_Heron_Deer -xyzlist /drives/hecla/.../2005_Heron_Deer/EM3000/ss_input.list -swap_dirs ss/ merged/ -addDN 34
2006_Heron_Deer -xyzlist /drives/hecla/.../2006_Heron_Deer/EM3002/ss_input.list -swap_dirs ss/ merged/ -addDN 34
2006_Heron_OAP -xyzlist /drives/duckie/.../OAP_2006/ss_input.list -swap_dirs ss/ merged/ -addDN 34
Again, the -xyzlist option provides the path to the ss_input.list file you created earlier. Of note in the example above is the DN shift that is applied to each dataset such that their mosaics are as close as possible in range of grey levels.

config/config.stencil_{backscatter,sidescan}: These files are lists of the source directories in order of preferred layover during the stencil operation that combines them into a single mapsheet. The stencil program simply overlays mosaiced mapsheets thus those that are stenciled first will be hidden if there is any overlapping data from other datasets. In the example below, the 1992 and 1995 Creed data are always stenciled first, this way they will be hidden by later surveys of (presumably) better quality. To completely ignore a data source in the compilation, you can comment it out in the config.stencil file, as shown below:

# 1992 and 1995 Creed are not worth adding
#1992_Creed_Passam
#1995_Creed_Passam
2000_Plover_Letang
2002_Heron_GrManan
2002_Plover_Passam
2002_Heron_StAnd
2004_Heron_Deer
2005_Heron_Deer
2006_Heron_Deer
2006_Heron_OAP


5. Gridding and Mosaicing Data for a New Source

Once you've update the 7 configuration files (and maybe created a new weights file in misc/weights/), you're now ready to grid. This is done using the make_grid script, as in the following example:
	make_grid 2007_Heron_OAP/bathy/*.r4.gz
Mosaicing is done similarly:
	make_mosaic 2007_Heron_OAP/{backscatter,sidescan}/*.mos.gz
Both of these scripts have a -skip option, this will skip files that already have a .r4 or .mos file created (implying they've been gridded and/or mosaiced before). More importantly, they will check the time stamp of the .header files versus the file that was passed to it (be it .r4. or .mos). If the .header file is older, the default behaviour is to SKIP that particular mapsheet (this saves regridding and remosaicing mapsheets that are not new). This behaviour can be overridden with the -force option.

Note that make_mosaic runs VERY slowly with sidescan data (not MB backscatter) if getBounds has not been run on the .ss files (a new feature as of Feb. 22, 2008). If you're not the owner of the .ss data, you'll need to contact the owner of the data files and get them to run getBounds on the .ss files.

You can now add the coverage achieved from this newly added source to the mapsheets in the maps/ directory. These scripts take the .{r4,mos} files in the maps/ directory as input and they use the config.sources, config.addWG and config.stencil_{backscatter,sidescan} files to instruct them how to combine the mapsheets from all the sources into a single mapsheet. Also, the mapsheets are trimmed by the lat/long bounds of the basemap if the TrimBaseMap option is set to YES in the config/config.map file. This is done by the make_trimmed_map script, which is described below in the supporting scripts section below.

	make_combo_grid maps/bathy/*.r4.gz
	make_combo_mosaic maps/{backscatter,sidescan}/*.mos.gz
These scripts will only update combo mapsheets when an equivalent mapsheet in a source directory is newer. Again, this saves unnecessary processing. This behaviour can be overridden by the -force switch.

6. Making Index Maps

Now that you've gridded and mosaiced all of your data, the next step is to create index maps. Before starting, it's time to examine a few parameters of interest in the config/config.map file. There are several configuration options stored in the config/config.map file, however, only those relevant to this discussion are shown in the example below.
# Size of the Index Mapsheets in minutes
IndexMapLatitudeSizeMinutes 10
IndexMapLongitudeSizeMinutes 10

# Size of .r4 for Index Mapsheet bathy backdrop
# should be significantly larger that Basemap resolution
IndexMapResolutionMetres 20
IndexMap8bitBackdrop misc/background_maps/SWNB.sun_315

# Index Mapsheet projection parameters (Lambert Conformal Conic)
# Has nothing to do with the basemap projection, simply for creation
# of the Index Mapsheet on the website.
IndexMapStdParallel1 45
IndexMapStdParallel2 44.66
IndexMapOriginLat 44
IndexMapOriginLon -66.5
IndexMapFalseEasting 500000
IndexMapFalseNorthing 500000

# This is the width of the resized index and overview map gif images
WebpageLeftColumnWidthPixels 250
Of immediate interest are the parameters IndexMapLatitudeSizeMinutes and IndexMapLongitudeSizeMinutes. Each Base Map has a parent "index map", whose name can be derived from the base map name and these two parameters (using the script get_index_mapsheet_name). The other options are self explanatory. Note that all index maps are created using a Lambert Conformal Conic projection. Since these maps are used for giving an overview, it doesn't matter if they match the base map projection. The projection parameters are set in the config/config.map file, as shown above. For new projects, you should update the origin latitude and longitude to be close to the center of your work area. Also, the standard parallels should fall within your work area.

Index maps are created and populated with data using a few scripts, namely:

  1. make_index_map: takes OMG index mapsheets from the maps_index/ directory and searches for the maps/ directory for basemaps that need to be patched into them. The script uses the time stamps of the basemaps and the index maps to determine which basemaps need to be repatched into the index maps, this behaviour can be overridden with the -force option. If you want to completely refresh the index maps, you can specify the -fresh_start option, this wipes the index mapsheet clean and then forces a repatch of all contributing basemaps into the index mapsheet, regardless of time stamps.
  2. make_index_gif: generates the gif index map image for the web portal. Paints the outlines of all basemaps that have been patched in to date. Also creates a .webmap file which is used during webpage creation to allow the index map to be a clickable webmap. The .gif is moved to the html/maps_index/ directory whereas the webmap file is kept in the maps_index/ directory. The width of the output .gif file is specified by the 'WebpageLeftColumnWidthPixels' parameter in the config/config.map file. If a gif file exists already, the time stamp is checked against the source mapsheet and it is skipped if the mapsheet is older than the gif file. This can be overridden with the -force option. This script calls the make_index_background script, which is described in the supporting scripts section below.
An example of their usage is shown below.
	make_index_map maps_index/*.{r4,mos}.gz
	make_index_gif maps_index/*.{r4,mos}.gz


7. Making the Overview map

As usual, there are a few configuration options that need to be specified in the config/config.map file:
# Path to the overview background map
OverviewMapBackdrop misc/background_maps/NB_nbstereo_dem.sun_315

# This is the width of the resized index and overview map gif images
WebpageLeftColumnWidthPixels 250
The 'make_overview_map' script is used to create an overview map (.gif) and .webmap for any given index mapsheet. The overview map is created from the backdrop image specified by the path in the config file. The outlines of index maps are drawn into the image, and a .webmap file is created, much in the same way as with the index maps. The output .gif image is stored in html/maps_index/{bathy,backscatter}/overview.gif, the overview.webmap file resides in the maps_index/ directory (along with overview.ppm). The script will generate a fresh image if -fresh_start is specified. The script is run as follows:
	make_overview_map maps_index/bathy/*.r4.gz
	make_overview_map maps_index/backscatter/*.mos.gz

8. Making Base Map Webpage (and associated files)

The webpage creation process involves three scripts. Each of the scripts takes base maps from the maps/ directory as input. All three scripts put their output in the html/ directory. Note that the maptype (bathymetry or backscatter) is inferred from the .r4 or .mos extension of the input mapsheets.
  1. make_map: Creates the gif mapsheet for the webpage (and it's thumbnail). If an existing image is found, its timestamp is compared to that of the source mapsheet so that it can skip over mapsheets whose gif images are newer than the source mapsheet. This behaviour can be overridden with the -force option. This script can take quite long to complete so it's best not to use -force unless absolutely necessary.
  2. make_webpage: Creates the webpage for the base map. Calls 'make_coverage_thumbs' to create coverage thumbnails for each source (see supporting scripts). A template webpage is stored in the misc/html directory, it can be tailored specifically for each base map project. The template is copied and modified by 'make_webpage' through the use of capitalized keywords in the template file. These should be unique words that aren't commonly found in html markup tags. Another restriction is that the keyword cannot be a substring of another keyword (e.g. keyword NORTH is not useable if keyword NORTHEAST is already used).
Quite a few parameters are parsed from the config/config.map file. For 'make_map', the following parameters are of interest:
# Size of the Basemap Mapsheets in minutes
BaseMapLatitudeSizeMinutes 2
BaseMapLongitudeSizeMinutes 2
BaseMapResolutionMetres 1

# A few parameters for the basemap GIFs
BaseMapLogoLeft misc/img/omg_banner.gif
BaseMapLogoRight misc/img/GGE_Crest.gif
BaseMapLogoRightXDimension 207

BaseMapColourBar misc/img/colourbar.gif
BaseMapGreyBar misc/img/greybar.gif
BaseMapSeriesName OMG/SWNB
BaseMapSeriesAuthor J.Beaudoin

# Used to stretch the base map backscatter imagery
BaseMapMinDN 115
BaseMapMaxDN 195

# Used to label the greybar in base map backscatter
# Should correspond to min DN and max DN!
BaseMapMinDB -40
BaseMapMaxDB -10
The 'make_coverage_thumbs' parses the following information:
# Size of the Basemap Mapsheets in minutes
BaseMapLatitudeSizeMinutes 2
BaseMapLongitudeSizeMinutes 2

# Used to colourcode the r4's in the index map bathy coverage.
# Does not affect the colourcoding of the basemap GIF files on the
# website...also used by the coverage thumbs
IndexMapMinDepth 0
IndexMapMaxDepth 50

# Used to stretch the index map backscatter imagery (and coverage thumbs)
IndexMapMinDN 115
IndexMapMaxDN 195

# This is the width of the coverage thumbnails
CoverageThumbnailsWidthPixels 200
Finally, 'make_webpage' uses the following:
# Size of the Basemap Mapsheets in minutes
BaseMapLatitudeSizeMinutes 2
BaseMapLongitudeSizeMinutes 2

# Size of the Index Mapsheets in minutes
IndexMapLatitudeSizeMinutes 10
IndexMapLongitudeSizeMinutes 10

# Path to the overview background map
OverviewMapBackdrop misc/background_maps/NB_nbstereo_dem.sun_315

# This is the width of the resized index and overview map gif images
WebpageLeftColumnWidthPixels 250
Phew...! An example of the running sequence is shown below.
	make_map maps/{bathy,backscatter}/*.{r4,mos}.gz
	make_webpage maps/{bathy,backscatter}/*.{r4,mos}.gz

9. Making 'Extra' Base Map Files

You can create ESRI grid files for each basemap, these are then available for download via the web portal. The script takes base maps from either the maps/ directory as input (when creating ESRI files for the combo mapsheets), or from the individual source directories. Output is placed in the html/grids/source/ directory, where source matches the directory passed to make_ESRI (e.g. make_ESRI maps/45_06_N_67_06_W.r4.gz will put the ESRI grid in html/grids/maps/, whereas make_ESRI 1992_Creed_Passam/45_06_N_67_06_W.r4.gz will put output in html/grids/1992_Creed_Passam/. The script creates a zipped .tar archive file which contains a .flt and .hdr file following the ESRI binary grid format. Grids can be created for either bathymetry (.r4) or backcatter (.mos) mapsheets. The script will skip mapsheets whose ESRI files are newer than the source mapsheet. This behaviour can be overridden with the -force switch. An example is shown below.
	make_ESRI maps/*.{r4,mos}.gz
If you decide not to offer ESRI grid files, you'll need to update the template.html file to remove the link.

10. Supporting Scripts

There are a few supporting scripts that are called by the main scripts above, they mostly do little bits of work behind the scenes that isn't worth mentioning above (mostly to cut down on the confusion that ensues when presented with too many details up front). This section serves as a bit of documentation, should the need arise to fix any of the supporting scripts.

11. Summary of Commands

For those who just want an overview/reminder (or who are the type to not read manuals), here's a summary of the relevant commands with minimal explanation.Note that I've placed this at the end so hopefully you've read at least a little bit of the manual before becoming fed-up and jumping to this part.
	# Wander over to wherever your .merged and .ss files are and create a few list files.
	cd /drives/hecla/disk1/NB_HydroData/SouthWest_NB/2002_Heron_GrManan/EM3000/
	ls merged/JD*/*.merged > merged_input.list
	ls ss/JD*/*.ss_bp > ss_input.list
	# Now do the sidescan
	cd /drives/hecla/disk1/NB_HydroData/SouthWest_NB/2002_Heron_GrManan/K200/
	ls ss/JD*/*.ss_bp > ss_input.list

	# Now wander over to your Base Map project directory and make .header and .{r4,mos} files for your new data source
	cd /drives/boothia/disk1/jonnyb/SWNB_basemaps/
	make_basemap_headers -data_type bathy -project 2002_Heron_GrManan/ -xyzlist /drives/hecla/disk1/NB_HydroData/SouthWest_NB/2002_Heron_GrManan/EM3000/merged_input.list
	make_basemap_headers -data_type backscatter -project 2002_Heron_GrManan/ -xyzlist /drives/hecla/disk1/NB_HydroData/SouthWest_NB/2002_Heron_GrManan/EM3000/ss_input.list
	make_basemap_headers -data_type sidescan -project 2002_Heron_GrManan/ -xyzlist /drives/hecla/disk1/NB_HydroData/SouthWest_NB/2002_Heron_GrManan/K200/ss_input.list

	# Make OMG mapsheets for all the header files
	make_swnb_mapsheet 2002_Heron_GrManan/{bathy,backscatter,sidescan}/*.header

	# Update the configuration files to include the new data source
	vi config/config.sources
	vi config/config.weigh_grid
	vi config/config.addWG
	vi config/config.mos2_backscatter
	vi config/config.mos2_sidescan
	vi config/config.stencil_backscatter
	vi config/config.stencil_sidescan

	# Now grid and mosaic all of your new mapsheets
	# Both of these only update the mapsheets whose .header files
	# are newer than the .r4/.mos files (use -force to do all sheets).
	make_grid 2002_Heron_GrManan/bathy/*.r4.gz
	make_mosaic 2002_Heron_GrManan/{backscatter,sidescan}/*.mos.gz

	# Incorporate them into the base map collection.  Again, only combo
	# mapsheets that have newer source files are recreated (use -force
	# to do all sheets).
	make_combo_grid maps/bathy/*.r4.gz
	make_combo_mosaic maps/{backscatter,sidescan}/*.mos.gz

	# Now patch the base map coverage into the index mapsheets
	make_index_map maps_index/{bathy,backscatter,sidescan}/*.{r4,mos}.gz

	# At this point, things are done from an OMG mapsheet production
	# point of view, the remainder of the commands are for 
	# updating the website.

	# Create GIF files from the index mapsheets.
	make_index_gif maps_index/{bathy,backscatter,sidescan}/*.{r4,mos}.gz

	# Now update the overview map with any new index maps that might
	# have been created
	make_overview_map maps_index/{bathy,backscatter,sidescan}/*.{r4,mos}.gz

	# Make the GIF mapsheet for the basemaps
	make_map maps/{bathy,backscatter,sidescan}/*.{r4,mos}.gz

	# Create ESRI grid files if you want to offer them on the website
	make_ESRI maps/{bathy,backscatter,sidescan}/*.{r4,mos}.gz

	# Finally, after all above web related commands have been run,
	# create the webpage.  It is important that you run this command last
	# since it depends on the output from the web related commands above.
	make_webpage maps/{bathy,backscatter,sidescan}/*.{r4,mos}.gz

	# Now that you're done, copy the entire html directory somewhere online.
	mkdir ~/public_html/SWNB_Basemaps/
	cp -r html/* ~/public_html/SWNB_Basemaps/

12. Common Problems


13. Future Additions...


14. Caveats & Known Limitations


15. Document History and Bug Fix Log

  1. February 7th, 2007: First set of rationalized scripts and documentation available for general consumption.
  2. Late March, 2007:
  3. November, 2007:
  4. December, 2007:
  5. November, 2008:
  6. February, 2009:
  7. March, 2009:

Last modified: February 23rd, 2010, by J. Beaudoin