Mapserver + NEXRAD Composite HOWTO Daryl Herzmann This HOWTO explains how to create "unprojected" geo-referenced NEXRAD imagery ready for use within GIS. As an example, is used to illustrate web mapping with this raster file. Updated on 28 Jul 2004 with various fixes to make the document current. Introduction

Almost every weather website on the Internet has some sort of . Unfortunately almost all of these sites are only fulfilling a single purpose, allowing the user to view an image. Geographical Information Systems (GIS) goes beyond this single purpose model to allow the user to view the image, but then also process it within a GIS system. Processing could involve working with the RASTER data, reprojecting the image, querying the image, or many other things.

This HOWTO covers a mechanism to produce geo-referenced NEXRAD imagery from . As a GIS web mapping example, we use to create a simple, but powerfull web application. This HOWTO assumes that you are running a modern version of Linux. Other UNIXs with installed should work as well.

This HOWTO is geared for those in the with an active feed and already have GEMPAK installed. If you don't have either of these, you will probably just be interested in the Mapserver sections. The radmapserver-files.tar.gz contains an example RADAR image ready for use with Mapserver or any other raster supporting GIS.

411 on abbreviations

Some of the abbreviations used in this HOWTO. IEM|Iowa Environmental Mesonet@ UNIDATA|University meteorological research group@ NEXRAD|Network of Next Generation (WSR-88D) RADAR used by the National Weather Service@ NWS|National Weather Service@ GEMPAK|General Meteorological Application Package. Useful for dealing with NWS produced formats
Abbreviations

Reading this HOWTO

Some of you reading this may have no interest in generating the composites, but wish to use the composites. Currently, I am not aware of a group generating this data source for free. I have started generating a national base composite reflectivity as a proof of concept.

The Quick and Dirty version

The files used in this howto are available here: .

Download and install Proj.4 from . Download and compile Mapserver with proj, TIFF and PNG support from . Place the resulting 'mapserv' binary in your webserver's CGI-BIN directory. Extract the radmapserver-files.tar.gz archive into your web root. Point your browser at .

Links

Requirements Software and Data Access

The following items must be installed and configured before any of this HOWTO could potentially work for you. Describing the installation of any of these packages is beyond the scope of this document. You should be using the Linux operating system, but there is no reason this won't work on Solaris or a BSD variant. GEMPAK, for converting the NIDS file format into RASTER Some mechanism for NIDS data access Realtime access via UNIDATA IDD using LDM Or FTP access to NIDS files via NOAA Or locally available NIDS files on your hard-drive. A webserver, which should be Apache. ImageMagick: used to convert the GIF files to TIFF.

The following items will be installed during this howto. The installation of Proj.4 will require privileged access to your system. The Mapserver components will require access to your webserver's CGI-BIN directory. Proj.4 Mapserver Simple Mapserver application

Hardware

Hardware requirements is a tricky topic. I have seen 300 MHz machines outperform 1 GHz machines when properly configured. Common sense applies here. If you want to generate nationwide composites every 5 minutes, you will need a healthy machine. If you are supporting a small intranet, a modest machine will do.

For our purposes, Mapserver will not introduce a performance bottleneck on your system. It is extremely fast and robust! Any slowness typically comes when you start reprojecting massive raster layers, but we aren't going to do that!

System Environment

You are going to need root privileges on your system. So either know your root password or be very kind to the system administrator! We will be installing Proj.4 into /usr/local, which is a standard place to put 3rd party software. Mapserver will be installed into the cgi-bin directory of your webserver.

Building the composites with GEMPAK

Probably the easiest component of this HOWTO is building the composite image within GEMPAK! If you are running a version of GEMPAK before 5.6.j, you will need to download and install the nex2img program. If you already have nex2img, then you can skip the next subsection.

GEMPAK is publicly available from the . Installing GEMPAK is way outside the scope of this HOWTO.

Nex2img was written by Steve Chiswell at the UNIDATA Program Center. His program is the key to making this whole system work. Before nex2img, it was extremely tricky to get the geo-referencing of RASTERs, produced in GEMPAK, correct for use within GIS.

Building nex2img

Building nex2img is very straightforward with GEMPAK already installed. $ cd $NAWIPS $ wget http://mesonet.agron.iastate.edu/docs/radmapserver/nex2img.tar.gz $ tar -zxvf nex2img.tar.gz $ cd unidata/programs/nex2img $ make clean $ make all $ make install $ make clean $ rehash $ which nex2img

With the nex2img program built, you are now ready to generate the composites.

Scripting the generation of the composite

This simple script will crank out an image. I am specifying a GRDAREA for Iowa and extended. You will want to modify for your own needs. Keep in mind that this application composites individual site NEXRADs, you can't build a nationwide composite without having all the individual sites. #!/bin/csh # Replace to fit your environment! source /home/nawips/Gemenviron set gtime=`date -u +'%y%m%d/%H%M'` rm -f radar.gif radar.tif >& /dev/null nex2img << EOF > nex2img.log GRDAREA = 37.00;-104.00;48.99;-87.01 PROJ = CED KXKY = 1700;1200 CPYFIL = GFUNC = n0r RADTIM = ${gtime} RADDUR = 15 RADFRQ = STNFIL = nexrad.tbl RADMODE = P RADFIL = radar.gif LUTFIL = upc_rad24.tbl list run exit EOF if (-e radar.gif) then convert radar.gif radar.tif cp radar.tif /var/www/htdocs/radmapserver/gisdata endif

The nex2img program will generate an image called "radar.gif". In order to work with most GIS systems, you will probably want to convert this file to another format namely TIFF, but you can choose JPG or PNG. We then place the radar.tif file in the webserver directory for later use by mapserver.

Without getting too long winded, lets discuss image formats. The easiest format for use within GIS is probably TIFF. Although very large, uncompressed tiff files are extremely fast within GIS. The reason is that the application can "seek" the file to only pull needed data. With compressed formats, an entire set of data needs to be uncompressed and then processed. Also, most people will probably try loading the image in ESRI's ArcView. ArcView likes TIFF files and has built in support for them.

Geo-referencing the composite image

Now that we have an image, we need to georeference it. Georeferencing of images is done with a world file. A world file is simply a file with 6 lines of meta data describing the navigation of an image. The format is for example: 0.010 (size of pixel in x direction) 0.000000000000 (rotation term for row) *Typically zero 0.000000000000 (rotation term for column) *Typically zero -0.010 (size of pixel in y direction) -95.0000 (x coordinate of centre of upper left pixel in map units) 42.300 (y coordinate of centre of upper left pixel in map units) Note that the units of the geo-referencing are in the same units of the projection. Since we are in lat-lon space, the units are decimal degrees. If you were in UTM space, it would probably be meters. For the image generated from my example of nex2img, my world file would look like 0.01000 0.00000 0.00000 -0.01000 -104.00000 49.00000

Naming the world file

Depending on your GIS, the naming convention of your world file may be different. Many systems use a generic .wld to describe any world file. Others use variants on the image format nomenclature, for instance, the world file for TIF formated files is .tfw and for PNG is .pngw. For our purposes, we will use .wld.

Lets pause for a second

If you have made it this far, you are probably wondering what to do with the RASTER image with no political borders or other denotations and this silly 6 line world file. The next section shows an example of taking this image and using the powerful Mapserver to produce a Web Map application. Trust me, this is not the only application of this data.

Building Proj.4 & Mapserver

Like many Open Source packages, Mapserver relies on external libraries to handle much of the heavy lifting. This is both a good thing and a bad thing. The good thing is that the Mapserver folks don't have to program all the needed functionality, the bad thing is that you can reach dependency-hell with this setup. For our purposes, we will only build with Proj.4 support, but you can rebuild mapserver later to link with other libraries for more functionality.

Building Proj.4

provides projection and reprojection support for Mapserver. Building and installing is hopefully simple, if you have ever built an open source package before. Make sure that /usr/local/lib is included in /etc/ld.so.conf, so that Mapserver can link to that library without special configuration. $ wget ftp://ftp.remotesensing.org/pub/proj/proj-4.4.8.tar.gz $ tar -zxvf proj-4.4.8.tar.gz $ cd proj-4.4.8 $ ./configure; make $ su -l # make install #(as root, to install in /usr/local!) # /sbin/ldconfig # exit $ make clean

If everything went well, you should notice a libproj in /usr/local/lib that mapserver can link to. Building Mapserver

As I write this (28 Jul 2004), Mapserver is at version 4.2.1 . It is preferable that you be running this version for this HOWTO, but it is not required. For our purposes, the only difference between the current version 4.2.1 and the old 3.6.x versions is the 'OFFSITE' keyword in the .map file. In Mapserver 4.x, this is a RGB value and in Mapserver 3.x, this is a color map index value. $ wget http://cvs.gis.umn.edu/dist/mapserver-4.2.1.tar.gz $ tar -zxvf mapserver-4.2.1.tar.gz $ cd mapserver-4.2.1 $ ./configure --with-proj --with-png $ make $ cp -i mapserv /var/www/cgi-bin/mapserv

Now, if the build process worked, you should have a binary called 'mapserv' if your current working directory. Verify that the mapserv binary is okay by executing $ ./mapserv -v This command will spit out the capabilities of Mapserver. For this demo, we need support for SHAPEFILES and the TIFF file format. The output from the above command should contain OUTPUT=PNG INPUT=TIFF INPUT=SHAPEFILE.

Lets pause again!

If you got this far with these instructions, you are much smarter than I am. Since I couldn't even get this to work following my own instructions. Anyway, we now have a CGI application that we can install on our webserver and begin building a web mapping application! Yeah!

Building your Web Mapping Application Getting Started

It is somewhat hard to keep this generic, when everyone does things differently. I will assume a base RPM install of apache on Red Hat Linux. If your environment is different, you should know how it is different!

Assuming that your web directory root is /var/www/htdocs and that your CGI-BIN directory is /var/www/cgi-bin, you will want to issue the following commands as a privileged user to these directories. $ cd /tmp $ wget http://mesonet.agron.iastate.edu/docs/radmapserver/radmapserver-files.tar.gz $ cd /var/www/htdocs $ mkdir mstmp $ chown apache mstmp $ tar -xzvf /tmp/radmapserver-files.tar.gz The mstmp directory is needed by mapserver to write temporary files to. The radmapserver directory is where we will build our application interface at.

The Mapserver .map file

Fundamental to the Mapserver application, is a .map configuration file. For our demo, we will build a simple map file called radmapserver.map in our /var/www/htdocs/radmapserver directory. The file is as follows. # Start of Mapfile (radmapserver.map) NAME radmapserver STATUS ON SIZE 450 350 EXTENT -104 37 -87 49 UNITS DD IMAGETYPE PNG # Map projection definition PROJECTION "proj=epsg:4326" END # Map interface definition WEB LOG "/var/www/htdocs/mstmp/radmapserver.log" TEMPLATE radmapserver.html IMAGEPATH "/var/www/htdocs/mstmp/" IMAGEURL "/mstmp/" END # Our Geo-referenced RADAR layer LAYER TYPE RASTER STATUS ON NAME radar DATA "/var/www/htdocs/radmapserver/gisdata/radar.tif" PROJECTION "proj=epsg:4326" END # For Mapserver 3.x, we would use # OFFSITE 0 OFFSITE 0 0 0 END # Our states shapefile, just to get some political boundaries LAYER TYPE POLYGON STATUS ON NAME states DATA "/var/www/htdocs/radmapserver/gisdata/states.shp" PROJECTION "proj=epsg:4326" END CLASS OUTLINECOLOR 255 0 0 END END END # End of mapfile radmapserver.map

In the Web definition of our mapfile, we referenced a file called radmapserver.html. This file is a HTML template mapserver uses to generate an interface. Our example radmapserver.html file is as follows. <html> <head> <title>RadMapserver Test</title> </head> <form method="GET" action="/cgi-bin/mapserv" name="mapserv"> <input type="hidden" value="[mapext]" name="imgext"> <!-- imgxy is set to half of your display image size --> <input type="hidden" value="225 175" name="imgxy"> <input type="hidden" value="[map]" name="map"> <input type="hidden" value="browse" name="mode"> <table border="1"> <tr><td> <input name="img" type="image" src="[img]" border="0"> </td><td> <p> <b> Select layers to display </b> <br><input type="checkbox" value="radar" name="layer" [radar_check] > RADAR layer <br><input type="checkbox" value="states" name="layer" [states_check] > States layer <p> Set your zoom option: <br> <select name="zoom" size="1"> <option value="4" [zoom_4_select]> Zoom in 4 times <option value="3" [zoom_3_select]> Zoom in 3 times <option value="2" [zoom_2_select]> Zoom in 2 times <option value="1" [zoom_1_select]> Recenter Map <option value="-2" [zoom_-2_select]> Zoom out 2 times <option value="-3" [zoom_-3_select]> Zoom out 3 times <option value="-4" [zoom_-4_select]> Zoom out 4 times </select> <p> <input type="submit" value="Make Map!"> <form> </td></tr></table> <html>

Believe it or not, we are ready to rock and roll! Get your favorite Open Source web browser loaded up and point it at your website with the following URL http://your.web.site/cgi-bin/mapserv?map=/var/www/htdocs/radmapserver/radmapserver.map&layers=states

GIS-Ready! NEXRAD information

This section contains links to folks already generating GIS-Ready products from NEXRAD data.

Iowa Environmental Mesonet

The IEM is generating a nationwide composite of base reflectivity every 5 minutes. This product is available in this directory: http://mesonet.agron.iastate.edu/data/gis/images/unproj/USCOMP/ The IEM is generating a PNG and TIFF formatted image. There is also a compressed TIFF image available in that directory as well. The file prefix n0r_0 contains the current data.

Scripting an automated download

Here is an example script which would allow you to automatically download files from the IEM website and then save them to a local directory #!/bin/csh set DATADIR="/set/me/to/something" cd /tmp wget -q http://mesonet.agron.iastate.edu/data/gis/images/unproj/USCOMP/n0r_0.tif.Z uncompress n0r_0.tif.Z cd ${DATADIR} foreach i (9 8 7 6 5 4 3 2 1 0) cp -f n0r_${num}.tif n0r_`echo ${num} + 1 | bc`.tif end mv /tmp/n0r_0.tif n0r_0.tif exit

Conclusions

This HOWTO demonstrated a very primitive component of GIS, web mapping. There is just so much more that can be done. For those of you familiar with PHP, Mapserver has an outstanding PHP interface to the Mapserver API. You can create some of the neatest applications using the Mapserver API and the power of PHP. The Iowa Environmental Mesonet has implemented most of its Mapserver interfaces using the PHP mapscript module. The source code for these applications is licensed under the GPL, but I just don't place the code on the website to download. Just email me if you would like a copy...

Future Work

I would suspect that most people will not have the ability to generate nationwide NEXRAD composites. One of the neat things Mapserver can do is act as a Web Map Service (WMS) server and client. Potentially a WMS server could be set up somewhere on the Internet and then your mapserver could make a remote WMS call to it get the image.

Another option, will be for either UNIDATA, the IEM, or somebody else to generate the nationwide composite (in 'unprojected' coordinates) and make that available on the IDD. This is probably the easiest way to get this data to the community.

Although never covered in this HOWTO, timestamps are a big issue with this system. I have built hacks using DBF files in order to get timestamps on the generated images. Here is an example of output using this 'hack'. At some point, I will add this hack to the HOWTO.

Contact info

If you have questions, shoot me an email (akrherz@iastate.edu) or give me a call (515.294.5978). Good luck!