********************************************************************* NOTE: THIS ARCHIVE IS BEING CHANGED....DISREGARD THIS INFORMATION! ********************************************************************* Please see the following ARL archive reference page: http://www.arl.noaa.gov/ss/transport/archives.html ------------------------------------------------------------- EDAS data are available in the files called edas.mmmyy.00#, where mmm is the month (e.g. jul) and yy is the year (97) and 00# refer to: 00#=001 - days 1-7 of the month 00#=002 - days 8-15 of the month 00#=003 - days 16- 22 00#=004 - rest of the month The data are archived on tape and eventually sent to the National Climatic Data Center (NCDC) for distribution to anyone. Note however, that the EDAS archive is fairly new and still undergoing changes and, as of March 1997, has not been sent ot NCDC. NCDC can be reached at: Climate Services Branch National Climatic Data Center 151 Patton Avenue Asheville, NC 28801 Email: orders@ncdc.noaa.gov Phone: 704-271-4800 Fax: 704-271-4876 ftp: ftp.ncdc.noaa.gov www: http://www.ncdc.noaa.gov The following is a document describing the EDAS ARCHIVE. ---------------------------------------------------------------- NCEP Model Output -- EDAS ARCHIVE TD-???? Archive began: January 1, 1997 Prepared for the National Climatic Data Center (NCDC) by Glenn D. Rolph NOAA-Air Resources Laboratory 1315 East-West Highway Silver Spring, MD 20910 (301 713-0295) February 26, 1997 EDAS ARCHIVE OVERVIEW The National Weather Service's National Centers for Environmental Prediction (NCEP) runs a series of computer analyses and forecasts operationally (Petersen and Stackpole, 1989). One of the operational systems is the Global Data Assimilation System (GDAS, Kanamitsu, 1989), which uses the spectral Medium Range Forecast model (MRF) for the forecast (Sela, 1980). Another system is the EDAS (Eta Data Assimilation System), covering the U.S. At NOAA's Air Resources Laboratory (ARL), NCEP model output are used for air quality transport and dispersion modeling. ARL archives both EDAS and GDAS data using a 1-byte packing method. Both archives contain basic fields such as the u- and v-wind components, temperature, and humidity. However, the archives differ from each other because of the horizontal and vertical resolution, as well as in the specific fields, provided by NCEP. ORIGIN OF THE DATA The 3-hourly archive data come from NCEP's EDAS. The EDAS was implemented into the operational early Eta model runs during 1995. The EDAS is an intermittent assimilation system consisting of successive 3-h Eta model forecasts and Optimum Interpolation (OI) analyses for a pre-forecast period (12-h for the early Eta) on a 38 level, 48 km grid. A 6-h forecast from the GDAS is used to to start the assimilation at 12-h prior to model start time. The following is a schematic for the 12Z cycle 6-h GDAS | 3-h eta | 3-h eta | 3-h eta | 3-h eta | 48-h Eta ---------------> |-------->|-------->|-------->|-------->|------------------....--> Forecast | fcst | fcst | fcst | fcst | forecast 00Z 03Z 06Z 09Z 12Z where; | | = Eta OI analysis | The 3-h analysis updates allow for the use of high frequency observations, such as wind profiler, NEXRAD, and aircraft data. ARL saves the successive 3-hour analyses, twice each day to produce a continuous data archive. Some fields such as precipitation and surface fluxes are not available in the analysis files, therefore these are taken from the successive 3-hour forecast files. The 48 km data are interpolated to a 40 km, Lambert Conformal Grid, covering the continental United States. ARL PROCESSING The ARL archiving program extracts every other grid point of the 3 hourly, 40 km data to produce a 3 hourly, 80 km dataset on pressure surfaces. NOTE: The EDAS Archive is subject to change in the near future due to the large amount of data that is being processed (currently 85 MB per week) and the recent availability of the data on Eta model surfaces as opposed to pressure surfaces. The data are put into weekly files and saved to tape for shipping to NCDC. About 6 months of EDAS data is also put online at ARL's web site (http://www.arl.noaa.gov/ss/transport/archives.html) for easy access via ftp. DATA DESCRIPTION The archive data file contains the data in synoptic time sequence, without any missing records (missing data will be represented by nulls and the forecast hour by negative 1). Therefore it is possible to position randomly to any point within a data file. Each file contains data for approximately one week: days one through 7, 8 through 15, 16 through 22, and 23 through the end of the month. At each time period, an index record and surface data come first, followed by all data in each pressure level from the ground up. Cartridge Specifications (beginning January 1, 1997) Two files, containing data for two weeks, are archived on a 3480 cartridge and sent to NCDC. Files are copied to cartridge with the UNIX "dd" command. TYPE 3480 cartridge, ASCII* RECORD LENGTH 30475 BLOCK SIZE 30475 * Note that each data record is composed of a 50-character header in ASCII, followed by the binary packed data. Therefore, an ASCII-EBCDIC conversion on the entire data record or cartridge file is not possible. Data Grid The data are on a 93 by 65 Lambert Conformal grid (Fig. 1). In Table 1, the data grid is identified by the model that produced the data, a grid identification number, the number of X and Y grid points, the Pole position (latitude and longitude) of the grid projection, a reference latitude and longitude, the grid spacing (km) which is true at the reference point, the orientation with respect to the reference longitude, the angle between the axis and the cone, and a point on the grid in grid units and latitude and longitude . The given pole position results in the lowest left grid point to have a value of (1,1). --------------------------------------------------------------------------------------- Table 1. Data Grid Specifications --------------------------------------------------------------------------------------- Model ID X Y Pole Pole Ref. Ref. Ref. Orient Cone Sync Sync Sync Sync Type # Max Max Lat. Lon. Lat. Lon. Grid Ang. X Y Lat. Lon. --------------------------------------------------------------------------------------- EDAS 28 93 65 90N 0W 35N 95W 80 0 25 52.5 25.5 35N 95W --------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------- Table 2. Meteorological Fields for EDAS Archive Data. --------------------------------------------------------------------------------------- Field Units Label Data Order ---------------------------------------------------------------------------------------- Pressure reduced to mean sea level hPa MSLP S1 Temperature at surface K TMPS S2 Accumulated precipitation (3 h accumulation) m TPP6 S3 Accumulated convective precipitation (3 h accumulation) m TPP3 S4 Soil temperature 0-10cm K SOLT S4 Volumetric soil moisture content frac. SOLW S5 Temperature at 2m AGL K T02M S6 Relative Humidity at 2m AGL % RH2M S7 U-component of wind at 10 m AGL m/s U10M S8 V-component of wind at 10 m AGL m/s V10M S9 Potential temperature at 10 m AGL K P10M S10 Pressure at surface hPa PRSS S11 Water equiv. of accumulated snow depth Kg/m2 WESD S12 Categorial snow (yes=1, no=0) - CSNO S13 Categorial ice pellets (yes=1, no=0) - CICE S14 Categorial freezing rain (yes=1, no=0) - CFZR S15 Categorial rain (yes=1, no=0) - CRAI S16 Latent heat net flux at surface (upward is negative!) W/m2 LHTF S17 Sensible heat net flux at surface (upward is negative!) W/m2 SHTF S18 Momentum flux, u-component at surface N/m2 UMOF S19 Momentum flux, v-component at surface N/m2 VMOF S20 Surface roughness length m RGHS S21 Low cloud cover % LCLD S22 Medium cloud cover % MCLD S23 High cloud cover % HCLD S24 Total cloud cover % TCLD S25 Downward short wave radiation flux W/m2 DSWF S26 Water temperature K WTMP S27 U-component of wind with respect to grid m/s UWND U1 V-component of wind with respect to grid m/s VWND U2 Geopotential height gpm* HGTS U3 Temperature K TEMP U4 Pressure vertical velocity hPa/s WWND U5 Relative humidity % RELH U6 Turbulent kinetic energy J/kg TKEN U7 ------------------------------------------------------------------------------------------ * geopotential meters ------------------------------------------------------------------------------------------ Meteorological Fields and Vertical Structure The archived data file only contains some of the fields normally produced by the model at NCEP. These were selected according to what is most relevant for transport and dispersion studies and disk space limitations. In Table 2, the fields are identified by a description, the units, and a unique four character identification label that is written to the header label (see Data Grid Unpacking Procedure in a later section) of each record. Data order in the file is given by a two digit code. The first digit indicates if it is a surface (or single) level variable (S) or an upper level variable (U). The second digit indicates the order in which that variable appears in the file. The upper level EDAS data are output on the following 26 pressure surfaces. Table 3 gives the level number corresponding to each data level, which is also written to each header label. ------------------------------------------------ Table 3. Description of Vertical Levels ------------------------------------------------ Level Height ------------------------------------------------ 26 50 hPa 25 100 hPa 24 150 hPa 23 200 hPa 22 250 hPa 21 300 hPa 20 350 hPa 19 400 hPa 18 450 hPa 17 500 hPa 16 550 hPa 15 600 hPa 14 650 hPa 13 700 hPa 12 725 hPa 11 750 hPa 10 775 hPa 9 800 hPa 8 825 hPa 7 850 hPa 6 875 hPa 5 900 hPa 4 925 hPa 3 950 hPa 2 975 hPa 1 1000 hPa 0 surface ------------------------------------------------------------------- Missing Data Missing data are written as an array of nulls with a forecast hour of -1 in the header label. Usually the associated field label is "NULL". Definition File The definition file given in Appendix A summarizes the grid specifications and data fields. The format is such that the first 20 characters are the dummy ID field followed by the data. Much of the information is written into the index record of each time period. Record 1 consists of a four character string that identifies the source of the meteorological data. Record 2 is the integer identification of the meteorological data grid (Table 1). Record 3 is an integer number that identifies the vertical coordinate system. Only four coordinate types are recognized: 1-pressure sigma; 2-pressure absolute; 3-terrain sigma; 4-hybrid sigma. Records 4 & 5 identifies the pole position of the grid projection. Most projections will either be defined at +90 or -90 depending upon the hemisphere. The longitude would be the point 180 degrees from which the projection is cut. Records 6 & 7 is the reference position at which the grid spacing is defined. Record 8 is the grid spacing in km at the reference position. Record 9 is the grid orientation or the longitude of the meridian which is parallel to the up-down direction of the grid. Record 10 is the angle between the axis and the surface of the cone. For regular projections it is equal to the latitude at which the grid is tangent to the earth's surface. A polar stereographic grid would be tangent at either 90 or -90, while a Mercator projection is tangent at 0 latitude. A Lambert Conformal projection would be in between the two limits. An oblique stereographic projection would have a cone angle of 90. Records 11 & 12 are used to equate a position on the grid with a position on the earth as given in Records 13 & 14. Any position is acceptable. It need not even be on the grid. Record 15 is not currently used. Records 16 & 17 identify the number of grid points in each direction. Record 18 is the number of levels in the vertical, including the surface level. Record 19, through the number of levels, identifies the height of each level in appropriate units according the definition of the vertical coordinate, the number of variables at that level, and the four character identification string for each variable. The height coordinate is as follows for each type of vertical coordinate: 1-sigma (fraction); 2-pressure (mb); 3-terrain (fraction); 4-hybrid (mb-offset.fraction) Index record The key to reading the meteorological files is decoding the ASCII index record, the first record of each time period. The first 50 characters of the index record contain the same "header" information as do the other records in the given time period. The four-character label is "INDX". The format for this record is given below. Complete descriptions are similar to the variables in the discussion above of the Definition File. Format of the Index Record A4 Data Source I3 Forecast hour I2 Minutes associated with data time 12F7. 1) Pole Lat, 2) Pole Long, 3) Tangent Lat, 4) Tangent Long, 5) Grid Size, 6) Orientation, 7) Cone Angle, 8) X-Synch pnt, 9) Y-Synch pnt, 10) Synch pnt lat, 11) Synch pnt long, 12) Reserved 3I3 1) Numb x pnts, 2) Numb y pnts, 3) Numb levels I2 Vertical coordinate system flag I4 Length in bytes of the index record, excluding the first 50 bytes LOOP ===> number of data levels F6. height of the first level I2 number of variables at that level LOOP ===> number of variables A4 variable identification I3 rotating checksum of the packed data 1X Reserved space for future use END LEVEL AND VARIABLE LOOPS Data Grid Unpacking NCEP typically saves their model output in GRIB format. However, here the data are stored differently because the ARL format is a bit more compact and it can be directly used on a variety of computing platforms with direct access I/O. The data array is packed and stored into one byte characters. To preserve as much data precision as possible the difference between the values at grid points is saved and packed rather than the actual values. The grid is then reconstructed by adding the differences between grid values starting with the first value, which is stored in unpacked ASCII form in the header record. To illustrate the process, assume that a grid of real data, R, of dimensions i,j is given by the example below. 1,j 2,j .... i-1,j i,j 1,j-1 2,j-1 .... i-1,j-1 i,j-1 .... .... .... .... .... 1,2 2,2 .... i-1,2 i,2 1,1 2,1 .... i-1,1 i,1 The packed value, P, is then given by Pi,j = (Ri,j - Ri-1,j)* (2**(7-N)), where the scaling exponent N = ln dRmax / ln 2 . The value of dRmax is the maximum difference between any two adjacent grid points for the entire array. It is computed from the differences along each i index holding j constant. The difference at index (1,j) is computed from index (1,j-1), and at 1,1 the difference is always zero. The packed values are one byte unsigned integers, where values from 0 to 126 represent -127 to -1, 127 represents zero, and values of 128 to 254 represent 1 to 127. Each record length is then equal in bytes to the number of array elements plus 50 bytes for the header label information. The 50 byte label field preceedes each packed data field and contains the following ASCII data: Field Format Description Year I2 Greenwich date for which data valid Month I2 " Day I2 " Hour I2 " Forecast* I2 Hours forecast, zero for analysis Level I2 Level from the surface up (see Table 3) Grid I2 Grid identification (see Table 1) Variable A4 Variable label (see Table 2) Exponent I4 Scaling exponent needed for unpacking Precision E14.7 Precision of unpacked data Value 1,1 E14.7 Unpacked data value at grid point 1,1 *Forecast hour is -1 for missing data. Sample Program In the sample FORTRAN program (Appendix B), the unpacking subroutine is used to read the first record after the index record. The value at grid point 1,1 from the header and the unpacked value are printed, along with the other header values. In the program the packed data array is in the variable QPACK and the unpacked real data array is returned in variable QVAR. NX and NY are the number of grid points in the horizontal and vertical directions, respectively. NXY is just the product of NX and NY. The variable NEXP and VAR1, are the packing exponent and value of the field at the 1,1 grid point. These values are obtained from the header label information. REFERENCES Kanamitsu, M., 1989: Description of the NMC Global Data Assimilation and Forecast System, Weather and Forecasting, 4 (335-342). Petersen, R.A. and J.D. Stackpole, 1989: Overview of the NMC Production Suite, Weather and Forecasting, 4 (313-322). Sela, J.G., 1980: Spectral modeling at the National Meteorological Center, Mon. Wea. Rev., 108 (1279-1292). ------------------------------------------------------------ Appendix A. Definition File - EDAS.CFG MODEL TYPE: EDAS GRID NUMB: 28 VERT COORD: 2 POLE LAT: 90. POLE LON: 0. REF LAT: 35. REF LON: -95. REF GRID: 80.0 ORIENTATION: 0. CONE ANGLE: 25. SYNC X: 52.5 SYNC Y: 25.5 SYNC LAT: 35. SYNC LON: -95. SPECIAL: 0. NUMB X: 93 NUMB Y: 65 NUMB LEVELS: 27 LEVEL 1: 0. 28 MSLP TMPS TPP3 CPP3 SOLT SOLW T02M RH2M U10M V10M P10M PRSS WESD CSNO CICE CFZR CRAI LHTF SHTF UMOF VMOF RGHS LCLD MCLD HCLD TCLD DSWF WTMP LEVEL 2: 1000. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 3: 975. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 4: 950. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 5: 925. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 6: 900. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 7: 875. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 8: 850. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 9: 825. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 10: 800. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 11: 775. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 12: 750. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 13: 725. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 14: 700. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 15: 650. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 16: 600. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 17: 550. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 18: 500. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 19: 450. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 20: 400. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 21: 350. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 22: 300. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 23: 250. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 24: 200. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 25: 150. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 26: 100. 07 UWND VWND HGTS TEMP WWND RELH TKEN LEVEL 27: 50. 07 UWND VWND HGTS TEMP WWND RELH TKEN Appendix B. Sample program to unpack data **************************************************************************** * SAMPLE PROGRAM TO UNPACK DATA ARRAY IN SECOND RECORD **************************************************************************** * QVAR = UNPACKED REAL DATA ARRAY * QPACK = PACKED DATA ARRAY * NX = MAXIMUM GRID DIMENSION IN X-DIRECTION * NY = MAXIMUM GRID DIMENSION IN Y-DIRECTION * NXY = NX * NY * LABEL = 50 CHARACTER ASCII HEADER, CONSISTING OF THE FOLLOWING: * IY = YEAR FOR WHICH DATA ARE VALID * IM = MONTH FOR WHICH DATA ARE VALID * ID = DAY FOR WHICH DATA ARE VALID * IHR = HOUR FOR WHICH DATA ARE VALID (GREENWICH) * IFHR = HOURS AFTER INITIALIZATION * LVL = LEVEL FROM THE SURFACE UP * IG = GRID ID * VARB = 4 CHARACTER VARIABLE LABEL * NEXP = PACKING EXPONENT * PREC = PRECISION OF UNPACKED DATA * VAR1 = UNPACKED DATA VALUE AT GRID POINT (1,1) **************************************************************************** PARAMETER (NX=129, NY=129, NXY=NX*NY) REAL*4 QVAR(NX,NY) CHARACTER*1 QPACK(NXY) CHARACTER*4 VARB CHARACTER*50 LABEL CHARACTER*80 FNAME LREC=NXY+50 WRITE(*,*) 'ENTER ARCHIVE FILENAME (within single quotes)' READ(*,*) FNAME OPEN(10,FILE=FNAME,ACCESS='DIRECT',RECL=LREC,FORM='UNFORMATTED') READ(10,REC=2) LABEL,QPACK READ(LABEL,100) IY,IM,ID,IHR,IFHR,LVL,IG,VARB,NEXP,PREC,VAR1 100 FORMAT(7I2,A4,I4,2E14.7) CALL UNPACK(QVAR,QPACK,NX,NY,NXY,NEXP,VAR1) WRITE(*,200) IY,IM,ID,IHR,IFHR,LVL,IG,VARB 200 FORMAT(1X,'YEAR=',I2,' MONTH=',I2,' DAY=',I2,' HOUR=',I2, : ' FORECAST HOUR=',I2,' LEVEL=',I2,' GRID=',I2, : ' VARIABLE=',A4) WRITE(*,300) VAR1 300 FORMAT(1X,'FROM HEADER, QVAR(1,1)= ',F7.1) WRITE(*,400) QVAR(1,1) 400 FORMAT(1X,'FROM UNPACKED DATA, QVAR(1,1)= ',F7.1) STOP END ************************************************************************ SUBROUTINE UNPACK(QVAR,QPACK,NX,NY,NXY,NEXP,VAR1) ************************************************************************ CHARACTER*1 QPACK(NXY) REAL*4 QVAR(NX,NY) SCALE=2.0**(7-NEXP) QOLD=VAR1 INDX=0 DO J=1,NY DO I=1,NX INDX=INDX+1 QVAR(I,J)=(ICHAR(QPACK(INDX))-127.)/SCALE+QOLD QOLD=QVAR(I,J) END DO QOLD=QVAR(1,J) END DO RETURN END