Data Definition Specification (DDS) SOLAR2000/E10.7 SET/SEC Hourly SWDS-Format Data Files Version f, 13 May 03 Prepared by Dave Bouwer, SET/SpaceWx Introduction The Hourly SWDS files represent all the nowcast information related to s2kops/SOLAR2000: both input and output (forecasts are defined in MFD documents available for SET/SpaceWx customers.) The original intent was to define permanent SQL-compliant archival database records daily for official NOAA/SEC products, but the Hourly SWDS files proved so useful (especially in the "RunStatusCode" field to determine nowcast/forecast errors), the content was expanded and became an integral part of the operational software, and the SWDS data files were made more widely available. Users are encouraged to use fields in the SWDS files to evaluate the specific SOLAR2000 data products (i.e., nowcast and forecast data and plot files) to achieve greater accuracy in subsequent analysis. For example, the RunStatusCode is assigned a value after autonomous operational software reads and interprets run logs from other operational programs, searching for error flags (performing about a hundred validation tests on operational software logs and data values), and if it's final value is greater than 3 (see definiton below), it would be unwise to use the data values at that particular time because the data quality is highly suspect. If the RunStatusCode = 0, one can have a high level of confidence in the data quality. Additionally, numerous date and time records associated with data and file times can be used to relate a more granular time series with input and output values or to detect potential errors. Some input proxy data is older than the final SOLAR2000 nowcast times. SOLAR2000 nowcasts are actually predictive in a limited sense during most of the day, i.e. "nowcasts" are true nowcasts, not based on just yesterday's values. For example, one of the input proxies, the NOAA/SBUV MgII measurement, used in SOLAR2000 is available early in the day, but the final F10 value is not available until the end of the day. SOLAR2000 runs every hour, and uses whatever the most recent input proxy data is available at run time, extrapolating output for the next 24 hours (based on a common linear estimation technique, which results in estimation errors at the 1-2% level.) As new input proxies become avaialbe (e.g., new satellite instruments become operational), the measurements will be added in near real-time to the input data streams of the s2kops/SOLAR2000 operational runs, further improving the granularity of the tme series. The SOLAR2000 output data - E10.7 and other irradiance estimates - represent a major improvement over the traditional radio flux in estimating solar variability and it's subsequent terrestrial effects; it estimates the actual energy input to the earth's thermosphere and ionosphere. Extensive publications and documentation on the SOLAR2000 irradiace specification tool, and related products, can be found at the www.SpaceWx.com web site. Specifications (0) This document version differs from version e (and earlier versions) in the addition of several data record definitions, the renaming of averages from "Avg" to "_81" to indicate averages are 81-day averages (calculated from a running triangular filter, that uses a linear estimation technique to predict future values at the end of a time series, in order to estimate averages up to current values), and in some of the descriptive text. (1) The s2kops nowcast data file (also referred to by the "hourly s2kops SWDS file"), is created 24 times a day (current actual I/O data values change only about two-four times a day based on Mg II and F10.7 inputs; the particular time may occur at any time during the day.) Typically, the last data input (currently F10) to SOLAR2000 occurs at about 21 UT. Assume an hourly cadence, but software or system failures may prevent a data file being created, in which case a "default" set of fields might be substituted with missing values in the database. These data need to be archived for at least 90 days at NOAA/SEC. SET/SpaceWx maintains triple-redundant online and off-line archives. (2) All the default (i.e., missing) min/max values are given below in the next file description. "data time" refers to the time the SOLAR2000/s2kops associates with data, e.g., F10.7 is defined at Penticton as Local noon, and it has been converted to the associated UT for the F10.7 "data time". "File Time" refers to the computer system time associated with the files (e.g., the time one would see from a directory listing). The time stamp in the filename is in "file time" which should correspond, within one hour, to "data time" if all is well. All times have a SQL-formatted date/time string and a double-precision Julian Day. The distinction between "file time" and "data time" is particularly important for validation software, and for other maintenance and analytical uses. Generally, end-users will be primarily interested with the data times. (3) In the Java code that creates the data file, a Hashtable class is used, converting it to a String with name/value pairs as a single line (see sample below.) The fields shown below do not follow the same order as the name/value pairs in the data file (an artifact of the Hashtable.toString() method in Java.) It is important that software that retrieves the data file do so on the basis of the name = value fields, not on the particular order (thus new fields can easily be added later.) (4) All date/time strings are in SQL format, e.g., yyyy-mm-dd hh:mm:ss.mmm, in UTC. Nearly all times are retrieved from the computer system time, which must be in (Greenwich noon) UTC! (5) The default, or missing, values in all dates and times are 1970-01-01 00:00:00.000, JD = 2440587.5. (6) All float values should have "-1.0" as a missing-value flag. (7) Integer values should have "-1" as a missing value flag. (8) String values, other than date/time, should have "-1" as a missing value flag. (9) The units of E10, F10, Ly-a are: a. F10, F81 are daily and 81-day solar 10.7 cm radio flux in x10-22 Watts per meter squared per Hertz b. Lya, L81 are daily and 81-day Lyman-alpha in x10+09 photons per centimeter squared per second c. E10, E81, are in x10-22 Watts per meter squared per Hertz. E10.7 is observed, integrated 1-105 nm EUV in F10.7 units (10) The hourly "nowcast" data file will written to a single file on the operational computers in the directory/filename: /spacewx/s2kops/sftp/swds/2k-nowcast-YYYY_MM_DD_HH_MM.txt The date/time fields in the filename will correspond to file time. Table 1: Hourly s2k-nowcast/SWDS Record Field Definitions 1. NowSQL, String*23, The time this file was created 2. F10inp, Float, The current F10.7 (uncorrected) value 3. Mg2inp, Float, The current NOAA/SBUV Mg II c/w value 4. F10, Float, The F10.7 daily value used in SOLAR2000 (extrapolated) 5. F10_81, Float, The F10.7 81-day average calculated by SOLAR2000 6. Lya, Float, The Ly-a value calculated by SOLAR2000 from Mg II 7. Lya_81, Float, The Ly-a 81-day average calculated by SOLAR2000 8. E10, Float, The E10.7 index calculated by SOLAR2000 9. E10_81, Float, The E10.7 81-day average calculated by SOLAR2000 10. RunStatusCode, Int, A flag (0-9) indicating data quality; 0 means all is OK 11. Mg2source, String*6, The satellite (e.g. NOAA16) used to measure the MgII c/w ratio 12. S2Kvers, String *40, A version ID for the s2kops and SOLAR2000 software 13. UTCfileJD, Double Float, The Julian Day the SOLAR2000 output file was last modified (created) 14. InpFileJD, Double Float, The Julian Day the input data file to SOLAR2000 was last modified 15. UTCdataJD, Double Float, The Julian Day associated with the SOLAR2000 data output values 16. FinpJD, Double Float, The Julian Day associated with the current F10.7 data value 17. MinpJD, Double Float, The Julian Day associated with the current MgII c/w data value 18. UTCfileSQLtime, String*23, The SQL Date/Time the SOLAR2000 output file was last modified 19. InpFileSQLtime, String*23, The SQL Date/Time input data file to SOLAR2000 was last modified 20. UTCdataSQLtime, String*23, The SQL Date/Time associated with SOLAR2000 data output values 21. MinpSQLtime, String*23, The SQL Date/Time associated with the current MgII c/w data value 22. FinpSQLtime, String*23, The SQL Date/Time associated with the current F10.7 data value *Note: in most instances, the file modification time is the file creation time. (11) A sample s2k-nowcast-yyyy_mm_dd_hh_mm.txt file: {MinpSQLtime=2003-05-13 00:21:00.000, MinpJD=2452772.5145833334, Lya=4.46, InpFileJD=2452773.034710648, FinpSQLtime=2003-05-12 12:00:00.000, UTCfileSQLtime=2003-05-13 13:09:59.000, F10=99.5, Mg2inp=0.27063, F10inp=94.0, RunStatusCode=0, S2Kvers=s2kopsV02_S2K_V1.24/256C78rumba, UTCdataSQLtime=2003-05-14 13:09:59.000, InpFileSQLtime=2003-05-13 12:49:59.000, UTCdataJD=2452774.048599537, E10_81=124.5, FinpJD=2452772.0, E10=106.9, Lya_81=4.68, Created=2003-05-13 13:26:01.000, Mg2source=noaa16, F10_81=124.5, UTCfileJD=2452773.048599537} (12) The data file will written to a single file in /spacewx/s2kops/sftp/s2k-nowcast-YYYY_MM_DD_HH_MM.txt. In other applications, files are overwritten with a single filename swdsNowcast.txt or in a similar convention. a. UTCsqlCreated, String*23 b. UTCsqlData (seconds and milliseconds are always 0), String*23 - Year: min = 1947, max = 2147, plotmin = 1970, plotmax = 2047 - Month: min = 1, max = 12, plotmin = 1, plotmax = 12 - Day: min = 1, max = 31, plotmin = 1, plotmax = 31 - Hour: min = 0, max = 60, plotmin = 0, plotmax = 59 - min = 0, max = 60, plotmin = 0, plotmax = 59 c. DataJulianDate, Double Float - min = 2405600.0, max = 2505600.0, plotmin = 2440587.0, plotmax = 2468712.0 d. F10, Float - min = 0.0, max = 400.0, plotmin = 25.0, plotmax = 350.0 e. F81, Float - min = 0.0, max = 400.0, plotmin = 25.0, plotmax = 350.0 f. Mg2 - min = 0.20, max = 0.35, plotmin = 0.260, plotmax = 0.295 h. Mg81 - min = 0.20, max = 0.35, plotmin = 0.260, plotmax = 0.295 j. Ly, Float - min = 1.0x10^11, max = 9.0x10^11, plotmin = 3.0x10^11, plotmax = 8.0x10^11 k. Lya81, Float - min = 1x10^11, max = 9x10^11, plotmin = 3x10^11, plotmax = 8x10^11 l. E10, Float - min = 0.0, max = 400.0, plotmin = 25.0, plotmax = 350.0 m. E81, Float - min = 0.0, max = 400.0, plotmin = 25.0, plotmax = 350.0 n. RunStatusCode, Int - min = 0, max = 9, plotmin = 0, plotmax = 9 (13) Notes: a. An open issue is how to index records in a SQL database. Data record times should monotonically increase, and only 1- minute resolution is required. There must be only one record per time. One suggestion is to use a large integer based on the YYYYMMDDHHMM. b. If the RunStatusCode is greater than 3, the data should not be considered reliable. When it is 2-3, it might be suspicious, but probably is adequate (e.g., an input proxy to SOLAR2000 is delayed, or one of the s2kops programs are currently running). If the RunStatusCode is 0-1, no errors have been detected in the validation software (which in itself only tests for the most obvious errors, e.g., min/max values). Full verification (i.e., data values fully meet expectations) is not performed automatically. Currently, the daily data values have had an SEC operator verify the SOLAR2000 E10.7 is "nominal", but only post-analysis with corrected input values (e.g. the corrected Penticton F10.7 daily values) and auxiliary measurements can fully verify SOLAR2000 data. c. Even if current data input is unavailable, s2kops will "gracefully fail", i.e., it will produce the last observed input data and predicted SOLAR2000 values. The s2kops software runs in crons, and automatically restarts and recovers from most error conditions. d. An open issue is that in the event of fatal system failures (e.g., prolonged downtime due to system crashes, network outages, etc.), how data records with missing data flags will be back-filled. SET/SpaceWx maintains triple-redundant archives. e. At some time in the future, it should be anticipated that other I/O data (e.g., NOA17 MgII, background X-rays) may be added. f. On rare occasion, date/time values may be zero or otherwise improbable due to operating system malfunctions: this is unavoidable. In such instances the data must be ignored and the problem promptly corrected. g. F10.7 is that used at the end of the day, and may be corrected later for final archival the the NOAA World Data Center. The corrected data typically occurs prior to the final archival of a months data at the WDC. These corrections may be due to correction of the measurement to remove solar radio bursts effects. (14) To meet specific SEC requirements, the /spacewx/s2kops/sftp/swds/s2kSWDShourlyTable nowcast test file, overwritten every hour, is now created from the hashtable/SQL version specified above, and is available to SEC operations. It is generally not recommended to use this specific file, as data definitions will change over time. An example (ignore linefeeds within each of the two lines below) is: UTCfileJD InpFileJD UTCdataJD FinpJD MinpJD F10inp Mg2inp F10 F10_81 Lya Lya_81 E10 E10_81 RunStatusCode UTCfileSQLtime UTCdataSQLtime InpFileSQLtime MinpSQLtime FinpSQLtime Mg2source S2Kvers 2003-05-13 15:23:59.000 2452773.1319444445 2452773.1180439815 2452774.1319444445 2452772.0 2452772.5145833334 94.0 0.27063 99.5 124.5 4.46 4.68 106.9 124.5 0 2003-05-13 15:10:00.000 2003-05-14 15:10:00.000 2003-05-13 14:49:59.000 2003-05-13 00:21:00.000 2003-05-12 12:00:00.000 noaa16 s2kopsV02_S2K_V1.24/256C78rumba