Object-Based Verification of a Prototype Warn-on-Forecast System

Skinner, P., D. Wheatley, K. Knopfmeier, A. Reinhart, J. Choate, T. A. Jones, G. Creager, D. Dowell, C. Alexander, T. Ladwig, L. Wicker, P. Heinselman, P. Minnis, and R. Palikonda (2018), Object-Based Verification of a Prototype Warn-on-Forecast System, Skinner Et Al., doi:10.1175/WAF-D-18-0020.1.

An object-based verification methodology for the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e) has been developed and applied to 32 cases between December 2015 and June 2017. NEWS-e forecast objects of composite reflectivity and 30-min updraft helicity swaths are matched to corresponding reflectivity and rotation track objects in Multi-Radar Multi-Sensor system data on space and time scales typical of a National Weather Service warning. Object matching allows contingency-table-based verification statistics to be used to establish baseline performance metrics for NEWS-e thunderstorm and mesocyclone forecasts. NEWS-e critical success index (CSI) scores of reflectivity (updraft helicity) forecasts decrease from approximately 0.7 (0.4) to 0.4 (0.2) over 3 h of forecast time. CSI scores decrease through the forecast period, indicating that errors do not saturate during the 3-h forecast. Lower verification scores for rotation track forecasts are primarily a result of a high-frequency bias. Comparison of different system configurations used in 2016 and 2017 shows an increase in skill for 2017 reflectivity forecasts, attributable mainly to improvements in the forecast initial conditions. A small decrease in skill in 2017 rotation track forecasts is likely a result of sample differences between 2016 and 2017. Although large case-to-case variation is present, evidence is found that NEWS-e forecast skill improves with increasing object size and intensity.

PDF of Publication: 
Download from publisher's website.
Research Program: 
Modeling Analysis and Prediction Program (MAP)