Stochastic catastrophe models price business based on thousands of generated events, but how do we know those modeled events and losses are realistic? A model’s “internal consistency” – broad agreement between observation and mathematical simulation - is one of the essential checks. A good stochastic model will show internal consistency at all stages of the modeling process. In this article our modelers explain this important check of a model’s ultimate reliability, using examples from PartnerRe’s CatFocus® tropical cyclone model.

What is meant by internal consistency?

Combined with strong underwriting experience, portfolio knowledge and a sound enterprise risk management framework, an advanced, stochastic catastrophe model is a highly reliable tool for natural hazard risk evaluation and assumption - now more so than ever given the increased scientific understanding of natural hazard events and advances in event data recording and modeling methodologies.

However, an important and essential requisite of a catastrophe model is that the thousands of stochastically modeled events show broad agreement with observed statistics – if this is the case, the model is said to display “internal consistency”. For example, for a large enough area with many observations, the model should reproduce a statistical distribution of wind speeds consistent with those observations. Internal consistency does not mean a perfect match between observed and modeled quantities at every location; we are modeling a phenomena which occurs relatively infrequently at any given location. Internal consistency is only expected when we have many observations to compare with our model.

Whether we “expect” it or not, internal consistency will of course only be achieved if the model is using sufficient, quality data and incorporating detailed understanding of all the model components, and if modeling biases are systematically identified and corrected. If all this is happening, the model will also serve the important purpose of providing information on the less frequently observed aspects of tropical cyclones and their associated losses.

“Even with the high quality observations available today, these alone cannot give the full picture of tropical cyclone intensity and frequency. It is important to incorporate the complexity of the physical processes at play and to understand the simplifications that might lead to model biases, which can lead to modeled events and losses for a region being inconsistent with the observed data.” Dr Erik Rüttener, Head of Research at PartnerRe.

Internal consistency checks must be made throughout the modeling process to prevent inaccuracies being passed on within the model, making the final output unreliable. Beyond checking internal consistency, modelers will also use independent data - data independent of the process used to build the model in the first place - such as cedant loss data, to validate a model’s output. This gives further reassurance of the ultimate quality of the model.

We now present an example of an advanced modeling technique from the CatFocus® tropical cyclone model that helps to ensure internal consistency. We will then discuss correction of model bias. Finally, we show how differences in modeled return periods will often persist in defined areas, but that these relate to insufficient observation data and corroborate the use of good stochastic models to “fill the data gap”.

Achieving internal consistency: comprehensively modeling the full wind field

Tropical cyclone “best track” data give the position of a tropical cyclone eye and intensity at 6-hourly intervals – this gives a smoothed path, but does not describe the complete wind field, information which is needed to fully understand the storm’s damage potential.

Two parameters, the radius of maximum sustained wind (Rmax) and the radius of gale strength wind (Rgale), are necessary to describe the full wind field. These parameters depend on the intensity, maximum sustained wind speed (Wmax) and latitude derived from best track data, e.g. McAdie et al 2009¹ and Demuth et al 2006². A key component of the CatFocus® wind field model is the way it comprehensively models the relationship between Rmax and Wmax (figure 1). This relationship is needed since many of the early best track data contain only Wmax, not Rmax. Modeling the mean Rmax/Wmax relationship (figure 1a and b, red curves) shows that there is a tendency for Rmax to be longer for low Wmax values and shorter for high Wmax values. However, the mean relationship does not explain the substantial variation in this relationship from one cyclone to another (gray shading indicates the density of observation points). CatFocus® uses information from the mean relationship and information about the residual probability density at each value of Wmax to produce Rmax for its stochastic storms.



Figure 1: Rmax as a function of Wmax (a) is needed to estimate the wind field of observed tropical cyclones in the Atlantic ocean basin prior to routine observation of Rmax and Rgale. CatFocus® models not just the mean relationship, but also the residuals (c). This full relationship is used as a basis to generate internally consistent Rmax values for the CatFocus® stochastic track wind fields (b and d).

Once we have modeled the many dependencies between tropical cyclone variables, including Rmax as a function of Wmax, a wind field model is used to approximate the radial shape of the wind field outside the Rmax at any given point in time. The wind field model incorporates features such as the rate at which wind speed reduces with distance from the cyclone eye, the effects of eyewall cycles, and how rotating winds are modified by the forward translation speed of a cyclone and by the underlying surface of sea or land. As a result, the CatFocus® tropical cyclone model shows good agreement for observed and generated wind fields.

Even so, modeled return periods are often markedly different from observation – does this negate the models?

Even with positive internal consistency checks and validation, tropical cyclone hazard models, whether stochastic or dynamical (modeling the physics of tropical cyclones), will often show inconsistency in the frequency of occurrence (return periods) of extreme wind speeds at a given location (or within a given region) compared to historical wind observations. This problem becomes greater for rarer extreme wind speeds. So should we stop believing the model numbers? The answer is no. There are two main reasons for the discrepency: model bias (predominantly correctable, see below) and most notably, lack of observation data. Once the former is corrected, a good internally consistent model is supplying us with the missing information - this information is the discrepancy.

Correcting model bias

An important cause of inconsistency is bias in the modeling method (such as incorrect assumptions) despite every effort to minimize such errors. These biases should as far as possible be corrected. For example, where we found regional biases in our tropical cyclone model we first recalculated the relevant parameters of the model using the latest best track data or corrected our track simulation parameters. In other cases we used statistical calibration techniques which minimize the frequency differences at various cyclone intensities based on extreme value analysis models.

Which leaves only a lack-of-data issue

Sampling issues play a major role in determining the frequency/intensity relationship. Essentially, observation reflects only short or inhomogeneous wind speed records and does not cover the full spectrum of possible events; physics-based models within a stochastic model, together with the statistical theory that models the occurrence of extreme values (in this case extreme wind speeds), can help us explore this range of possible events and identify the important processes involved, thereby filling the observation gaps.

Return period example

Figure 2 shows historical wind speed values and modeled stochastic wind speed values from the CatFocus® wind field model for selected 50 km grid boxes centered at the positions marked by the central sub-figure. The stochastic generated wind speeds are based on advanced hazard modeling methodologies and model bias has as far as possible been corrected. The solid lines show the Generalized Pareto Distribution³ fitted to maximum wind gust speed data from the historic (black) and stochastic (red) event sets.

For most locations there is reasonable agreement between the curves at wind gust speeds of between 80 and 120 knots. There is also good agreement between the distributions below 80 knots, not shown. Above 120 knots there are only between 1 and 7 observations at the given locations over the last 110 years; estimating frequencies from such small samples would not be reliable. The fact that CatFocus® shows relatively good agreement between the historic and stochastic event sets at these high wind speeds is a good test of the model’s internal consistency. However, at more extreme wind speeds there is more variation between the historic and stochastic event sets. In (a) the stochastic model (red curve) indicates that there is substantially more risk of wind speeds above 140 knots than indicated by the historic event set (black curve). This is most likely to be due to sampling issues associated with taking a relatively small area and data set to build the historical climatology, highlighting the value of using a stochastic event set based on track simulations to help inform us of the risk of extreme wind speeds. The opposite situation is shown for Grand Cayman Island (g) where the stochastic set indicates that the historic event set overestimates the wind speed risk.



Figure 2: A comparison of historical (black) and stochastic (red) modeled wind gust speeds for grid boxes in the Atlantic Ocean basin according to the central map. CatFocus® shows good agreement between the historic and stochastic event sets at high wind speeds, a good test of the model’s internal consistency. However, there is more variation at more extreme wind speeds as a result of sampling issues - highlighting the added value of using a stochastic event set based on track simulations to help inform us of the risk of extreme wind speeds.

To summarize, stochastic track simulations avoid sampling errors at small spatial scales and can also model loss correlations in many different territories in a physically consistent way.⁴ The advantages of sound, stochastic models far outweigh the drawbacks of possible biases (which can in many cases be corrected, as discussed).

PartnerRe at the forefront of catastrophe modeling

The CatFocus® tropical cyclone model is an advanced catastrophe model used by PartnerRe alongside commercial models and in combination with strong underwriting skills to provide a reliable estimation of risk. By developing the CatFocus® suite of proprietary catastrophe models, and by working closely with the scientific community, PartnerRe remains at the forefront of this field with an understanding of the benefits and limitations of varying approaches. We can thus offer our clients an alternative, informed view of risk and remain a trusted discussion partner in natural hazard risk evaluation.

Related PartnerRe Content

For a more detailed overview of tropical cylone modeling, please refer to our 2010 publication, “The CatFocus® Tropical Cyclone Model: Advanced Innovation and Validation”. For a detailed overview of earthquake modeling at PartnerRe, refer to “The CatFocus® Earthquake Model: High-Quality Risk Analysis” at www.partnerre.com/research/reports.

¹ C. J. McAdie et al, “Tropical Cyclones of the North Atlantic Ocean, 1851-2006.” Historical Climatology Series 6-2, Prepared by the National Climatic Data Center, Asheville, NC in cooperation with the National Hurricane Center, Tech. Rep., 2009.
² J. L. Demuth, M. DeMaria, and J. A. Knaff, “Improvement of Advanced Microwave Sounding Unit Tropical Cyclone Intensity and Size Estimation Algorithms,” Journal of Applied Meteorology and Climatology, vol. 45, no. 11, pp. 1573–1581, 2006.
³ A probability distribution function based on extreme value theory which is suitable for modeling the frequency/intensity of extreme wind speeds.
⁴ T. M. Hall and S. Jewson, “Comparison of Local and Basinwide Methods for Risk Assessment of Tropical Cyclone Landfall,” Journal of Applied Meteorology and Climatology, vol. 47, pp. 361–367, 2008.