2.6.1.5.2 Qualitative uncertainty analysis


The major assumptions and model choices underpinning the Gloucester subregion surface water model are listed in Table 8. The goal of the table is to provide a non-technical audience with a systematic overview of the model assumptions, their justification and effect on predictions, as judged by the modelling team. This table will also assist in an open and transparent review of the modelling.

Each assumption is scored on four attributes as ‘low’, ‘medium’ or ‘high’. The data column is the degree to which the question ‘If more or different data were available, would this assumption/choice still have been made?’ would be answered positively. A ‘low’ score means that the assumption is not influenced by data availability while a ‘high’ score would indicate that this choice would be revisited if more data were available. Closely related is the resources attribute. This column captures the extent to which resources available for the modelling, such as computing resources, personnel and time, influenced this assumption or model choice. A ‘low’ score indicates the same assumption would have been made with unlimited resources, while a ‘high’ score indicates the assumption is driven by resource constraints. The third attribute deals with the technical and computational issues. ‘High’ is assigned to assumptions and model choices that are predominantly driven by computational or technical limitations of the model code. These include issues related to spatial and temporal resolution of the models. The final, and most important column, is the effect of the assumption or model choice on the predictions. This is a qualitative assessment of the modelling team of the extent to which a model choice will affect the model predictions, with ‘low’ indicating a minimal effect and ‘high’ a large effect.

A detailed discussion of each of the assumptions, including the rationale for the scoring, follows Table 8.

Table 8 Qualitative uncertainty analysis as used for the Gloucester subregion surface water model


Assumption / model choice

Data

Resources

Technical

Effect on predictions

Selection of calibration catchments

medium

low

low

low

High-flow and low-flow objective function

low

low

high

low

Selection of goodness-of-fit function for each hydrological response variable

low

low

low

low

Selection of acceptance threshold for uncertainty analysis

medium

high

medium

medium

Interaction with the groundwater model

medium

medium

high

low

Implementation of the coal resource development pathway

high

medium

low

high

No streamflow routing

low

low

low

low

Selection of calibration catchments

The parameters that control the transformation of rainfall into streamflow are adjusted based on a comparison of observed and simulated historical streamflow. Only a limited number of the receptor nodes have historical streamflow. To calibrate the surface water model, a number of catchments are selected outside the Gloucester subregion. The parameter combinations that achieve an acceptable agreement with observed flows are deemed acceptable for all receptor catchments in the subregion.

The selection of calibration catchments is therefore almost solely based on data availability, which results in a ‘medium’ score for this criterion. As it is technically trivial to include more calibration catchments in the calibration procedure and as it would not appreciably change the computing time required, both the resources and technical columns have a ‘low’ score.

The regionalisation methodology is valid as long as the selected catchments for calibration are not substantially incompatible with those in the prediction domain in terms of size, climate, land use, topography, geology and geomorphology. The majority of these assumptions can be considered valid (see Section 2.6.1.6) and the overall effect on the predictions is therefore deemed to be small.

High-flow and low-flow objective function

The AWRA-L landscape model simulates daily streamflow. High-streamflow and low-streamflow conditions are governed by different aspects of the hydrological system and it is difficult for any streamflow model to find parameter sets that are able to adequately simulate both extremes of the hydrograph. In recognition of this issue, two objective functions are chosen, one tailored to high flows and another one tailored to low flows.

Even with more calibration catchments and more time available for calibration, a high-flow and low-flow objective would still be necessary to find parameter sets suited to simulate different aspects of the hydrograph. Data and resources are therefore scored ‘low’, while the technical criterion is scored ‘high’.

The high-streamflow objective function is a weighted sum of the Nash–Sutcliffe efficiency (NSE) and the bias. The former is most sensitive to differences in simulated and observed daily and monthly streamflow, while the latter is most affected by the discrepancy between long-term observed and simulated streamflow. The weighting of both components represents the trade-off between simulating short-term and long-term streamflow behaviour. It also reflects the fact that some parameters are more sensitive to daily behaviour and some are more sensitive to long-term hydrology.

The low-streamflow objective is achieved by transforming the observed and simulated streamflow through a Box-Cox transformation (see Section 2.6.1 .4). By this transformation, a small number of large discrepancies in high streamflow will have less prominence in the objective function than a large number of small discrepancies in low streamflow. Like the high-streamflow objective function, the low-streamflow objective function consists of two components, the NSE transformed by a Box-Cox power of 0.1 and bias, which again represent the trade-off between daily and mean annual accuracy.

The choice of the weights between both terms in both objective functions is based on the experience of the modelling team (Viney et al., 2009). The choice is not constrained by data, technical issues or available resources. While different choices of the weights will result in a different set of optimised parameter values, experience in the Water Information Research And Development Alliance (WIRADA) Project in which the AWRA-L is calibrated on a continental scale, has shown the calibration to be fairly robust against the weights in the objective function (Vaze et al., 2013).

While the selection of objective function and its weights is a crucial step in the surface water modelling process, the overall effect on the predictions is marginal through the uncertainty analysis, hence the ‘low’ scoring.

Selection of goodness-of-fit function for each hydrological response variable

The goodness-of-fit function for each hydrological response variable for uncertainty analysis has a very similar role to the objective function in calibration. Where the calibration focusses on identifying a single parameter set that provides an overall good fit between observed and simulated values, the uncertainty analysis aims to select an ensemble of parameter combinations that are best suited to make the chosen prediction.

Within the context of the BA, the calibration aims at providing a parameter set that performs well at a daily resolution, while the uncertainty analysis focusses on specific aspects of the yearly hydrograph.

The goodness-of-fit function is tailored to each hydrological response variable and averaged over the four calibration catchments that contribute to flow in the Gloucester subregion. This ensures parameter combinations are chosen that are able to simulate the specific part of the hydrograph relevant to the hydrological response variable, at a local scale.

Like the objective function selection, the choice of summary statistic is primarily guided by the predictions and to a much lesser extent by the available data, technical issues or resources. This is the reason for the ‘low’ score for these attributes.

Selection of acceptance threshold for uncertainty analysis

The acceptance threshold ideally is independently defined based on an analysis of the system (see companion submethodology M09 (as listed in Table 1) for propagating uncertainty through models (Peeters et al., 2016)). For the surface water hydrological response variables such an independent threshold definition can be based on the observation uncertainty, which depends on an analysis of the rating curves for each observation gauging station as well as at the receptor locations. There are limited rating curve data available, hence the ‘medium’ score. Even if this information were to be available, the operational constraints within the bioregional assessment (BA) prevent such a detailed analysis – although it is technically feasible. The resources column therefore receives a ‘high’ score while the technical column receives a ‘medium’ score.

The choice of setting the acceptance threshold equal to the 90th percentile of the summary statistic for a particular hydrological response variable (i.e. selecting the best 10% of replicates) is a subjective decision made by the modelling team. By varying this threshold through a trial and error procedure in the testing phase of the uncertainty analysis methodology, the modelling team learned that this threshold is an acceptable trade-off between guaranteeing enough prediction samples and overall good model performance. While relaxing the threshold will lead to larger uncertainty intervals for the predictions, the median predicted values are considered robust to this change. A formal test of this hypothesis has not yet been carried out. The effect on predictions is therefore scored ‘medium’.

Interaction with the groundwater model

The coupling between the results of the groundwater models and the surface water model, described in the model sequence section (Section 2.6.1.1), represents a pragmatic solution to account for surface water – groundwater interactions at a regional scale. Like the majority of rainfall-runoff models, the current version of AWRA-L does not allow an integrated exchange of groundwater related fluxes during runtime. Even if this capability were available, the differences in spatial and temporal resolution would require non-trivial up- and downscaling of spatio-temporal distributions of fluxes.

The choice of the coupling methodology is therefore mostly a technical choice, hence the ‘high’ score for this attribute. The data and resources columns are scored ‘medium’ as even when it is technically possible to couple both models in an integrated fashion, the implementation would be constrained by the available data and the operational constraints. This warrants the ‘medium’ score for both resources and data.

The integration of a change in baseflow from the groundwater model into AWRA-L does mean that the overall water balance is no longer closed in AWRA-L. This method of coupling both models is therefore only valid if the exchange flux is small compared to the other components of the water balance. The exchange flux (see companion product 2.5 for the Gloucester subregion (Herron et al., 2018)) does show that for the Gloucester subregion the change in baseflow under baseline and under CRDP is much smaller than the other components of the surface water balance.

Another risk in this methodology is that consistency between both models is not guaranteed. In the Gloucester subregion, the major river reaches represented in the groundwater model are all considered to be gaining. The implementation of these in the MODFLOW model as drainage boundary conditions ensures these reaches will always be simulated to be gaining. By design in the current version of AWRA-L, all simulated reaches are gaining. This means the construction of both models in this subregion guarantees consistency. The groundwater modelling results presented in companion product 2.6.2 for the Gloucester subregion (Peeters et al., 2018) show that it is very unlikely that the river system will change from gaining to losing as a result of coal resource development.

The overall effect on the predictions is assumed to be small, as the change in baseflow due to coal development is small compared to the other components of the water balance and the effect of rainfall interception by mine sites (see companion product 2.5 for the Gloucester subregion (Herron et al., 2018)).

Implementation of the coal resource development pathway

The CRDP is implemented through the interaction with the groundwater models and by removing the fraction of runoff of the catchment that is intercepted by the mine footprint from the total catchment runoff.

In catchments in which the mine footprint is only a small fraction of the total area of the catchment, the precise delineation of the spatial extent of the mine footprint is not crucial to the predictions. In catchments in which the footprint is a sizeable fraction, the effect of precise delineation of mine footprint spatial extent does become very important.

Similarly, the temporal evolution of the mine footprints is crucial as it will determine how long the catchment will be affected. This is especially relevant for the post-mining rehabilitation of mine sites, when it becomes possible again for runoff generated within the mine footprint to reach the streams.

In the Gloucester subregion, the accuracy with which mine footprints are represented, depends fully on the resolution of the planned mine footprints provided by the mine proponents. This therefore is one of the crucial aspects of the surface water model as it potentially has a high impact on predictions and it is driven by data availability rather than availability of resources or technical issues. The data attribute is therefore scored ‘high’, while the resources and technical columns score ‘low’. The effect on predictions is scored ‘high’.

No streamflow routing

Streamflow routing is not taken into account in the Gloucester subregion as the system is unregulated and sufficiently small that lags in streamflow due to routing will be within a daily time-step. The effect of not incorporating routing is therefore minimal on the prediction. Seeing the small potential for impact, resourcing the development of a river routing model for this region was not warranted. All attributes are scored ‘low’ as it is technically feasible and within the operational constraints of the BA to carry out streamflow routing. Doing so would only minimally affect the predictions.

Last updated:
31 October 2018
Thumbnail of the Gloucester subregion

Product Finalisation date

2018