2.6.1.5.2 Qualitative uncertainty analysis


The major assumptions and model choices underpinning the Galilee subregion surface water model are listed in Table 8. Each assumption is scored on four attributes as ‘low’, ‘medium’ or ‘high’.

The data column is the degree to which the question ‘if more or different data were available, would this assumption/choice still have been made?’ would be answered positively. A ‘low’ score means that the assumption is not influenced by data availability, while a ‘high’ score indicates that this choice would be revisited if more data were available. Closely related is the resources attribute. This column captures the extent to which resources available for the modelling, such as computing resources, personnel and time, influenced this assumption or model choice. Again, a ‘low’ score indicates the same assumption would have been made with unlimited resources, while a ‘high’ score indicates the assumption is driven by resource constraints. The third attribute deals with the technical and computational issues. ‘High’ is assigned to assumptions and model choices that are dominantly driven by computational or technical limitations of the model code. These include issues related to spatial and temporal resolution of the models. The final and most important column is the effect of the assumption or model choice on the predictions. This is a qualitative assessment of the modelling team of the extent to which a model choice will affect the model predictions, with ‘low’ indicating a minimal effect and ‘high’ a large effect.

A detailed discussion of each of the assumptions, including the rationale for the scoring, follows Table 8. The goal of the table is to provide a non-technical audience with a systematic overview of the model assumptions, their justification and effect on predictions, as judged by the modelling team. This table also seeks to assist in an open and transparent review of the modelling.

Table 8 Qualitative uncertainty analysis as used for the Galilee subregion surface water model


Section*

Assumption/model choice

Data

Resources

Technical

Effect on predictions

2.6.1.4

Selection of calibration catchments

Medium

Low

Low

Low

2.6.1.4

High-flow and low-flow objective function

Low

Low

High

Low

2.6.1.5

Selection of goodness-of-fit function for each hydrological response variable

Low

Low

Low

Low

2.6.1.5

Selection of acceptance threshold for uncertainty analysis

Medium

High

Low

Medium

2.6.1.3

Interaction with the groundwater model

Medium

Medium

High

Medium

2.6.1.3

Implementation of the coal resource development pathway

High

Low

Low

High

2.6.1.1

No streamflow routing

Medium

Low

Low

Low

* Section of this product that contains more details on the assumption/model choice

2.6.1.5.2.1 Selection of calibration catchments

The parameters that control the transformation of rainfall into streamflow are adjusted based on a comparison of observed and simulated historical streamflow. Only a limited number of the model nodes have historical streamflow. The parameter combinations that achieve an acceptable agreement with observed flows are deemed acceptable for all model nodes in the subregion.

The selection of calibration catchments is therefore almost solely based on data availability, which results in a medium score for this criterion. As it is technically trivial to include more calibration catchments in the calibration procedure and as it would not appreciably change the computing time required, both the resources and technical columns are scored low.

The regionalisation methodology is valid as long as the selected catchments for calibration are not substantially incompatible with those in the prediction domain in terms of size, climate, land use, topography, geology and geomorphology. The majority of these assumptions can be considered valid for the Galilee subregion and the effect on predictions is therefore deemed to be small.

While the regionalisation assumption is valid, the availability of additional calibration catchments may further constrain the predictions. However, the overall effect of the choice of calibration catchments on the predictions is still considered to be low.

2.6.1.5.2.2 High-flow and low-flow objective function

The AWRA-L simulates daily streamflow. High-streamflow and low-streamflow conditions are governed by different aspects of the hydrological system and it is difficult for any streamflow model to find parameter sets that are able to adequately simulate both extremes of the hydrograph. In recognition of this issue, two objective functions are chosen, one tailored to medium and high flows and another one tailored to low flows.

Even with more calibration catchments and more time available for calibration, a high-flow and low-flow objective would still be necessary to find parameter sets suited to simulate different aspects of the hydrograph. Data and resources are therefore scored low, while the technical criterion is scored high.

The high-streamflow objective function is a weighted sum of the Nash–Sutcliffe efficiency (E) and the bias. The former is most sensitive to differences in simulated and observed daily and monthly streamflow, while the latter is most affected by the discrepancy between long-term observed and simulated streamflow. The weighting of both components represents the trade-off between simulating short-term and long-term streamflow behaviour. It also reflects the fact that some parameters are more sensitive to daily behaviour and some are more sensitive to long-term hydrology.

The low-streamflow objective is achieved by transforming the observed and simulated streamflow through a Box-Cox transformation (see Section 2.6.1.4). By this transformation, a small number of large discrepancies in high streamflow will have less prominence in the objective function than a large number of small discrepancies in low streamflow. Like the high-streamflow objective function, the low-streamflow objective function consists of two components, the E transformed by a Box-Cox power of 0.1 and bias, which again represent the trade-off between short-term and long-term accuracy.

The choice of the weights between both terms in both objective functions is based on the experience of the modelling team (Viney et al., 2009). The choice is not constrained by data, technical issues or available resources. While different choices of the weights will result in a different set of optimised parameter values, experience in the Water Information Research and Development Alliance (WIRADA) project in which the AWRA-L is calibrated on a continental scale has shown the calibration to be fairly robust against the weights in the objective function (Vaze et al., 2013).

While the selection of objective function and its weights is a crucial step in the surface water modelling process, the overall effect on the predictions is marginal through the uncertainty analysis, hence the low score.

2.6.1.5.2.3 Selection of goodness-of-fit function for each hydrological response variable

The goodness-of-fit function for each hydrological response variable for uncertainty analysis has a very similar role to the objective function in calibration. Where the calibration focuses on identifying a single parameter set that provides an overall good fit between observed and simulated values, the uncertainty analysis aims to select an ensemble of parameter combinations that are best suited to make the chosen prediction.

Within the context of the bioregional assessment (BA), the calibration aims at providing a parameter set that performs well at a daily resolution, while the uncertainty analysis focuses on specific aspects of the yearly hydrograph.

The goodness-of-fit statistic is tailored to each hydrological response variable and averaged over the calibration catchments that contribute to flow to the modelling domain. This ensures parameter combinations are chosen that are able to simulate the specific part of the hydrograph relevant to the hydrological response variable, at a local scale. There are other ways to summarise the difference between observed and simulated values.

Like the objective function selection, the choice of summary statistic is primarily guided by the predictions and to a much lesser extent by the available data, technical issues or resources. This is the reason for the low scores for these attributes.

The impact on the predictions is deemed minimal (low score) as it is an unbiased estimate of model mismatch and because it summarises the same aspect of the hydrograph as is needed for the prediction.

2.6.1.5.2.4 Selection of acceptance threshold for uncertainty analysis

The acceptance threshold ideally is independently defined based on an analysis of the system (see companion submethodology M09 (as listed in Table 1) for propagating uncertainty through models (Peeters et al., 2016)). For the surface water hydrological response variables such an independent threshold definition can be based on the observation uncertainty, which depends on an analysis of the rating curves for each observation gauging station as well as at the model nodes. The resources required to carry out such an analysis are not justifiable within the BAs.

The choice of setting the acceptance threshold equal to the 90th percentile of the summary statistic for a particular hydrological response variable (i.e. selecting the best 10% of replicates) is a subjective decision made by the modelling team. By varying this threshold through a trial-and-error procedure in the testing phase of the uncertainty analysis methodology, the modelling team learned that this threshold is an acceptable trade-off between guaranteeing enough prediction samples and overall good model performance. While relaxing the threshold will lead to larger uncertainty intervals for the predictions, the median predicted values are considered robust to this change. A formal test of this hypothesis has not yet been carried out. The effect on predictions is therefore scored medium.

2.6.1.5.2.5 Interaction with the groundwater model

The coupling between the results of the groundwater model and the surface water model, described in the model sequence section (Section 2.6.1.1), represents a pragmatic solution to account for surface water – groundwater interactions at a regional scale. Like the majority of rainfall-runoff models, the current version of AWRA-L does not allow an integrated exchange of groundwater-related fluxes during runtime. Even if this capability were available, the differences in spatial and temporal resolution would require non-trivial upscaling and downscaling of spatio-temporal distributions of fluxes. The choice of the coupling methodology is therefore mostly a technical choice and is scored high in the table.

Most of the streams in the model domain are ephemeral and considered to be disconnected from groundwater. Only the main channel of the Belyando River is considered to be connected with the alluvial groundwater system. The river system is mostly considered to be losing water to groundwater. In the weeks and months after major flood events, the groundwater sustains river flow. There are, however, very few joint observations of river flow and groundwater head in the alluvial groundwater systems to test these hypotheses. The data scarcity therefore warrants a medium score for the data column.

Even with more observations to infer the river – groundwater connection status, considerable resources would be required to generalise or establish a more detailed connection status, as most of the observations have only a very limited spatial support. This explains the medium score for resources.

The main assumption in the groundwater model is that the river is always connected and always able to provide water to groundwater. This is an overestimate of the available water and thus the estimated change in surface water – groundwater flux is also an overestimate.

In the tributaries of the stream network that are not included in the groundwater model (see Table 4 in Section 2.6.1.3), if the assumption of the disconnected status is not valid, the surface water – groundwater flux obviously is under estimated. A comparison of magnitude of the estimated fluxes compared to the magnitude of the direct impact of mines due to interception of runoff shows that the potential effect of underestimation of the surface water – groundwater flux is minor. The overall effect, therefore, is scored medium to reflect that while the perceived impact is minor, the large uncertainty in connection status warrants further research.

2.6.1.5.2.6 Implementation of the coal resource development pathway

The coal resource development pathway (CRDP) is implemented through the interaction with groundwater models and by removing the fraction of runoff of the catchment that is intercepted by the mine footprint from the total catchment runoff. The key choices that are made in implementing the CRDP are (i) determining which mining developments are included, and (ii) deciding on the spatial and temporal development of their hydrological footprints.

In catchments in which the mine footprint is only a small fraction of the total area of the catchment, the precise delineation of the spatial extent of the mine footprint is not crucial to the predictions. In catchments in which the footprint is a sizeable fraction, the effect of precise delineation of mine footprint spatial extent becomes very important.

Similarly, the temporal evolution of the mine footprints is crucial as it will determine how long the catchment will be affected. This is especially relevant for the post-mining rehabilitation of mine sites, when it becomes possible again for runoff generated within the mine footprint to reach the streams.

In the Galilee subregion, the accuracy with which mine footprints are represented depends fully on the resolution of the planned mine footprints provided by the mine proponents. This, therefore, is one of the crucial aspects of the surface water model as it potentially has a high impact on predictions and it is driven by data availability rather than availability of resources or technical issues. The data attribute is therefore scored high, while the resources and technical columns score low. The effect on predictions is scored high.

2.6.1.5.2.7 No streamflow routing

Streamflow routing is not taken into account in the Galilee subregion as the system is unregulated. This also means that lags in streamflow as water moves down the river are not taken into account.

Inferring routing parameters from river channel characteristics is difficult due to the limited observation dataset so the data column is scored medium. With this information it would be possible, within the operational constraints and technical ability of the modelling team to simulate streamflow routing (hence low scoring in both columns).

Routing of flow, however, would only change the simulated daily flow. The effect on hydrological response variables, which are annual summaries, is insignificant. The effect on predictions is therefore scored low.

Last updated:
6 December 2018