2.6.1.5.2 Qualitative uncertainty analysis


The major assumptions and model choices underpinning the Namoi subregion surface water model are listed in Table 12. The goal of this qualitative uncertainty analysis is to provide a non-technical overview of the model assumptions, their justification and effect on predictions, as judged by the modelling team. This will also assist in an open and transparent review of the modelling.

Each assumption in Table 12 is rated against three attributes (data, resources and technical) and their effect on predictions.

  1. The data rating is the degree to which the question ‘If more or different data were available, would this assumption or choice still have been made?’ would be answered positively. A low rating means that the assumption is not influenced by data availability, while a high rating indicates that this choice would be revisited if more data were available.
  2. The resources rating reflects the extent to which resources available for the modelling, such as computing resources, personnel and time, influenced this assumption or model choice. Again, a low rating indicates the same assumption would have been made with unlimited resources, while a high rating indicates the assumption is driven by resource constraints.
  3. The technical rating reflects the extent to which the assumption is influenced by technical and computational issues. A high rating is assigned to assumptions and model choices that are predominantly driven by computational or technical limitations of the model code. These include issues related to spatial and temporal resolution of the models.

The most important rating relates to the effect of the assumption or model choice on the predictions. This is a qualitative assessment by the modelling team of the extent to which a model choice will affect the model predictions, with low indicating a minimal effect and high a large effect.

A detailed discussion of each of the assumptions, including the rationale for the scoring, follows.

Table 12 Qualitative uncertainty analysis as used for the Namoi subregion surface water model


Assumption or model choice

Data

Resources

Technical

Effect on predictions

Selection of calibration catchments

Medium

Low

Low

Low

High-flow and low-flow objective function

Low

Low

High

Low

Selection of goodness-of-fit function for each hydrological response variable

Low

Low

Low

Low

Selection of acceptance threshold for uncertainty analysis

Medium

High

Medium

Medium

Interaction with the groundwater model

Medium

Medium

High

Low

Implementation of the coal resource development pathway

High

Low

Low

High

Selection of calibration catchments

The parameters that control the transformation of rainfall into streamflow are adjusted based on a comparison of observed and simulated historical streamflow. Only a limited number of the model nodes have historical streamflow. To calibrate the surface water model, a number of catchments are selected outside the Namoi subregion. The parameter combinations that achieve an acceptable agreement with observed flows are deemed acceptable for all catchments in the subregion.

The selection of calibration catchments is therefore almost solely based on data availability, which results in a medium rating for this criterion. As it is technically trivial to include more calibration catchments in the calibration procedure and as it would not appreciably change the computing time required, both the resources and technical columns have a low rating.

The regionalisation methodology is valid as long as the selected catchments for calibration are not substantially incompatible with those in the prediction domain in terms of size, climate, land use, topography, geology and geomorphology. The majority of these assumptions can be considered valid and the overall effect on the predictions is therefore deemed to be low.

High-flow and low-flow objective function

AWRA-L simulates daily streamflow. High-streamflow and low-streamflow conditions are governed by different aspects of the hydrological system and it is difficult for any streamflow model to find parameter sets that are able to adequately simulate both extremes of the hydrograph. In recognition of this issue, two objective functions are chosen: one tailored to medium and high flows and another one tailored to low flows.

Even with more calibration catchments and more time available for calibration, a high-flow and low-flow objective function would still be necessary to find parameter sets suited to simulate different aspects of the hydrograph. Data and resources are therefore scored low, while the technical criterion is scored high.

The high-streamflow objective function is a weighted sum of the Nash–Sutcliffe efficiency and the bias. The former is most sensitive to differences in simulated and observed daily and monthly streamflow, whereas the latter is most affected by the discrepancy between long-term observed and simulated streamflow. The weighting of both components represents the trade-off between simulating short-term and long-term streamflow behaviour. It also reflects the fact that some parameters are more sensitive to daily behaviour and some are more sensitive to long-term hydrology.

The low-streamflow objective function is achieved by transforming the observed and simulated streamflow through a Box-Cox transformation (see Section 2.6.1.4). By this transformation, a small number of large discrepancies in high streamflow will have less prominence in the objective function than a large number of small discrepancies in low streamflow. Like the high-streamflow objective function, the low-streamflow objective function consists of two components, the efficiency

The choice of the weights between both terms in both objective functions is based on the experience of the modelling team (Viney et al., 2009). The choice is not constrained by data, technical issues or available resources. Although different choices of the weights will result in a different set of optimised parameter values, experience in the Water Information Research and Development Alliance (WIRADA) project, in which the AWRA-L is calibrated on a continental scale, has shown the calibration to be fairly robust against the weights in the objective function (Vaze et al., 2013).

Although the selection of objective function and its weights is a crucial step in the surface water modelling process, the overall effect on the predictions is marginal through the uncertainty analysis, hence the low rating.

Selection of goodness-of-fit function for each hydrological response variable

The goodness-of-fit function for each hydrological response variable for uncertainty analysis has a very similar role to the objective function in calibration. Where the calibration focuses on identifying a single parameter set that provides an overall good fit between observed and simulated values, the uncertainty analysis aims to select an ensemble of parameter combinations that are best suited to make the chosen prediction.

Within the context of the bioregional assessment (BA), the calibration aims to provide a parameter set that performs well at a daily resolution, whereas the uncertainty analysis focuses on specific aspects of the yearly hydrograph.

The goodness-of-fit function is tailored to each hydrological response variable and averaged over a number of selected catchments that contribute to flow in the Namoi subregion modelling domain. This ensures parameter combinations are chosen that are able to simulate the specific part of the hydrograph relevant to the hydrological response variable, at a local scale.

Like the objective function selection, the choice of summary statistic is primarily guided by the predictions and to a much lesser extent by the available data, technical issues or resources. This is the reason for the low rating for these attributes.

The impact on the predictions is deemed minimal (low rating) as it is an unbiased estimate of model mismatch and because it summarises the same aspect of the hydrograph as is needed for the prediction.

Selection of acceptance threshold for uncertainty analysis

The acceptance threshold ideally is independently defined based on an analysis of the system (see companion submethodology M09 (as listed in Table 1) for propagating uncertainty through models (Peeters et al., 2016)). For the surface water hydrological response variables, such an independent threshold definition can be based on the observation uncertainty, which depends on an analysis of the rating curves for each observation gauging station as well as at the model nodes. There are limited rating curve data available, hence the medium rating. Even if this information were available, the operational constraints within the BA prevent such a detailed analysis – although it is technically feasible. The resources column therefore receives a high rating while the technical column receives a medium rating.

The choice of setting the acceptance threshold equal to the 90th percentile of the summary statistic for a particular hydrological response variable (i.e. selecting the best 10% of replicates) is a subjective decision made by the modelling team. By varying this threshold through a trial-and-error procedure in the testing phase of the uncertainty analysis methodology, the Assessment team learned that this threshold is an acceptable trade-off between guaranteeing enough prediction samples and overall good model performance. Although relaxing the threshold may lead to larger uncertainty intervals for the predictions, the median predicted values are considered robust to this change. A formal test of this hypothesis has not yet been carried out. The effect on predictions is therefore scored a medium rating.

Interaction with the groundwater model

The coupling between the results of the groundwater model and the surface water model, described in the model sequence section (Section 2.6.1.1), represents a pragmatic solution to account for surface water – groundwater interactions at a regional scale. Even if a suitable algorithm for integrated coupling of fluxes between the surface water and groundwater models were available, the differences in spatial and temporal resolution would require non-trivial upscaling and downscaling of spatio-temporal distributions of fluxes. For these reasons and also for practical reasons related to run times and computational storage issues, the modelling methodology for the Namoi subregion involves a one-directional feed of changes in the groundwater flux to streams from the groundwater model, rather than a fully coupled implementation. Thus the rating for the technical attribute is high.

The data and resources columns are rated medium because even if it were technically feasible to fully integrate the models, the implementation would be constrained by the available data and the operational constraints. In an integrated model, a simulation would likely involve multiple iterations between the groundwater and stream components and increase the computational load significantly.

The overall effect on the predictions is assumed to be small, as the change in baseflow due to coal mining is small compared to the other components of the water balance and the effect of rainfall interception by mine sites (see companion product 2.5 for the Namoi subregion (Crosbie et al., 2018)).

Implementation of the coal resource development pathway

The CRDP is implemented through the interaction with the groundwater model and by removing the fraction of runoff in the catchment that is intercepted by the mine footprint from the total catchment runoff. The key choices that are made in implementing the CRDP are (i) determining which mining developments are included, and (ii) deciding on the spatial and temporal development of their hydrological footprints.

In catchments in which the mine footprint is only a small fraction of the total area of the catchment, the precise delineation of the spatial extent of the mine footprint is not crucial to the predictions. In catchments in which the footprint is a sizeable fraction, accurate delineation of mine footprint becomes very important.

Similarly, the temporal evolution of mine footprints is crucial as it will determine how long the catchment will be affected. This is especially relevant for the post-mining rehabilitation of mine sites, when it becomes possible again for runoff generated within the mine footprint to reach the streams.

In the Namoi subregion, the accuracy with which mine footprints are represented in the model depends largely on the accuracy of the planned mine footprints published or provided by the mine proponents. This therefore is one of the crucial aspects of the surface water model as it potentially has a high impact on predictions and it is driven by data availability rather than availability of resources or technical issues. The data attribute is therefore rated high, while the resources and technical columns are rated low. The effect on predictions is rated high.

Last updated:
6 December 2018
Thumbnail of the Namoi subregion

Product Finalisation date

2018

ASSESSMENT