Given the hydrological scenarios defined in the previous stage, the predicted response for the is elicited from the external experts at the receptor impact modelling workshop. The expert education and preparation process leading up to and including the receptor impact modelling workshop is detailed in Section 6.1. The elicitation procedure used at the workshop to elicit expert judgement conditional on the hydrological scenarios, which were developed in the previous stage, is detailed in Section 6.2. Incorporating additional or alternative experts would likely lead to alternative . The quality of a constructed receptor impact model can only be assessed by evaluating the expert assessment against empirical data. The statistical nature of the receptor impact models (Section 5.1) and scenario construction (Section 5.2) explicitly enables this empirical evaluation of expert contributions by allowing incorporation of independent empirical data through the function within a generalised linear model. This section discusses the expert preparation, elicitation structure and probability assessment that ensure the potential coherent assimilation of the contributed expert assessments with empirical data, while accommodating for expert in the relationship between and the receptor impact variable.
The protocol for eliciting is structured around each defined within each or . The fundamental question ‘How might selected receptor impact variables change under various scenarios of change for the hydrological response variables?’ is addressed by the experts. A very important step in the process therefore is the formalisation and specification of the relevant impact and response variables and the relationships among these variables (Chapter 4).
The protocol methodology asks experts to provide estimates of the chosen at a range of values of the relevant . It then uses statistical modelling to build a relationship between the hydrological response variables and the receptor impact variables, where the former are treated as covariates and the latter as the response of a linear model with potentially non-linear basis functions (sensu Hosack et al., 2016). This approach provides a repeatable protocol to develop these models via experts.
The elicitation session is designed to tackle a challenging problem, and it depends on the collaboration and cooperation of the experts. For this session, experts are asked to contribute their knowledge and expertise in a small group setting, although sometimes individually when only a single expert is available. The group format permits experts to confer and seek a consensus opinion when responding to presented scenarios. A group approach not only allows for the entire group to contribute but also permits the opportunity for feedback and group learning while responding to hypothetical scenarios.
The elicitations take place at receptor impact modelling workshops, which are separately held for each bioregion and subregion. The attending experts represent a wide range of expertise and experience. At the workshop, parallel sessions are run with experts grouped by expertise. The groups may range in size from a single individual up to a small number of individuals. The expert preparation for these group elicitation sessions is as follows.
The following steps were involved prior to the receptor impact modelling workshop:
- Experts are selected for the elicitation process. The choice of experts and their invitation into the process ensures that appropriate expertise is identified and included in the process. Relevant experts are contacted by staff collaborating among the Office of Water, the Bureau of Meteorology and the ecology discipline teams for each bioregion or subregion. Ideally, experts will have previously attended the relevant qualitative modelling workshop for each bioregion or subregion. This allows experts to take advantage of their previous exposure to and discussion about the bioregion or subregion, the relevant landscape classes and the receptors and hydrological response variables identified in the qualitative models. This continuity helps experts smoothly transition into a quantitative assessment for one or more receptor impact variables, while understanding the given to a particular receptor impact variable by its relationship with respect to the landscape class and . As noted above, the qualitative models help experts identify both potential direct and indirect hydrological impacts. Also, experts that have attended the qualitative modelling workshop will have had previous exposure to the scope and objectives of the Bioregional Assessment Programme, which leads to further gains in efficiency.
- Prior to the first meeting the experts receive material outlining the Programme, the nature and purpose of the receptor impact models and a description of the landscape classes, hydrological response variables and potential receptor impact variables. It also describes how the experts’ responses are to be used to develop the receptor impact model.
- The quality of the group elicitation depends on a collegial, open-minded and focused atmosphere. The experts are provided information on the group elicitation approach. In particular, the following guidelines are provided:
- Respect alternative opinions. Everyone has different points of view and experience that will be called upon over the course of the session.
- Practice patience. The problem is new and challenging. It will take some thought and work to assess and talk about, which can take time. Occasionally, points will need to be considered again (and even again).
- Ask questions of the session facilitators. Assessing expert opinion is a communication exercise, and please ask if you are unsure about what is being asked of you. We much prefer to address any confusion as it arises.
- Ask questions of your fellow experts. Do not assume that ‘you know what they mean’, even if you have worked together closely in the past. Everyone has different ways for expressing their views, and clarifying questions can help resolve where points of view are both different and similar.
- Recognise the minority opinion. Please speak up! The group elicitation approach depends on everyone participating. If you are uncertain, that is ok, this is accommodated by the elicitation approach that will be used in this session.
At the receptor impact modelling workshop, the following steps are completed before beginning the elicitation sessions:
- The experts are convened and given a brief introduction about BAs, landscape class definitions and hydrological response variable definitions. They are then encouraged to ask questions about the process.
- Before undergoing an elicitation session, experts are first educated about subjective probability, trained on heuristics and biases commonly encountered in expert elicitation exercises (O’Hagan et al., 2006).
- The elicitation procedure is presented (Section 6.2) to the experts. Practice examples are worked through until experts are comfortable with the method before beginning the elicitation.
- The experts are reminded of the group elicitation approach and its accompanying challenges and guidelines for expected group behaviour, as documented above. The experts are then split into their parallel sessions by expertise.
- Within each parallel session, the facilitator uses their experience of the process to highlight challenges experts may have and strategies to overcome them. The choice of receptor impact variables is discussed to identify any fundamental conceptual modelling modifications from the preceding qualitative modelling workshop.
Throughout each parallel elicitation session, a BA contact is made available to respond to all queries from the experts about questions of hydrology, ecology or the BA process.
The target of the elicitation is the unknown value of the , . The receptor impact variable is defined such that it has a direct interpretation relative to potential observables and expert knowledge (Chapter 4). For example, in a Bernoulli response model considered at the ith hydrological scenario, a (hypothetical) observation , corresponds to either a presence or absence, and is interpreted as the probability of presence. In a Poisson response model where corresponds to a count, is interpreted as an intensity that may relate to the annual average abundance over a defined spatial scope. The experts provide subjective describing the receptor impact variable estimates conditional on a hydrological scenario summarised within the design point (see Section 5.2.2). The elicited subjective probability distributions are assumed independent conditional on the hydrological scenarios.
Conditional on the design point , the goal is to elicit a normal distribution for with the mean and variance parameters summarised into the parameter vector . An elicitation of fractiles (equivalently, percentiles or quantiles) for the target , given the monotonic link function, directly translates into fractiles for the conditional normal distribution of the linear predictor, . The experts are asked for each design point to perform judgements of equal odds in three steps:
- What value do you believe gives a chance that the true is lower? (This obtains a prediction of the median, )
- Assume that the true value is really below . Given this information, what value do you believe gives a chance of being above or below the true value of the receptor impact variable? (This is the first quartile, )
- Assume that the true value is really above . Given this information, what value do you believe gives a chance of being above or below the true value of the receptor impact variable? (This is the third quartile, ).
The parameters are then chosen to minimise the information lost by approximating the elicited fractiles by a normal distribution. The elicited fractiles are assembled into the vector, where . The extreme fractiles and (possibly infinite) are determined by the support of . Let the probability intervals determined by these elicited fractiles be denoted by for . Let denote the histogram constructed by these elicited fractiles.
Fitted probability intervals for the approximating normal distribution are obtained by:
The Kullback-Leibler divergence of from the elicited , where subscripts denote dependence on and , is approximated by:
where is absolutely continuous with . The parameters are chosen so as to minimise the approximate Kullback-Leibler divergence of from :
For each scenario, the values of the design point are portrayed and the group discusses the potential ecological response. The goal is to develop a probability distribution that is an acceptable representation of the experts’ beliefs. Of course, with unlimited time resources a suite of distribution families could be presented for consideration by experts. Resources in are not unlimited, however, and for consistency and efficiency the parametric distribution considered by the experts is defined by the models specified and described in Section 5.1. The following steps are then performed by the experts:
- Initial fractile assessments are elicited using the quartile method. These are plotted graphically as vertical dashed lines. Note that more fractiles than free parameters are elicited in an approach that is referred to as ‘overfitting’ (O’Hagan et al., 2006), which will permit feedback between the model representation and the group’s final probabilistic statement as detailed in the following steps.
- The corresponding fitted fractiles and density curve from the fitted probability density function are plotted as three vertical blue dashed lines and blue curve, respectively. The overfitting approach uses the parametric model to average across multiple probability statements. The parametric model is unlikely to exactly match the elicited fractiles from Step 1, and so this process encourages the group to evaluate the parametric model with respect to their beliefs. If the values for the matching fractiles are not acceptable, then the group returns to Step 1 and adjusts the elicited fractiles. These two steps are repeated as often as necessary until the group accepts the parametric model quantiles as acceptable.
- The extreme deciles from are plotted as dashed blue lines. The group considers these new predictions and returns to Step 1 if the predictions are unacceptable.
- The group is allowed to consider other fractiles or cumulative probabilities as predictions from the parametric model. The group returns to Step 1 if these predictions are unacceptable.
- After completing the above feedback steps, the group is allowed to accept the elicited subjective probability distribution as a reasonable assessment of the expert opinion.
This process thus elicits from the group a subjective probability distribution of ecological response given the covariate values that make up the defined scenario. Note that the elicitation focuses on the distribution, not the raw fractile assessments (e.g. Step 1 above). The raw fractiles are used by the experts as ‘parameters’ to iteratively build a probability distribution that is an acceptable representation of their beliefs. The probability predictions made by the elicited probability distribution provide the final products that are assessed by experts for either acceptance or rejection and further iteration until the experts accept the probability distribution as a reasonable model of their beliefs. This process is repeated for each design point.
METHODOLOGY FINALISATION DATE
- 1 Background and context
- 2 Identification of potentially impacted landscape classes (stage 1)
- 3 Qualitative mathematical modelling (stage 2)
- 4 Identification of hydrological response variables and receptor impact variables (stage 3)
- 5 Development of scenarios for receptor impact model expert elicitation (stage 4)
- 6 Receptor impact modelling workshop (stage 5)
- 7 Receptor impact model estimation (stage 6)
- 8 Receptor impact model prediction (stage 7)
- 9 Content for product 2.7 (receptor impact modelling)
- Contributors to the Technical Programme
- About this submethodology