Information Theory for Model Diagnostics: Structural Error is Indicated by Trade-Off Between Functional and Predictive Performance

摘要

Because of the possibility of getting the right answers for the wrong reasons, the predictive performance of a complex systems model is not by itself a reliable indicator of hypothesis quality for the purposes of scientific learning about processes. The predictive performance of a structurally adequate model should be an emergent property of its functional performance. In this context, any Pareto trade-off between measures of predictive performance versus functional performance indicates process-level error in the model; this trade-off, if it exists, indicates that the model's predictions are right for the wrong functional reasons. This paper demonstrates a novel concept based on information theory that is capable of attributing observed errors to specific processes. To demonstrate that the concept and method hold true for models and observations of real systems, we employ a minimal single-parameter-variation sensitivity analysis using a sophisticated ecohydrology model, MLCan, for a well-monitored field site (Bondville IL Ameriflux Soybean). We identify both functional and predictive error in MLCan, and also evidence of the hypothesized tradeoffs between the two. This trade-off indicates structural error within MLCan. For example, the sensible heat flux process can be calibrated to achieve good predictive performance at the cost of poor functional performance. In contrast, we find little structural error for processes driven by solar radiation, which appear "right for the right reasons." This method could be applied broadly to pinpoint process error and structural error in a wide range of system models, beyond the ecohydrological scope demonstrated here.