Scorecard template

[tennlee]

It would be ideal for PyEarthTools to support generation of standardised metrics scorecards for both training and evaluation. It would be very helpful to have a basket of metrics generated per-epoch or per-n-training-steps so that the effect of training on verification function performance could be visualised and understood. Such a scorecard approach might also help point the way towards multi-objective loss functions in future.

There are a number of questions to be dealt with here, including:

• Setting appropriate comparison benchmarks, like persistence, climatology, and a comparison model (if available)
• What metrics to include by default for various model types
• Whether to generate just data for each epoch, or whether to generate an artefact such as images, PDF, html etc for posterity

It would be useful to identify some of the most common headline metrics typically used, and the most common stratifications used (e.g. performance in particular geographic regions, etc) and write that up as a scorecard spec, which we can then implement and hook into the training cycle.

If anyone has any opinions, please feel free to add them to this issue.

[nicholasloveday]

It would be useful to identify some of the most common headline metrics typically used, and the most common stratifications used (e.g. performance in particular geographic regions, etc) and write that up as a scorecard spec, which we can then implement and hook into the training cycle.

I have shapefiles that are of different Australian regions which were designed by consultation with a group of meteorologists around Australia for geographical areas for verification. They may be useful for Australian-focused evaluation. Let me know if you want me to add these to PET (and where). It would be nice if people could supply their own shapefiles for their particular geographical regions.

Other thoughts:

If you want default sets of metrics, then I suggest different metrics based on forecast goal (and coming up with a list of them). E.g.,

• Expected (mean-valued) forecast
• Quantile (including median) forecast
• Deterministic forecast of a realistic physical scenario.
• Binary forecast
• Probability of binary event forecast
• CDF forecast
• Ensemble forecast

Did I miss any? Then we can design a set of suitable default metrics for each forecast type.

What are the speed requirements?

Do we want confidence intervals (e.g., probability that the new epoch performed better than the last)?