Metrics #210
Description
What
Hook up DataFusion metrics.
Why
Users probably want insights into how their UDFs are performing.
How
I'm not entirely sure if DataFusion has a good story of per-UDF metrics. It seems that metrics only exists for physical plans, but UDFs are created beforehand (or at least they exist both in the logical and physical realm). So this might need some upstream work.
Some example metrics:
- HTTP requests:
- request count
- number of uploaded bytes
- number of downloaded bytes
- number of epochs consumed by UDF
- number of max bytes (this is probably already exposed by the DF memory pool)