Skip to content

Allow passing in additional plugins for evals #4147

@allenli873

Description

@allenli873

Right now EvaluationGenerator just has the two baked in plugins _RequestIntercepterPlugin and EnsureRetryOptionsPlugin. I would like to also have the ReflectAndRetryPlugin here.

Describe the solution you'd like
Allow for arbitrary plugins to be added for evals.

Describe alternatives you've considered
Add ReflectAndRetry on the eval runner. Not as robust and some might not want this.

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluation

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions