Dealing with VLM

[UlrickBL]

Hello,

When dealing with VLM, the content of the prompt / chat template is not pure text but contains PIL images or base64 images. So the prompt formater that deals with the reflective LLM is limited since it transforms the output in text. Is there a way to overwrite this function easily ? I guess it would be even better to adapt this function so it outputs a chat template format that is then fed to the reflective LLM (open AI format) instead of pure string.

[fgn]

I'd suggest looking into the MultiModalInstructionProposer in DSPy's documentation, particularly since it supports image + text inputs. See dspy.ai - GEPA_Advanced when-to-use-custom-instruction_proposer
This is a good starting point as it also contains info how to write a custom instruction proposer in case you need to make tweaks.