Add centralized config file for data pipeline

[giovanni-vaccarino]

What feature do you want to see added?

Currently the configs of the data pipeline(e.g. the chunk size in the chunking part) are written directly into the .py scripts. It would be good to have a centralized config file (a YAML could be used), so that if anyone has to run the data pipeline can change in a unique point all the possible parameters that can be tuned. Especially in combination with #25, this would provide a solid enhancement.

Upstream changes

No response

Are you interested in contributing this feature?

No response

[Zaki-Mohd]

I’d like to take this issue. Please assign it to me ;)

[giovanni-vaccarino]

@Zaki-Mohd Feel free to open a PR. No assignment is required