The Datalake serves as the central data backbone for the TEXTaiLES project, designed to unify all incoming data from sensors, drones, robots, and user uploads. It provides a simple API endpoint for all project tools to submit artifacts and their metadata. The architecture ensures robust and flexible data handling: large files are securely stored in MinIO object storage, while all metadata is organized in a Postgres database for detailed querying. A central Kafka message bus reliably notifies other tools of new data, allowing components like analysis or simulation tools to react instantly. This decoupled approach makes the entire system highly scalable, allowing new tools and data producers to be easily integrated at any time. 

Github Repo: https://github.com/textailes/artifact-pipeline

Manual/Tool description: https://github.com/TEXTaiLES/HESTIA/blob/main/README.md