Combining research projects implies sharing data between companies. To see the interaction between systems involved in the projects, data must be exchanged automatically from day one and in real time. To make this interaction possible, The Green Village has launched a project together with the ICT innovation department of TUD and SURFsara. The development of a data sharing system became possible thanks to the expertise of database management, authorization, storage and archiving. The platform is based on the specific needs of the required hardware of the area.
Ultimately a system was developed that brings together the various data streams, saves them safely and at the same time makes them quickly available for the many users and applications. For this purpose the open-source software Apache Kafka was used. Because the data streams can become very large, the storage and processing capacity has to be able to grow as needed. Redundancy is also important. If for example a machine fails, no information may be lost. The chosen open source system Kubernetes ensures that data collection is automatically resumed by another machine in the cluster. In addition, the accessibility and reliability of the data must be guaranteed, as must the legal frameworks within data can be stored and shared.
The system has now been tested with producers and users of the data. This test has shown that the platform functions as desired. The next step is to continue using the data platform and then take on the challenge to visualize the relationships between the projects at The Green Village in a dynamic way.
(text SURF Magazine)
