Providing computing solutions for exascale challenges
Mini-LOBCDER is a VM that allows you to deploy the LOBCDER data-service micro-infrastructure locally. The setup includes a VM which is provisioned with Kubernetes, core containers for LOBCDER and a user micro-infrastructure containers.
Moving data in the exascale era is very time consuming because moving data is bound by the underlying physical limitations. One exabyte of data takes around 3 years to transfer on the faster, state of the art interconnects. At the same time projects like the Square Kilometer Array are expected to generate an exabyte of data every day or so. This puts a lot of strain on how to handle and process the data which is globally distributed and difficult to move.
Our approach, in PROCESS, is that the data services can not be viewed as just software components but a more holistic approach is needed which takes into consideration the underlying infrastructure, the flow of data, the processing and the services. This leads to a smart infrastructure where we can, for example, move the compute to the data instead of moving the data to compute or split the processing of data several parts each done on different parts of the network.
LOBCDER introduces the concept of micro-infrastructure which uses container technologies to combine data services with infrastructure to create a smart data infrastructure. This means that development of services, now, consists of two parts; the description of the underlying container infrastructure (micro-infrastructure) and the development of the different software services that fit into the micro-infrastructure.