The ever-increasing quantity of data generated by modern industrial and business processes poses an enormous challenge for organisations seeking to glean knowledge and understanding from the data. Combinations of HPC, Cloud and Big Data technologies are key to meeting the increasingly diverse needs of large and small organisations alike, however, access to powerful computing platforms for SMEs which has been difficult due to both technical and financial considerations may now be possible.
The LEXIS (Large-scale EXecution for Industry & Society) project is building an advanced engineering platform at the confluence of HPC, Cloud and Big Data, which leverages large-scale geographically-distributed resources from the existing HPC infrastructure, employs Big Data analytics solutions and augments them with Cloud services.
Driven by the requirements of several pilot test cases, the LEXIS platform relies on best-in-class data management solutions (EUDAT) and advanced, distributed orchestration solutions (TOSCA), augmenting them with new, efficient hardware and platform capabilities (e.g. in the form of Data Nodes and federation, usage monitoring and accounting/billing support). Thus, LEXIS realises an innovative solution aimed at stimulating the interest of the European industry and at creating an ecosystem of organisations that benefit from the LEXIS platform and its well-integrated HPC, HPDA and Data Management solutions.
The consortium is thus developing a demonstrator with a significant Open Source dimension, including validation, test and documentation. It will be validated in large-scale pilots – in industrial and scientific sectors (Aeronautics, Earthquake and Tsunami, Weather and Climate), where significant improvements in KPIs including job execution time and solution accuracy are anticipated. LEXIS will organise a specific call that will stimulate adoption of the project framework, increasing stakeholders’ engagement with the targeted pilots.
LEXIS will promote this solution to the HPC, Cloud and Big Data sectors, maximising impact through targeted and qualified communication. LEXIS brings together a consortium with the skills and experience to deliver a complex multi-faceted project, spanning a range of complex technologies across seven European countries, including large industry, flagship HPC centres, industrial and scientific compute pilot users, technology providers and SMEs.
During the first 18 months of the project, the main focus was on the individual components of the LEXIS platform and on establishing a common model for their integration. A high-level architecture of the basic LEXIS platform components and their relationships has been devised. Thanks to the collaboration of the consortium partners, the crucial technical components such as the distributed data infrastructure, the authentication layer, and the federated orchestration layer, are in the late stages of deployment and are being integrated together.
Indispensable parts of the LEXIS design process are three pilots with different requirements. In the case of the aeronautics pilot, work was mainly focused on the improvement of the code for the Turbomachinery use-case and the improvement of the advanced hydrodynamics simulation capabilities used to predict fluid flow around complex geometries under turbulent motion in the Rotating Parts use-case. Additionally, it will be possible to visualise the results after the computation.
In the Earthquake and Tsunami pilot, the main achievement was a clear formalisation of the complex complete workflow, which run under time-constrained conditions. Difficulties in the definition were raised due to non-linearity in the workflow but were overcome to fit the LEXIS solution. The computational efficiency of the tsunami simulation code was improved, therefore delivering results of a potential tsunami wave inundation in less than a minute enabling faster and more accurate emergency warnings.
In the weather pilot, the models have been virtualised and prepared as ready-to-run containers and virtual machines compatible with the LEXIS platform. The procedures for handling the data and data quality assurance from different sources have been set up. Prototypes of the workflows were created and are being implemented and tested on the LEXIS infrastructure.
To accommodate easy access to these technologies, a LEXIS Portal is being developed which facilitates the management of various LEXIS objects such as projects, member roles, and available data. It enables the users to manage the results of their computations, to apply for computational resources at HPC centres and to deploy and run applications to the LEXIS platform. To deliver the best user experience, typical user scenarios were collected together with the pilot partners giving invaluable feedback. Based on these, the mock-ups of the portal were designed. At present, there are prototype views for managing users in the projects, for accessing and listing available datasets, for deployment and running of applications (with the possibility to run visualisation of the results), and for monitoring and billing. These are connected to the technical solutions mentioned earlier.
In the LEXIS project, a specialised platform aimed at lowering the entry barriers to the worlds of supercomputing, cloud, and big data without sacrificing the execution performance is being created. Based on the LEXIS pilots, we identified several needs shared by SMEs and industry within a number of different fields:
- dynamic data-aware orchestration of complex workflows,
- data sharing between Cloud and HPC resources and distributed data management with appropriate data backend to the orchestration solutions,
- access to HPC/BigData/Cloud resources for SMEs and Industry, and
- easy control of workflows and data through a user-friendly web interface, with seamless integration of remote visualisation services.
The central aim is to increase the usability of currently-available resources, allowing customers to easily access and leverage heterogeneous hardware at multiple sites without the need for advanced computing architecture knowledge, all while upholding the Industry level standard of security. The solutions provided by LEXIS will enable improved cooperation between industry and academia. The new platform will ensure that SMEs and industries are able to use the appropriate resources for their applications in a user-friendly manner.
To increase the quality of the LEXIS platform, an Open Call is planned during the LEXIS project, in which partners from academia and business will be asked to test their applications on the LEXIS platform and provide feedback. This should also increase the impact of the project and establish a presence in various communities of potential users.