About
Welcome to the website of the LEXIS (Large-scale EXecution for Industry & Society) project which lasted from 2019 to 2021. If you are looking for the main outcome of the LEXIS project, please continue to LEXIS Platform documentation. The prototype of the LEXIS Platform, developed during the LEXIS project, is now being finalised and operated by IT4Innovations and LRZ within the sustainability of the project.
The ever-increasing quantity of data generated by modern industrial and business processes poses an enormous challenges for organisations seeking to glean knowledge and understanding from the data. Combinations of HPC, Cloud and Big Data technologies are key to meeting the increasingly diverse needs of large and small organisations alike, however, access to powerful compute platforms for SMEs which has been difficult due to both technical and financial considerations may now be possible.
The LEXIS (Large-scale EXecution for Industry & Society) project has built an advanced engineering platform at the confluence of HPC, Cloud and Big Data, which leverages existing, geographically distributed large-scale resources in a federation of established EU supercomputing centres. It employs Big Data Analytics solutions, and augments them with Cloud services. Driven by the requirements of several pilot test cases, the LEXIS Platform relies on best-in-class data management solutions (EUDAT) and advanced, distributed orchestration solutions (TOSCA), augmenting them with new, efficient hardware and platform capabilities (e.g. in the form of data nodes and federation, usage monitoring and accounting/billing support). Thus, LEXIS has realised an innovative solution aimed at stimulating the interest of European industry and at creating an ecosystem of organisations that benefit from the LEXIS platform and its well-integrated HPC, HPDA and Data Management solutions.
The consortium has thus developed a demonstrator with a significant Open Source dimension, including validation, test and documentation. It has been validated in large-scale pilots – in the industrial and scientific sectors (Aeronautics, Earthquake and Tsunami, Weather and Climate), where significant improvements in KPIs including job execution time and solution accuracy have been achieved. LEXIS has also organised a specific Open Call for platform usage, which has stimulated adoption of the project framework and increased stakeholders’ engagement with the targeted pilots.
The collaboration has promoted and will keep promoting this solution to the HPC, Cloud and Big Data sectors, maximising impact through targeted and qualified communication. LEXIS has brought together a consortium with the skills and experience to deliver a complex multi-faceted project, spanning a range of complex technologies across seven European countries, including large industry, flagship HPC centres, industrial and scientific compute pilot users, technology providers and SMEs.
Work performed during the project
The LEXIS Platform allows orchestration of complex scientific and industrial workflows which can be easily run on distributed HPC and Cloud resources. Thanks to its API and modern web based UI, powerful HPC resources and complex Cloud deployments can be easily accessed and challenging workflows can be executed. It uses YORC (Ystia Orchestrator) to handle the workflows, an iRODS-based data management solution with asynchronous staging APIs, and integration with EUDAT services to handle extensive datasets in a secure and convenient way.
The individual LEXIS software modules have been implemented, deployed and integrated in the resulting LEXIS Platform, which has been successfully validated, tested and exposed on a public endpoint. The LEXIS Portal is the web based UI through which users can use all the features of the platform, including overview of consumed resources and its cost. The LEXIS Platform implements multi-tenancy by grouping users in different roles to organisations and projects through which the compute resources are consumed.
In the LEXIS project, a specialised platform aimed at lowering the entry barriers to the worlds of supercomputing, Cloud, and Big Data without sacrificing the execution performance was created. Based on the LEXIS pilots, we identified several needs shared by SMEs and industry within a number of different fields:
- Dynamic data-aware orchestration of complex workflows,
- Data sharing between Cloud and HPC resources and distributed data management with appropriate data backend to the orchestration solutions,
- Access to HPC/BigData/Cloud resources for SMEs and Industry, and
- Easy control of workflows and data through a user-friendly web interface, with seamless integration of remote visualisation services.
To accommodate easy access to these technologies, a LEXIS Portal was developed which facilitates the management of various LEXIS objects such as projects, member roles, and available data. It enables the users to manage the results of their computations, to apply for computational resources at HPC centres and to deploy and run applications to the LEXIS platform. To deliver the best user experience, typical user scenarios were collected together with the pilot partners giving invaluable feedback.
The central aim is to increase the usability of currently-available resources, allowing customers to easily access and leverage heterogeneous hardware at multiple sites without the need for advanced computing architecture knowledge, all while upholding the Industry level standard of security. The solutions provided by LEXIS enable improved cooperation between industry and academia. The new platform ensures that SMEs and industries are able to use the appropriate resources for their applications in a user-friendly manner.
Validation of the LEXIS Platform
An indispensable part of the LEXIS design process were three pilots with different requirements, focusing on aeronautics, earthquakes and tsunamis, and – last but not least – weather and climate.
In the case of the Aeronautics pilot, work was mainly focused on the improvement of the code for the Turbomachinery use-case and the improvement of the advanced hydrodynamics simulation capabilities used to predict fluid flow around complex geometries under turbulent motion in the Rotating Parts use-case. Additionally, the possibility to visualise the results after the computation was considered.
In the Earthquake and Tsunami pilot, the main achievement was a clear formalisation of the complex complete workflow, which runs under time-constrained conditions. Difficulties in the definition were raised due to non-linearity in the workflow but were overcome to fit the LEXIS solution. The computational efficiency of the tsunami simulation code was improved, therefore delivering results of a potential tsunami wave inundation in less than a minute enabling faster and more accurate emergency warnings.
In the Weather and Climate pilot, the models have been virtualised and prepared as ready-to-run containers and virtual machines compatible with the LEXIS platform. The procedures for handling the data and data quality assurance from different sources have been set up. Prototypes of the workflows were created and were implemented and tested on the LEXIS infrastructure.
As part of the LEXIS Platform validation efforts a LEXIS Open Call has been established to attract third-party users. Based on an extensive review process, a set of various applicants from both SME and academia domains have been selected. The participants provided a valuable feedback on the LEXIS Platform functionality and features in exchange for access to various compute resources and close collaboration on deployment of their use-cases. The LEXIS Open Call allowed us to jump-start many fruitful collaborations which can lead to further exploitation of the LEXIS Platform.
