{"id":2211,"date":"2020-11-23T12:58:00","date_gmt":"2020-11-23T12:58:00","guid":{"rendered":"https:\/\/lexis-project.eu\/web\/?p=2211"},"modified":"2021-02-19T19:14:43","modified_gmt":"2021-02-19T19:14:43","slug":"lexis-distributed-data-infrastructure-ddi-gets-operational","status":"publish","type":"post","link":"https:\/\/lexis-project.eu\/web\/news\/lexis-distributed-data-infrastructure-ddi-gets-operational\/","title":{"rendered":"LEXIS Distributed Data Infrastructure (DDI) gets operational"},"content":{"rendered":"\n<p>In the LEXIS project, we are developing&nbsp;an advanced and user-friendly&nbsp;computing environment that converges Big Data Analytics, High\nPerformance Computing and Cloud Computing capabilities of European\nsupercomputing centres. Three main layers \u2013 the infrastructure layer, the data management layer and a workflow\norchestration layer \u2013 constitute the main building blocks of LEXIS. With the\nLEXIS platform and portal, we enable users in science, industry and society\nto&nbsp;automate&nbsp;and run their compute-and data-intensive workflows efficiently,\nand thus to accelerate their Research and Development.<\/p>\n\n\n\n<p>The LEXIS Data System allows users to consistently manage input, output and temporary data of their workflows. Its core is the Distributed Data Infrastructure (DDI), which federates the storage systems of the LEXIS infrastructure layer and can be conveniently addressed via REST APIs (i.e., interfaces based on web technology) Figure 1. The DDI is based on the &#8220;Integrated Rule-Oriented Data System&#8221; (iRODS) and B2SAFE of the &#8220;European Data Collaborative Data Infrastructure&#8221; (EUDAT CDI). Technically, this means that LEXIS data can be accessed and managed in a uniform way, independently of where the data are physically located. From a collaboration perspective, the integration of LEXIS with EUDAT is one step towards a unified, European research data management following the FAIR principles (Wilkinson et al., 2016,&nbsp;<a href=\"https:\/\/doi.org\/10.1038\/sdata.2016.18\">https:\/\/doi.org\/10.1038\/sdata.2016.18<\/a>). It gives us straightforward possibilities to federate our data system with more European data centres and projects.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><a href=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/LEXIS-Infrastructure-highlighting-the-core-components-of-data-management-layer-DDI-API.jpg\"><img loading=\"lazy\" width=\"898\" height=\"394\" src=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/LEXIS-Infrastructure-highlighting-the-core-components-of-data-management-layer-DDI-API.jpg\" alt=\"LEXIS Infrastructure highlighting the core components of data management layer, the DDI and API (source: LRZ) \" class=\"wp-image-2212\" srcset=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/LEXIS-Infrastructure-highlighting-the-core-components-of-data-management-layer-DDI-API.jpg 898w, https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/LEXIS-Infrastructure-highlighting-the-core-components-of-data-management-layer-DDI-API-300x132.jpg 300w, https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/LEXIS-Infrastructure-highlighting-the-core-components-of-data-management-layer-DDI-API-768x337.jpg 768w\" sizes=\"(max-width: 898px) 100vw, 898px\" \/><\/a><figcaption> Figure 1: LEXIS Infrastructure highlighting the core components of data management layer, the DDI and API (source: LRZ) <\/figcaption><\/figure><\/div>\n\n\n\n<p>In practice, all project members can interact with their data through\nthe LEXIS Portal. They can utilize the data within their LEXIS workflows and\niRODS automatically manages cross-site data transfer wherever necessary. The\nnovel Burst Buffer systems in LEXIS can be used to prefetch remote data, if\ndata transfer would take too long. Alternatively, immediate data availability\nand increased data security can be obtained by activating a convenient\ncross-site replication functionality implemented with iRODS\/B2SAFE.<\/p>\n\n\n\n<p>Recently, the first results from &#8220;Weather and Climate Large Scale Pilot&#8221; workflows exploiting the LEXIS Computing and Distributed Data Infrastructure were published (Parodi et al., 2020,&nbsp;<a href=\"https:\/\/doi.org\/10.1007\/978-3-030-50454-0_25\">https:\/\/doi.org\/10.1007\/978-3-030-50454-0_25<\/a>). We are proud to disseminate these at CISIS 2020, and at SC 2020 with a poster highlighting the DDI (Figure 2 and 3). <\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><a href=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Forest-fire-prediction-and-prevention-workflow-as-executed-by-the-orchestrator-1024x423-1.png\"><img loading=\"lazy\" width=\"1024\" height=\"423\" src=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Forest-fire-prediction-and-prevention-workflow-as-executed-by-the-orchestrator-1024x423-1.png\" alt=\"Forest fire prediction and prevention workflow as executed by the orchestrator (source: CIMA, ATOS, LRZ) \" class=\"wp-image-2213\" srcset=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Forest-fire-prediction-and-prevention-workflow-as-executed-by-the-orchestrator-1024x423-1.png 1024w, https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Forest-fire-prediction-and-prevention-workflow-as-executed-by-the-orchestrator-1024x423-1-300x124.png 300w, https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Forest-fire-prediction-and-prevention-workflow-as-executed-by-the-orchestrator-1024x423-1-768x317.png 768w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><figcaption> Figure 2: Forest fire prediction and prevention workflow as executed by the orchestrator (source: CIMA, ATOS, LRZ) <\/figcaption><\/figure><\/div>\n\n\n\n<div class=\"wp-block-image is-style-default\"><figure class=\"aligncenter size-large\"><a href=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Workflow-results-visualized-with-Dewetra-platform-1024x524-1.jpg\"><img loading=\"lazy\" width=\"1024\" height=\"524\" src=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Workflow-results-visualized-with-Dewetra-platform-1024x524-1.jpg\" alt=\"Workflow results visualized with Dewetra platform (Italian Department of Civil Protection, CIMA) \" class=\"wp-image-2214\" srcset=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Workflow-results-visualized-with-Dewetra-platform-1024x524-1.jpg 1024w, https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Workflow-results-visualized-with-Dewetra-platform-1024x524-1-300x154.jpg 300w, https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/Workflow-results-visualized-with-Dewetra-platform-1024x524-1-768x393.jpg 768w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><figcaption>Figure 3: Workflow results visualized with Dewetra platform (Italian Department of Civil Protection, CIMA) <\/figcaption><\/figure><\/div>\n\n\n\n<p>Currently, we are working on benchmarking the DDI system, taking into  account different network speeds between LEXIS sites. The LEXIS  Orchestration System will be aware of physical data locations and  typical transfer speeds. Thus, it can select suitable storage and  computing sites for executing a given workflow at the best performance.  Mastering the challenges in integrating IT4I, LRZ and further  infrastructure within LEXIS, we\u00a0keep our focus on providing\u00a0an optimised  system with immediate benefits to the users. Besides convenient  usability via the portal, high performance and speed gains are key  points for an optimum uptake of the DDI and the LEXIS platform as a  whole. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the LEXIS project, we are developing&nbsp;an advanced and user-friendly&nbsp;computing environment that converges Big Data Analytics, High Performance Computing and Cloud Computing capabilities of European supercomputing centres. Three main layers \u2013 the infrastructure layer, the data management layer and a workflow orchestration layer \u2013 constitute the main building blocks of LEXIS. With the LEXIS platform [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v16.6.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\r\n<title>LEXIS Distributed Data Infrastructure (DDI) gets operational - LEXIS project \/ LEXIS Platform<\/title>\r\n<meta name=\"robots\" content=\"noindex, follow\" \/>\r\n<meta property=\"og:locale\" content=\"en_GB\" \/>\r\n<meta property=\"og:type\" content=\"article\" \/>\r\n<meta property=\"og:title\" content=\"LEXIS Distributed Data Infrastructure (DDI) gets operational - LEXIS project \/ LEXIS Platform\" \/>\r\n<meta property=\"og:description\" content=\"In the LEXIS project, we are developing&nbsp;an advanced and user-friendly&nbsp;computing environment that converges Big Data Analytics, High Performance Computing and Cloud Computing capabilities of European supercomputing centres. Three main layers \u2013 the infrastructure layer, the data management layer and a workflow orchestration layer \u2013 constitute the main building blocks of LEXIS. With the LEXIS platform [&hellip;]\" \/>\r\n<meta property=\"og:url\" content=\"https:\/\/lexis-project.eu\/web\/news\/lexis-distributed-data-infrastructure-ddi-gets-operational\/\" \/>\r\n<meta property=\"og:site_name\" content=\"LEXIS project \/ LEXIS Platform\" \/>\r\n<meta property=\"article:published_time\" content=\"2020-11-23T12:58:00+00:00\" \/>\r\n<meta property=\"article:modified_time\" content=\"2021-02-19T19:14:43+00:00\" \/>\r\n<meta property=\"og:image\" content=\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/LEXIS-Infrastructure-highlighting-the-core-components-of-data-management-layer-DDI-API.jpg\" \/>\r\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\r\n<meta name=\"twitter:label1\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\r\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/lexis-project.eu\/web\/#website\",\"url\":\"https:\/\/lexis-project.eu\/web\/\",\"name\":\"LEXIS project \/ LEXIS Platform\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/lexis-project.eu\/web\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/lexis-project.eu\/web\/news\/lexis-distributed-data-infrastructure-ddi-gets-operational\/#primaryimage\",\"inLanguage\":\"en-GB\",\"url\":\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/LEXIS-Infrastructure-highlighting-the-core-components-of-data-management-layer-DDI-API.jpg\",\"contentUrl\":\"https:\/\/lexis-project.eu\/web\/wp-content\/uploads\/2020\/11\/LEXIS-Infrastructure-highlighting-the-core-components-of-data-management-layer-DDI-API.jpg\",\"width\":898,\"height\":394},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/lexis-project.eu\/web\/news\/lexis-distributed-data-infrastructure-ddi-gets-operational\/#webpage\",\"url\":\"https:\/\/lexis-project.eu\/web\/news\/lexis-distributed-data-infrastructure-ddi-gets-operational\/\",\"name\":\"LEXIS Distributed Data Infrastructure (DDI) gets operational - LEXIS project \/ LEXIS Platform\",\"isPartOf\":{\"@id\":\"https:\/\/lexis-project.eu\/web\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/lexis-project.eu\/web\/news\/lexis-distributed-data-infrastructure-ddi-gets-operational\/#primaryimage\"},\"datePublished\":\"2020-11-23T12:58:00+00:00\",\"dateModified\":\"2021-02-19T19:14:43+00:00\",\"author\":{\"@id\":\"https:\/\/lexis-project.eu\/web\/#\/schema\/person\/8ff0c5622bd460c9dc16c68e0dc490e3\"},\"breadcrumb\":{\"@id\":\"https:\/\/lexis-project.eu\/web\/news\/lexis-distributed-data-infrastructure-ddi-gets-operational\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/lexis-project.eu\/web\/news\/lexis-distributed-data-infrastructure-ddi-gets-operational\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/lexis-project.eu\/web\/news\/lexis-distributed-data-infrastructure-ddi-gets-operational\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/lexis-project.eu\/web\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"LEXIS Distributed Data Infrastructure (DDI) gets operational\"}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/lexis-project.eu\/web\/#\/schema\/person\/8ff0c5622bd460c9dc16c68e0dc490e3\",\"name\":\"atanas.pushkarov\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/lexis-project.eu\/web\/#personlogo\",\"inLanguage\":\"en-GB\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/94e8725bff84278405e6675244addd57?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/94e8725bff84278405e6675244addd57?s=96&d=mm&r=g\",\"caption\":\"atanas.pushkarov\"},\"url\":\"https:\/\/lexis-project.eu\/web\/author\/atanas-pushkarov\/\"}]}<\/script>\r\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/posts\/2211"}],"collection":[{"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/comments?post=2211"}],"version-history":[{"count":6,"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/posts\/2211\/revisions"}],"predecessor-version":[{"id":2339,"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/posts\/2211\/revisions\/2339"}],"wp:attachment":[{"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/media?parent=2211"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/categories?post=2211"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lexis-project.eu\/web\/wp-json\/wp\/v2\/tags?post=2211"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}