Home Tags Posts tagged with "data analysis"
Tag:

data analysis

Moderator at the ITDACH Roundtable meeting on Artificial Intelligence and IoT on April 25th, 2024

by

 I’m thrilled to announce that I’ll be moderating the upcoming roundtable ITDACH Roundtable meeting on Artificial Intelligence and IoT on April 25th, 2024. We’ll be delving into some fascinating topics, including Generative AI in the Enterprise, enhancing creativity and content generation, Natural Language generation (NLG) for business communication, and AI-driven product and service Innovation. We’ll also be exploring the importance of personalization and customer experience, and how to make the most of new technology in corporate organizations.

 

Speaker at the Global TechOps Leadership Team of Incyte on Generative AI – 20th February 2024 in Morges, Switzerland

by


I will speak at the next Leadership Meeting at Incyte, 14th February 2024 | Morges, Switzerland.

Impact and Applications of Generative AI in the Pharmaceutical Industry

I am pleased to present an overview of the presentation I will make to the Leadership Team of Incyte, a Pharmaceutical Company based in Morges, Switzerland.

I will focus on the transformative role of Generative AI across various sectors in the pharmaceutical industry, including its impact on workforce and key applications in drug development, clinical and commercial supply, manufacturing, quality assurance, and regulatory submissions.

Generative AI and Its Impact on Workforce and Employees: This section will explore how generative AI is reshaping the workforce landscape. We will delve into the potential for enhanced productivity, the creation of new roles, and the necessity for reskilling and upskilling employees to adapt to AI-driven environments.

Application to Drug Development: We will examine how generative AI accelerates drug discovery processes, from identifying new drug candidates to predicting drug-target interactions. The focus will be on the reduction of time and costs, and the increase in the efficiency of R&D processes.

Application to Clinical and Commercial Supply – Planning & Forecasting: This segment will cover how generative AI can revolutionize supply chain management. We will discuss AI’s role in demand forecasting, inventory optimization, and enhancing the agility of supply chain responses to market changes.

Application to Manufacturing: Here, we will explore the integration of generative AI in manufacturing processes. The emphasis will be on predictive maintenance, process optimization, and how AI can aid in scaling production while maintaining high-quality standards.

Application to Quality Assurance: This part will address the utilization of generative AI in quality assurance, focusing on automated quality checks, real-time monitoring, and predictive analytics to anticipate and prevent quality issues.

Application to Regulatory Submission (CMC Submission): Finally, we will discuss the application of generative AI in regulatory submissions, especially in Chemistry, Manufacturing, and Controls (CMC). The focus will be on how AI can streamline data analysis, enhance documentation accuracy, and ensure compliance with evolving regulatory standards. This application promises to significantly reduce the time and resources required for regulatory submissions, while maintaining the highest levels of accuracy and compliance.

Link to the website of the Incyte: https://incyte.com/


Unlocking the Power of Data Standardization: A Guide to CDISC and FAIRplus Cookbook for Streamlined Research. Transforming data management to foster adoption of FAIR principles

by

Looking for a resource that provides guidance on FAIR data management in Life Sciences? Look no further than the FAIR Cookbook! This resource is a collection of recipes that offer a deep dive into the technical aspects of FAIR data management and the infrastructure needed. It also provides applied examples of FAIRification with clinical trial, epidemiological, and molecular data, making it a valuable resource for data stewards, data managers, and data curators. One of the recipe of this great resource, if the application of FAIR principle in a context of CDISC-SDTM, a standard model for organizing, annotating, and formatting data from clinical trials.

The FAIR Cookbook is a live resource that provides guidance on how to implement the FAIR Principles in the Life Sciences. It is intended for a wide range of audiences, including researchers, data stewards, software developers, policymakers, and trainers.  The recipes are a combination of guidance, technical, hands-on, background, and review types, and are classified according to the intended audience. It is based on a grant funded by UE and IMI. It is an international effort that included the Countries listed on the right side. 

The FAIR Cookbook is a community-driven resource that is being populated and improved iteratively in an open manner. It is funded by the IMI FAIRplus project, which comprises a coalition of Europe’s leading experts in data interoperability, standards, pre-clinical to clinical translation, and long-term sustainable data repositories. The Cookbook also links to relevant community, mature, and complementary resources in the Life Sciences, such as the RDMkit, FAIRsharing, biotools, and TeSS, as well as the Pistoia Alliance’s FAIR Toolkit for Life Science Industry and more generic resources, such as ‘The Turing Way’ handbook for reproducible data science.

The FAIR principles provide guidelines for making data more easily findable, accessible, interoperable, and reusable. These principles have been widely adopted by funding agencies, industry leaders, and organizations in various fields. 

The FAIRplus Cookbook uses these principles as an organizing principle and offers atomic recipes for implementing them.

Ensuring that data is FAIR is important for facilitating data sharing and reuse, which can lead to more efficient and effective scientific research. However, there are also ethical considerations surrounding data sharing and reuse, such as ensuring data privacy and security and obtaining informed consent from study participants.  These issues are explored in the FAIRplus Cookbook.

The first time I have been in touch with this precious service was during the mapping of clinical trial data to CDISC (Clinical Data Interchange Standards Consortium). 
CDISC-SDTM is a standard model and framework for organizing, annotating, and formatting data from clinical trials. Regulatory agencies such as the US FDA require clinical trial results to be submitted in this format. The SDTM Implementation Guide (SDTMIG) provides expanded guidance on implementing SDTM for specific use cases or “domains”. CDISC standards provide procedures and guidelines for encoding project-specific data that might not fit into the existing domains. 
Mapping a non-conforming data dictionary to CDISC STDM standard and CDISC Vocabulary presents some challenges. Given the complexity of CDISC standards, any project team intending to convert their datasets to SDTM at any point in the project life cycle should aim to align with the standard as early on in the process as possible. Data dictionaries and data collection instruments should, where possible, be aligned to the relevant CDISC standards to facilitate data conversions later on. The FAIRcookbook site was very useful to initiate the project for the tabulation model. 
CDISC provides a comprehensive framework for the standardization of data collection, organization, and analysis in clinical trials, with the aim of improving data quality, consistency, and interoperability.
The main parts in CDISC (Clinical Data Interchange Standards Consortium) are the following:
– CDASH (Clinical Data Acquisition Standards Harmonization): CDASH provides guidelines for the collection of data during clinical trials, with a focus on data quality and consistency. It aims to standardize the collection of data across multiple trials, making it easier to compare and combine data from different sources.
– SDTM (Study Data Tabulation Model): SDTM is a standard format for organizing and presenting data from clinical trials. It provides a framework for the tabulation of clinical trial data in a consistent and standardized way, making it easier to analyze and compare data from different sources.
– ADaM (Analysis Data Model): ADaM provides a standard format for the analysis of clinical trial data. It provides a framework for the creation of analysis datasets that can be used for statistical analysis, reporting, and submission to regulatory agencies.
– Define-XML: Define-XML is an XML file that provides metadata for the datasets used in a clinical trial. It describes the variables, datasets, and relationships between them, providing a clear and comprehensive overview of the data used in the trial.
– Controlled Terminology: Controlled Terminology provides a standardized set of codes and definitions for data elements used in clinical trials. It aims to ensure consistency and accuracy in the use of terminology across different trials and data sources.

Another part of the CDISC standard I worked on is the SEND, which provides a way to exchange non-clinical data between organizations, such as pharmaceutical companies and regulatory agencies. The SEND Implementation Guide defines the structure and content of non-clinical data that should be submitted to regulatory agencies in order to support the evaluation of a drug’s safety and efficacy. The SEND Implementation Guide specifies the format and content of datasets for non-clinical studies, including general toxicology, reproductive toxicology, carcinogenicity, and other studies. The guide provides detailed instructions for the organization of data, variable naming conventions, controlled terminology, and other elements necessary for standardized data exchange. The SEND Implementation Guide is intended to be used in conjunction with other CDISC standards, including the Study Data Tabulation Model (SDTM), which provides a standard format for clinical trial data. 

Together, these standards help ensure that data is submitted in a consistent and standardized format, which facilitates data exchange, analysis, and evaluation.

 

Total Data Quality Management: A Comprehensive Approach to Data Quality

by

Total Data Quality Management (TDQM) is a comprehensive approach to data quality that covers every aspect of data management, from data governance and data strategy to data modelling and data analysis.

TDQM focuses on ensuring that data is accurate, complete, consistent, timely, relevant, accessible, usable, transparent, and secure, and that it meets the needs of the business.

Total Data Quality Management is a comprehensive approach to managing data quality that ensures that data is accurate, complete, and consistent. It involves a range of activities, including data profiling and analysis, data cleansing, data governance, and data quality monitoring. TDQM helps organizations to improve their data quality, reduce data-related risks, and make better-informed business decisions. By implementing TDQM, organizations can ensure that their data is a strategic asset that supports their business objectives and enables them to compete in today’s data-driven business environment.

 

TDQM involves several key components, including data governance, data management, data strategy, data modeling, data analysis, data integration, data security, data privacy, data accuracy, data completeness, data consistency, data timeliness, data relevancy, data accessibility, data usability, data transparency, and data lineage. These components are interrelated and work together to ensure that data quality is managed effectively.

Data Governance

Data governance is the foundation of TDQM. It is the process of defining and implementing policies, procedures, and standards for data management. Data governance involves identifying the stakeholders, roles, responsibilities, and processes for managing data. It also involves defining the rules for data quality and ensuring that they are adhered to.

Data Management

Data management involves the processes and technologies for managing data throughout its lifecycle. This includes data integration, data modeling, data analysis, data quality assurance, data quality control, and data quality improvement. Data management ensures that data is accurate, complete, consistent, timely, relevant, accessible, usable, transparent, and secure.

Data Strategy

Data strategy is the process of defining the goals and objectives for data management. It involves identifying the business needs for data, the data requirements, and the data sources. Data strategy also involves defining the metrics for data quality and measuring the performance of data management.

Data Modeling

Data modeling is the process of defining the structure and relationships of data. It involves creating a data model that represents the data in a way that is understandable to the business. Data modeling ensures that data is consistent and that it meets the business requirements.

Data Analysis

Data analysis is the process of analyzing data to gain insights into the business. It involves using statistical methods and data visualization to identify patterns and trends in the data. Data analysis helps to identify data quality issues and to improve the quality of data.

Data Integration

Data integration is the process of combining data from different sources into a single view of the data. It involves identifying the data sources, mapping the data, and transforming the data to ensure that it is consistent. Data integration helps to ensure that data is accurate and complete.

Data Security and Privacy

Data security and privacy are critical components of TDQM. Data security involves protecting data from unauthorized access, use, disclosure, or destruction. Data privacy involves protecting personal information from unauthorized access or use. Data security and privacy are essential for maintaining the trust of customers and stakeholders.

 

Data Profiling and Analysis
Data profiling and analysis is an essential component of TDQM that involves the examination of data to understand its structure, content, and relationships. Data profiling is a process of gathering and analyzing metadata to understand the characteristics and quality of data. It provides a better understanding of the data to identify quality issues and helps to create a baseline for data quality improvement efforts. Data analysis, on the other hand, involves the use of statistical methods to detect patterns, trends, and anomalies in the data. The results of data analysis help to identify root causes of data quality problems, develop appropriate solutions, and monitor the effectiveness of the implemented solutions.

Data Cleansing
Data cleansing, also known as data scrubbing, is the process of detecting and correcting or removing corrupt or inaccurate data from a database. It is a critical step in TDQM that ensures that data is accurate, consistent, and complete. Data cleansing involves identifying and correcting errors such as misspellings, duplicates, and inconsistencies in the data. The process of data cleansing may involve manual or automated methods, depending on the nature and complexity of the data.

Data Governance
Data governance is the process of managing the availability, usability, integrity, and security of the data used in an organization. It involves the development of policies, procedures, and standards for data management, as well as the establishment of roles, responsibilities, and processes for data quality management. Data governance ensures that data is managed consistently across the organization, and that data quality is monitored and improved continuously.

Data Quality Monitoring
Data quality monitoring is the process of continuously monitoring and measuring the quality of data to ensure that it meets the required standards. It involves the use of metrics, indicators, and key performance indicators (KPIs) to measure the quality of data and identify areas for improvement. Data quality monitoring helps to ensure that data quality is maintained over time, and that data is fit for the intended purpose.

Data Management Plans (DMP)

by

A Data management platform (DMP) is a complex piece of software used to collect, store, classify, analyze, and distribute large quantities of data. It is a cornerstone technology for larger organizations when it comes to advertising data management, with rapidly increasing adoption.  Data collection is a core capability of every data management platform. Being able to pull data from various disparate sources into one place may unlock enormous value.  Typically, DMPs can ingest first-party data, second-party data from contracted partners, as well as third-party data from external providers.What differentiates various DMPs is the range of available data sources and integrations out of the box, data collection implementation, and speed of data transfer. The best DMPs have a large number of reliable (ideally lossless) and fast data integrations with other technology and data vendors. In addition, they offer an easy implementation with customization options.

(altro…)

Research on Climate Change with statistics methods

by

Predicting climate change effects

In the past I supported agrobiodiversity scientists demonstrating the causal effects of climate change on the living of the plants. 

Scientists monitor and track more than 15.000 different plant species worldwide.

More than 1.300 Institutes worldwide collect “observations” of plants in a huge portal, the Global Biodiversity Information Facility GBIF https://www.gbif.org/

Today GBIF stores more than 1,3 billion of records catalogued using the data-standard Darwin Core, including observational data (GPS coordinates, altitude, etc.). 

 

GBIF

Challenges in statistics

How can you find evidence that climate changes is moving plants and/or change their dispersion on the territory. For example, some plants disappeared from where they were observed during ‘60s or appeared in new zones or at higher altitudes. Some other plants spread over mountains because the climate was more favorable due to the increase of temperature. Example: the plant Melampyrum nemorosum which lives in temperate climates has been observed now in Croatia mountains, which has very cold winters.

I collected the observation data needed, described data with descriptive statistics and performed regression analysis to find correlation among plants behavior and changes of the climate.

 

The Dataset

Datasets I worked on:

– Plants: GBIF portal, 250 million of observations made from 1920 to today

– Climate data from World Meteorological Organization GCOS: I used about 200k records of historical climate data observations

Variables considered:

– 152 variables from plants’ observations (Darwin Core data standard + Observational data)

– 13 variables from climate observations: precipitation, climate air temperature, humidity, wind speed, pressure, lightning, upper-air temperature, carbon dioxide (CO2), ozone, date, location (GPS coordinates)

Features

0 plants' observations
0 climate's observations

Data challenges

  • Direct plants’ observations made in 60 years were affected by many quality issues:
  1. Human errors: name of the plants wrong, recorded date invalid, date invalid, geodetic datum invalid, elevation non-numeric or swapped, etc.
  2. Technology for observation changed, like the GPS tools (just few units in the 60s, today quite common with smartphones)
  3. Process to create observations changed: Initially scientists catalogued plants in printed forms; then scientists used Word documents, and, only in the last 30/40 years, relational databases have been adopted – I led a project to convert data from thousands printed forms to database of observations with metadata (presentation at TDWG 2010 conference)
Sampling was affected by many (non-sampling) biases. Some examples I remember:
  1. Undercoverage bias:
    • Number of observations of plants in a specific area depends on the funds allocated to the Institutes in doing this process. Therefore, the presence of plants in a location depended on the number of observations made in that area. It means only some Institutes had good historical data.
    • Scientists observed species only in some areas, probably close to their institutes, and at different times. This means I had to group by time the observations of plants in a map in order to consider which plants live in that area.
  2. Information bias:
    1. Each Institute uses its own set of tools and procedures to observe data. This means descriptive statistics are difficult to compare each other
    2. Observations happened at different seasons worldwide; in extreme climates (like cold winters or hot months) many plants naturally die so frequency of observations of course change.
    3. Climate data refers to the place where is located the meteorological station 
  3. Time varying confounders: external factors to the climate change can modify the presence of plants in a zone, like pollution and human activities
 

Methodology for the problem

For some operations, like interpolation of climate data or calculation of the centroids of plants in a geographical map, I used the methodologies commonly used by agrobiodiversity scientists and climate specialists.

The methodology I followed was tailored for the data, and required in some transformations and assumptions:

  1. GROUPING OF OBSERVATIONS BY DATE: To calculate correctly descriptive measures, I grouped observations of plants by date of observation: all plants were considered “living organism” in the quarter of the year where observation was recorded. So, if a plant was observed in February 1998 then it became part of the group of plants living in the quarter January-March 1998 in that area.
  2. IDENTIFICATION OF CENTRES OF MASS OF SAMPLED PLANTS: I considered the geographical distribution of each sampled plant as concentrated in a single point, the centroid; for each plant sampled, I calculated also its mean and dispersion considering 10 hectares around its centroid. I considered the distance in meters centroids moved in a quarter. The hypothesis was that centroids moved because plants grow differently because of changes of climate factors.
  3. INTERPOLATION OF CLIMATE DATA: For the climate data, since the observations are done in meteorological stations, I performed interpolation of climate data to identify values in the exact location of the plants sampled. I used a deterministic interpolation algorithm called Inverse Distance Weighting (IDW), commonly used by climate scientists.

Unfortunately, I had to consider only those variables having good historical data, in the geographical areas identified after the sampling of the plants: humidity, temperature, pressure, precipitation.

Implementation

 

A standard process of statistic was followed, considering the quality issues and biases described above:

Multistage samplings of plants observations

Stage 1: Cluster sampling based on Institute names; only some Institutes collected good historical data

Stage 2: Stratified sampling based on qualitative variables of the plants (like kingdom, family name, genus name, taxonRank) – made by biologists

Stage 3: Systematic sampling – this avoided manipulation of the whole frame of Stage 2

The sample size resulted 1.2 million of samples of about 1250 plant species.

Definition of response variables

  1. Variation of GPS position of the centroids in meters (∆centroid) from quarter to quarter: the centroid is the centre of mass of the plants in a map, in a period of a quarter of year. I found the centroids using haversine formula with GPS coordinates
  2. Number of occurrences found in the area of 10 hectares around the centroid
  3. Variation of the Quartile Coefficient of Dispersion (∆QCD): the dispersion of plants in a region of 10 hectares around their centroids were measured by the Quartile Coefficient of Dispersion (QCD). Please note that the computation complexity for the QCD algorithm is excellent: O(n log n), n=number of units observed from a sampled plant

Inverse Distance Weighting (IDW) algorithm

  • The dataframe was created locally considering the response variables above defined
  • I accessed the historical observations of climate data from the GCOS institute and applied Inverse Distance Weighting (IDW) algorithm to interpolate the climate data in the exact location of the centroids of the sampled plants
  • I determined the mean values of the climate data in that point in a quarter period (very time-consuming iterations; today accelerating adaptive inverse distance weighting algorithm could be used)
  • I created then a dataframe joining data of sampled plants, climate data, observations data, date expressed in quarter/year

Data analysis

  • Analysis of the dataframe and some findings:
    • From time plots of samples, I found seasonality and some trends on number of plants in a region and their spread in that region
    • With Polar seasonal plots and scatterplot matrices, I found good correlations of some groups of plants with the climate data: with the help of biologists, we identified group of plants which responded more to variations of climate data. Example: plants that live at high altitudes or plants with good autochthone resilience showed high correlation of dispersion with temperature and humidity
    • Forecasts from regression model highlighted that at ground level plants should increase dispersion in their region (∆QCD) and their occurrences in their region should drastically drop
    • Some plants found the habitat at different locations: this was evident mostly in dried areas where the ∆centroid values were higher
    • Putting altitude of observations as a response variable, I found some plants changed their altitude because of the change of climate in their region. So even if their dispersion in the territory was low (∆QCD) they actually spread vertically over mountains and hills

Getting data

  • Data accessed and manipulated via dedicated RESTFUL API of the data provider GBIF
  • Data engineering of local datasets done in dedicated on-premises servers
  • Development language: python and some R scripts, T-SQL for some data manipulation functions
  • Languages for data engineering: SQL and stored procedures in Microsoft SQL Server
  • Geographical tools: ArcGIS, google maps
  • Development tools: Eclipse

 

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More