Ensuring Compliance of Generative AI: Assessing LLMs’ Adherence to EU AI Regulations | David Sweenor | August 2023

Ensuring Compliance of Generative AI: Assessing LLMs’ Adherence to EU AI Regulations | David Sweenor | August 2023

Introduction:

With the increasing prominence of generative artificial intelligence (AI), there is a growing demand for regulations to govern its use. The potential negative impacts of AI, such as discrimination, privacy violations, and negative biases, highlight the need for robust regulations. While the US has recently announced voluntary AI guidelines, it still lags behind the European Union (EU) in terms of AI regulations. In June 2023, the EU passed the world’s first draft regulation on AI, focusing on ensuring the safety, transparency, and non-discrimination of AI systems. This blog explores how well large language models (LLMs) comply with the draft EU AI Act, highlighting the need for further improvement in various areas such as copyright issues, risk mitigation, evaluation and auditing, and energy consumption reporting.

Full Article: Ensuring Compliance of Generative AI: Assessing LLMs’ Adherence to EU AI Regulations | David Sweenor | August 2023

How Well Do LLMs Comply with the EU AI Act?

Generative artificial intelligence (AI) has become a subject of concern due to its potential to negatively impact society. It has the ability to perpetuate stereotypes, violate privacy, discriminate, and undermine human values. To address these concerns, the US government introduced voluntary AI guidelines in June 2023, which were followed by prominent companies. However, the European Union (EU) has always been at the forefront of AI regulations.

In June 2023, the EU passed the world’s first draft regulation on AI, building upon the AI ethics framework ratified in 2019. The primary goal of the EU is to ensure that AI systems used in the EU are safe, transparent, traceable, non-discriminatory, and environmentally friendly. The EU emphasizes the importance of human involvement in AI systems, preventing companies from letting AI and automation run on its own.

The draft EU AI Act categorizes AI into three different levels of risk, each requiring a different level of regulation. The three risk tiers identified by the EU are unacceptable risk, high risk, and limited risk. Unacceptable risk refers to the use of harmful technology that poses a threat to human beings, such as cognitive influence or social ranking based on AI. High risk AI systems that could have a detrimental effect on safety or basic rights and freedoms are divided into two categories. The first category includes AI embedded in retail products, while the second category requires registration in an EU database. Limited risk AI systems, at the very least, must meet transparency and openness standards to allow users to make informed decisions.

The EU AI Act is currently in the negotiation phase among member countries, with the aim of ratification by the end of 2023. In light of this legislation, researchers at Stanford University’s Center for Research on Foundation Models (CRFM) and Institute for Human-Centered Artificial Intelligence (HAI) conducted a study to assess how well current large language models (LLMs) comply with the draft act.

The researchers selected twelve requirements from the draft act and created a rubric to evaluate the compliance of various foundation model providers. The analysis included models from OpenAI, Anthropic, Google, Meta, Stability.ai, and others. The results showed significant variability in model compliance across providers, with some scoring less than 25% and only one provider scoring above 75%.

The study identified several persistent challenges in LLM compliance with the draft act. These challenges include ambiguous copyright issues related to training data, lack of risk mitigation disclosure by providers, evaluation and auditing deficits in assessing model performance, and inconsistent reporting of energy consumption by foundation model providers.

In conclusion, while the EU is taking significant steps towards regulating AI with the draft AI Act, there is still room for improvement in LLM compliance. Addressing challenges such as copyright issues, risk mitigation, evaluation, and energy consumption reporting will be crucial for ensuring the safe and ethical use of AI.

Summary: Ensuring Compliance of Generative AI: Assessing LLMs’ Adherence to EU AI Regulations | David Sweenor | August 2023

Generative artificial intelligence (AI) technology has raised concerns about discrimination, privacy violations, and negative biases. While the US has announced voluntary AI guidelines, the European Union (EU) is at the forefront of AI regulation. The EU has passed the first draft regulation on AI, prioritizing safety, transparency, and non-discrimination. The proposed law categorizes AI into three risk levels – unacceptable risk, high risk, and limited risk – each with different regulations. Researchers at Stanford University evaluated compliance of current large language models (LLMs) with the draft act and found variability across providers. Persistent challenges include copyright issues, lack of risk mitigation disclosure, evaluation and auditing deficits, and inconsistent energy consumption reports. Providers and adopters of LLMs still have a long way to go in ensuring compliance with AI regulations. Collaboration among law makers, designers, governments, and organizations is crucial in addressing these challenges.

Frequently Asked Questions:

Q1: What is data science and why is it important?
A1: Data science is a multidisciplinary field that involves extracting insights and knowledge from complex and large datasets using various scientific methods, algorithms, and tools. It combines statistical analysis, data visualization, and machine learning to uncover patterns, trends, and correlations. Data science plays a crucial role across industries as it supports informed decision-making, predictive modeling, and the development of innovative solutions to complex problems.

Q2: What are the key skills required to become a successful data scientist?
A2: To excel in data science, individuals need a combination of technical, analytical, and communication skills. Proficiency in programming languages like Python and R is essential for data manipulation and analysis. Strong statistical knowledge is necessary for hypothesis testing and predictive modeling. Additionally, skills in machine learning, data visualization, and data manipulation tools such as SQL are crucial. Effective communication skills, both written and verbal, are vital to effectively present insights and findings to non-technical stakeholders.

Q3: Which industries benefit the most from data science?
A3: While data science has wide-ranging applications, some industries have particularly benefited from its implementation. Finance institutions utilize data science for fraud detection, risk assessment, and algorithmic trading. Healthcare organizations employ data science to develop personalized treatments, predict disease outbreaks, and improve patient care. E-commerce companies leverage data science to enhance customer experience, optimize pricing models, and target marketing campaigns. Other sectors, such as transportation, energy, retail, and telecommunications, also greatly benefit from the insights derived from data science.

Q4: What is the process of data science and its main steps?
A4: Data science encompasses a systematic approach that involves several key steps. These steps include defining the problem statement, data collection and preprocessing, exploratory data analysis, feature engineering and selection, model building and evaluation, and finally, deploying and assessing the model. Each phase of the data science process is crucial in understanding the problem, preparing the data, analyzing patterns, and developing models to derive actionable insights.

Q5: How does data science contribute to business decision-making?
A5: Data science plays a critical role in business decision-making by providing evidence-based insights and predictions. By analyzing large volumes of data from various sources, businesses gain valuable information about customer behavior, market trends, and operational efficiency. Data science techniques can assist in optimizing marketing strategies, improving supply chain management, predicting customer churn, and identifying potential areas for cost reduction. Informed decision-making driven by data science leads to better outcomes, increased efficiency, and a competitive advantage in today’s data-driven business landscape.