SafeGPT

In recent years, large language models (LLMs) have become increasingly prevalent, with platforms like ChatGPT transforming how we interact with artificial intelligence. However, this technological advancement raises significant questions about reliability, bias, and ethical considerations. To address these challenges, companies like SafeGPT and Giskard have developed sophisticated quality assurance platforms that monitor LLM outputs in real-time. This article examines how these systems work, their technical foundations, and their importance in the context of emerging AI regulations. From detecting factual errors to ensuring compliance with European AI directives, the technologies discussed represent crucial developments in making AI systems safer and more trustworthy.

SafeGPT Platform Overview

The SafeGPT platform offers comprehensive quality assurance capabilities through two main components: a browser extension for end-users and a monitoring dashboard for developers. The browser extension enables real-time identification of wrongful answers, reliability issues, and ethical biases across any Large Language Model (LLM), including ChatGPT. The monitoring dashboard features robust alerting and root-cause analysis capabilities, allowing users to filter issues by queries, topics, and users, while the developer platform provides a quality assurance framework for LLM debugging and testing.

Giskard's technical foundation enables scalable and secure AI systems through its open-source components, including Giskard Vision for model testing and the Giskard Open-Source toolkit for data scientists. The company's quality assurance platform automatically generates comprehensive test sets to evaluate answer accuracy for RAG agents, while its browser extension and monitoring dashboard components offer straightforward solutions for preventing errors and privacy issues in LLM outputs.

The platform addresses critical AI risks including hallucinations (factual errors), privacy leaks, ethical biases, and solution inconsistencies across providers. Operating on a foundation of state-of-the-art research and real-time data, SafeGPT ensures independent third-party assessments of LLM safety through its comprehensive test catalog covering reliability, bias, performance, and fairness metrics. These protections align with emerging European regulations, particularly the AI Act, which requires robust testing frameworks and independent oversight for high-risk AI systems.

Giskard Company Background

Giskard's development traces back to 2011 when its co-founders began creating AI systems for large enterprises. The company's expertise in AI safety and model quality emerged in response to both technological advancements and growing regulatory demands. Recognizing the escalating complexity of AI applications in critical sectors, Giskard developed specialized software solutions to address the increasing need for responsible AI deployment.

The company's technical foundation spans both private development for enterprise clients and open-source initiatives. Giskard employs a team of experienced engineers and data scientists who have worked together for over a decade to develop advanced AI safety systems. Their current workforce of 12 professionals is located in Paris, where the company continues to innovate in AI risk management.

At its core, Giskard operates at the intersection of AI technology and regulatory compliance. The founders identified several primary risks associated with AI deployment, including societal impact, ethical considerations, and technical errors. These risks are particularly acute in public services and financial sectors, where AI can perpetuate existing biases or create new forms of discrimination. The company's approach stands out through its dual focus on technical accuracy and ethical integrity, offering solutions that help maintain trust in AI while ensuring its safe integration into various industries.

Giskard's AI safety framework addresses multiple critical areas through its portfolio of products and services. The company's technical solutions include Giskard Vision, a proprietary tool for model testing, and Giskard Open-Source, which provides integrated testing capabilities directly within Python notebooks and IDEs. These tools cover essential aspects of AI validation, from automatic testing and bias detection to robustness evaluation across different model implementations. Through these platforms, Giskard empowers both developers and users to implement comprehensive quality assurance practices that align with emerging regulatory standards.

The company plays a pivotal role in shaping the future of AI through its collaboration with the ArGiMi consortium on next-generation French language models. This work represents a broader commitment to open-source development and cross-industry standards in AI technology. Giskard's approach reflects the growing recognition that responsible AI deployment requires shared standards and collaborative development frameworks. Their technical solutions and regulatory expertise position them as key contributors to the evolving landscape of AI safety and compliance.

AI Risk Management and Regulation

The European Union's AI Act represents the world's first comprehensive regulatory framework for artificial intelligence, building upon the success of the General Data Protection Regulation (GDPR) enacted in 2015. Published as a proposal in April 2021, the Act has undergone significant refinement to address emerging challenges, particularly those posed by general-purpose AI tools like ChatGPT.

Risk Categorization and Requirements

The AI Act establishes a risk-based regulatory structure encompassing four categories of AI applications: Unacceptable Risk, High-Risk, Limited Risk, and Minimal Risk. Unacceptable Risk applications, including government social scoring systems and mass surveillance tools, have been explicitly prohibited. High-Risk AI systems, which can profoundly impact health, safety, fundamental rights, or the environment, are subject to stringent deployment requirements.

High-risk applications are categorized into two groups: safety components of regulated products (subject to third-party assessment under sector-specific legislation) and specialized systems in critical sectors including education, employment, essential services, and law enforcement. Limited Risk systems, which interact with humans, detect emotions, or produce manipulated content, face transparent obligations. Minimal Risk applications, which carry minimal or no risk, are permitted without restrictions, though providers are encouraged to adopt voluntary codes of conduct.

Regulatory Framework and Implementation

The governance structure for AI implementation spans multiple levels of European governance. At the EU level, the Act creates the European Artificial Intelligence Board, comprising representatives from Member States, the European Commission, and the European Data Protection Supervisor. This body coordinates national supervisory authorities and provides expertise to the Commission, facilitating best practice sharing among Member States.

At the national level, Member States designate one or more national competent authorities, with the national supervisory authority responsible for monitoring AI implementation. These authorities oversee processes for assessing, designating, and monitoring conformity assessment bodies—informally known as notified bodies. These bodies perform essential functions including conformity assessment, testing, certification, and inspection.

Market surveillance authorities are established to enforce compliance across the EU market. The Act introduces significant penalties for non-compliance, calculated either as fixed sums or percentages of the offender's total worldwide annual turnover. The penalty structure distinguishes between large companies and small- and medium-sized enterprises (SMEs), with lower fines applied to the latter.

Technical Requirements and Testing

Technical requirements for AI systems emphasize robust risk management and comprehensive testing protocols. Providers of high-risk AI systems must establish quality management systems covering data governance, risk management, record keeping, technical documentation, and human oversight mechanisms. Continuous testing using state-of-the-art methods and automated protocols through open-source software like Giskard is mandated throughout the AI system's lifecycle.

For foundation models, including those supporting ChatGPT, additional obligations include fundamental rights protection guarantees, health and safety standards, and environmental compliance measures. Providers must implement comprehensive risk management systems capable of identifying and mitigating known and foreseeable risks associated with their AI systems.

The EU's AI Act represents a significant step toward responsible AI development, integrating technical standards with robust governance frameworks. While implementation challenges remain, particularly for small businesses and innovative startups, the Act's comprehensive approach provides a foundation for safe and ethical AI deployment across Europe.

SafeGPT for ChatGPT Users

The SafeGPT platform offers two main components for managing risks associated with Large Language Models (LLMs) like ChatGPT: a browser extension and a monitoring dashboard. The browser extension enables real-time monitoring of LLM outputs, while the dashboard provides comprehensive analysis tools for identifying and responding to errors and biases.

Browser Extension Features

The browser extension serves as an essential tool for ChatGPT users, providing immediate feedback on the reliability and accuracy of generated content. This user-facing component analyzes LLM outputs and flags potential issues, including factual errors, privacy violations, ethical concerns, and solution inconsistencies. By integrating with popular browsers, the extension offers seamless protection for users interacting with various LLM-powered applications.

Monitoring Dashboard

The monitoring dashboard represents the central management interface for SafeGPT users and LLM developers. It features sophisticated alerting and root-cause analysis capabilities, enabling users to filter issues by query type, specific topics, or individual users. The dashboard's design prioritizes usability and actionable insights, helping both technical and non-technical stakeholders understand and address identified risks.

Technical Integration and Data Handling

SafeGPT leverages state-of-the-art research methodologies and real-time data processing to maintain its effectiveness. The platform's technical foundation includes robust data governance practices and continuous testing protocols, ensuring that its monitoring capabilities remain reliable and responsive to emerging risks.

Giskard's Technical Solutions

Giskard has developed a comprehensive suite of technical solutions for AI quality assurance, built on their expertise in both proprietary enterprise applications and open-source software development. Their platform encompasses both model testing and developer tools, providing foundational capabilities for responsible AI deployment.

The company's primary testing solution is Giskard Vision, specifically designed for model evaluation across various applications. This tool enables users to identify and correct biases in AI models through visual diagnostics, providing transparent insights into model performance and reliability. The platform's open-source nature allows for flexibility in deployment, supporting multiple development environments and integration points.

Giskard's technical capabilities extend to automated testing through their Giskard Open-Source toolkit. This component offers two core features for data scientists and developers:

1. Direct integration into Python notebooks and IDEs, allowing users to identify vulnerabilities affecting LLM performance, fairness, and security through simple code modifications.

2. Automated testing capabilities for model evaluation, including the generation of realistic test sets and comprehensive answer accuracy assessments for RAG agents.

The company's technical solutions address multiple dimensions of AI risk through their platform architecture:

• The browser extension provides real-time monitoring of LLM outputs across any provider, including direct integration with popular browsers for seamless user protection.

• The monitoring dashboard offers sophisticated alerting and root-cause analysis capabilities, enabling users to filter and respond to issues based on specific query types, topics, and user interactions.

• The Giskard platform supports both ready-made and custom test creation, covering reliability, bias detection, performance, and fairness metrics across all stages of model development and deployment.

Giskard has developed a comprehensive approach to AI risk management that aligns with emerging European regulatory standards:

• The platform automatically generates comprehensive test sets to evaluate answer accuracy for RAG agents, ensuring continuous model performance across multiple implementation contexts.

• Real-time data processing and state-of-the-art research methodologies support the platform's effectiveness, maintaining rigorous performance standards through automated protocol adherence.

• The company's approach enables scalable deployment while maintaining high standards of security and trustworthiness, positioning them at the forefront of AI quality assurance development.