Berri.ai

Berri.ai has developed a sophisticated platform that combines multiple tools to manage language models and content safety. At its core is the LiteLLM system, which provides advanced management features including pass-through endpoints, single sign-on key management, and detailed budget tracking. This comprehensive solution includes robust content safety mechanisms, fine-grained access controls, and multiple layers of security and privacy protection. The platform supports both self-hosted and cloud environments and includes comprehensive features for team management, budget tracking, and integration with various observability systems. This technical overview examines the key components of the Berri.ai platform, including its security architecture, access controls, and integration capabilities.

Platform Overview

The Berri.ai platform combines multiple tools to manage language models and content safety. At its core is the LiteLLM system, which provides advanced management features including pass-through endpoints, single sign-on key management, and detailed budget tracking.

Content Safety and Guardrails

Content safety is enforced through the Lakera system, which employs guardrails to detect and prevent prompt injection attacks. The system distinguishes between jailbreak and prompt injection categories, applying configurable threshold values of 0.1 for each. Guardrails functionality in LiteLLM enables custom security mechanisms through three operational modes - pre-call, during-call, and post-call - allowing for precise control over model interactions.

Access Control and Organization Management

The platform implements role-based access controls (RBAC) with multiple layers of authentication and authorization. The RBAC structure includes admin roles for platform and organization management, as well as internal user roles with varying levels of access. The process for creating organizations requires the proxy_admin role and involves specifying details such as organization alias, models, and maximum budget.

Security and Data Privacy

Role-Based Access Controls (RBAC)

The platform implements Role-Based Access Control (RBAC) with multiple layers of authentication and authorization. The RBAC structure includes several key roles:

• proxy_admin: Full platform and organization management capabilities. This role can view and manage all keys and budget across the platform.

• org_admin: Complete control over organization elements including team creation and user management within their specific organization.

• internal_user: Basic access allowing users to view and manage their own keys and spending.

• internal_user_viewer: Limited access allowing users to view their own keys and spending, but with no ability to create or delete keys.

Organizations must be created using the proxy_admin role, requiring specific details including organization alias, models, and maximum budget. This tiered access system allows for precise control over platform and organizational resources.

Cookie and Browser Security

Minimal cookie usage ensures efficient user authentication and session management through the LiteLLM Admin UI. These cookies, stored as JSON Web Tokens (JWT) in web browsers, expire when the browser is closed, logged out, or after 24 hours. No third-party cookies are utilized, and the platform does not access or share cookie data beyond basic login authentication.

Infrastructure and Data Regionalization

LiteLLM operates across multiple data regions for both self-hosted and cloud environments. The platform supports US data centers in Northern California and Virginia, with European operations based in Germany/Frankfurt. All data is encrypted using the LITELLM_MASTER_KEY, with secure transmission via TLS.

Data Collection and Usage

Cloud users can track LLM usage data, with access limited to proxy administrators within their organization. This data, stored locally in the user's environment, includes spend tracking and request logs but does not store message or response content. Self-hosted users maintain full control over their infrastructure, with no data transmitted to or stored on LiteLLM servers.

Vulnerability Management and Security Audits

Berri.ai follows a rigorous security framework including CodeQL for system management and ongoing SOC-2 Certification through Drata. The company maintains detailed audit logs for authentication and usage events. For security incident management, the platform provides alerting and monitoring tools while maintaining a vulnerability disclosure policy and active scanning services.

Enterprise Features

The platform implements comprehensive access control through role-based authentication with multiple layers of authorization. At the platform level, the proxy_admin role manages all aspects of organization and key management, while the proxy_admin_viewer role has read-only access to budgets and spend data but no capability to create or delete keys or add users.

Organizations are created through the proxy_admin role and require specific details including organization alias, models, and maximum budget. The RBAC structure includes:

• org_admin: Complete control over organization elements, including team creation and user management within their specific organization

• internal_user: Basic access allowing users to view and manage their own keys and spending

• internal_user_viewer: Limited access allowing users to view their own keys and spending, but with no ability to create or delete keys

Model Access and Team Management

The platform supports detailed team management through virtual keys, which can be assigned to specific teams or model access groups. This system allows for fine-grained control over model usage and access across multiple teams and projects.

Budget Tracking and Usage Reporting

Self-Hosted users have full control over their infrastructure with no data transmitted to or stored on LiteLLM servers. However, LiteLLM Cloud users can track usage through the platform's logging system, with all spend and usage data stored locally in their environment. Proxy administrators are the only users with access to this data, and it is limited to their organization's usage.

The platform provides three pricing tiers:

1. Open Source: Free

2. Self-Hosted: 100+ LLM Provider Integrations

3. Langfuse, Langsmith, OTEL Logging

4. Virtual Keys, Budgets, Teams

Security and Compliance

The company maintains SOC-2 Certification through Drata and follows a rigorous information security management system with CodeQL. LiteLLM's security framework includes:

• Data encryption using LITELLM_MASTER_KEY for all stored data

• Secure transmission via TLS for data in transit

• Infrastructure running on GCP and AWS with NeonDB management

• Data stored in multiple regions including US (Northern California, Virginia) and EU (Germany/Frankfurt)

• Support for OAuth 2.0 Single Sign-On through Google, Okta, Microsoft, and KeyCloak

• Logging with retention policies for authentication and usage events

• Access controls via IP address whitelisting

• Vulnerability management with email-based reporting to support@berri.ai

• Active scanning services for security vulnerabilities

Logging and Integration

The platform integrates with multiple observability systems including S3, Datadog, OTEL, and Langfuse for logging purposes. For self-hosted users, the system stores all telemetry data locally without transmitting any information to LiteLLM servers. The platform also provides a self-serve key management portal for team management and key creation.

Guardrails Integration

The LiteLLM platform integrates multiple security and policy enforcement mechanisms through its Guardrails feature. These mechanisms operate in three distinct modes:

Pre-Call Mode

In pre-call mode, guardrails run before the LLM call on the input data. This allows for early detection and prevention of potentially problematic requests before they reach the model.

During-Call Mode

During-call mode operates in parallel with the LLM call, monitoring the input data in real-time. This mode enables precise control over the interaction flow and can enforce rules dynamically based on the ongoing request.

Post-Call Mode

Post-call mode runs after the LLM call, examining both the input and output data. This allows for comprehensive validation of the model's response, ensuring that it meets the defined security and policy requirements.

Configuration and Implementation

To configure guardrails, the platform uses a YAML-based system. These configurations can include multiple guardrails, each specifying the model to apply the rule to, the guardrail provider (Lakera, Aporia, or Bedrock), the operating mode, and required API keys.

Guardrails Providers

The platform supports three primary guardrails providers:

• Lakera: The primary provider, requiring configuration parameters including model_name, litellm_params, api_key, api_base, and category_thresholds for prompt_injection and jailbreak detection.

• Aporia: Another supported provider, though specific configuration details are not provided in the documentation.

• Bedrock: The third provider option, with similar configuration requirements to Lakera.

Practical Implementation

A practical implementation example from the documentation demonstrates how to configure guardrails in the LiteLLM config.yaml file. This includes specifying model parameters, guardrail provider details, and operating mode.

Testing and Validation

The platform provides tools for testing guardrails configurations. A sample curl command demonstrates how to send a request with specific guardrails enabled, allowing developers to verify the configuration's effectiveness in real-time scenarios.

Logging and Observability

The platform manages logging through a callback system that supports multiple observability platforms including Lunary, Langfuse, Helicone, Promptlayer, Traceloop, and Slack. This system allows users to track LLM usage data while maintaining privacy by storing all telemetry data locally in the user's environment when using self-hosted instances. For LiteLLM Cloud users, the platform enables logging to these observability systems while ensuring that no message or response content is accessed or stored on LiteLLM servers.

Configuration and Setup

To enable logging observability, users must configure environment variables for their preferred observability system and set the appropriate success and failure callbacks in their LiteLLM configuration. The basic setup involves installing the required dependencies and setting up configuration files that define which observability systems to use. For example, to integrate with Lunary, users need to set the LUNARY_PUBLIC_KEY environment variable and enable the Lunary callback using litellm.success_callback = ["lunary"].

Data Collection and Storage

The platform collects telemetry data including request logs and spend information, which is stored locally in the user's environment for both self-hosted and Cloud users. When using self-hosted instances, all data remains on the user's infrastructure without transmission to LiteLLM servers. Cloud users can track LLM usage through the platform's logging system, but the system does not store message or response content. Access to this usage data is limited to proxy administrators within their organization.

Integration with External Systems

LiteLLM provides several integration methods for managing and using language models, including support for LangChain ChatLiteLLM, prompt templates, custom chains, and the LiteLLM proxy server. This flexibility allows users to choose the integration method that best fits their development workflow while maintaining consistent logging and observability. The platform maps exceptions across all supported providers to OpenAI's exception types, making it easy to implement consistent error handling across different APIs.