MedARC

The rapid advancement of artificial intelligence (AI) has profound implications for medical research and practice, yet significant challenges remain in developing AI systems tailored for healthcare applications. MedARC, founded by Tanishq Mathew Abraham and Jeremy Howard, is at the forefront of this evolving landscape, combining cutting-edge AI research with clinical expertise to develop foundation models for medical AI. This article explores the center's innovative approaches to AI development, contamination detection, and its impact on medical research and practice.

MedARC's Founding and Leadership

At the helm of MedARC is Tanishq Mathew Abraham, a five-year PhD candidate in Biomedical Engineering at the University of California, Davis. His journey into medical AI research began as a 14-year-old completing his biomedical engineering degree, driven by a lifelong passion for medicine and technology that solidified during his doctoral studies. Abraham's research focuses on applying generative AI to microscopy and digital pathology, a field he expanded upon through presentations at prestigious conferences like SPIE Photonics West and ICML workshops.

Jeremy Howard, MedARC's President, brings decades of experience in AI research and development to the table. As a founding researcher at fast.ai and an honorary professor at the University of Queensland, he has led groundbreaking initiatives that have shaped the landscape of AI innovation. Prior to his role at MedARC, Howard founded Enlitic and served as a Distinguished Research Scientist at the University of San Francisco, demonstrating his expertise in both academic research and commercial applications of AI technology.

The center's leadership draws inspiration from successful open-source initiatives in deep learning, particularly EleutherAI's development of the Pile dataset and GPT-NeoX-20B, and OpenBioML's replication of AlphaFold results. This approach emphasizes the potential for collaborative, decentralized research methods to advance medical AI, maintaining flexibility to address a wide range of AI research topics while serving as a catalyst for progress in the field.

Research Focus and Methods

MedARC's research methodology emphasizes the development of foundation models specifically tailored for medical applications, addressing the current limitations of domain-agnostic AI models. The center operates as an open and collaborative community, drawing from successful decentralized initiatives like EleutherAI's Pile dataset and GPT-NeoX-20B, and OpenBioML's AlphaFold replication efforts.

Research initiatives are spearheaded by interdisciplinary teams comprising clinicians with medical expertise and machine learning researchers/engineers. This collaborative approach has proven successful through partnerships like WAMRI.ai, resulting in multiple startups, publications, and notable contributions to Nature Methods.

Current projects span a range of medical applications, including real-time fMRI reconstructions and fine-tuning efforts on medical image generation. Notable initiatives like MindEye employ advanced techniques for fMRI-to-image reconstruction, utilizing parallel submodules for retrieval and reconstruction, specialized training methods, and models trained with large-scale parameter counts.

The center's computational infrastructure is supported by partnerships with Stability AI, which provides essential resources for research and development. Through initiatives like the MindEye2 project, which achieved state-of-the-art performance with reduced training data requirements, MedARC demonstrates its commitment to advancing medical AI through innovative research methodologies.

Current Projects and Achievements

Current projects at MedARC focus on building specialized AI models for medicine through interdisciplinary research teams. Recent progress includes real-time fMRI reconstructions and fine-tuning efforts on medical image generation, demonstrating the center's progress in AI for medical applications.

One of their key projects, MindEye2, represents significant advancements in fMRI-to-image reconstruction. This system achieves state-of-the-art performance with just 2.5% of the previously required training data, demonstrating substantial improvements in data efficiency. The architecture consists of two parallel submodules: one for retrieval using contrastive learning and another for reconstruction using a diffusion prior. Notably, MindEye2 outperforms other methods in both reconstruction and retrieval tasks when using just 1 hour of training data.

The development process emphasizes specialized training techniques and large-scale parameter usage. The system maps fMRI brain activity to CLIP image space through an MLP backbone consisting of a linear layer followed by four residual blocks and a final linear projector. These embeddings are fed into an MLP projector and a diffusion prior in parallel, with the entire pipeline trained end-to-end. To produce image reconstructions, the system maps voxels to the embedding space of Stable Diffusion's VAE, generating blurry reconstructions that serve as input for the final image generation process.

Interdisciplinary Team and Research Partnerships

The center's research approach draws from successful open-source initiatives in deep learning, particularly EleutherAI's development of the Pile dataset and GPT-NeoX-20B, and OpenBioML's replication of AlphaFold results. This collaborative model enables the center to maintain flexibility while addressing diverse AI research topics.

MedARC operates as an open and collaborative research community, sharing models and datasets when possible and maintaining transparent communication through a Discord server. The team structure includes clinicians with medical expertise and machine learning researchers/engineers, following successful models established through partnerships like WAMRI.ai and the Stanford Center for Artificial Intelligence in Medicine & Imaging (AIMI).

AI Model Development Process

The development process at MedARC centers on building foundation models specifically tailored for medical applications, addressing the current limitations of domain-agnostic AI models. The team employs an open and collaborative research model, drawing inspiration from successful decentralized initiatives like EleutherAI's Pile dataset and GPT-NeoX-20B, and OpenBioML's AlphaFold replication efforts. This approach enables the center to maintain flexibility while addressing diverse AI research topics.

The center's computational infrastructure is supported by partnerships with Stability AI, providing crucial resources for research and development. This partnership has facilitated significant advancements, exemplified by projects like MindEye2, which has achieved state-of-the-art performance using just 2.5% of the previously required training data.

Research initiatives at MedARC are spearheaded by interdisciplinary teams, combining expertise in medicine and machine learning. These teams have demonstrated success through partnerships like WAMRI.ai, resulting in multiple startups, publications, and notable contributions to Nature Methods. The team structure emphasizes collaboration between clinicians with medical expertise and machine learning researchers/engineers.

The development process at MedARC employs specialized submodules for retrieval and reconstruction, along with advanced training techniques and large-scale parameter usage. Notably, the MindEye2 project employs an architecture consisting of an MLP backbone with residual blocks and a diffusion prior. This system maps fMRI brain activity to CLIP image space embeddings, enabling accurate image reconstruction through a process that combines supervised learning with generative AI techniques.

The computational pipeline involves two primary steps: retrieval and reconstruction. During retrieval, subject-specific ridge regression maps fMRI activity to a shared latent space, while subsequent mapping to CLIP image space facilitates reconstruction. The model training process utilizes a multi-subject approach, where each batch contains inputs from multiple subjects during pre-training. This shared-subject model architecture enables improved out-of-subject generalization with limited training data, setting it apart from single-subject methods in the field.

The system produces its outputs through a structured pipeline that begins with ground truth embeddings for image reconstruction. The process generates blurry image reconstructions using a VAE decoder and further refines these outputs through an img2img process, maintaining low-level structural integrity while optimizing high-level semantic accuracy. The final reconstruction step employs a diffusion process starting from noised encodings of the initial blurry reconstructions, using a UniPCMultistep noise scheduling algorithm for optimized denoising.

Contamination Detection in LLMs

MedARC has developed five methods for detecting contamination in large language models (LLMs), specifically tailored for medical applications. These methods operate under the assumption that contamination occurs when models generate responses based on their training data exposure rather than their intended medical knowledge.

The first method involves splitting test questions in half and measuring similarity between model-generated completions and actual completions using inverse length-normalized Levenshtein distance. A similarity score of 0.95 or higher indicates likely training set exposure, suggesting contamination.

The second approach calculates token-level log-likelihoods for each test set example and averages the lowest K% scores. A Min-K% Prob score exceeding -7.3523 (tuned on WikiMIA validation set) indicates contamination likelihood.

A third method generates three paraphrases of each test case using GPT-4 and asks the model to identify the original. Model performance is measured using Cohen's kappa for quiz accuracy; lower scores correlate with higher contamination risks.

The fourth method employs paraphrase data from the quiz to compute average log likelihood differences between original and paraphrase responses. Values below 0.5 suggest higher contamination risk without requiring explicit threshold classification.

The final approach instructs models to generate questions for test set inputs using both "Guided" (matching original test set instructions) and "General" (any question) prompts. Statistical significance in "Guided" score increases indicates potential contamination.

The center has tested these methods across four top-performing models - Yi-34b, Mixtral-7b, Llama-2-70b, and Qwen-72b - on medical QA and MMLU tasks. Key findings include:

• No evidence of contamination using MELD method

• No evidence of contamination using Guided Instruction method

• Yi-34b shows potential contamination through Quiz Accuracy and Neighborhood Loss Delta metrics

• Qwen-72b shows potential contamination through Neighborhood Loss Delta metric

• No clear evidence of contamination across all metrics for PubMedQA tasks

These findings highlight the complex challenges in evaluating medical knowledge across different LLM architectures while demonstrating the effectiveness of MedARC's contamination detection methods.