KARMA

KARMA is an evaluation framework for medical AI systems, providing a toolkit to assess models using various datasets. It is built for researchers and healthcare organizations to ensure reliable performance testing. The system supports multi-modal evaluation and is model-agnostic. **Use cases**: Evaluating medical AI model performance, Testing models on Indian healthcare datasets, Benchmarking text, image, and audio models, Integrating custom evaluation metrics, Running efficient batch evaluations **Key features**: Extensible toolkit for custom models and datasets; Supports text, image, and audio evaluation; Smart caching for faster re-evaluations; Registry-based architecture for auto-discovery; Rich CLI with progress bars and outputs; Model-agnostic with unified interface; Integrated with HuggingFace and local evaluation **Who it's for**: Researchers, developers, and healthcare organizations that need to evaluate medical AI systems reliably. It is especially useful for those working in or with the Indian healthcare environment.

KARMA is an evaluation framework for medical AI systems, providing a toolkit to assess models using various datasets. It is built for researchers and healthcare organizations to ensure reliable performance testing. The system supports multi-modal evaluation and is model-agnostic.