deepeval

Python toolkit for Chinese LLMs, with flexible batch capacity, structured real-time visulization and automated accumulation for streaming, and explicit feedback on vendor-native parameter validation.

Learn GenAI and Agentic AI from Zero to Production

26-framework SDET monorepo: Playwright, Selenium, Cypress, Cucumber, Postman, FastAPI, Pact, AI/LLM eval (DeepEval), agentic AI (LangChain, LangGraph, DSPy, Claude), k6, flakiness detection, site monitoring, failure triage, QMS evidence (ISO 9001/SOC 2), dependency audit · GitHub Actions · DataDog · Terraform IaC

Advanced RAG pipelines for medical (HealthBench, MedCaseReasoning, MetaMedQA, PubMedQA) and financial (FinanceBench, Earnings Calls) QA. LangGraph orchestration + BAML structructed generation, Milvus Hybrid search (Dense + BM25 + RRF), three-layer Metadata Enrichment, Contextual AI instruction-following reranker, and DeepEval evaluation.

Advanced RAG pipeline optimization framework using DSPy. Implements modular RAG pipelines with Query-Rewriting, Sub-Query Decomposition, and Hybrid Search via Weaviate. Automates prompt tuning and few-shot selection using GEPA, SIMBA, MIPRO, COPRO, and BootstrapFewShot optimizers on datasets like FreshQA, HotpotQA, TriviaQA, Wikipedia and PubMedQA.

An example that combines Behave (BDD testing) with DeepEval (LLM evaluation) to create human-readable, stakeholder-friendly tests for AI Agents / chatbots.

Self-Reflective Question Answering for Biomedical Reasoning. GRPO fine-tuning via QLoRA & Unsloth with rewards for correctness, relevance, groundness, utility & XML structure. Structured think → answer → self-reflection with context grading, relevance assessment & groundness evaluation. DeepEval LLM-as-a-Judge (GEval, Faithfulness, Relevancy).

🚀 Production-ready modular RAG monorepo: Local LLM inference (vLLM) • Hybrid retrieval with Qdrant • Semantic caching • Docling document parsing • Cross-encoder reranking • DeepEval evaluation • Full observability with Langfuse • Open WebUI chat interface • OpenAI-compatible API • Fully Dockerized

Universal Agent Evaluation Framework (UAEF) is a framework-agnostic evaluation system for AI agents. Invoke any agent and score it on multiple metrics spanning tool calling, response quality, safety, performance, and reasoning. Track experiments against baselines, catch regressions automatically, and get LLM-generated insights.

面向开发者的中文 Eval Harness 源码教材：解析 lm-evaluation-harness、Inspect AI、OpenAI Evals、Promptfoo、DeepEval 与 Harbor，覆盖任务、运行、评分、统计与发布门禁。

Multi-agent LangGraph app turning raw requirements into backlog-ready user stories, ACs, and test cases. Live Gemini-Flash CI eval gate + nightly N=10 adversarial regression.

Production-grade prompt engineering curriculum — 7 modules covering foundations, reasoning, RAG, security, and evaluation. Model-agnostic (OpenAI/Anthropic/Ollama), runnable notebooks, mini-projects, Mermaid diagrams. Not prompt tips — prompt engineering as a discipline.

AI research assistant for prosthetics, orthotics & rehabilitation robotics papers. RAG pipeline with streaming citations, structured summaries, terminology extraction, and paper comparison. Built with Next.js 15, Claude, pgvector. Includes observability tracing and DeepEval evaluation pipeline.

RAG assistant for accurate, cited search over credential/certificate documents — built with formal evaluation and hallucination-resistant retrieval

Agent 测试转型手册：资深自动化测试工程师的 AI QE 学习路径 + 练手代码

TermDraw-Bench: a benchmark for whether LLMs and VLMs can generate and edit structured ASCII diagrams. 80 tasks across 4 categories, scored by a DeepEval LLM judge.

Pytest plugin that records, gates and tracks your LLM evals in the CI you already have.

BNS-LexAI is an AI-powered legal information and case understanding assistant.

Automated LLM red-teaming framework testing 520+ adversarial prompts across 6 failure categories using DeepEval, LangSmith & Groq