Zum Inhalt springen
KS

Fallstudie

AI Tutor at Coschool

An AI tutor built from zero during my internship and shipped to 1,000+ beta users, with evaluation pipelines to keep its answers on topic.

Alle Projekte
Status
Fertig
Zeitraum
03/2023 to 08/2023, Hyderabad
Rolle
Machine Learning Intern
Technik
  • RAG
  • Milvus
  • GPT

1,000+

beta users

80%+

fewer off-topic answers

500+

test cases

Problem

The tutor had to stay on topic. Incorrect or off-topic answers were the gap I measured.

Approach

Treat the off-topic rate as a number. Evaluation pipelines flagged unsafe or off-topic outputs, and 500+ test cases showed whether a change helped.

Implementation

  • Evaluation pipelines that catch unsafe or off-topic outputs.
  • GPT models fine-tuned with guardrails.
  • A RAG pipeline in front of the model. Documents are split with semantic chunking and stored in Milvus.
  • An ingestion path that covered 10K+ documents.

What I measured

  • 80%+ fewer incorrect or off-topic answers across 500+ test cases
  • 35%+ higher contextual accuracy after adding the RAG pipeline
  • 40% lower retrieval latency after semantic chunking and Milvus ingestion

These figures come from internal evaluation during the internship in 2023. The test sets are not public.

Current status

Shipped to 1,000+ beta users during the internship, which ended in August 2023.

What I learned

A guardrail is only real if I measure it. The 500+ test cases were how I knew the off-topic rate had dropped.