한국어 Contact us

Home Services LLM and agent evaluation

Korean LLM and agent evaluation

We evaluate Korean-language LLM answers and AI agent runs for global AI developers and AI data and evaluation platforms. Every judgment names the criterion behind it, so your team can check our work.

Contact us See examples

What you receive

  • 01

    Scores and rankings with reasons

    Each answer scored on every criterion or ranked against the others, with a short reason that names the criterion and quotes the problem.

  • 02

    Step labels for agent runs

    A pass or fail for each run, the step number and error type for every issue, and a mark on the first step that blocked completion.

  • 03

    Evaluation tasks with verifiers

    Task prompts, expected outcomes and verifiers, run against correct and plausible wrong solutions before we hand them over.

  • 04

    Batch notes and issue log

    Recurring error patterns in the batch, plus the unclear guidelines, broken tasks and conflicting rubric items we found.

How a project runs

  1. 1

    Scope by email

    Send the model or agent, task type, rough volume and any guidelines. We reply with questions and propose a sample batch.

  2. 2

    Calibrate on a sample

    We judge a small sample first and review it with you. Each disagreement becomes a written ruling.

  3. 3

    Same standard, whole batch

    Every item is judged by the same rulings. Unclear guidelines and broken tasks go back to you as questions, not guesses.

  4. 4

    Audit before submission

    Our own tools check format, steps, citations and numbers across the batch before anything is submitted.

  5. 5

    Deliver and follow up

    Results arrive in your format with the issue log. We answer questions about any judgment with its reason.

What we check

LLM answers

  • Facts, numbers and names match the source text
  • No claims the source does not support
  • Every instruction met: length, format, count, language
  • Answers the question that was actually asked
  • Rankings follow the rubric, not answer length

Korean language

  • Polite and casual speech mixed in one answer
  • Honorifics that do not fit the person referred to
  • Translationese: stiff, word-for-word phrasing from English
  • Particle, word spacing and spelling errors
  • Number unit slips: 3.5 million is 350만, not 35만
  • Swapped date order or 오전/오후 (a.m./p.m.)

Agent steps

  • The right tool or action at each step
  • Arguments match the request: dates, times, names, amounts
  • Each result read correctly before the next step
  • No actions the user did not ask for
  • Asks before acting when the request is unclear
  • Final answer matches what actually happened

Tasks and verifiers

  • One clear, checkable outcome per task
  • Solvable with the tools and files provided
  • Instructions do not give away the answer
  • Verifier accepts every correct solution
  • Verifier rejects plausible wrong solutions

Worked examples

Synthetic data, made for this page.

Example

Recipe summary checked against its source

Synthetic example: a user pastes a recipe and asks for a three-step summary, and we score the answer against the recipe.

  1. Source (excerpt)

    양념장: 간장 2큰술, 설탕 1큰술, 물 100ml를 섞는다.

    Sauce: mix 2 tablespoons soy sauce, 1 tablespoon sugar and 100 ml water.

  2. Prompt

    초보자도 따라 할 수 있게 세 단계로 정리해 주세요.

    Please put this into three steps a beginner can follow.

  3. Model answer

    1. 간장 2작은술, 설탕 1큰술, 물 100ml를 섞어 양념장을 만들어요.

    1. Mix 2 teaspoons soy sauce, 1 tablespoon sugar and 100 ml water to make the sauce.

  4. Accuracy

    Fail. The source says 2 tablespoons (큰술). The answer says 2 teaspoons (작은술).

  5. Instruction following

    Pass. Three steps, as asked.

  6. Tone consistency

    Pass. The same polite style in all three steps.

VerdictFails on accuracy

Accuracy: 2 tablespoons became 2 teaspoons, so a cook would add a third of the soy sauce. Fluent wording and the right format do not make up for a changed quantity.

Example

Wrong time argument in a tool call

Synthetic example: a phone assistant agent can check the weather and set alarms, and we score its run step by step.

  1. Request

    내일 서울에 비 오면 아침 7시에 우산 챙기라고 알람 맞춰 줘.

    If it rains in Seoul tomorrow, set an alarm for 7 in the morning to remind me to take an umbrella.

  2. Step 1 · tool call

    get_weather(city="Seoul", date="tomorrow") → "rain, 90% chance"

  3. Step 2 · tool call

    set_alarm(time="19:00", label="우산 챙기기")

    label: "Take an umbrella"

  4. Final answer

    내일 아침 7시에 우산 알람을 맞춰 두었어요.

    I've set an umbrella alarm for 7 tomorrow morning.

  5. Step check

    Step 1 passes. Step 2 fails: 19:00 is 7 p.m., but the user said 아침 (morning). The final answer reports a time that was never set.

VerdictFails at step 2

Tool-call accuracy: the alarm was set for 19:00, not 7 in the morning. The final answer hides the error, so we check each call against the request instead of trusting the summary.

Data and formats

  • Single and multi-turn chats
  • Answer sets for scoring and ranking
  • Answers with their source documents
  • Agent trajectories and tool-call logs
  • Mobile and desktop screenshots
  • Code diffs and terminal output
  • JSON, JSONL, CSV and spreadsheets
  • Tasks inside your annotation tool

Questions

Can you work inside our annotation tool and follow our rubric?

Yes. We usually work in the client's own tool and apply its rubric as written. If a criterion is hard to apply to Korean, we ask during the sample stage instead of guessing.

Which kinds of agents can you evaluate?

Mobile, coding and computer-use agents. We read each run step by step: the action or tool call, its arguments, the result it returned, and whether the final answer matches what happened.

How do you keep judgments consistent across a batch?

We settle each ruling once, write it down and apply it to every item. Before delivery, our own tools check the batch for format errors and steps that do not line up.

Can you design new evaluation tasks, not only score outputs?

Yes. We write tasks with one checkable outcome and build a verifier for each. We run the verifier on correct and plausible wrong solutions, so it neither rejects good work nor passes bad work.

Can we get the rationales in English?

Yes. Rationales can be in English, Korean or both. Korean words at issue are quoted as written, with an English gloss, as in the examples on this page.

Other services

RLHF and post-training data Red teaming and safety evaluation Localization QA In-house QA tools

Contact

Tell us about your project

Write in English or Korean, and include these four points if you can.

  1. 1Model, agent or product
  2. 2Type of work
  3. 3Rough volume and timeline
  4. 4Existing guidelines, if any
contact@marudata.io

The address opens a draft in your mail app. We then agree scope and schedule with you.