In-house tools that check our work

Seven kinds of checks

01
Trajectory parsing

Reads an agent's run record and splits it into numbered steps: action, tool call, result. Flags records it cannot read in full. Outputs a step list for review.
02
Step reconciliation

Reads the parsed steps, the agent's final answer and our rating. Flags where they disagree, such as a 'completed' rating when a tool call failed. Outputs the steps involved.
03
Citation and number checks

Reads a written rationale and its source text. Flags quotes not found word for word in the source and numbers that fail a recount. Lists each mismatch.
04
Format checks

Reads delivery files (XLSX, TSV, JSON, sheet exports). Flags empty required fields, labels outside the allowed set, scores that contradict rankings and text over length limits. Outputs each row and field.
05
Silence and audio quality

Reads audio files with their segment times. Measures silence and level, and flags long pauses and segment edges that do not match the speech. Outputs timestamps and level readings.
06
Speaker consistency

Compares the voices in recordings labelled as the same speaker. Flags files that may be a different person and recordings where a second voice appears. Outputs a list to relisten.
07
Pre-submission batch audit

Reads a whole batch before submission and runs the checks that apply. Flags items that fail any of them. Outputs one report: items ready, items held and why.

Where they fit in a project

1
After the work is done

We finish the labeling, evaluation or writing under the project guideline. The tools then read the finished files.
2
Checks before submission

Before a batch goes out, we run the checks that fit it: trajectories, text and data, or speech.
3
A report of flags

Each flag names the item, the exact place and the rule. A flag is a question for a person, not a verdict.
4
Fix and recheck

A person reviews each flag and corrects the item if needed. We run the check again before the batch is submitted.

What they look at

Agent trajectories

Steps split into action, tool call and result
Final answer claims compared with tool results
Ratings and cited steps checked against the record
Records that are cut off or missing steps

Text and data

Quotes found word for word in the source
Numbers and counts recomputed from the text
Required fields filled, labels from the allowed set
Scores consistent with rankings
Delivery sheet matches our working file, cell by cell

Speech and audio

Silence measured inside and around each segment
Segment edges compared with where speech starts and ends
Zero-length or very short segments
Speaker labels compared with who is speaking
Level, noise floor and clipping measured

Worked examples

Synthetic data, made for this page.

Example

Draft rating checked against the agent's steps

A synthetic scheduling agent run, cut to three steps, with our draft rating; step reconciliation compares the rating with each step's tool result.

Input · step 1 · request
목요일 오후 3시로 회의실 잡아 줘.

Book a meeting room for Thursday at 3 p.m.
Input · step 2 · tool call
book_room("Thu 15:00") → {"status": "error", "reason": "room_taken"}
Input · step 3 · final answer
목요일 오후 3시로 회의실을 잡았습니다.

I booked a meeting room for Thursday at 3 p.m.
Input · draft rating
{"item": "05", "completed": true, "issues": []}
Finding
The draft rates the run completed with no issues, but step 2 returned an error and step 3 still reports a booking.
Report
$ python audit.py example_batch/ ! reconcile item 05 rated completed, but step 2 returned error → item 05 held for recheck
Reviewer decision
Checked the record. Changed the rating to not completed, citing steps 2 and 3.

VerdictRating corrected

The agent claimed a booking that failed, and the draft rating took the claim at face value. The check compared the rating with the step results, and a reviewer corrected it before submission.

Example

A segment that starts before the speech

A synthetic two-speaker clip with our timed segments; the silence check measures where speech starts inside each one.

Input · audio
9.0 s clip, two speakers, three timed segments
Input · segment 02
start 3.20 s · end 5.90 s · B"네, 목요일 오후 세 시 괜찮습니다."

"Yes, Thursday at 3 p.m. works for me."
Finding
Speech starts at 3.62 s, so the segment opens with 0.42 s of silence.
Report
$ python audit.py example_batch/ ! audio item 07 seg 02 0.42 s of silence before speech starts → item 07 held for recheck
Reviewer decision
Listened and agreed. Moved the start to 3.62 s; the recheck passed.

VerdictBoundary corrected

Segments in this example must start with speech, not silence. The check measured the gap, and a person listened before moving the edge.

Files they read

JSON and JSONL
CSV and TSV
Spreadsheets (XLSX)
Agent run logs and HTML exports
Audio files
Transcripts with timestamps and speaker labels
Markdown and plain text

Questions

Do the tools replace human review?

No. They check files and flag items that may need another look. A person reviews each flag against the source and decides what, if anything, to change.

Can you adapt the checks to our guidelines?

Yes. The rules come from your guideline: required fields, allowed labels, length limits and timing tolerances. We set them up for each project and update them when the guideline changes.

What do we receive from these checks?

You receive the deliverables agreed for the project. If useful, we can explain which checks ran on a batch.

Why check these things with scripts?

Fields, quotes, counts and timings follow fixed rules, so a script checks them the same way on every item. Reviewers can then give their attention to judgment calls such as tone, accuracy and safety.

Other services

LLM and agent evaluation RLHF and post-training data Red teaming and safety evaluation Localization QA

Contact

Tell us about your project

Write in English or Korean, and include these four points if you can.

1Model, agent or product
2Type of work
3Rough volume and timeline
4Existing guidelines, if any

contact@marudata.io

The address opens a draft in your mail app. We then agree scope and schedule with you.

In-house tools that check our work

Seven kinds of checks

Trajectory parsing

Step reconciliation

Citation and number checks

Format checks

Silence and audio quality

Speaker consistency

Pre-submission batch audit