Evaluating conversation analytics across voice and digital channels

Separate speech analytics, Conversation Analytics and Data Analytics

Vendor terminology overlaps. Parloa's full overview describes conversation content analysis across voice and digital channels and compares several market labels. Use that as terminology context, not a universal taxonomy or evidence that every vendor handles every channel. Text analytics can also be multi-turn, and a product name alone does not establish its actual input, context or output contract. [1]

This guide evaluates conversation-content outputs across voice and digital channels. It does not design ingestion pipelines, ETL, a warehouse or dashboard metric reconciliation. It also stops before individual CRM approval/writeback: a good summary score does not confer permission to save a record or trigger an action.

Start with a specific decision and a reviewable label contract

Replace 'measure sentiment accuracy' with a concrete question: can the output help an authorised quality reviewer find an unresolved sign-in problem without misrepresenting what was said? Specify the evaluation unit—a turn, speaker, interaction or permitted linked journey—and the evidence the reviewer may see. A turn-level classification is not automatically a valid conversation-level conclusion.

• Define the task separately: intent/topic classification, expressed sentiment, extraction of facts, source-grounded summary or prioritisation for review.
• Write label meanings, examples and boundaries. If mixed or unclear evidence is possible, define how to represent it rather than force certainty.
• Define missing input, abstention, unsupported language, truncated context and processing failure. None should silently become a successful neutral result.
• Identify consequential errors and the person responsible for review. Set acceptance criteria for the actual risk and use case, not a generic percentage.
• Record the model/version, prompt, taxonomy, preprocessing, context window, sampling settings, transcript source and evaluator/rubric revision for a comparison.

The fictional sentiment exercise below uses positive, neutral and negative as candidate labels. Its invented annotation roles may also mark needs-review; that is an unresolved annotation state, not an emotion category. For the exercise, unresolved service-problem language outweighs polite thanks when assigning the task's sentiment label. This local teaching rule is not an objective measure of the person's feelings.

Build a test set whose limits are visible

A synthetic set is useful for testing known edge cases and rubric consistency. It cannot replace an approved, representative operational evaluation. Microsoft's sentiment transparency note explicitly warns that synthetic tests which do not reflect the end-to-end scenario may be insufficient; performance depends on domain and input variation. Obtain appropriate permission, purpose, access and retention decisions before using operational material. This article collects none. [2]

• Separate development/calibration examples from a held-out evaluation set. Keep duplicates, fragments of the same conversation and closely related journeys from leaking across the split.
• Slice by channel, supported language/locale, task, interaction length and relevant input conditions. English success does not establish another language or dialect.
• For voice, examine noise, codec/recording conditions, overlapping speakers and diarisation. For chat/email, examine quoted history, short messages, typos, missing turns, attachment-only information and asynchronous context.
• Include rare but consequential cases, not just the most common easy label. Declare how sampling and weighting relate to the intended population.
• Record how many examples are scorable, disputed, missing, unsupported or failed in each slice. A slice with no evidence is not a zero-error success.
• Report denominators and use an appropriate uncertainty method for the sampling design. Correlated turns are not necessarily independent observations; a small fixture cannot justify a narrow confidence interval.

Do not infer demographics or sensitive attributes to fill a fairness spreadsheet. Any such evaluation needs an independently approved purpose, lawful handling and qualified review. Language, geography and demographic identity are not interchangeable. The examples here are English text only and establish no subgroup performance.

Use an original rubric that does not average away critical errors

Use pass, review, fail or not-assessable for each applicable dimension, with an evidence span and a reason. Define which failures block the intended use before comparing candidates. A polished summary with the wrong outcome should not pass because its readability score compensates for the error. The rubric below is an original proposed checklist, not an approved industry scorecard or a numerical benchmark.

Not-assessable is not pass. Keep missing evidence visible and report it separately. Record candidate failures, not just an average score; acceptance remains a decision by the responsible evaluation and operational owners.

Measure transcript errors and sentiment disagreement separately

For real speech recognition evaluation, Microsoft's current guide requires audio with corresponding human-labelled transcripts and defines word error rate as (substitutions + deletions + insertions) divided by reference words. Tokenisation and normalisation matter. WER is a transcription metric, not sentiment accuracy, factual correctness or permission to act. This guide does not adopt the vendor page's generic quality thresholds as acceptance criteria for your use case. [3]

The first fictional example deletes 'not' from six reference words. The text-edit calculation is one deletion divided by six words: 16.67%. No audio was supplied and no ASR engine ran. The point is that a single meaning-critical error needs its own review even when an aggregate transcription score appears acceptable.

Where authorised, compare downstream analytics on an approved reference transcript and on the actual ASR output, then review the end-to-end result. That helps separate transcript-induced changes from analytics-model errors; it does not make reference text infallible. Digital channels avoid an ASR stage but can still lose speakers, quoted history and relevant turns.

Microsoft's sentiment note explains that its text model can struggle with domain mismatch, underrepresented language/dialects, sentence importance and sarcasm. It also distinguishes confidence in a classification from intensity of sentiment. These are documented limits of that service, not a universal calibration rule for every model. A sentiment score is not verified emotion, satisfaction, truthfulness or intent. [2]

In a real evaluation, have qualified annotators apply the same rubric independently and retain initial disagreements before adjudication. Blind candidate identity/order where practical. Report raw agreement and choose any additional agreement statistic with its assumptions stated; agreement is not proof of truth. Do not quietly drop disputed items to make a headline accuracy number look better.

Evaluate grounded claims and missing facts independently

A summary can contain only supported sentences and still omit the fact the reviewer most needs. Evaluate source support, attribution, contradictions, must-cover facts and uncertainty separately. Define required facts from the task, not from whichever summary happened to be generated.

Microsoft's current GroundednessEvaluator reference describes checking whether claims are substantiated by provided context, noting that a factually correct statement can still be ungrounded if that source does not verify it. Groundedness is not completeness or real-world truth: a response can agree with an erroneous transcript. The manual annotations below do not execute that evaluator or use its scoring scale. [4]

For a bounded reference, distinguish what the person requested, what they reported, what another speaker promised and what an authoritative system actually confirmed. If the source establishes no completed action, do not invent one. Summary evaluation does not authorise individual writeback or consequential automation.

Inspect six annotated examples—not customer evidence

Every line, candidate output and annotation below is invented. A and B are hypothetical annotation roles, not actual employees, SMEs or people who performed review. Voice-reference-text means written text standing in for voice, with no recording, speaker segmentation or acoustic test. Candidate labels and summaries were hand-authored, not generated by a vendor model.

The exercise intentionally exposes a mismatch between fluent summaries and useful review evidence. It contains no adversarial benchmark suite, representative channel distribution or native-language validation. The critical-omission annotations are teaching decisions for this local rubric, not legal or production release requirements.

Reproduce only the fixture arithmetic offline

Save the following as analytics-fixture.mjs and run node analytics-fixture.mjs. It has no dependencies, network, model, account, recordings or customer data. Its annotation arrays encode manual teaching decisions; the code tallies them, not language understanding. All percentages below are calculated fixture results, never vendor accuracy benchmarks, customer outcomes or acceptance thresholds.

const fixture = [
  { id: 'SYN-E01', channel: 'voice-reference-text', reference: 'Please do not cancel my plan.', candidateSummary: 'The person requested cancellation and the plan was cancelled.', a: 'neutral', b: 'neutral', expected: 'neutral', predicted: 'neutral', required: ['no-cancellation-request', 'no-confirmed-action'], covered: [], criticalMissing: ['no-cancellation-request'], unsupported: ['cancellation-request', 'completed-cancellation'] },
  { id: 'SYN-E02', channel: 'chat', reference: 'Thanks for trying, but I still cannot sign in.', candidateSummary: 'The sign-in problem remains unresolved.', a: 'negative', b: 'negative', expected: 'negative', predicted: 'positive', required: ['signin-unresolved'], covered: ['signin-unresolved'], criticalMissing: [], unsupported: [] },
  { id: 'SYN-E03', channel: 'email', reference: 'Please email the reset steps. I cannot take a phone call.', candidateSummary: 'The person requests reset instructions.', a: 'neutral', b: 'neutral', expected: 'neutral', predicted: 'neutral', required: ['reset-request', 'email-not-phone'], covered: ['reset-request'], criticalMissing: ['email-not-phone'], unsupported: [] },
  { id: 'SYN-E04', channel: 'chat', reference: 'Great, another reset loop.', candidateSummary: 'The person reports another reset loop.', a: 'negative', b: 'needs-review', expected: null, predicted: 'positive', required: ['repeat-reset-loop'], covered: ['repeat-reset-loop'], criticalMissing: [], unsupported: [] },
  { id: 'SYN-E05', channel: 'voice-reference-text', reference: 'I can sign in now. Thank you.', candidateSummary: 'The person reports that access now works.', a: 'positive', b: 'positive', expected: 'positive', predicted: 'positive', required: ['user-reports-access-restored'], covered: ['user-reports-access-restored'], criticalMissing: [], unsupported: [] },
  { id: 'SYN-E06', channel: 'email', reference: 'This is the third incorrect bill. The issue is still open.', candidateSummary: 'The billing issue has been resolved.', a: 'negative', b: 'negative', expected: 'negative', predicted: 'neutral', required: ['reported-repeat-bill-error', 'billing-unresolved'], covered: [], criticalMissing: ['billing-unresolved'], unsupported: ['billing-resolved'] }
];
const rate = (numerator, denominator) => ({ numerator, denominator, percent: denominator ? (100 * numerator / denominator).toFixed(2) + '%' : null });
const sum = (rows, field) => rows.reduce((n, r) => n + r[field].length, 0);
const scorable = fixture.filter(r => r.expected !== null);
const negative = scorable.filter(r => r.expected === 'negative');
const predictedNegative = scorable.filter(r => r.predicted === 'negative');
const trueNegativeLabel = negative.filter(r => r.predicted === 'negative').length;
const labels = ['negative', 'neutral', 'positive'];
const confusion = labels.map(actual => labels.map(predicted => scorable.filter(r => r.expected === actual && r.predicted === predicted).length));
const tokens = text => text.toLowerCase().replace(/[.!?,]/g, '').trim().split(/\s+/);
function editDistance(a, b) {
  let row = Array.from({ length: b.length + 1 }, (_, i) => i);
  for (let i = 1; i <= a.length; i++) {
    const next = [i];
    for (let j = 1; j <= b.length; j++)
      next[j] = Math.min(row[j] + 1, next[j - 1] + 1, row[j - 1] + (a[i - 1] === b[j - 1] ? 0 : 1));
    row = next;
  }
  return row[b.length];
}
const referenceWords = tokens(fixture[0].reference);
const alteredWords = tokens('Please do cancel my plan.');
const metrics = {
  fictionalAnnotationAgreement: rate(fixture.filter(r => r.a === r.b).length, fixture.length),
  referenceScorableCoverage: rate(scorable.length, fixture.length),
  sentimentCorrectnessOnScorable: rate(scorable.filter(r => r.predicted === r.expected).length, scorable.length),
  negativeLabelRecall: rate(trueNegativeLabel, negative.length),
  negativeLabelPrecision: rate(trueNegativeLabel, predictedNegative.length),
  requiredFactCoverage: rate(sum(fixture, 'covered'), sum(fixture, 'required')),
  summariesWithCriticalOmissions: rate(fixture.filter(r => r.criticalMissing.length).length, fixture.length),
  unsupportedOrContradictedClaimCount: sum(fixture, 'unsupported'),
  textEditWER: rate(editDistance(referenceWords, alteredWords), referenceWords.length),
  confusion: { rows: labels, columns: labels, counts: confusion },
  slices: [...new Set(fixture.map(r => r.channel))].map(channel => ({ channel, examples: fixture.filter(r => r.channel === channel).length, scorable: scorable.filter(r => r.channel === channel).length }))
};
console.log(JSON.stringify(metrics, null, 2));

The code also emits a confusion matrix with actual labels as rows and candidate labels as columns, plus channel sample counts. Each channel has only two invented examples; chat has just one scorable sentiment reference. No per-language, acoustic or channel performance conclusion is supported. Keep the denominators and unresolved items alongside any reported metric, and never convert an undefined ratio into a pass.

Calibrate automated grading against appropriate human review

OpenAI's current evaluation guide recommends task-specific tests, representative datasets and human feedback to calibrate automated scoring; it discusses disagreement and judge position/verbosity bias. Use those methodological principles rather than copying its illustrative target numbers or assuming one judge is an independent authority. Any provider-hosted evaluation feature's lifecycle must be checked separately; this guide depends on no hosted evaluation API. [5]

If an LLM judge is later approved, give it a bounded rubric and permitted evidence, validate its agreement and failure cases against qualified review, and test candidate-order effects. Record judge/model/prompt versions and repeated-run variability. Do not let a transcript instruct the judge to ignore the rubric. A generated rationale is not proof that a judgement is right, and using the same model family to generate and grade can leave shared blind spots.

Human review is also fallible. Use native/domain-competent reviewers for the intended languages, training/calibration examples and a documented route to resolve or retain disputes. Do not present a machine-generated translation reviewed only in English as validation of the original language's nuance.

State inappropriate uses and the evidence needed after changes

Do not treat inferred sentiment as a diagnosis, a lie detector, proof of satisfaction or an automatic basis for consequential decisions about employees or customers. Microsoft's sentiment note specifically cautions against automatic high-impact actions without human intervention. Responsible legal and operational owners must assess the actual use, affected people and applicable requirements; this guide does not provide a universal legal rule. [2]

• Before use: approve the task, permitted evidence, rubric, failure consequences, receiving workflow and review ownership.
• Before a comparison: freeze the held-out set and label definitions, document exclusions, and compare candidates on the same applicable evidence.
• Before a consequential next step: require the relevant verification, human decision and system permission; an analytics score alone grants none.
• After model, prompt, ASR, taxonomy or context changes: rerun approved regression slices and inspect critical failures, not just the aggregate average.
• During an approved operating review: check changed input mix, language/channel gaps, reviewer disagreement and correction patterns with minimal permitted diagnostics.
• If evidence is missing or disputed: narrow the supported use, require review or withhold the output from the affected decision rather than label it reliable.

Logging, monitoring or a vendor's ability to process all captured interactions is not permission to record everything. Decide collection, access, redaction limits, retention and any reuse with responsible privacy/security owners. Redaction is not a guarantee that material is anonymous. This article adds no tracking, data upload, capture pipeline or automatic training loop.

Resources

• Conversation Analytics engineering capabilities →
• Operational Data Analytics capabilities →
• Discuss channels, data boundaries and evaluation requirements →