Open-Ended Score
Discover why the Open-Ended Score is the most effective method for detecting poor data quality.
What is the Open-Ended Score?
ReDem enables precise evaluation of open-ended responses by classifying them into nine distinct categories, each scored accordingly. Our hybrid approach combines the accuracy of expert manual reviews with the efficiency of a GPT-4-powered AI model, ensuring reliable and comprehensive results you can trust.
How is the Open-Ended Score calculated?
Each response is first classified into one of our quality categories. Each category is then assigned a score from 0 to 100, reflecting the response’s quality. These scores are aggregated to calculate an overall Open-Ended Score (OES) for each respondent.
How does ReDem classify responses?
We classify each response into distinct quality categories, providing a clear and comprehensive assessment of the respondent’s score. These categories encompass all critical aspects of open-ended response quality. Our criteria are continuously refined to adapt to evolving needs and standards.
Wrong Topic
Identifies responses that deviate from the question or topic by evaluating their context against relevant keywords and the question itself. The context check can be enabled or disabled when importing data.
When adding keywords, ensure they broadly represent the context to minimize false positives from overly narrow interpretations. Both the question and keywords must be in a supported language.
Responses failing to align with the expected context are classified as “Wrong Topic” and assigned an OES (Overall Evaluation Score) of 30.
Enable this option only if your questions are meaningful and contain relevant keywords.
AI Generated Answer
model analyzes open-ended responses to determine if they were artificially generated by evaluating the pattern of a wide range of variables, including grammar, structure, phrasing, syntax, word choice, sentence length, complexity and predictability. Responses identified as AI generated content are assigned a score of 0.
Nonsense
Enabling nonsense detection identifies gibberish, random numbers, and meaningless statements. Such responses are classified as “Nonsense” and assigned a score of 10.
Wrong Language
Responses in an unexpected language are categorized as “Wrong Language” and assigned a score of 20. You can define the expected languages; without this, the language check remains inactive. Open-Ended Score supports over 100 languages, including English, German, French, Spanish, Chinese, Japanese, Swedish, and more.
Duplicate Respondent
The optional duplicate check detects potentially fraudulent responses, identifying both exact duplicates and partial matches.
- Duplicate Respondents in a Single Question: This check detects repeated responses to the same question. Single duplicates are classified as “Duplicate Respondent” and assigned a score of 50, while multiple duplicates receive a score of 0.
- Duplicate Respondents Across Multiple Questions: Our duplicate check also detects responses repeated across multiple questions. Such responses are classified as “Duplicate Respondent” and assigned a score of 10.
Duplicate Answer
We verify if a respondent’s answers are repeated or partially repeated across multiple questions. These responses are classified as “Duplicate Answer” and scored 50 for a single duplicate or 0 for multiple duplicates.
Bad Language
Responses containing swear words or offensive language are classified as “Bad Language” and are assigned a score of 10.
Generic Answers
Responses with generic statements like “good,” “ok,” “anything,” or “yes” are classified as “Generic Answers” and are assigned a score of 50.
No Information
Responses lacking meaningful content, such as “no idea,” “nothing,” “no comment,” or “I don’t know,” are classified as “No Information” and are assigned a score of 60.
Valid Answer
Valid responses are those that do not fall into any other quality category. Each response is also evaluated for its level of detail. “Valid Answers” are assigned a score between 70 and 100, based on their detail quality.
Use of GPT-4 for Open-Ended Score
ReDem leverages OpenAI’s GPT-4, one of the most advanced Large Language Models (LLM), to power its quality categorization. This cutting-edge technology enables highly sophisticated evaluation of open-ended responses, ensuring precise and reliable results. To uphold the highest data protection standards, GPT-4 integration in the ReDem OES is implemented as follows:
Individual Responses & Anonymity
Each open-ended response is sent to OpenAI individually, using a fully anonymized ID. Only the individual response is shared per API query, ensuring that full survey datasets are never accessible to OpenAI.
Exclusive Interaction with OpenAI
ReDem is the sole user interacting with OpenAI. OpenAI is never informed about the origin or source of the imported data.
Data Storage & Retention
Data sent through the API is stored by OpenAI for a maximum of 30 days before being permanently deleted. This data is not used for training AI models under any circumstances.
GDPR-Compliant Data Transfers
ReDem and OpenAI operate under a “Data Processing Agreement” based on EU Standard Contractual Clauses (SCC), ensuring all data transfers comply with GDPR regulations, including those involving personal data in open-ended responses.
This robust implementation provides the power and security you need to confidently assess the quality of open-ended survey responses.