Coherence Score
Do’s and Don’ts
To ensure the Coherence Score functions reliably, please follow the recommendations below.1. Select the right questions
The total number of questions is not critical; even for long questionnaires, a maximum of approximately 100 well-chosen questions is usually sufficient. The primary selection criteria are that the full question wording and fully labeled answer options are provided (see section 2) and that the questions have been answered by the majority of study participants (80% or more). In addition, it is important to include logically connected question sequences that allow the detection of contradictions and incoherence, such as questions on brand awareness followed by questions on brand usage, where claiming unawareness of a brand while reporting its use would constitute a clear inconsistency. Similarly, contradictions may arise when respondents make conflicting attitudinal and behavioral statements, for example claiming to be vegan while later reporting regular meat consumption. Furthermore, our experience shows that sociodemographic questions are particularly effective for identifying contradictions and implausible response patterns and should therefore be included whenever possible. A typical example of a sociodemographic contradiction would be a respondent stating that they are 18–24 years old while later indicating that they have been in full-time employment for more than 15 years, which is not plausibly compatible. Sociodemographic information may also conflict with behavioral statements, such as a respondent claiming to use the metro every day while reporting that they live and work in a city that does not have a metro system.2. Use only complete question texts and labeled answers
For the AI to detect contradictions effectively, it must be provided with the full question wording, and fully labeled answer options. Do not include questions that are only partially formulated (e.g. brand name is missing) or questions whose answer options are numeric codes (e.g. 0, 1) instead of explicit labels (not selected, selected).3. Account for the AI Model’s Knowledge Cutoff
Even the most recent AI models are trained on data that is not fully up to date. Avoid questions that rely on recent information, such as: “Which smartphone model do you currently use?” Recently released products may not yet exist in the model’s knowledge base and can therefore lead to unjustified Coherence Score deductions.4. Make Questionnaire Logic and Context Explicit
The Coherence Score is designed to minimize false positives and to account for plausible explanations. However, when contextual information is missing or implicit, misclassification becomes unavoidable. Clear and explicit context is therefore the single most effective way to maximize Coherence Score accuracy. The Coherence Score can only correctly interpret routing, filters, skips, and conditional logic if this information is clearly visible in the data provided. Ensure full contextual visibility by embedding answer options and explanatory notes directly into the question text. In addition, because the Coherence Score expects a list of actual question–answer pairs, include only questions that were actually shown to the respondent. If questions are skipped due to routing but still appear in the data, the Coherence Score may interpret this as inconsistency or inattention, resulting in a lower score.Grid Question Score
Do’s and Don’ts
To ensure the Grid Question Score functions reliably, please follow the recommendations below.1. Minimum requirements
To reliably detect click patterns, a grid question must contain at least seven rows and four columns. In general, the more rows and columns, the more reliable the results.2. Item arrangement
To avoid incorrectly penalizing legitimate response patterns, ensure that the item arrangement does not naturally encourage uniform answering. Avoid, for example, placing all positive items on one side of the scale and all negative items on the other, or formulating all statements exclusively in either a positive or a negative direction. In such cases, respondents with genuinely positive or negative attitudes toward the research object may be incorrectly flagged as straightliners simply because they consistently select the most positive or most negative response option. Instead, occasionally invert the direction of selected items so that consistently choosing the same scale point becomes implausible or logically contradictory, making true click patterns easier to identify.3. Scale
The Grid Question Score does not strictly require an interval scale. In certain cases, an ordinal scale may also be suitable. For example, in closed questions where respondents report how frequently they engage in specific behaviors, a uniform response pattern may be highly implausible. In such cases, grid questions using an ordinal scale can also be effectively analyzed.Time Score
Do’s and Don’ts
To ensure the Time Score functions reliably, please follow the recommendations below.1. Total Duration
Only use the total interview duration/length of interview (loi) as a datapoint if it is not significantly influenced by routing or filtering. If respondents see different numbers of questions, total duration becomes unreliable.2. Interruptions
When using total interview duration, ensure that it excludes interruptions. The measured time should reflect only the active time spent completing the interview, not breaks or pauses.3. Time per Question / Page
We strongly recommend not relying on total interview duration. Instead, use the response time per individual question or page that a respondent actually saw. This approach is more reliable because it is not affected by routing or filtering. In addition, fraudsters and automated bots can easily manipulate total completion time by waiting at the end of the questionnaire, whereas abnormally fast response times at the question level remain detectable. If response times for all individual questions are not available, use sections containing multiple questions that were answered by all respondents. Avoid basing the Time Score on only a few individual questions. Response-time variance at the question level is naturally higher, making results less reliable than when evaluating complete sections or the full set of questions.Open Ended Score
Do’s and Don’ts
To ensure the Open-Ended Score operates reliably, please follow the recommendations below.1. Obligatory questions
Make the open-ended questions used for the quality check mandatory. Fraudsters tend to answer only those questions they are required to answer. If open-ended questions are optional, low-quality or fraudulent respondents may skip them and remain undetected.2. Use questions with a defined context
Select open-ended questions or reformulate them so that the AI can clearly evaluate whether an answer fits the question. Avoid overly generic questions such as “What else do you want to tell us?” In this case, virtually any response would be valid and cannot be meaningfully evaluated. Instead, use clearly contextualized questions. For example: Rather than “What was the message of the commercial you have just seen?”, use “What was the message of the insurance commercial you have just seen?” Alternatively, add key words (e.g. “insurance”) in the keywords section of the open-ended check.3. Use unaided recall questions sparingly
Use unaided brand recall questions only if no other open-ended questions are available. If they refer to specific brands that are not globally known or are limited to certain countries, add contextual information. This includes specifying the relevant country and providing a few example brands in the keywords section of the open-ended check. Doing so helps the AI correctly interpret plausible answers.4. Use multiple open-ended questions
Include two or more open-ended questions for quality control. This improves overall reliability of the Open-Ended Score and enables to detect duplicates within interviews.5. Add open-ended questions
If a questionnaire contains no suitable open-ended questions, add at least one specifically for quality-check purposes. These questions do not need to be substantively analyzed; they are used solely to help identify inattentive or fraudulent interviews. Open-ended questions added for this purpose should be thematically aligned with the research topic.6. Duplicate check
Keep the duplicate answer and respondent check enabled in most cases. Deactivate it only if you expect very similar or identical answers within interviews or across respondents (for example, short factual answers or standardized phrases).If you have any questions, please contact [email protected]