> ## Documentation Index
> Fetch the complete documentation index at: https://docs.redem.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Cleaning

> Discover how ReDem automates and simplifies data cleaning.

## Why data cleaning matters?

Data cleaning helps ensure your results are based on valid, high-quality responses. ReDem automates this process to help you detect and exclude unreliable data.

ReDem’s data cleaning feature automates and streamlines this process, providing a standardized and transparent approach grounded in ReDem’s comprehensive evaluation framework.

## How Cleaning Works

Every respondent evaluated by ReDem undergoes a series of quality checks. These checks generate data points, classification labels and scores, which form the basis of the cleaning logic. When cleaning, you define what is acceptable or unacceptable by setting thresholds for these elements.

**OR Condition:**

* All cleaning options operate as OR conditions.
* You must select at least one score as a cleaning condition - usually the ReDem Score, since it is a comprehensive metric covering all selected quality checks.
* You may also add other scores in an OR condition if needed.

<Note> **Example:** <br /> If you set an R-Score threshold of 60, all respondents below 60 are flagged. If you also apply an OR condition for Time Score \< 30, then even respondents with a valid R-Score (e.g., 70) are flagged if their Time Score is below 30 (speeding).</Note>

**Data Points:** <br />
Data points indicate the number of measurements used to calculate a score.

<Note> **Example:** <br /> If you set the Open-Ended Score threshold to 40 with two data points, then any interview with at least two open-ended responses and an overall Open-Ended Score below 40 is excluded. </Note>

**Default Settings:** <br />
To simplify the process, ReDem provides recommended default settings that work well for many projects. You can, however, adjust them to match the specific needs of your study. The next two sections first describe the default settings and then explain how to select your own.

### ReDem Recommended Cleaning Settings

Our default settings apply best-practice thresholds to the following metrics:

* **ReDem Score (R-Score)**: Respondents with an R-Score below 60 are excluded.
* **Open-Ended Score (OES) & Response Categories:**
  Respondents with an OES below 40 are excluded if they provide at least two open-ended responses.

  Respondents flagged in at least two open-ended responses for wrong language, bad language, AI suspect, gibberish or off topic are excluded.
* **Coherence Score (CHS)**: Respondents with a CHS below 30 are excluded.
* **Grid-Question Score (GQS)**: Respondents with a GQS below 20 and at least two valid grid-question responses are excluded.
* **Time Score (TS)**: Respondents with a TS below 30 are excluded.
* **Behavioral Analytics Score (BAS)**: Respondents with a BAS below 20 and at least two valid BAS data points are excluded.

### Custom Cleaning Settings

You can define your own thresholds for each quality metric. This enables fine-tuned control over what qualifies as low-quality data based on your specific needs.

Customizable elements include:

* **ReDem Score (R-Score):** Threshold
* **Open-Ended Score (OES):** Threshold + min. number of open-ended responses + category-based exclusion logic
  * **Open-Ended Response Categories:**
    * **Bad Language:**
    * **No Answer:**
    * **Duplicate Respondent:**
    * **Duplicate Answer:**
    * **Gibberish:**
    * **Wrong Language:**
    * **Off Topic:**
    * **AI Suspect:**
* **Time Score (TS):** Threshold + min. number of time data points
* **Grid-Question Score (GQS):** Threshold + min. valid grid questions
* **Coherence Score (CHS):** Threshold + min. number of coherence data points
* **Behavioral Analytics Score (BAS):** Threshold + min. valid BAS data points + category-based exclusion logic
  * **BAS Categories:**
    * **Unnatural Typing:**
    * **Copy and Paste:**

Here’s an example of a cleaning settings object that can be used to customize the cleaning process via the API:

```javascript theme={null}
  "redemScore": 60,
  "OES": {
    "activate": true,
    "score": 60,
    "minDataPoints":2,
    "categories": {
      "NO_ANSWER": {"activate": true, "minDataPoints":3},
      // ... other categories ...
    }
  },
  "CHS": {"activate": true,"score": 50},
  "GQS": {"activate": true,"score": 40, "minDataPoints":2},
  "TS": {"activate": true,"score": 30},
  "BAS": {
    "activate": true,
    "score": 60, 
    "minDataPoints":2,
    "categories": {
      "UNNATURAL_TYPING": {"activate": true, "minDataPoints":2},
      "COPY_AND_PASTE": {"activate": true, "minDataPoints":2},
    }
  }
```

## Changing cleaning settings

For **imported** and **live** (API-connected) projects, changing cleaning settings in the ReDem app works the same way: open the survey **results** page, use **Cleaning Settings**, update the thresholds, then **Apply Cleaning** to reprocess and update exclusions for respondents already in the project.

For **live surveys** that are still in the field, you should also update the **programming or settings in your survey platform** so it stays aligned with your chosen rules. Integrations send **cleaning settings per respondent** with each submission (for example in the `cleaningSettings` field of the addRespondent request). ReDem applies the settings included in **each** request to that respondent. **New respondents** therefore follow whatever you send on each call—if you only change settings inside the ReDem app but not in your fieldwork script, new completes may still be sent with the old `cleaningSettings` until you update the integration.

## What Is the Outcome of the Cleaning Process?

The cleaning process classifies each response as either **Included** or **Excluded**, with clear reasons provided for exclusions. Only one exclusion condition needs to be met for a respondent to be removed.

**Reasons for Exclusion** (only one needs to be true):

* **ReDem Score Threshold**: Respondent’s ReDem Score is below the default (60) or a custom threshold.
* **Open-Ended Score Threshold**: Respondent’s OES is below the default (40) or a custom threshold.
* **Open Ended Category**: Respondent exceeds the defined category threshold.
* **Time Score Threshold**: Respondent’s TS is below the default (30) or a custom threshold.
* **Grid-Question Score Threshold**: Respondent’s GQS is below the default (20) or a custom threshold.
* **Coherence Score Threshold**: Respondent’s CHS is below the default (30) or a custom threshold.
* **Behavioral Analytics Score Threshold**: Respondent’s BAS is below a custom threshold.
* **Behavioral Analytics Category**: Respondent exceeds the defined BAS category threshold.

This structured reasoning provides clear insights into exclusions, empowering users to refine their criteria based on the analysis.

## View Exclusion Reason Breakdown

The exclusion reason breakdown shows how many excluded respondents were removed for each cleaning criterion. It is available at two levels:

* **Company level** on the **Surveys** page: in the metrics row, click the **info icon** next to **Excluded Respondents** when at least one respondent is excluded. **Employees** see a breakdown across all surveys they have access to; **Admins** see a breakdown for the whole company.
* **Survey level** on the survey **results** page: in the **ReDem Score** card, click the **info icon** next to **Excluded Respondents** when at least one respondent in that survey is excluded.

The breakdown includes:

* A chart showing the share of exclusions per main category (for example, ReDem Score threshold, Open-Ended Score threshold, Coherence Score threshold)
* A table with the count and percentage for each category
* Expandable rows for **Open-Ended** and **BAS** category breakdowns (for example, AI Suspect, Copy and Paste, Unnatural Typing)

<Steps titleSize="h3">
  <Step title="Open the Surveys page (company level)">
    Go to **Surveys**. In the metrics row at the top, find **Excluded Respondents**. If any respondents are excluded, click the **info icon** next to that label.
  </Step>

  <Step title="Open the survey results page (survey level)">
    Go to **Surveys** and open the survey you want to review.
  </Step>

  <Step title="Open the exclusion breakdown">
    In the **ReDem Score** card, find **Excluded Respondents**. If any respondents are excluded, click the **info icon** next to that label.
  </Step>

  <Step title="Review the breakdown">
    In the dialog, review the chart and table. Expand a category row to see sub-categories where available.
  </Step>
</Steps>

Per-respondent exclusion reasons are still available in the respondent table (**Reasons for Exclusion** column) and in each respondent's **Cleaning** tab. For a plain-language AI explanation of those reasons (table, details, and download), see [Respondent Insights](/features/respondent-insights#ai-generated-respondent-insights).

## Example of How the Cleaning Process Works

As an example, let's consider the cleaning settings applied to a specific respondent:

```javascript theme={null}
  "redemScore": 60,
  "OES": {
    "activate": true,
    "score": 60,
    "minDataPoints":2,
    "categories": {
      "NO_ANSWER": {"activate": true, "minDataPoints":2},
      "BAD_LANGUAGE": {"activate": false, "minDataPoints":2},
      "GIBBERISH": {"activate": false, "minDataPoints":2},
      "DUPLICATE_ANSWER": {"activate": false, "minDataPoints":2},
      "DUPLICATE_RESPONDENT": {"activate": false, "minDataPoints":2},
      "OFF_TOPIC": {"activate": false, "minDataPoints":2},
      "WRONG_LANGUAGE": {"activate": false, "minDataPoints":2},
      "AI_SUSPECT": {"activate": true, "minDataPoints":2}
    }
  }
```

To better understand how cleaning settings are applied, let's consider different cases of respondent data and determine whether they should be excluded and what should be the reason for exclusion.

<AccordionGroup>
  <Accordion title="Respondent with Low ReDem Score and Low OES Score">
    **Input:**

    The respondent has a **ReDem Score of 50** and an **OES Score of 60**. They have provided **4 valid answers for OES data points**, **`3 of which are categorized as AI_SUSPECT`**.

    **Output:**

    The respondent is **excluded** because their **ReDem Score falls below the threshold**. Additionally, their **OES Score is below the defined threshold**, and they have provided **more than 2 valid answers for OES data points**, with **`3 categorized as AI_SUSPECT`**, exceeding the threshold set in the cleaning settings.
  </Accordion>

  <Accordion title="Respondent with Low OES Score">
    **Input:**

    The respondent has a **ReDem Score of 70** and an **OES Score of 30**. They have provided **2 valid answers for OES data points**.

    **Output:**

    The respondent is **excluded** because their **Open-Ended Score is below the threshold** and they have provided **2 valid answers for OES data points**.
  </Accordion>

  <Accordion title="Respondent with more `AI_SUSPECT` responses than the threshold">
    **Input:**

    The respondent has a **ReDem Score of 70** and an **OES Score of 50**. They have provided **4 valid answers for OES data points**, with **`3 categorized as AI_SUSPECT`**.

    **Output:**

    The respondent is **excluded** despite their **ReDem Score and Open-Ended Score being above the threshold**, because **`more than 2 of their answers are categorized as AI_SUSPECT`**, exceeding the threshold defined in the cleaning settings.
  </Accordion>

  <Accordion title="Respondent who should not be excluded">
    **Input:**

    The respondent has a **ReDem Score of 70** and an **OES Score of 30**. They have provided only **one valid OES data point**.

    **Output:**

    The respondent should **not be excluded** because they **do not meet the minimum valid OES data points requirement for cleaning**, even though their **OES Score is below the threshold**.
  </Accordion>
</AccordionGroup>