Skip to content

7b. Create Development Dataset

What is a Development Dataset?

A Development Dataset is a collection of source tables that serve as substitutes for production source tables. It is used during feature ideation to accelerate exploratory data analysis (EDA) and feature selection. Development datasets are especially valuable when the original tables are extremely large, and only a subset of the data is needed for analysis.

How to create a Development Dataset ?

You can create a Development Dataset in two ways:

  • Manually: by mapping production tables to smaller, existing development tables.
  • Automatically: from the EDA observation table of a Use Case, combined with a feature lookback to ensure sufficient history for feature aggregation.

This guide explains how to create a Development Dataset from a Use Case's EDA Observation Table. We will create a Development Dataset from 50K applications, the EDA table created earlier for Loan Default by client.

Step 1: Navigate to Development Dataset Catalog

From the menu, go to the Formulate section and select the Development Dataset catalog.

Empty Development Dataset Catalog

Step 2: Create Development Dataset from an Observation Table

  1. Click Image.
  2. Select the Create New Tables tab and set the Use Case as Loan Default by Client. Name
  3. Review the suggested settings. In this example, we choose to set the Feature Lookback to 25 months. Name
  4. Click Image. This will create a Development Dataset with the status DRAFT. Name

What is Entity Selection in Settings?

Entity Selection is automatically suggested based on your use case. The selection defines, per table, the analysis level of features that may be generated during feature ideation. In most cases, the entity of the use case is recommended, or one of its parent entities if the entity can not be joined to the table.

You can extend the selection to any eligible parent entities. This may result in additional features being generated, including similarity features. In this example, no additional parent entities can be selected.

Step 3: Review the SQL to create Distinct IDs tables.

  1. Click on the Development Dataset and navigate to the SQL Plan tab. Name

  2. Review the SQL that generates Distinct IDs tables.

  3. These tables are used to materialize samples of the source tables.

  4. The complexity of the SQL script may vary depending on your data model and the entity selection.

  5. After review, click Image next to the Draft status. This will compute the Sample-to-Full ratio for each table, helping you decide whether to create new development tables.

Step 4: Review the Plan to Materialize Development Tables.

  1. Navigate to the Distinct IDs Tables tab and review the tables and their lineage. Name

  2. Go to the Development Tables tab and review the plan. In this case, all materializations are disabled by default because the expected Sample-to-Full ratio for all tables is greater than 5%. Name

  3. Navigate to the Settings tab, set Max Sample-to-Full Ratio to 0.15 (15%), and click Image to activate materialization for all tables. Name

  4. Return to the Development Tables tab and review the updated plan. Name

  5. Go back to the SQL Plan tab to review the updated SQL script. Name

  6. After review, click Image next to the Entity Sampling status.

  7. Confirm the materialization settings. Name

Step 5: Review the Development Dataset

  1. Navigate to the Development Tables tab and review the development tables. Name

  2. Click on one development table to view more details. Name

You’ve successfully created a Development Dataset and it’s ready to use for feature ideation!