Skip to content

7. Create Observation Tables

What is an Observation Table?

An Observation Table is a structured collection of historical data points that acts as the foundation for training datasets. By adding features, you can create Feature Tables that can be used to train and validate Machine Learning models.

Each data point represents a specific historical moment for a particular entity and may also include target values. Observation Tables are often utilized across experiments within the same use case, even if selected features and models vary.

How to create an Observation Table?

You can either upload an Observation Table from a parquet or csv file or create one from a Source Table or an existing observation table.

This guide explains how to configure Observation Tables from a Source Table, derive Observation Table from existing one and link them to our Credit Default context and use case.

We will first create two Observation Tables:

  1. Applications with Credit Default target: Credit Default Observations with information whether the loan default the next 6 months.
  2. 50K applications: Credit Default Observations for EDA .

Then we will derive three new Observation Tables:

  1. Applications up to Dec 2024: Credit Default Observations for training up to Dec 2024.
  2. Applications Q1 2025: Credit Default Observations for validation (Q1 2025).
  3. Applications Preview: 50 Credit Default Observations for Feature PREVIEW.

Step 1: Navigate to Observation Table Catalog

From the menu, navigate to the 'Formulate' section:

  1. Select the Observation Table catalog. Empty Observation Table Catalog

  2. Select the Use Case or Context for which you want to create observation tables. Select Use Case

Step 2: Create Observation Tables from a Source Table

  1. Click Image.

  2. Select 'Derive from Source Table' tab and click Image Name

  3. In the Source Table listing, select OBSERVATIONS_WITH_TARGET under the DEMO_DATASETS database and the CREDIT_DEFAULT schema and click Image. Name

  4. Set the table as follows and click Image to save the table: Name

    • Name: "Applications with Credit Default target"
    • Description: "Credit Default Observations with information whether the loan default the next 6 months."
    • Purpose: Training
    • Sample Rows: 0 (to disable sampling)
    • Sampling Date Range: leave it blank as we will do time partitioning later

    • Columns to Include:

      1. Original Column Name: POINT_IN_TIME --> New Column Name: POINT_IN_TIME
      2. Original Column Name: SK_ID_CURR --> New Column Name: SK_ID_CURR
      3. Original Column Name: Loan_Default --> New Column Name: Loan_Default (as Target)

  5. Select OBSERVATION_EDA_TABLE and click Image. Name

  6. Set the table as follows and click Image to save the table: Name

    • Name: "50K applications"
    • Description: "Credit Default Observations for EDA."
    • Purpose: EDA
    • Sample Rows: 0 (to disable sampling)
    • Sampling Date Range: leave it blank as we will do time partitioning later

    • Columns to Include:

      1. Original Column Name: POINT_IN_TIME --> New Column Name: POINT_IN_TIME
      2. Original Column Name: SK_ID_CURR --> New Column Name: SK_ID_CURR
      3. Original Column Name: Loan_Default --> New Column Name: Loan_Default (as Target)

Step 3: Derive Training and Validation tables

  1. Navigate to the Observation Table catalog under the Formulate section: Name

  2. Select the Applications with Credit Default target table and go the About tab. Scroll down and click Image Name

  3. Create Applications up to Dec 2024 and click Image. Name

    • Name: "Applications up to Dec 2024"
    • Description: "Credit Default Observations for training up to Dec 2024."
    • Purpose: Training
    • Sample Rows: 0
    • Sampling Date Range: April 1, 2019 - Jan 1, 2025

  4. Select again the Applications with Credit Default target table and create Applications Q1 2025: Name

    • Name: "Applications Q1 2025"
    • Description: "Credit Default Observations for validation (Q1 2025)"
    • Purpose: Validation-Test
    • Sample Rows: 0
    • Sampling Date Range: January 1, 2025 - April 1, 2025

Step 4: Derive Preview table (Optional)

  1. From the Observation Table catalog, select the 50K Applications table and go the About tab. Scroll down and click Image

  2. Create Applications Preview and click Image. Name

    • Name: "Applications Preview"
    • Description: "Credit Default Observations for preview."
    • Purpose: Preview
    • Sample Rows: 50

Step 5: Check Observation Tables

Check successful registration by reviewing the Observation Table Catalog.

Name

Step 6: Check the Use Case

  1. Navigate to the Use Case Catalog (you may to refresh the browser) and select the "Loan Default by client" use case.

    Name

  2. In the 'About' tab, review the linked Observation Tables.

    Name