Skip to content

8. Ideate Features

FeatureByte provides two primary methods for feature creation:

  • Manual Creation: Utilize the SDK declarative framework for complete control over feature engineering.
  • Automatic Creation: Accelerate feature engineering through Feature Ideation an AI-powered, automated approach.

How FeatureByte Ideates Features

Feature Ideation uses an agentic approach to tailor features to your use case. It can:

  1. Analyze tables and relationships to identify relevant data.
  2. Fill in missing semantic tags using column metadata.
  3. Suggest useful column transformations, such as time deltas, ratios, and differences.
  4. Recommend key filters to isolate important events.
  5. Highlight key columns for advanced feature engineering.
  6. Propose relevant time windows for feature aggregation.
  7. Analyze event frequencies to identify timing signals.
  8. Recommend features and evaluate their semantic relevance to your use case.
  9. Ensure feature reuse by detecting and avoiding duplicate features in the Catalog.

Each step is documented, ensuring complete transparency and traceability throughout the process

Modes of Operation

Feature Ideation offers two modes of operation:

  • Fully automated mode: Automatically executes the entire workflow.
  • Semi-automated mode: Enables you to review and refine recommendations step by step.

After ideating features, leverage exploratory data analysis (EDA) and advanced feature selection techniques to finalize your feature set.

This tutorial demonstrates the fully automated mode of Feature Ideation. A detailed guide to the semi-automated mode will follow in the next section.

Note

If you want to learn how to manually create features, please consult our SDK tutorials.

Step 1: Select Your Use Case

  1. Navigate to Feature Ideation from the 'Ideate' section of the menu.

  2. Select the use case: "Loan Default by client".

Name

Step 2: Initiate a New Feature Ideation Workflow

  1. Click New Ideation Button to start the Feature Ideation process. Name

  2. Edit the Feature Ideation name and description by clicking Edit Button.

    Name

Step 3: Start Automated Mode

  1. Begin the automated Feature Ideation workflow by clicking Auto Run Button.

  2. Optionally, you can stop the process at any stage if needed.

    Name

Once the process is initiated, you’ll see confirmation that the run has started:

Name

Step 4: Review the Feature Ideation Report

After the process completes, a table of ideated features will be displayed for your review.

Name

  1. Access the Detailed Report, describing each step of the ideation process, by clicking Report Button next to the Ideation name "Automated Mode".

    Name

  2. Visualize the full report with an indexed view in a new tab, by clicking Report tab Button.

    Name

Step 5: Run EDA

  1. Select All Ideated Features by clicking Select All Button.

    Name

  2. Initiate EDA. Scroll to the bottom of the ideated features table and click EDA Button to begin the Exploratory Data Anaylsis (EDA) process.

    Name

  3. Sort by Predictive Score. After EDA completes, each feature will have a univariate Predictive Score. Sort the features by predictive score to identify those with the highest predictive power.

    Name

Step 6: View Detailed EDA for a Feature

To explore the EDA results for a specific feature:

  1. Click Filter Button to filter the list of features.

  2. Select the filter criteria: In our example, "PRIOR_APPLICATIONS" for the Primary Table and "most_frequent" for the Signal Type.

    Name

  3. Click on the feature name to open its details (here: 'CLIENT_Prior_Application_PRODUCT_COMBINATION_with_Highest_sum_of_Prior_Applications_AMT_CREDITs_104w'), then navigate to the 'EDA' tab.

    Name

  4. Interact with EDA Plots. Within the 'EDA'' tab, click on the plot to activate tooltips for additional insights.

    Name

  5. Clear Filters and Close the Panel by clicking Clear Filter Button to remove the filter and Close Filter Button to close the filter panel in Close Filter Button.

Step 7: Run Feature Selection

  1. Start Feature Selection by clicking on the Magic Wand Magic Wand.

  2. Select the SHAP-Based mode.

    Screen candidates

    1. Set the pre-filtering options:

      • No. of Top Features Overall: 2000
      • No. of Top Features per Theme: None

    2. Choose the options to:

      • exclude Low Added Value Features
      • but keep Dictionary and Vector Features.

    Obtain a Small but Predictive Feature Selection

    1. Number of l1 rounds = 1:

      • The round will train a LGBM and applies L1 regularization on its SHAP values
      • This eliminates features with minimal contribution or high collinearity.

    2. Number of importance rounds to 1:

      • The round trains a LGBM and keep only top-performing features that meet the SHAP importance threshold of 0.95.

    Name

    Name

  3. Once the selection is complete, ensure that the new feature selection is properly selected..

    Name

  4. For more details, navigate to the 'Feature Selection' tab and click on the selection.

    Name

    Name

    Name

Step 8: Review Features

Review Individual Features

  1. Click on a feature to open its details. You can use the filter or the search, to find a specific feature.

  2. Check Semantic Relevance in the 'About' tab of the feature.

    Name

  3. Explore Feature Lineage by switching to the 'Lineage' tab. Click Lineage Button to trace the feature's origin and transformations.

    Name

    Name

  4. Analyze Feature Distribution and its relationship with the Target in the 'EDA' tab.

    Name

Review SDK Code for Feature Selection

  1. Clear the filter or search (if you used it).

  2. Ensure the feature selection is properly selected.

    Name

  3. Select features in the feature list by clicking Select All Button.

    Name

  4. Download the notebook to inspect the feature declaration code by clicking Download Notebook.

    Name

  5. Open the notebook to review the feature engineering code.

    Name

Step 9: Add Features to the Feature Catalog

  1. Follow the same steps as in Review SDK Code for Feature Selection.

  2. Instead of clicking Download Notebook, click Save Feature List.

  3. Call the feature list "SHAP selection from Automated Mode".

    Name

After completion, the selected features are added to the catalog, and their readiness status updates to 'DRAFT' instead of 'NEW'.

Name