7. Create Observation Tables

Create Observation Table¶

FeatureByte has two important concepts related to feature materialization:

• Observation Set - a collection that combines specific moments in history (timestamps) with related entity key values, used to determine feature values for those moments.
• Observation Table - representation of observation set in the feature store.

In this tutorial we will:

• create 2 observation tables for a use case requiring predictions about a grocery customer whenever an invoice event occurs. One will be used to preview features for the customer entity. One will be used to get historical features for training with the customer entity.
• compute values for the target we previously created.
• create one observation table at the item level that can be used to preview features for any entity.

In [1]:

import featurebyte as fb
import pandas as pd

# Set your profile to the tutorial environment
fb.use_profile("tutorial")

catalog_name = "Grocery Dataset Tutorial"
catalog = fb.Catalog.activate(catalog_name) 

groceryinvoice_view = catalog.get_view("GROCERYINVOICE")

import featurebyte as fb import pandas as pd # Set your profile to the tutorial environment fb.use_profile("tutorial") catalog_name = "Grocery Dataset Tutorial" catalog = fb.Catalog.activate(catalog_name) groceryinvoice_view = catalog.get_view("GROCERYINVOICE")

21:58:26 | INFO     | Using configuration file at: /Users/gxav/.featurebyte/config.yaml
21:58:26 | INFO     | Active profile: tutorial (https://tutorials.featurebyte.com/api/v1)
21:58:26 | WARNING  | Remote SDK version (0.5.0.dev6) is different from local (0.5.0.dev1). Update local SDK to avoid unexpected behavior.
21:58:26 | INFO     | No catalog activated.
21:58:26 | INFO     | 6 feature lists, 31 features deployed
21:58:26 | INFO     | Using profile: tutorial
21:58:27 | INFO     | Using configuration file at: /Users/gxav/.featurebyte/config.yaml
21:58:27 | INFO     | Active profile: tutorial (https://tutorials.featurebyte.com/api/v1)
21:58:27 | WARNING  | Remote SDK version (0.5.0.dev6) is different from local (0.5.0.dev1). Update local SDK to avoid unexpected behavior.
21:58:27 | INFO     | No catalog activated.
21:58:28 | INFO     | 6 feature lists, 31 features deployed
21:58:28 | INFO     | Catalog activated: Grocery Dataset Tutorial

Let's create 2 observation tables:

• one for preview (with the size of 10)
• one for training (with the size of 1K)

In [2]:

cond = (groceryinvoice_view["Timestamp"] >= pd.to_datetime("2022-07-01")) & (
    groceryinvoice_view["Timestamp"] < pd.to_datetime("2023-07-01")
)
groceryinvoice_1y_view = groceryinvoice_view[cond].copy()

for size in [10, 1000]:
    if size == 10:
        table_name = "Preview Table with 10 Customers"
    else:
        table_name = f"1K Customers at time of purchase 22S2-23S1"
    observation_table = groceryinvoice_1y_view.create_observation_table(
        name=table_name,
        sample_rows=size,
        columns=["Timestamp", "GroceryCustomerGuid"],
        columns_rename_mapping={
            "Timestamp": "POINT_IN_TIME",
            "GroceryCustomerGuid": "GROCERYCUSTOMERGUID",
        },
    )
    observation_table.update_description(
        f"{size} customers at time of purchase between 01-Jul-2022 and 30-Jun-2023"
    )

cond = (groceryinvoice_view["Timestamp"] >= pd.to_datetime("2022-07-01")) & ( groceryinvoice_view["Timestamp"] < pd.to_datetime("2023-07-01") ) groceryinvoice_1y_view = groceryinvoice_view[cond].copy() for size in [10, 1000]: if size == 10: table_name = "Preview Table with 10 Customers" else: table_name = f"1K Customers at time of purchase 22S2-23S1" observation_table = groceryinvoice_1y_view.create_observation_table( name=table_name, sample_rows=size, columns=["Timestamp", "GroceryCustomerGuid"], columns_rename_mapping={ "Timestamp": "POINT_IN_TIME", "GroceryCustomerGuid": "GROCERYCUSTOMERGUID", }, ) observation_table.update_description( f"{size} customers at time of purchase between 01-Jul-2022 and 30-Jun-2023" )

Done! |████████████████████████████████████████| 100% in 6.8s (0.15%/s)         
Done! |████████████████████████████████████████| 100% in 6.8s (0.15%/s)         

Add a target to observation tables¶

In [3]:

target = catalog.get_target("CUSTOMER_Sum_of_invoice_Amount_next_14d")

target = catalog.get_target("CUSTOMER_Sum_of_invoice_Amount_next_14d")

In [4]:

# get new observation table with the target populated
observation_table = catalog.get_observation_table("1K Customers at time of purchase 22S2-23S1")
target.compute_target_table(observation_table, "1K Customers Spending next 2 weeks at time of purchase 22S2-23S1")

# get new observation table with the target populated observation_table = catalog.get_observation_table("1K Customers at time of purchase 22S2-23S1") target.compute_target_table(observation_table, "1K Customers Spending next 2 weeks at time of purchase 22S2-23S1")

Done! |████████████████████████████████████████| 100% in 10.2s (0.10%/s)        

Out[4]:

Observation Table

name	1K Customers Spending next 2 weeks at time of purchase 22S2-23S1
created_at	2023-09-11 13:58:53
updated_at	None
description	None
type	observation_table
feature_store_name	playground
table_details	database_name TUTORIAL schema_name TUTORIAL_PROD table_name OBSERVATION_TABLE_64ff1d19ceb61c9075514a2e

In [5]:

training_table = catalog.get_observation_table(
    "1K Customers Spending next 2 weeks at time of purchase 22S2-23S1"
)
training_table.to_pandas().head()

training_table = catalog.get_observation_table( "1K Customers Spending next 2 weeks at time of purchase 22S2-23S1" ) training_table.to_pandas().head()

Downloading table |████████████████████████████████████████| 1000/1000 [100%] in

Out[5]:

	POINT_IN_TIME	GROCERYCUSTOMERGUID	CUSTOMER_Sum_of_invoice_Amount_next_14d
0	2022-07-06 10:33:23	ef39897d-3562-4b1d-aa0e-3398d4e62084	134.35
1	2022-12-09 14:22:05	ef39897d-3562-4b1d-aa0e-3398d4e62084	161.83
2	2022-09-13 16:33:51	b407e089-7b44-4493-ac17-afe3a7ace716	164.24
3	2022-11-26 05:52:26	b407e089-7b44-4493-ac17-afe3a7ace716	43.52
4	2022-12-26 10:06:56	b407e089-7b44-4493-ac17-afe3a7ace716	10.99

Create Preview observation table at the item level¶

This observation table can be used to materialize target, features and feature lists with primary entities that are parents of the item entity. This includes:

• item
• product
• productgroup
• invoice
• customer
• frenchstate
• customer, product
• ...

In [6]:

invoiceitems_view = catalog.get_view("INVOICEITEMS")

invoiceitems_view = catalog.get_view("INVOICEITEMS")

In [7]:

# filter the view to extract items during a specific period of time
cond = (invoiceitems_view["Timestamp"] >= pd.to_datetime("2022-07-01")) & (
    invoiceitems_view["Timestamp"] < pd.to_datetime("2023-07-01")
)
invoiceitems_1y_view = invoiceitems_view[cond].copy()

observation_table = invoiceitems_1y_view.create_observation_table(
    name=f"Preview Table with 10 items",
    sample_rows=10,
    columns=["Timestamp", "GroceryInvoiceItemGuid"],
    columns_rename_mapping={
        "Timestamp": "POINT_IN_TIME",
        "GroceryInvoiceItemGuid": "GROCERYINVOICEITEMGUID",
    },
)
observation_table.update_description(
    f"Preview Table with 10 items between 01-Jul-2022 and 30-Jun-2023"
)

# filter the view to extract items during a specific period of time cond = (invoiceitems_view["Timestamp"] >= pd.to_datetime("2022-07-01")) & ( invoiceitems_view["Timestamp"] < pd.to_datetime("2023-07-01") ) invoiceitems_1y_view = invoiceitems_view[cond].copy() observation_table = invoiceitems_1y_view.create_observation_table( name=f"Preview Table with 10 items", sample_rows=10, columns=["Timestamp", "GroceryInvoiceItemGuid"], columns_rename_mapping={ "Timestamp": "POINT_IN_TIME", "GroceryInvoiceItemGuid": "GROCERYINVOICEITEMGUID", }, ) observation_table.update_description( f"Preview Table with 10 items between 01-Jul-2022 and 30-Jun-2023" )

Done! |████████████████████████████████████████| 100% in 6.7s (0.15%/s)         

In [8]:

# preview target with the preview table we just created
target.preview(observation_table.to_pandas())

# preview target with the preview table we just created target.preview(observation_table.to_pandas())

Downloading table |████████████████████████████████████████| 10/10 [100%] in 0.1

Out[8]:

	POINT_IN_TIME	GROCERYINVOICEITEMGUID	CUSTOMER_Sum_of_invoice_Amount_next_14d
0	2022-07-17 08:44:18	15791978-119b-4ba7-aaac-9f558c29fd21	54.14
1	2022-07-23 14:44:03	d29f9a20-bbb3-489f-893c-bda11c3983e7	65.12
2	2022-09-09 18:50:53	fe1d3b61-ae8a-4f47-8a0e-35a217e16474	0.00
3	2022-10-30 13:09:35	aa1cbf16-f3f2-45b2-ae16-928534a0418d	22.92
4	2022-11-01 09:03:04	a1cd660a-d25f-4436-b7d2-59ff4b0cec93	0.00
5	2022-11-18 15:18:22	a6a06ac6-6da1-4d8b-be91-5f89906195e2	0.00
6	2022-12-12 18:24:29	24e3c6b4-93ac-4ecb-95c0-b8b6a2036bbb	0.00
7	2023-01-11 12:49:07	234180c7-b12f-490f-8247-4fd143d55e59	92.67
8	2023-02-24 19:09:19	01e34a6b-a07d-48fa-b81d-c33f9f295a3a	123.54
9	2023-03-08 13:02:36	ff51f5f0-79a6-4e55-9efe-9652ea744714	3.00

List observation tables in catalog¶

In [9]:

catalog.list_observation_tables()

catalog.list_observation_tables()

Out[9]:

	id	name	type	shape	feature_store_name	created_at
0	64ff1d2b66704a9790300a44	Preview Table with 10 items	view	[10, 2]	playground	2023-09-11T13:59:11.124000
1	64ff1d18c0038ba1e425262a	1K Customers Spending next 2 weeks at time of ...	observation_table	[1000, 3]	playground	2023-09-11T13:58:53.315000
2	64ff1d0f66704a9790300a42	1K Customers at time of purchase 22S2-23S1	view	[1000, 2]	playground	2023-09-11T13:58:43.075000
3	64ff1d0666704a9790300a41	Preview Table with 10 Customers	view	[10, 2]	playground	2023-09-11T13:58:33.781000

Concepts in this tutorial¶

• Observation Set
• Serving Entity
• Feature Primary Entity
• Feature List Primary Entity

SDK reference for¶

• Observation Table
• View.create_observation_table()

In [ ]: