12. Use Embeddings
Use embeddings¶
In this tutorial, we'll use product group embeddings to compare a customer's latest invoice with their past purchases from the last 26 weeks.
To learn how to create a SQL Embedding User-Defined Function (UDF), check out the 'Bring Your Own Transformer' tutorials.
For our hosted tutorials, we have pre-configured a SQL UDF using the SBERT Transformer model on our Snowflake data warehouse. We'll register this UDF in the Catalog and apply it to analyze the ProductGroup descriptions.
In [1]:
Copied!
import featurebyte as fb
# Set your profile to the tutorial environment
fb.use_profile("tutorial")
catalog_name = "Grocery Dataset Tutorial"
catalog = fb.Catalog.activate(catalog_name)
import featurebyte as fb
# Set your profile to the tutorial environment
fb.use_profile("tutorial")
catalog_name = "Grocery Dataset Tutorial"
catalog = fb.Catalog.activate(catalog_name)
10:54:12 | WARNING | Service endpoint is inaccessible: http://featurebyte-server:8088/ 10:54:12 | INFO | Using profile: tutorial 10:54:12 | INFO | Using configuration file at: /Users/gxav/.featurebyte/config.yaml 10:54:12 | INFO | Active profile: tutorial (https://tutorials.featurebyte.com/api/v1) 10:54:12 | INFO | SDK version: 2.0.1.dev67 10:54:12 | INFO | No catalog activated. 10:54:13 | INFO | Catalog activated: Grocery Dataset Tutorial 16:09:06 | INFO | Using configuration file at: /Users/gxav/.featurebyte/config.yaml 16:09:06 | INFO | Active profile: tutorial (https://tutorials.featurebyte.com/api/v1) 16:09:06 | WARNING | Remote SDK version (1.1.0.dev7) is different from local (1.1.0.dev1). Update local SDK to avoid unexpected behavior. 16:09:06 | INFO | No catalog activated. 16:09:06 | INFO | Catalog activated: Grocery Dataset Tutorial
Register the F_SBERT_EMBEDDING UDF to the Catalog¶
In [2]:
Copied!
fb.UserDefinedFunction.create(
name='embedding',
sql_function_name='F_SBERT_EMBEDDING',
function_parameters=[fb.FunctionParameter(name="x", dtype=fb.enum.DBVarType.VARCHAR)],
output_dtype=fb.enum.DBVarType.EMBEDDING,
is_global=False,
)
fb.UserDefinedFunction.create(
name='embedding',
sql_function_name='F_SBERT_EMBEDDING',
function_parameters=[fb.FunctionParameter(name="x", dtype=fb.enum.DBVarType.VARCHAR)],
output_dtype=fb.enum.DBVarType.EMBEDDING,
is_global=False,
)
Out[2]:
User Defined Function
name | embedding | ||||||||||
created_at | 2024-12-18 02:54:13 | ||||||||||
updated_at | None | ||||||||||
description | None | ||||||||||
sql_function_name | F_SBERT_EMBEDDING | ||||||||||
function_parameters |
|
||||||||||
signature | embedding(x: str) -> embedding | ||||||||||
output_dtype | EMBEDDING | ||||||||||
feature_store_name | playground | ||||||||||
used_by_features | [] |
Apply the embedding UDF instance to ProductGroup¶
In [3]:
Copied!
# Get embedding UDF instance.
embedding_udf = catalog.get_user_defined_function("embedding")
# Get embedding UDF instance.
embedding_udf = catalog.get_user_defined_function("embedding")
In [4]:
Copied!
# Get view from GROCERYPRODUCT dimension table.
groceryproduct_view = catalog.get_view("GROCERYPRODUCT")
# Apply embedding to ProductGroup column in GROCERYPRODUCT view.
groceryproduct_view["ProductGroup_embedding"] = embedding_udf(groceryproduct_view["ProductGroup"])
# Get view from GROCERYPRODUCT dimension table.
groceryproduct_view = catalog.get_view("GROCERYPRODUCT")
# Apply embedding to ProductGroup column in GROCERYPRODUCT view.
groceryproduct_view["ProductGroup_embedding"] = embedding_udf(groceryproduct_view["ProductGroup"])
Get other views¶
In [5]:
Copied!
# Get view from GROCERYINVOICE event table.
groceryinvoice_view = catalog.get_view("GROCERYINVOICE")
# Get view from INVOICEITEMS item table.
invoiceitems_view = catalog.get_view("INVOICEITEMS")
# Get view from GROCERYINVOICE event table.
groceryinvoice_view = catalog.get_view("GROCERYINVOICE")
# Get view from INVOICEITEMS item table.
invoiceitems_view = catalog.get_view("INVOICEITEMS")
Join views¶
In [6]:
Copied!
# Join GROCERYPRODUCT view to INVOICEITEMS view.
invoiceitems_view = invoiceitems_view.join(groceryproduct_view, rprefix="product_")
# Join GROCERYPRODUCT view to INVOICEITEMS view.
invoiceitems_view = invoiceitems_view.join(groceryproduct_view, rprefix="product_")
Get the mean vector of an invoice's Product Group descriptions¶
In [7]:
Copied!
# Group invoiceitems_view by invoice entity (GroceryInvoiceGuid).
invoiceitems_view_by_invoice = invoiceitems_view.groupby("GroceryInvoiceGuid")
# Group invoiceitems_view by invoice entity (GroceryInvoiceGuid).
invoiceitems_view_by_invoice = invoiceitems_view.groupby("GroceryInvoiceGuid")
In [8]:
Copied!
# Mean vector of product_ProductGroup_embedding for the invoice.
invoice_mean_vector_of_item_product_productgroup_embedding = invoiceitems_view_by_invoice.aggregate(
"product_ProductGroup_embedding", method=fb.AggFunc.AVG,
feature_name="INVOICE_Mean_vector_of_item_product_ProductGroup_embedding"
)
# Mean vector of product_ProductGroup_embedding for the invoice.
invoice_mean_vector_of_item_product_productgroup_embedding = invoiceitems_view_by_invoice.aggregate(
"product_ProductGroup_embedding", method=fb.AggFunc.AVG,
feature_name="INVOICE_Mean_vector_of_item_product_ProductGroup_embedding"
)
Get the mean vector of the Customer's latest invoice¶
In [9]:
Copied!
# Add INVOICE_Mean_vector_of_item_product_ProductGroup_embedding feature to the GROCERYINVOICE view as a column.
groceryinvoice_view = groceryinvoice_view.add_feature(
"INVOICE_Mean_vector_of_item_product_ProductGroup_embedding",
invoice_mean_vector_of_item_product_productgroup_embedding
)
# Add INVOICE_Mean_vector_of_item_product_ProductGroup_embedding feature to the GROCERYINVOICE view as a column.
groceryinvoice_view = groceryinvoice_view.add_feature(
"INVOICE_Mean_vector_of_item_product_ProductGroup_embedding",
invoice_mean_vector_of_item_product_productgroup_embedding
)
In [10]:
Copied!
# Group GROCERYINVOICE view by customer entity (GroceryCustomerGuid).
groceryinvoice_view_by_customer = groceryinvoice_view.groupby(['GroceryCustomerGuid'])
# Group GROCERYINVOICE view by customer entity (GroceryCustomerGuid).
groceryinvoice_view_by_customer = groceryinvoice_view.groupby(['GroceryCustomerGuid'])
In [11]:
Copied!
# Get Latest Mean vector of item product_ProductGroup_embedding for the customer
customer_latest_invoice_mean_vector_of_item_product_productgroup_embedding =\
groceryinvoice_view_by_customer.aggregate_over(
"INVOICE_Mean_vector_of_item_product_ProductGroup_embedding", method="latest",
feature_names=["CUSTOMER_Latest_INVOICE_Mean_vector_of_item_product_ProductGroup_embedding"],
windows=[None]
)["CUSTOMER_Latest_INVOICE_Mean_vector_of_item_product_ProductGroup_embedding"]
# Get Latest Mean vector of item product_ProductGroup_embedding for the customer
customer_latest_invoice_mean_vector_of_item_product_productgroup_embedding =\
groceryinvoice_view_by_customer.aggregate_over(
"INVOICE_Mean_vector_of_item_product_ProductGroup_embedding", method="latest",
feature_names=["CUSTOMER_Latest_INVOICE_Mean_vector_of_item_product_ProductGroup_embedding"],
windows=[None]
)["CUSTOMER_Latest_INVOICE_Mean_vector_of_item_product_ProductGroup_embedding"]
Get the mean vector for the Customer's Product Group descriptions over past 26 weeks¶
In [12]:
Copied!
# Group INVOICEITEMS view by customer entity (GroceryCustomerGuid).
invoiceitems_view_by_customer = invoiceitems_view.groupby(['GroceryCustomerGuid'])
# Group INVOICEITEMS view by customer entity (GroceryCustomerGuid).
invoiceitems_view_by_customer = invoiceitems_view.groupby(['GroceryCustomerGuid'])
In [13]:
Copied!
# Get Mean vector of product_ProductGroup_embedding for the customer over time.
feature_group = invoiceitems_view_by_customer.aggregate_over(
"product_ProductGroup_embedding", method="avg",
feature_names=[
"CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w"
],
windows=["26w"],
)
# Get CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w object from feature group.
customer_mean_vector_of_item_product_productgroup_embedding_26w =\
feature_group["CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w"]
# Get Mean vector of product_ProductGroup_embedding for the customer over time.
feature_group = invoiceitems_view_by_customer.aggregate_over(
"product_ProductGroup_embedding", method="avg",
feature_names=[
"CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w"
],
windows=["26w"],
)
# Get CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w object from feature group.
customer_mean_vector_of_item_product_productgroup_embedding_26w =\
feature_group["CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w"]
Derive Similarity between latest invoice and 26 weeks purchases¶
In [14]:
Copied!
# Derive Similarity feature from cosine similarity between
# CUSTOMER_Latest_INVOICE_Mean_vector_of_item_product_ProductGroup_embedding
# and CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice =\
customer_latest_invoice_mean_vector_of_item_product_productgroup_embedding.vec.cosine_similarity(
customer_mean_vector_of_item_product_productgroup_embedding_26w
)
# Give a name to new feature
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice.name = \
"CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w_vs_latest_invoice"
# Derive Similarity feature from cosine similarity between
# CUSTOMER_Latest_INVOICE_Mean_vector_of_item_product_ProductGroup_embedding
# and CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice =\
customer_latest_invoice_mean_vector_of_item_product_productgroup_embedding.vec.cosine_similarity(
customer_mean_vector_of_item_product_productgroup_embedding_26w
)
# Give a name to new feature
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice.name = \
"CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w_vs_latest_invoice"
Preview feature¶
In [15]:
Copied!
# Get observation table: 'Preview Table with 10 items'
preview_table = catalog.get_observation_table("Preview Table with 10 items")
# Get observation table: 'Preview Table with 10 items'
preview_table = catalog.get_observation_table("Preview Table with 10 items")
In [16]:
Copied!
# Preview customer_mean_vector_of_item_product_productgroup_embedding_26w
customer_mean_vector_of_item_product_productgroup_embedding_26w.preview(preview_table)
# Preview customer_mean_vector_of_item_product_productgroup_embedding_26w
customer_mean_vector_of_item_product_productgroup_embedding_26w.preview(preview_table)
Out[16]:
POINT_IN_TIME | GROCERYINVOICEITEMGUID | CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w | |
---|---|---|---|
0 | 2023-03-21 13:49:48 | 84225bbb-adb8-451e-98c0-897c83c2fad9 | [-0.053134586908131005, 0.038257234091453006, ... |
1 | 2023-03-07 17:53:55 | cb478a4e-9266-4523-8ee0-e205881cc5f5 | [-0.05567531036902901, 0.034300674646403004, -... |
2 | 2023-06-06 16:13:44 | bdc6b6bb-a23a-48c8-bb3a-622d9161c0e8 | [-0.052150058938217, 0.024087210394162003, -0.... |
3 | 2022-08-22 06:10:42 | 9267d0dd-9685-4667-8f06-1761abe73c4d | [-0.043878268099474003, 0.026860922980375002, ... |
4 | 2023-05-21 13:59:44 | 989643fe-1377-4f5f-8f38-f349a611da0c | [-0.066463998863818, 0.028307016924553, -0.011... |
5 | 2022-09-23 18:09:33 | 459a1b6e-1239-46d1-9e40-539c7e895483 | [-0.05568020304793601, 0.037980580083815, -0.0... |
6 | 2023-01-06 08:49:29 | dad86e21-3af4-4b5b-8058-60f946b6dac5 | [-0.056112752006346, 0.02670271428755, -0.0204... |
7 | 2022-07-23 17:33:25 | 849454c5-6640-419d-871d-1f0895a1c3b4 | [-0.049964407368356, 0.014002430175121, -0.008... |
8 | 2022-09-07 09:16:15 | 53eac49e-7ecd-4cb6-a1c0-38e3900efd7a | [-0.045220346770172004, 0.022812435395463, -0.... |
9 | 2022-12-30 16:42:05 | 4dd1487a-0379-4eef-b200-97ac1bb1164f | [-0.052932139672271006, 0.033954532991289005, ... |
In [17]:
Copied!
# Preview customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice.preview(preview_table)
# Preview customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice.preview(preview_table)
Out[17]:
POINT_IN_TIME | GROCERYINVOICEITEMGUID | CUSTOMER_Mean_vector_of_item_product_ProductGroup_embedding_26w_vs_latest_invoice | |
---|---|---|---|
0 | 2023-03-07 17:53:55 | cb478a4e-9266-4523-8ee0-e205881cc5f5 | 0.636492 |
1 | 2023-03-21 13:49:48 | 84225bbb-adb8-451e-98c0-897c83c2fad9 | 0.830780 |
2 | 2023-05-21 13:59:44 | 989643fe-1377-4f5f-8f38-f349a611da0c | 0.523136 |
3 | 2022-08-22 06:10:42 | 9267d0dd-9685-4667-8f06-1761abe73c4d | 0.621796 |
4 | 2023-06-06 16:13:44 | bdc6b6bb-a23a-48c8-bb3a-622d9161c0e8 | 0.883115 |
5 | 2022-09-23 18:09:33 | 459a1b6e-1239-46d1-9e40-539c7e895483 | 0.757104 |
6 | 2022-07-23 17:33:25 | 849454c5-6640-419d-871d-1f0895a1c3b4 | 0.700455 |
7 | 2023-01-06 08:49:29 | dad86e21-3af4-4b5b-8058-60f946b6dac5 | 0.905406 |
8 | 2022-09-07 09:16:15 | 53eac49e-7ecd-4cb6-a1c0-38e3900efd7a | 0.931595 |
9 | 2022-12-30 16:42:05 | 4dd1487a-0379-4eef-b200-97ac1bb1164f | 0.842517 |
Save feature¶
In [18]:
Copied!
# Save feature
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice.save()
# Save feature
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice.save()
Done! |████████████████████████████████████████| 100% in 6.1s (0.17%/s) Done! |████████████████████████████████████████| 100% in 6.1s (0.17%/s)
Add description¶
In [20]:
Copied!
# Add description
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice.update_description(
"Compare the customer's 4w Mean vector of item "
"product_ProductGroup_embedding with the customer's most recent "
"invoice. This comparison is done using the Cosine Similarity metric to"
" measure how similar these mean vector embeddings are."
)
# Add description
customer_mean_vector_of_item_product_productgroup_embedding_26w_vs_latest_invoice.update_description(
"Compare the customer's 4w Mean vector of item "
"product_ProductGroup_embedding with the customer's most recent "
"invoice. This comparison is done using the Cosine Similarity metric to"
" measure how similar these mean vector embeddings are."
)