11. Derive Similarity Features from Bucketing

Create similarity features¶

In this tutorial we will look into another way of deriving features from other features - creating similarity features.

We will create a feature that compares the customer purchase patterns across product groups to all customers purchase patterns. We will first do bucketing at the customer level and at the overall level. Then we will compare the 2 dictionaries with the cosine similarity.

In [1]:

import featurebyte as fb

# Set your profile to the tutorial environment
fb.use_profile("tutorial")

catalog_name = "Grocery Dataset Tutorial"
catalog = fb.Catalog.activate(catalog_name)

import featurebyte as fb # Set your profile to the tutorial environment fb.use_profile("tutorial") catalog_name = "Grocery Dataset Tutorial" catalog = fb.Catalog.activate(catalog_name)

16:08:44 | WARNING  | Service endpoint is inaccessible: http://featurebyte-server:8088
16:08:44 | INFO     | Using profile: tutorial
16:08:44 | INFO     | Using configuration file at: /Users/gxav/.featurebyte/config.yaml
16:08:44 | INFO     | Active profile: tutorial (https://tutorials.featurebyte.com/api/v1)
16:08:44 | WARNING  | Remote SDK version (1.1.0.dev7) is different from local (1.1.0.dev1). Update local SDK to avoid unexpected behavior.
16:08:44 | INFO     | No catalog activated.
16:08:44 | INFO     | Catalog activated: Grocery Dataset Tutorial

In [2]:

# Get view from GROCERYPRODUCT dimension table.
groceryproduct_view = catalog.get_view("GROCERYPRODUCT")
# Get view from INVOICEITEMS item table.
invoiceitems_view = catalog.get_view("INVOICEITEMS")

# Get view from GROCERYPRODUCT dimension table. groceryproduct_view = catalog.get_view("GROCERYPRODUCT") # Get view from INVOICEITEMS item table. invoiceitems_view = catalog.get_view("INVOICEITEMS")

Join views¶

In [3]:

# Join GROCERYPRODUCT view to INVOICEITEMS view.
invoiceitems_view = invoiceitems_view.join(groceryproduct_view, rsuffix="")

# Join GROCERYPRODUCT view to INVOICEITEMS view. invoiceitems_view = invoiceitems_view.join(groceryproduct_view, rsuffix="")

Create distribution features across |product groups¶

In [4]:

# Group INVOICEITEMS view by customer entity (GroceryCustomerGuid) across different ProductGroups.
invoiceitems_view_by_customer_across_productgroup =\
invoiceitems_view.groupby(
    ['GroceryCustomerGuid'], category="ProductGroup"
)

# Group INVOICEITEMS view by customer entity (GroceryCustomerGuid) across different ProductGroups. invoiceitems_view_by_customer_across_productgroup =\ invoiceitems_view.groupby( ['GroceryCustomerGuid'], category="ProductGroup" )

In [5]:

# Create Buckets representing the cumulative TotalCost of item, categorized by ProductGroup,
# for the customer over the past 26 weeks.
# The result is presented as a dictionary where the ProductGroup serves as the key
# and its corresponding sum of item TotalCost for the customer forms the value.
feature_group =\
invoiceitems_view_by_customer_across_productgroup.aggregate_over(
    "TotalCost", method=fb.AggFunc.SUM,
    feature_names=[
        "CUSTOMER_item_TotalCost_across_product_ProductGroups_26w"
    ],
    windows=["26w"]
)
# Get CUSTOMER_item_TotalCost_across_product_ProductGroups_26w object from feature group.
customer_item_totalcost_across_product_productgroups_26w =\
feature_group["CUSTOMER_item_TotalCost_across_product_ProductGroups_26w"]

# Create Buckets representing the cumulative TotalCost of item, categorized by ProductGroup, # for the customer over the past 26 weeks. # The result is presented as a dictionary where the ProductGroup serves as the key # and its corresponding sum of item TotalCost for the customer forms the value. feature_group =\ invoiceitems_view_by_customer_across_productgroup.aggregate_over( "TotalCost", method=fb.AggFunc.SUM, feature_names=[ "CUSTOMER_item_TotalCost_across_product_ProductGroups_26w" ], windows=["26w"] ) # Get CUSTOMER_item_TotalCost_across_product_ProductGroups_26w object from feature group. customer_item_totalcost_across_product_productgroups_26w =\ feature_group["CUSTOMER_item_TotalCost_across_product_ProductGroups_26w"]

In [6]:

# Group INVOICEITEMS view across different ProductGroups.
invoiceitems_view_by_overall_across_productgroup =\
invoiceitems_view.groupby([], category="ProductGroup")

# Group INVOICEITEMS view across different ProductGroups. invoiceitems_view_by_overall_across_productgroup =\ invoiceitems_view.groupby([], category="ProductGroup")

In [7]:

# Create Buckets representing the cumulative TotalCost of item, categorized by ProductGroup,
# for ALL customers over the past 26 weeks.
# The result is presented as a dictionary where the ProductGroup serves as the key
# and its corresponding sum of item TotalCost forms the value.
feature_group =\
invoiceitems_view_by_overall_across_productgroup.aggregate_over(
    "TotalCost", method=fb.AggFunc.SUM,
    feature_names=[
        "OVERALL_item_TotalCost_across_product_ProductGroups_26w"
    ],
    windows=["26w"]
)
# Get OVERALL_item_TotalCost_across_product_ProductGroups_26w object from feature group.
overall_item_totalcost_across_product_productgroups_26w =\
feature_group["OVERALL_item_TotalCost_across_product_ProductGroups_26w"]

# Create Buckets representing the cumulative TotalCost of item, categorized by ProductGroup, # for ALL customers over the past 26 weeks. # The result is presented as a dictionary where the ProductGroup serves as the key # and its corresponding sum of item TotalCost forms the value. feature_group =\ invoiceitems_view_by_overall_across_productgroup.aggregate_over( "TotalCost", method=fb.AggFunc.SUM, feature_names=[ "OVERALL_item_TotalCost_across_product_ProductGroups_26w" ], windows=["26w"] ) # Get OVERALL_item_TotalCost_across_product_ProductGroups_26w object from feature group. overall_item_totalcost_across_product_productgroups_26w =\ feature_group["OVERALL_item_TotalCost_across_product_ProductGroups_26w"]

Derive Similarity feature across entities¶

In [8]:

# Derive Similarity feature from cosine similarity between
# CUSTOMER_item_TotalCost_across_product_ProductGroups_26w
# and OVERALL_item_TotalCost_across_product_ProductGroups_26w
customer_vs_overall_item_totalcost_across_product_productgroups_26w =\
customer_item_totalcost_across_product_productgroups_26w.cd.cosine_similarity(
        overall_item_totalcost_across_product_productgroups_26w
    )
# Give a name to new feature
customer_vs_overall_item_totalcost_across_product_productgroups_26w.name = \
"CUSTOMER_vs_OVERALL_item_TotalCost_across_product_ProductGroups_26w"

# Derive Similarity feature from cosine similarity between # CUSTOMER_item_TotalCost_across_product_ProductGroups_26w # and OVERALL_item_TotalCost_across_product_ProductGroups_26w customer_vs_overall_item_totalcost_across_product_productgroups_26w =\ customer_item_totalcost_across_product_productgroups_26w.cd.cosine_similarity( overall_item_totalcost_across_product_productgroups_26w ) # Give a name to new feature customer_vs_overall_item_totalcost_across_product_productgroups_26w.name = \ "CUSTOMER_vs_OVERALL_item_TotalCost_across_product_ProductGroups_26w"

Preview feature¶

In [9]:

# Get observation table: 'Preview Table with 10 items'
preview_table = catalog.get_observation_table("Preview Table with 10 items")

# Get observation table: 'Preview Table with 10 items' preview_table = catalog.get_observation_table("Preview Table with 10 items")

In [10]:

# Preview CUSTOMER_item_TotalCost_across_product_ProductGroups_26w
customer_item_totalcost_across_product_productgroups_26w.preview(
    preview_table
)

# Preview CUSTOMER_item_TotalCost_across_product_ProductGroups_26w customer_item_totalcost_across_product_productgroups_26w.preview( preview_table )

Out[10]:

	POINT_IN_TIME	GROCERYINVOICEITEMGUID	CUSTOMER_item_TotalCost_across_product_ProductGroups_26w
0	2023-04-11 17:23:57	6084f39f-9d2c-4111-b1cc-502e1559c0c0	{'Adoucissants et Soin du linge': 5.98, 'Autre...
1	2023-02-07 11:04:26	fd1caae1-77e6-4667-8c83-df13f05bf2f5	{'Adoucissants et Soin du linge': 4.99, 'Aide ...
2	2023-03-17 11:15:09	1b627a25-7eb4-4f61-b243-c93db487bff0	{'Adoucissants et Soin du linge': 28.540000000...
3	2023-05-28 19:27:14	15973b2f-2256-4caa-b65b-cbbfdff0905b	{'Aide à la Pâtisserie': 2.29, 'Autres Produit...
4	2022-09-18 18:52:36	ac7edfb5-63ed-49fb-9b89-76b0288ed2f8	{'Adoucissants et Soin du linge': 34.02, 'Anim...
5	2023-03-31 18:50:00	213ef7d3-c27b-43e0-bc0a-57d6c7c254b0	{'Adoucissants et Soin du linge': 1.0, 'Aide à...
6	2022-12-26 15:01:07	264f79fd-c24a-47cc-8a68-fe3753a4d74b	{'Adoucissants et Soin du linge': 44.82, 'Aide...
7	2022-08-17 19:13:52	40a07ca4-a991-4d21-b5cf-74ee61220f96	{'Aide à la Pâtisserie': 1.29, 'Autres Produit...
8	2023-05-05 08:00:42	57ca0770-eb8b-4769-8e67-eb1b7cc0a934	{'Adoucissants et Soin du linge': 14.87, 'Aide...
9	2022-12-10 21:08:26	77d02174-f1e1-41c1-9fb9-01c6246b0009	{'Beurre': 3.49, 'Biscuits': 2.0, 'Biscuits ap...

In [11]:

# Preview OVERALL_item_TotalCost_across_product_ProductGroups_26w
overall_item_totalcost_across_product_productgroups_26w.preview(
    preview_table
)

# Preview OVERALL_item_TotalCost_across_product_ProductGroups_26w overall_item_totalcost_across_product_productgroups_26w.preview( preview_table )

Out[11]:

	POINT_IN_TIME	GROCERYINVOICEITEMGUID	OVERALL_item_TotalCost_across_product_ProductGroups_26w
0	2023-04-11 17:23:57	6084f39f-9d2c-4111-b1cc-502e1559c0c0	{'Adoucissants et Soin du linge': 1225.44, 'Ai...
1	2022-08-17 19:13:52	40a07ca4-a991-4d21-b5cf-74ee61220f96	{'Adoucissants et Soin du linge': 1133.67, 'Ai...
2	2023-05-05 08:00:42	57ca0770-eb8b-4769-8e67-eb1b7cc0a934	{'Adoucissants et Soin du linge': 1232.67, 'Ai...
3	2023-03-17 11:15:09	1b627a25-7eb4-4f61-b243-c93db487bff0	{'Adoucissants et Soin du linge': 1206.23, 'Ai...
4	2023-02-07 11:04:26	fd1caae1-77e6-4667-8c83-df13f05bf2f5	{'Adoucissants et Soin du linge': 1203.5, 'Aid...
5	2022-12-10 21:08:26	77d02174-f1e1-41c1-9fb9-01c6246b0009	{'Adoucissants et Soin du linge': 1148.13, 'Ai...
6	2023-05-28 19:27:14	15973b2f-2256-4caa-b65b-cbbfdff0905b	{'Adoucissants et Soin du linge': 1168.2, 'Aid...
7	2023-03-31 18:50:00	213ef7d3-c27b-43e0-bc0a-57d6c7c254b0	{'Adoucissants et Soin du linge': 1234.72, 'Ai...
8	2022-09-18 18:52:36	ac7edfb5-63ed-49fb-9b89-76b0288ed2f8	{'Adoucissants et Soin du linge': 1068.47, 'Ai...
9	2022-12-26 15:01:07	264f79fd-c24a-47cc-8a68-fe3753a4d74b	{'Adoucissants et Soin du linge': 1178.83, 'Ai...

In [12]:

# Preview CUSTOMER_vs_OVERALL_item_TotalCost_across_product_ProductGroups_26w
customer_vs_overall_item_totalcost_across_product_productgroups_26w.preview(
    preview_table
)

# Preview CUSTOMER_vs_OVERALL_item_TotalCost_across_product_ProductGroups_26w customer_vs_overall_item_totalcost_across_product_productgroups_26w.preview( preview_table )

Out[12]:

	POINT_IN_TIME	GROCERYINVOICEITEMGUID	CUSTOMER_vs_OVERALL_item_TotalCost_across_product_ProductGroups_26w
0	2023-02-07 11:04:26	fd1caae1-77e6-4667-8c83-df13f05bf2f5	0.736515
1	2023-04-11 17:23:57	6084f39f-9d2c-4111-b1cc-502e1559c0c0	0.733969
2	2023-03-17 11:15:09	1b627a25-7eb4-4f61-b243-c93db487bff0	0.655646
3	2023-05-28 19:27:14	15973b2f-2256-4caa-b65b-cbbfdff0905b	0.603565
4	2022-09-18 18:52:36	ac7edfb5-63ed-49fb-9b89-76b0288ed2f8	0.705695
5	2022-12-26 15:01:07	264f79fd-c24a-47cc-8a68-fe3753a4d74b	0.836482
6	2023-03-31 18:50:00	213ef7d3-c27b-43e0-bc0a-57d6c7c254b0	0.840879
7	2023-05-05 08:00:42	57ca0770-eb8b-4769-8e67-eb1b7cc0a934	0.697578
8	2022-12-10 21:08:26	77d02174-f1e1-41c1-9fb9-01c6246b0009	0.794200
9	2022-08-17 19:13:52	40a07ca4-a991-4d21-b5cf-74ee61220f96	0.576912

Save feature¶

In [13]:

# Save feature
customer_vs_overall_item_totalcost_across_product_productgroups_26w.save()

# Save feature customer_vs_overall_item_totalcost_across_product_productgroups_26w.save()

Done! |████████████████████████████████████████| 100% in 6.1s (0.17%/s)         

As always, add description and view definition file¶

In [14]:

# Add description
customer_vs_overall_item_totalcost_across_product_productgroups_26w.update_description(
    "Similarity between the customer and all customers measured by the "
    "Cosine Similarity between the Distribution representing the cumulative"
    " TotalCost of item,  categorized by their respective product's "
    "ProductGroup, over 26w for both entities."
)
# See feature definition file
customer_vs_overall_item_totalcost_across_product_productgroups_26w.definition

# Add description customer_vs_overall_item_totalcost_across_product_productgroups_26w.update_description( "Similarity between the customer and all customers measured by the " "Cosine Similarity between the Distribution representing the cumulative" " TotalCost of item, categorized by their respective product's " "ProductGroup, over 26w for both entities." ) # See feature definition file customer_vs_overall_item_totalcost_across_product_productgroups_26w.definition

Out[14]:

# Generated by SDK version: 1.1.0.dev7
from bson import ObjectId
from featurebyte import DimensionTable
from featurebyte import FeatureJobSetting
from featurebyte import ItemTable


# dimension_table name: "GROCERYPRODUCT"
dimension_table = DimensionTable.get_by_id(ObjectId("666956c78080c62d0dc616e2"))
dimension_view = dimension_table.get_view(
    view_mode="manual", drop_column_names=[], column_cleaning_operations=[]
)

# item_table name: "INVOICEITEMS", event_table name: "GROCERYINVOICE"
item_table = ItemTable.get_by_id(ObjectId("666956c58080c62d0dc616e1"))
item_view = item_table.get_view(
    event_suffix=None,
    view_mode="manual",
    drop_column_names=["record_available_at"],
    column_cleaning_operations=[],
    event_drop_column_names=["record_available_at"],
    event_column_cleaning_operations=[],
    event_join_column_names=[
        "Timestamp",
        "GroceryInvoiceGuid",
        "GroceryCustomerGuid",
        "tz_offset",
    ],
)
joined_view = item_view.join(
    dimension_view, on="GroceryProductGuid", how="left", rsuffix="", rprefix=""
)
grouped = joined_view.groupby(
    by_keys=["GroceryCustomerGuid"], category="ProductGroup"
).aggregate_over(
    value_column="TotalCost",
    method="sum",
    windows=["26w"],
    feature_names=["CUSTOMER_item_TotalCost_across_product_ProductGroups_26w"],
    feature_job_setting=FeatureJobSetting(
        blind_spot="120s", period="3600s", offset="120s"
    ),
    skip_fill_na=True,
    offset=None,
)
feat = grouped["CUSTOMER_item_TotalCost_across_product_ProductGroups_26w"]
grouped_1 = joined_view.groupby(
    by_keys=[], category="ProductGroup"
).aggregate_over(
    value_column="TotalCost",
    method="sum",
    windows=["26w"],
    feature_names=["OVERALL_item_TotalCost_across_product_ProductGroups_26w"],
    feature_job_setting=FeatureJobSetting(
        blind_spot="120s", period="3600s", offset="120s"
    ),
    skip_fill_na=True,
    offset=None,
)
feat_1 = grouped_1["OVERALL_item_TotalCost_across_product_ProductGroups_26w"]
feat_2 = feat.cd.cosine_similarity(other=feat_1)
feat_2.name = (
    "CUSTOMER_vs_OVERALL_item_TotalCost_across_product_ProductGroups_26w"
)
output = feat_2
output.save(_id=ObjectId("6669578ded0c9d417ba58fff"))

Concepts in this tutorial¶

• View Joins
• Feature Transforms
• Cross Aggregate features

SDK reference for¶

• Feature.cd.cosine_similarity()