Skip to content

Connect to BigQuery

This guide will help you set up a FeatureByte feature store in a BigQuery Data Warehouse.

Before You Begin

Gather the credentials for connecting to your BigQuery account, specifically:

  • Name of the project you're connecting to
  • Name of the dataset used to store data
  • Google service account credentials

You'll also want to ensure that the user you're connecting with has the relevant privileges that are required. Specifically, the role should have the following privileges:

  • BigQuery Data Editor
  • BigQuery Job User
  • BigQuery Read Session User

Refer to BigQuery Access Control for more details.

Why are these privileges needed? These privileges are needed for featurebyte to write some metadata into your BigQuery data warehouse. This is used internally by our application to track some metadata, and perform some optimizations to make your experience better.

Setup Guide

FeatureByte Installation

Make sure that you have FeatureByte installed. See installation for more details.

Step 1: Test that your connection works

We can now try to see if your connection works by trying to create a new feature store. We can do so by running the following commands (either in a notebook, or a python interactive shell).

  • If you know that a feature store exists already, we can try to list the existing feature stores on BigQuery. 
    import featurebyte as fb

fb.FeatureStore.list()

# Get a feature store if it exists
feature_store = fb.FeatureStore.get(name="<feature_store_name>")
  • Alternatively, try to create a feature store. 
    # Name of the feature store that we want to create/connect to
feature_store = fb.FeatureStore.get_or_create(
    name="<feature_store_name>",
    source_type=fb.SourceType.BIGQUERY,
    details=fb.BigQueryDetails(
        project_name="<project_name>",
        dataset_name="<dataset_name>",
    ),
    database_credential=fb.GoogleCredential(
        service_account_info={"type": "service_account", "private_key": "<private_key>"}
    ),
)

Step 2: Connect to your BigQuery feature store

# List all databases in the feature store
feature_store.get_data_source().list_databases()

Congratulations! You have successfully connected to your BigQuery data warehouse if you are able to run these commands without any errors!

Next Steps

Now that you've connected to your data, feel free to try out some tutorials!