System Architecture¶
FeatureByte is developed to integrate seamlessly with data warehouses, enhancing security and efficiency by bypassing large-scale outbound data transfers. This integration allows feature calculations to be performed within the data warehouse, leveraging scalability, stability, and efficiency.
Components¶
| Component | Packaging | Purpose | Comments |
|---|---|---|---|
| Featurebyte SDK | Python Package | Connects to the API service to provide feature authoring and management functionality through python classes and functions. | |
| API Service | Docker Container | REST-API service that validates and executes requests, queries data warehouses, and stores data. | Horizontally scalable for high availability. |
| Worker | Docker Container | Executes asynchronous or scheduled tasks. | Horizontally scalable for high availability. |
| MongoDB | Docker Container | Store metadata for created assets. | |
| Redis | Docker Container | Broker and queue for workers, messenger service for publishing progress updates. | |
| Query Graph Transpiler | Python Package | Construct data transformation steps as a query graph, which can be transpiled to platform-specific SQL. | |
| Source Tables | Data Warehouse | Tables used as data sources for feature engineering. | |
| Feature Store | Data Warehouse | Database that store data used to support feature serving. |