Job compute optimization
Job compute optimization uses advanced machine learning (ML) algorithms to provide automated compute optimization recommendations for organizations running their data infrastructure on CPUs or GPUs in the cloud.
⚠️
Job compute optimization costs are calculated differently from other costs shown in Slingshot.
- For DBU costs, job compute optimization calculates costs based on job compute size and runtime, whereas the rest of Slingshot uses system tables from Databricks.
- For infrastructure (cloud provider) costs, job compute optimization calculates costs based on cloud provider list prices and compute monitoring (e.g. AWS Eventbridge, Azure Eventgrid) / timeline (e.g. when a node joined / left the compute), whereas the rest of Slingshot uses cloud provider APIs (e.g. AWS Cost Explorer API), which accounts for customer discounts.
- Additionally, job compute optimization only shows costs for job computes onboarded to job compute optimization and the rest of Slingshot shows costs for all jobs in onboarded metastores.