Job cluster optimization
Job cluster optimization uses advanced machine learning (ML) algorithms to provide automated compute optimization recommendations for organizations running their data infrastructure on CPUs or GPUs in the cloud.
⚠️
Job cluster optimization costs are calculated differently from other costs shown in Slingshot.
- For DBU costs, job cluster optimization calculates costs based on job cluster size and runtime, whereas the rest of Slingshot uses system tables from Databricks.
- For infrastructure (cloud provider) costs, job cluster optimization calculates costs based on cloud provider list prices and cluster monitoring (e.g. AWS Eventbridge, Azure Eventgrid) / timeline (e.g. when a node joined / left the cluster), whereas the rest of Slingshot uses cloud provider APIs (e.g. AWS Cost Explorer API), which accounts for customer discounts.
- Additionally, job cluster optimization only shows costs for job clusters onboarded to job cluster optimization and the rest of Slingshot shows costs for all jobs in onboarded metastores.