View projects
The job cluster optimization Projects page displays summary metrics for your job cluster optimization projects, a cost optimization chart, and a table of all your optimization projects.
To access the Projects page:
- Navigate to Recommendations > Databricks from the Slingshot left sidebar.
⚠️
Job cluster optimization costs are calculated differently from other costs shown in Slingshot.
- For DBU costs, job cluster optimization calculates costs based on job cluster size and runtime, whereas the rest of Slingshot uses system tables from Databricks.
- For infrastructure (cloud provider) costs, job cluster optimization calculates costs based on cloud provider list prices and cluster monitoring (e.g. AWS Eventbridge, Azure Eventgrid) / timeline (e.g. when a node joined / left the cluster), whereas the rest of Slingshot uses cloud provider APIs (e.g. AWS Cost Explorer API), which accounts for customer discounts.
- Additionally, job cluster optimization only shows costs for job clusters onboarded to job cluster optimization and the rest of Slingshot shows costs for all jobs in onboarded metastores.
Summary metrics
Summary metrics for your optimization projects include:
- Savings to date: The total amount of savings since loading your first project.
- Projected annual savings: The total amount of savings expected across all your projects for the next 12 months, based on activity up to the current day.
- Engineering hr savings: The total time saved on manually training job clusters on your projects, as a result of enabling job cluster optimization.
- Total cost: The total cost of running all your projects since optimization started.
- SLAs met: The percentage of successful runs that completed within the Service Level Agreement (SLA) requirements, calculated as a ratio of total runs across all your projects.ℹ️When a project is first loaded, the SLA is set to 2x the most recent run (i.e., the first run where data is being collected). You can adjust the SLA by selecting Edit optimization settings from the project summary settings.
Cost chart
The Cost chart on the Projects page summarizes your total job cluster costs by DBU cost and Infra cost.
You can customize the data in the chart by:
- Date picker: Select a custom data range or use one of the preset date ranges.
- Time increment: Select a time increment from the dropdown. Depending on the date range you selected, your time increment can be Daily, Weekly, or Monthly.
Projects table
The table on the Projects page lists all your job cluster optimization projects by Project Name, App ID, Insights, Last monitored, Last run spark cost, Optimization phase, and Last run status.
- Select a column heading to sort the table by that field.
- Select Filters or use the Search bar to filter the table by a specific field value, such as a partial string match for a project name.