https://ift.tt/J6hnx1X Databricks Workflows is the fully-managed orchestrator for data, analytics, and AI. Today, we are happy to announce...
Databricks Workflows is the fully-managed orchestrator for data, analytics, and AI. Today, we are happy to announce several enhancements that make it easier to bring the most demanding data and ML/AI workloads to the cloud.
Workflows offers high reliability across multiple major cloud providers: GCP, AWS, and Azure. Until today, this meant limiting the number of jobs that can be managed in a Databricks workspace to 1000 (number varied based on tier). Customers running more data and ML/AI workloads had to partition jobs across workspaces in order to avoid running into platform limits. Today, we are happy to announce that we are significantly increasing this limit to 10,000. The new platform limit is automatically available in all customer workspaces (except single-tenant).
Thousands of customers rely on the Jobs API to create and manage jobs from their applications, including CI/CD systems. Together with the increased job limit, we have introduced a faster, paginated version of the jobs/list API and added pagination to the jobs page.
List of jobs with pagination
The higher workspace limit also comes with a streamlined search experience which allows searching by name, tags, and job ID.
Streamlined search by name, tag or job ID.
Put together, the new features allow scaling workspaces to a large number of jobs. For rare cases where the changes in behavior above are not desired, it is possible to revert to the old behavior via the Admin Console (only possible for workspaces with up to 3000 jobs). We strongly recommend that all customers switch to the new paginated API to list jobs, especially for workspaces with thousands of saved jobs.
To get started with Databricks Workflows, see the quickstart guide. We’d also love to hear from you about your experience and any other features you’d like to see.
Learn more about:
--
Try Databricks for free. Get started today.
The post Orchestrating Data and ML Workloads at Scale: Create and Manage Up to 10k Jobs Per Workspace appeared first on Databricks.
from Databricks https://ift.tt/VYkvbHd
via RiYo Analytics
ليست هناك تعليقات