Databricks has been actively enhancing its AI and ML capabilities, particularly with the introduction of new functions and tools in the last couple years. These advancements aim to simplify the integration of AI into existing workflows, making it more accessible for data analysts and enhancing their ability to derive insights from data. Here are some key AI/ML functions and tools that analysts should be aware of:

AI Functions in SQL

Databricks has introduced AI Functions that allow users to access large language models (LLMs) directly from SQL. This feature simplifies the incorporation of AI into data workflows by enabling tasks such as natural language query generation, data documentation, and custom logic creation using SQL commands.

Key Functions:

• ai_generate_text(): Generates text based on a given prompt.

• ai_analyze_sentiment(): Performs sentiment analysis on text data.

• ai_classify(): Classifies text into predefined categories.

• ai_translate(): Translates text from one language to another.

• ai_summarize(): Summarizes long pieces of text.

• ai_query(): Queries a serving endpoint for custom models.

These functions enable SQL users to perform advanced AI tasks without needing deep expertise in machine learning or programming  .

Enhancing Data Analysis with Databricks' AI Functions in SQL

MLflow 2.5 and AI Gateway

MLflow is an open-source platform for managing the machine learning lifecycle. The latest release, MLflow 2.5, includes several enhancements:

• MLflow AI Gateway: Allows organizations to manage credentials for SaaS models and model APIs, providing access-controlled routes for querying models.

• Prompt Tools: No-code visual tools to compare various models’ outputs based on a set of prompts, automatically tracked within MLflow.

These tools help streamline the deployment and monitoring of AI models, ensuring efficient management of resources and enhancing model governance .

Databricks Model Serving

Databricks Model Serving has been optimized for high performance, including GPU-based inference support. This feature enables the deployment of scalable AI models with minimal configuration. Key benefits include:

• Low-latency inference: Ensures fast response times for model predictions.

• Cost optimization: Scales up and down based on demand, reducing operational costs.

• End-to-end monitoring: Tracks all requests and responses to ensure model performance and data integrity.

Model Serving is essential for deploying production-quality AI models efficiently and cost-effectively  .

Vector Search

This function allows users to perform similarity searches using vectors, enabling tasks such as finding similar documents or images. Vector search leverages advanced indexing techniques to ensure efficient querying and retrieval of similar items.

This function is particularly useful for applications involving recommendation systems, content retrieval, and clustering similar data points .

Databricks Runtime for Machine Learning

Databricks Runtime for Machine Learning includes pre-configured environments with common ML and DL libraries such as TensorFlow, PyTorch, and Keras. It supports GPU-enabled clusters for deep learning applications and integrates seamlessly with Databricks’ data and model management features.

Key features:

• Pre-configured libraries: Simplifies setup for deep learning and machine learning projects.

• GPU support: Enhances performance for training and inference of complex models.

• Integration with MLflow: Facilitates tracking and managing ML experiments.

This runtime environment is ideal for developing, training, and deploying machine learning models in a scalable and efficient manner .

Conclusion

Databricks has made significant strides in enhancing its AI and ML capabilities, providing tools that democratize access to advanced analytics and machine learning. By leveraging these functions and tools, data analysts can elevate their workflows, integrate sophisticated AI models, and derive deeper insights from their data. These advancements make Databricks a powerful platform for any organization looking to harness the full potential of AI and machine learning in their data operations.

For more detailed information on these features, you can explore the Databricks documentation and Microsoft Learn.

‍