%201.svg){kind=icon}
Discover the transformative tools and techniques in data analytics, from AI and application development to task automation and semantic layers, empowering analysts to drive substantial value.
In the past decade of being a data practitioner, I have witnessed a tremendous evolution in the tools and technologies available to us. The acceleration has been particularly exponential in the last couple of years, driven by the AI revolution that has made data the focal point of innovation and excitement. The integration of artificial intelligence (AI), application development platforms, task and script automation tools, and semantic layers has transformed how the best teams approach data analytics. These advancements have not only simplified complex data processes but have also empowered analysts to create powerful, interactive, and highly valuable data products. This blog aims to share insights and tools that can help data analysts transcend traditional roles, enabling them to deliver substantial value to their organizations. Whether you're looking to enhance your business intelligence capabilities, augment data workflows, or ensure consistency and governance, this guide will provide you with the knowledge to level up quickly and drive impactful business outcomes.
In today's rapidly evolving data landscape, the capabilities available to data analysts have expanded exponentially. Modern data warehouses now offer powerful AI functions that allow analysts to build, deploy, and manage machine learning models directly within their familiar environments. This section highlights how platforms like Snowflake Cortex, Databricks, and Google BigQuery ML are revolutionizing data analytics and enabling analysts to extract deeper insights with minimal effort.
Here are some example use-cases and code
Text Classification Tools using Common Data Warehouses
Snowflake’s Cortex is a prime example of AI seamlessly integrated into a data platform. It allows analysts to build and deploy machine learning models directly within Snowflake, using familiar SQL queries. This means analysts can perform advanced analytics without needing to switch between multiple tools or platforms. By embedding AI into their workflows, analysts can gain deeper insights from their data and streamline their analytical processes.
For more examples using specific functions, here are a couple of pages that may be interesting:
Snowflake Cortex Use-Cases and Tools
Leveraging Snowflake Cortex COMPLETE Function for Prompt Engineering
Leveraging Snowflake Cortex for Advanced AI and ML: New Functionalities for Analytics Teams
Creating and Using Custom Models in Snowflake
Databricks provides robust support for AI and machine learning, making it easier for analysts to leverage these technologies. With Databricks, analysts can create and manage ML models, automate workflows, and integrate these models into their data pipelines. The platform’s integration with popular frameworks like TensorFlow and PyTorch, along with its collaborative workspace, simplifies the process of developing and deploying AI-driven insights.
If you’re curious for more Databricks and AI content, here’s a few introductory articles to learn more:
Recent AI/ML Function Types in Databricks to Help Analysts Level Up
Unlocking Advanced Data Analysis with Databricks' Latest Tools
Supercharging Machine Learning with Databricks Runtime for Machine Learning
Google’s BigQuery ML allows analysts to build and execute machine learning models using standard SQL queries. This integration means that analysts can create sophisticated models without extensive programming knowledge. BigQuery ML supports various types of models, including linear regression, k-means clustering, and more, enabling analysts to incorporate machine learning directly into their data queries and business intelligence tools.
Enhancing Data Analysis with Google BigQuery's AI/ML Functions
Integrating Vertex AI with Google BigQuery for Continuous Image Analysis
Other databases, such as PostgreSQL and traditional SQL databases, can also be augmented with AI and machine learning functionalities through hosted Python scripts and application tools. This approach allows analysts to extend the capabilities of these databases without needing extensive engineering support, enabling them to perform advanced text and image analysis.
Here’s some explainers for using Python and Postgres to help get you started
Using LLMs with Python, Jupyter and PostgreSQL
Steps to Implement Image Classification with PostgreSQL
Integrating AI into existing data tools enhances the capabilities of analysts, enabling them to perform advanced analytics without extensive programming knowledge. Modern BI tools, SQL interfaces, and data transformation layers now embed AI functions, allowing for predictive analytics, enriched data workflows, and the democratization of machine learning insights across organizations. This section explores how to leverage AI within business intelligence tools like Tableau and Power BI, embed AI functions in SQL queries using Snowflake and Databricks, and integrate AI into transformation layers with tools like dbt.
Modern BI tools like Tableau and Power BI are increasingly integrating AI capabilities. By connecting these tools with AI-embedded platforms like Snowflake and Databricks, analysts can enhance their visualizations and dashboards with predictive analytics and advanced analysis. This integration helps organizations make data-driven decisions faster and more accurately.
Example with Snowflake and Tableau:
Integrate Tableau with Snowflake to use predictive models directly within your dashboards:
-- Creating a predictive model in Snowflake
CREATE OR REPLACE MODEL sales_model
PREDICTING sales
USING
SELECT
total_spent,
num_items,
customer_age,
customer_income,
sales
FROM
sales_data;
-- Using this model in Tableau via SQL
SELECT
total_spent,
num_items,
customer_age,
customer_income,
PREDICT(sales_model, total_spent, num_items, customer_age, customer_income) AS predicted_sales
FROM
new_sales_data;
AI functions embedded in SQL interfaces allow analysts to perform complex analyses within their existing workflows. For instance, using Databricks, analysts can embed machine learning models directly into their queries, making it easier to share AI-driven insights across the organization. This approach democratizes access to advanced analytics, enabling more team members to benefit from AI.
Example with Databricks:
# Training a model in Databricks
from pyspark.ml.regression import LinearRegression
# Load training data
data = spark.read.format("libsvm").load("/path/to/data")
# Create and train model
lr = LinearRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)
lr_model = lr.fit(data)
# Save the model
lr_model.write().overwrite().save("/path/to/save/model")
# Using the model in SQL
spark.sql("""
SELECT
total_spent,
num_items,
customer_age,
customer_income,
PREDICT(lr_model, total_spent, num_items, customer_age, customer_income) AS predicted_sales
FROM
new_sales_data
""")
Data transformation tools like dbt (data build tool) have inherited AI support because of the integration of AI and machine learning models into the data warehouses where dbt is traditionally run. By embedding AI functions into dbt models, analysts can transform and enrich their data in new ways. This is particularly valuable for processing unstructured data, such as text or images, which traditional systems struggled to handle. With AI, analysts can now synthesize and analyze these data types more effectively, uncovering insights that were previously out of reach.
Example with dbt and Snowflake:
Use Snowflake's built-in text analysis functions to perform sentiment analysis or text classification within a dbt model.
-- dbt model (my_model.sql)
WITH sentiment_analysis AS (
SELECT
id,
text,
SENTIMENT(text) AS sentiment_score
FROM
source_table
)
SELECT
st.id,
st.text,
sa.sentiment_score
FROM
source_table st
LEFT JOIN
sentiment_analysis sa ON st.id = sa.id;
By integrating AI functions into these tools, data analysts can enhance their workflows and derive deeper insights from their data. These examples demonstrate how to leverage AI capabilities within existing tools and platforms, enabling more sophisticated and impactful data analysis.
Leveraging hosted scripts allows data analysts to run custom Python scripts without needing extensive data engineering expertise. These platforms simplify the integration of APIs and custom code, enabling data augmentation and enrichment tasks seamlessly. Here are some key providers:
Pipedream is a powerful platform for connecting APIs and running custom workflows. It supports a wide range of integrations and allows you to run custom Python code in response to various triggers. Automating data workflows, integrating third-party APIs, and running custom scripts for data processing and augmentation.
Google Cloud Functions is a serverless execution environment for building and connecting cloud services. It allows you to run code in response to events and integrates seamlessly with other Google Cloud services. You can use Google cloud functions for real-time data processing, triggering workflows based on changes in BigQuery, and integrating with Google Maps for geocoding addresses.
AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. It can be triggered by various AWS services or HTTP requests. You can use AWS lambda for data transformation tasks, invoking third-party APIs for data enrichment, and automating data pipelines.
Azure Functions is a serverless compute service that enables you to run event-driven code without having to manage infrastructure. It integrates well with other Azure services. Use Azure functions for automating data workflows, integrating with Azure services for data processing, and calling APIs for data augmentation.
Zapier is an automation tool that connects your apps and services. It allows you to automate tasks by creating workflows called “Zaps” that can include custom Python code. Zapier is a very common tool for operations heavy teams and can allow data teams to help increase the flow of information across systems, automating and improving the quality of their company’s tasks. You can use it for integrating various SaaS applications, automating repetitive tasks, and running custom code for specific data processing needs.
Integromat, now known as Make, is an integration platform that allows you to connect apps and automate workflows. It supports custom scripting for more advanced automation needs. Use Integromat for automating data flows between different systems, enriching data with external APIs, and running custom scripts for data manipulation.
These platforms provide flexible solutions for running custom Python scripts and integrating various APIs into your data workflows. By leveraging these tools, data analysts can perform complex data augmentation tasks without the need for a full-time data engineer, enhancing their ability to derive insights and add value to their data analysis processes.
The rise of new application-building software has enabled data analysts to go beyond traditional dashboards and create more customized, interactive experiences for users.
Retool is a powerful platform that allows analysts to quickly build internal tools and applications. With its drag-and-drop interface and integrations with various data sources, analysts can create read-write applications that provide a more dynamic and interactive user experience. This shift enables analysts to move from simply presenting data to actively engaging users with tailored applications that can drive more significant insights and actions.
Integrating Vertex AI with Google BigQuery for Continuous Image Analysis in Retool
Streamlit is another tool that has revolutionized how analysts can build applications. It allows analysts to create custom web applications for data science projects with minimal coding. Streamlit applications can be used to prototype new ideas, share insights interactively, and even deploy machine learning models in a user-friendly interface. This capability turns data analysts into product creators, enabling them to offer highly valuable tools to their organizations.
Integrating Vertex AI with Google BigQuery for Continuous Image Analysis in a Streamlit Application
Semantic layers are a powerful tool for ensuring consistency and accuracy in data analysis. They provide a validated definition layer that standardizes metrics and dimensions across an organization, enabling reliable and consistent insights. By modeling the business through a semantic layer, data analysts and AI tools can work with a unified view of data, ensuring that all analyses are based on the same definitions and logic.
Semantic layers act as an interface for AI, allowing custom AI tools or native tools like Push.ai to produce reliable numbers and insights. Here's how:
Push.ai, a tool that leverages the semantic layer, can provide accurate and reliable AI-driven insights by interacting with the semantic layer. Here’s how:
Here are some leading providers that help organizations model their business effectively through semantic layers:
Description: Cube offers a headless BI platform with a consistent and secure semantic layer for metrics and data models. It connects to data sources, transforms the data, and makes it accessible through APIs for BI tools and applications. Cube ensures that all data interactions are based on a single source of truth, enhancing data governance and reliability.
Website: Cube
Description: dbt is a powerful transformation tool that enables data analysts and engineers to transform data within their warehouses more effectively. The latest version of the dbt Semantic Layer allows organizations to centrally define business metrics, ensuring consistent and reliable metrics across various data platforms and tools. With integrations with tools like Tableau, Google Sheets, and Push.ai, dbt facilitates seamless access to accurate data metrics.
Website: dbt
Description: AtScale provides an adaptive analytics platform that enables enterprises to create a universal semantic layer. It supports dynamic and interactive queries, delivering consistent metrics and KPIs across BI tools. AtScale's platform is designed to integrate seamlessly with existing data architectures, providing a scalable solution for data analytics and AI.
Website: AtScale
Integrating AI with a semantic layer transforms how organizations interact with data, providing a reliable and consistent foundation for advanced analytics. By leveraging the semantic layer, tools like Push.ai can produce accurate and validated insights, empowering data analysts to drive significant business outcomes. This approach not only simplifies data access and analysis but also ensures that all users and tools are aligned on the same definitions and metrics, fostering trust and enhancing decision-making across the organization.
With these new tools, data analysts can now produce AI products that deliver substantial value to their organizations. By leveraging new platforms, analysts can:
These capabilities position analysts to take on roles similar to product managers, where they can package data into powerful systems that drive significant business outcomes. As a result, analysts are no longer just data interpreters but are becoming key players in developing strategic AI products that can revolutionize their organizations.
The rapid evolution of AI technologies and application-building tools has significantly lowered the barriers for data analysts to incorporate advanced analytics into their workflows. By leveraging tools like Snowflake Cortex, Databricks, BigQuery ML, Retool, and Streamlit, analysts can quickly stand up initial prototypes or full use cases with minimal effort. These integrations not only enhance the capabilities of traditional BI tools and SQL queries but also enable the creation of sophisticated applications that drive significant value. As AI continues to evolve, the opportunities for data analysts to drive value within their organizations will only grow, making now the perfect time to start exploring these powerful tools.
With the ability to develop interactive applications, enhance user experiences, drive both internal and external value, perform advanced data engineering tasks, and leverage powerful AI functions, data analysts are uniquely positioned to build powerful data products. These tools empower analysts to harness advanced techniques, transforming raw data into strategic assets and revolutionizing their roles within organizations.
Everything that a data leader needs to understand and deploy metrics at scale
Download The Full Guide
Core Semantic Layer Concepts
Benefits and ROI
Implementation Steps
Create an account to connect your business and elevate how your operate.
Britton is the CTO of Push.ai and oversees Product, Design, and Engineering. He's been a passionate builder, analyst and designer who loves all things data products and growth. You can find him reading books at a coffee shop or finding winning strategies in board games and board rooms.
Stay up to date with the latest product updates and insights sent straight to your inbox!
