An open source framework for data engineers and analysts

The Dataform SDK helps data teams manage and automate SQL-based operations in your data warehouse.
Read the docs or view on GitHub. github logo

How it works

You develop queries, operations, and assertions in SQL with Dataform's built-in templating syntax and APIs

Develop

You develop queries, operations, and assertions in SQL with Dataform's built-in templating syntax and APIs

Dataform compiles your project into a DAG (Directed Acyclic Graph) of every action in SQL

Compile

Dataform compiles your project into a DAG (Directed Acyclic Graph) of every action in SQL

Dataform connects to your data warehouse and executes the DAG (or the actions you specify)

Run

Dataform connects to your data warehouse and executes the DAG (or the actions you specify)

You receive detailed exection logs of the datasets created and the other actions performed

View logs

You receive detailed exection logs of the datasets created and the other actions performed

SELECT
  country,
  device_type,
  country,
  SUM(revenue) AS revenue,
  SUM(sessions) AS sessions,
  SUM(pageviews) AS pageviews

FROM raw_data

GROUP BY 1, 2
logo
table
dataform.my_table
FieldTypeDescription
countrySTRINGDescription
device_typeSTRINGDescription
revenueDOUBLEDescription
sessionsDOUBLEDescription
pageviewsDOUBLEDescription
SELECT
  country,
  device_type,
  country,
  SUM(revenue) AS revenue,
  SUM(sessions) AS sessions,
  SUM(pageviews) AS pageviews

FROM raw_data

GROUP BY 1, 2
Each .sql file in your definitions/ folder contains a SQL query or statement enriched with Dataform's template and configuration syntax. Dataform then generates the boilerplate required to create corresponding tables and views in your data warehouse.

Benefits

Write less SQL thanks to reusable code snippets and an integrated suite of APIs

Highly productive

Write less SQL thanks to reusable code snippets and an integrated suite of APIs

Develop reusable functions in JavaScript - to the benefit of your entire team

Fully extensible

Develop reusable functions in JavaScript - to the benefit of your entire team

View compiled SQL with real-time compilation

Iterate quickly

View compiled SQL with real-time compilation

Build a data warehouse you can be proud of

With Dataform, you can publish your first dataset in just 5 minutes

Already using Dataform? Log in