A framework for ambitious data engineers and analysts

The Dataform SDK is an open source framework for managing SQL based data operations in your warehouse.

How does Dataform work?

You develop queries, operations, and assertions in SQL with Dataform's built-in templating syntax and APIs

Dataform compiles your project into a DAG (Directed Acyclic Graph) of every action in SQL

Dataform connects to your warehouse and executes the DAG or just the actions that you specify

You receive detailed exection logs

select
    country,
    device_type,
    country,
    sum(revenue) as revenue,
    sum(sessions) as sessions,
    sum(pageviews) as pageviews

from raw_data

group by 1,2
table
dataform.my_table
FieldTypeDescription
countrySTRINGDescription
device_typeSTRINGDescription
revenueDOUBLEDescription
sessionsDOUBLEDescription
pageviewsDOUBLEDescription
select
    country,
    device_type,
    country,
    sum(revenue) as revenue,
    sum(sessions) as sessions,
    sum(pageviews) as pageviews

from raw_data

group by 1,2
Each .sql file in your definitions/ folder contains a SQL query or statement enriched with Dataform's template and configuration syntax. Dataform takes care of generating the boilerplate required to create corresponding tables and views in your data warehouse.

Benefits

Highly productive

Write dramatically less SQL with Dataform's integrated suite of APIs and reusable code snippets

Fully extensible

Develop reusable functions in plain Javascript that benefit your entire team

Iterate quickly

View compiled SQL and any compilation errors in real time

Build a data warehouse you can be proud of

Connect to your warehouse and publish your first dataset in 5 minutes

Already using Dataform? Log in