New and improved dependency tree! | Dataform
Product Update

New and improved dependency tree!

During compilation, Dataform builds a dependency tree of all actions to be run in the warehouse. We’ve made some changes to the dependency tree so that it is far easier to navigate.

Features
picture of author

Josie Hall on June 4, 2020

Managing dependencies in Dataform

Dataform provides methods that enable you to easily reference another dataset in your project using the ref () function.

This speeds up development as you don’t have to provide the full SQL dataset name and any dataset that is referenced by a query will be automatically added to that query’s dependencies. Dependency queries are always executed before dependent queries to ensure correctness.

During compilation Dataform resolves the ref () function onto the name of the table that will be created and checks for errors including missing or circular dependencies. Dataform then builds the dependency tree of all actions to be run in the warehouse; including tables and views, assertions and other SQL operations that need to run.

Dataform updates the entire dependency tree of your project with every save so you can get a quick visual overview of how your tables, views, and other actions depend on each other while developing.

The dependency tree helps with things such as onboarding new team members as they can easily get up to speed with where all the tables live and how they relate to each other. Livup scaled their data team from 6 to 20 people using the dependency tree as key traning material for new members of the team.

Updates to dependency tree

We’ve made some changes to the dependency tree so that it is far easier to navigate:

  • Merged the dependency tree with the data catalog so when you search for a dataset you can see where it sits in the dependency tree.
  • Added action types to actions on the graph so you can now see if a node is a dataset, a declaration or an operation.
  • Clicking on an action, either in the list or in the DAG itself, shows action details and all dependant and dependencies of that action.
  • You can now click and drag to scroll across the DAG making it far easier to navigate (especially if like us you have a big dependency tree)!
  • Hovering over an action in the catalog list will take you to the action in the dependency tree.
  • Searching for an action using the search bar will only show items that match the search criteria in the DAG as well as the catalog list.
  • It's a lot more colourful!

You can read more about managing dependencies in our docs.

As always if you have any feedback on features please message us on Slack or email us at dataform-support@google.com

More content from Dataform

Dataform is joining Google Cloud

Announcement

Dataform is joining Google Cloud

Today we're excited to announce that we've joined Google Cloud. Over the course of the past several months, our partnership with Google Cloud has deepened and we believe that our new combined efforts can make our customers and partners even more successful.
Learn more
Using Javascript to Create an Acquisition Funnel Dataset illustration
Tutorial

Using Javascript to Create an Acquisition Funnel Dataset

How to use a combination of SQL and Javascript to create an acquisition funnel dataset.
Learn more
How to add a new warehouse adapter to Dataform illustration
Guide

How to add a new warehouse adapter to Dataform

I recently wrote a Presto adapter for Dataform during a week long quarterly hackathon. Here's a guide on how I'd do it again.
Learn more

Learn more about the Dataform data modeling platform

Dataform brings open source tooling, best practices, and software engineering inspired workflows to advanced data teams that are seeking to scale, enabling you to deliver reliable data to your entire organization.