New and improved dependency tree! | Dataform
Product Update

New and improved dependency tree!

During compilation, Dataform builds a dependency tree of all actions to be run in the warehouse. We’ve made some changes to the dependency tree so that it is far easier to navigate.

Features
picture of author

Josie Hall on June 4, 2020

Managing dependencies in Dataform

Dataform provides methods that enable you to easily reference another dataset in your project using the ref () function.

This speeds up development as you don’t have to provide the full SQL dataset name and any dataset that is referenced by a query will be automatically added to that query’s dependencies. Dependency queries are always executed before dependent queries to ensure correctness.

During compilation Dataform resolves the ref () function onto the name of the table that will be created and checks for errors including missing or circular dependencies. Dataform then builds the dependency tree of all actions to be run in the warehouse; including tables and views, assertions and other SQL operations that need to run.

Dataform updates the entire dependency tree of your project with every save so you can get a quick visual overview of how your tables, views, and other actions depend on each other while developing.

The dependency tree helps with things such as onboarding new team members as they can easily get up to speed with where all the tables live and how they relate to each other. Livup scaled their data team from 6 to 20 people using the dependency tree as key traning material for new members of the team.

Updates to dependency tree

We’ve made some changes to the dependency tree so that it is far easier to navigate:

  • Merged the dependency tree with the data catalog so when you search for a dataset you can see where it sits in the dependency tree.
  • Added action types to actions on the graph so you can now see if a node is a dataset, a declaration or an operation.
  • Clicking on an action, either in the list or in the DAG itself, shows action details and all dependant and dependencies of that action.
  • You can now click and drag to scroll across the DAG making it far easier to navigate (especially if like us you have a big dependency tree)!
  • Hovering over an action in the catalog list will take you to the action in the dependency tree.
  • Searching for an action using the search bar will only show items that match the search criteria in the DAG as well as the catalog list.
  • It's a lot more colourful!

You can read more about managing dependencies in our docs.

As always if you have any feedback on features please message us on Slack or email us at team@dataform.co

More content from Dataform

What can data teams learn from Google’s State of DevOps report? illustration
Opinion

What can data teams learn from Google’s State of DevOps report?

Google's research shows us that elite engineering teams have a lot in common when it comes to DevOps. We think similar principles apply to data teams or, more precisely, DataOps.
Learn more
Cut data warehouse costs with run caching illustration
Product Update

Cut data warehouse costs with run caching

How to save time and money by using our run caching feature
Learn more
Building the Dataform VS Code extension illustration
Guide

Building the Dataform VS Code extension

How we made our own extension for Visual Studio Code.
Learn more

Learn more about the Dataform data modeling platform

Dataform brings open source tooling, best practices, and software engineering inspired workflows to advanced data teams that are seeking to scale, enabling you to deliver reliable data to your entire organization.