Managing dependencies in Dataform
Dataform provides methods that enable you to easily reference another dataset in your project using the ref () function.
This speeds up development as you don’t have to provide the full SQL dataset name and any dataset that is referenced by a query will be automatically added to that query’s dependencies. Dependency queries are always executed before dependent queries to ensure correctness.
During compilation Dataform resolves the ref () function onto the name of the table that will be created and checks for errors including missing or circular dependencies. Dataform then builds the dependency tree of all actions to be run in the warehouse; including tables and views, assertions and other SQL operations that need to run.
Dataform updates the entire dependency tree of your project with every save so you can get a quick visual overview of how your tables, views, and other actions depend on each other while developing.
The dependency tree helps with things such as onboarding new team members as they can easily get up to speed with where all the tables live and how they relate to each other. Livup scaled their data team from 6 to 20 people using the dependency tree as key traning material for new members of the team.
Updates to dependency tree
We’ve made some changes to the dependency tree so that it is far easier to navigate:
- Merged the dependency tree with the data catalog so when you search for a dataset you can see where it sits in the dependency tree.
- Added action types to actions on the graph so you can now see if a node is a dataset, a declaration or an operation.
- Clicking on an action, either in the list or in the DAG itself, shows action details and all dependant and dependencies of that action.
- You can now click and drag to scroll across the DAG making it far easier to navigate (especially if like us you have a big dependency tree)!
- Hovering over an action in the catalog list will take you to the action in the dependency tree.
- Searching for an action using the search bar will only show items that match the search criteria in the DAG as well as the catalog list.
- It's a lot more colourful!
You can read more about managing dependencies in our docs.
As always if you have any feedback on features please message us on Slack or email us at dataform-support@google.com
