Schedules can be used to run any selection of your datasets at a user-specified frequency. Running your code on a repeating schedule ensures that output data is always kept up to date. You can also run data quality tests as part of a schedule and check the data meets expectations. This means you can spot if something is wrong, and implement solutions before clients are impacted.
Dataform provides detailed run logs- what SQL was executed, at what time, and by who - to enable speedy debugging of failing queries and tests. You can also consult the exact SQL code that was run in your warehouse. All current and historical scheduled runs can be found in the Run logs page which lives in the left menu bar. Updates to run logs page
We’ve re-designed the run logs page, making it easier to quickly debug failing queries and view all runs:
- We’ve separated run lists and run views onto different pages
- All dates/times now in user local time
- Run logs now include a graph view
Run view:
- Has a table showing all actions in a run. You can filter this by status or action name.
- Start time and duration are shown for each action.
- Filters go into the URL meaning you can share a filtered view with another member of your team.
- Clicking the ‘Details’ button will show the compiled SQL that is being run and for failing actions will show any errors.
- We tag actions with their type.
- Clicking on an action name will take you to the SQLx file where it is defined.
Run list:
- The environment of a run is now shown as a column in the run list.
- You can search for runs by environments or triggers.
- The action field can now be a selection of tags.
Read more about run logs in our docs here.
As always if you have any feedback on features please message us on Slack or email us at dataform-support@google.com
