We are back with a brand new release from Designer Cloud (formerly Trifacta) that delivers more capabilities towards better data engineering. Here are the highlights from the latest 8.7 release from Designer Cloud.
Data Pipelines and Slack Channels
We now facilitate two powerhouses to work with each other. You can create tasks from Designer Cloud Plans, the command center for orchestrating data pipelines to deliver messages to specific Slack channels. This enables effective communication with stakeholders across the organization about the orchestration and execution of data pipelines.
A Slack task is a message from Designer Cloud to a Slack channel and can now be part of the defined set of tasks within a data pipeline. Messages are posted to defined Slack channels that you have access to, and authentication is done through OAuth. Once you create a new task within the plan view of Designer Cloud and define it as a Slack task, you can post the task as a message in the Slack channel. You can also use the metadata from the tasks in the body of the message. As an example, you can include {{$plan.name}}, {{$flow_xyz[‘Output_name’].rowCount}} within the message.
Click here to learn all about posting Slack tasks from Designer Cloud to your channels.
Pre-built Templates are now easier to use
Earlier in the summer, we launched pre-built templates from Designer Cloud to provide a headstart in your data engineering journey. Visit the Designer Cloud Template Gallery for a list of all available templates. These templates provide all the required elements to help transform your data quickly and easily for a wide range of use cases. Now, we make it easier to use these templates directly from Designer Cloud’s Flow View. You can now access the pre-built templates from the Flows page.
On this page, you simply select the template that you wish to use and it is immediately opened in the familiar Designer Cloud Flow View for your use. You can learn all about how to access and use Templates from this article here.
Strict Type Matching for BigQuery
BigQuery supports four types for storing datetimes; DateTime, Time, Date, and Timestamp. In addition to supporting Time and Datetime, Designer Cloud now supports Date and Timestamp data types in BigQuery. This capability ensures BigQuery publications from Designer Cloud are validated for all date/time subtypes such as specific dates against specific times. With this enhancement, invalid data is not published and information is not lost while publishing data into BigQuery.
Datetime values in Dataprep can be published to BigQuery when creating a table using the Create and Drop options, and when updating an existing table using the Append, Truncate, and Merge options. Click here to learn all about BigQuery Data Type conversions supported in Dataprep by Trifacta.
Additional fields to flag recipe steps
One of our popular features with many customers is the ability to flag specific steps in the recipe for review with experts and approvers. Flagging recipe steps helps you re-evaluate the data, provide additional inputs for updates, and confirm before the jobs are executed based on the recipe steps.
To recap, a flagged step in a recipe is marked for review in the recipe panel and must be reviewed before you can move forward with editing follow-on steps in the recipe, or run jobs. The recipe icon is highlighted in the Flow View for easy readability. The Flow View page header summarizes the number of flagged steps and recipes that need review. You can use this functionality to mark steps as reviewed, mark steps as pending review, rename review steps, and unflag for review to clear review for that particular step.
We now support additional fields to flag recipe steps both in the user interface (UI) and the API call. Names of recipe reviewers can be included in the description value of the UI for easy viewing and quick reference. Additionally, your collaborators will also be able to use flows containing flagged steps to help them with clear guidelines as part of their flows. Through the API, you can retrieve the step number of a flagged step in the recipe. These additional fields make recipe reviews easier and more flexible. Learn more from this helpful guide.
That’s it for now, we’ll see you next month with more capabilities as part of our innovation journey.