27 May 2023
Query Optimizer
Masthead Data has recently introduced a new feature that enables you to optimize your SQL queries using OpenAI GPT. With this latest update, you now have the option to select any query captured by Masthead and utilize the AI Optimizer to receive recommendations on how to improve it.
Â
With this feature you can:Â
Â
The AI Optimizer ensures that your table contents remain private and are not shared with the AI agent. It only shares metadata with OpenAI to enhance the query-writing capabilities of GPT.
Select any SQL query from the “Compute Costs” tab and review it. If you wish to improve this query, simply click the “AI Optimizer” button.
Masthead sends a request to OpenAI GPT and provides you with a response containing a recommendation for enhancing the query. It is entirely up to you to decide whether to use the recommended query.
Note that Masthead Data cannot guarantee the accuracy of the output in every instance. Hence, we strongly advise you to thoroughly review the recommended query before executing it.
27 Apr 2023
Column Data Lineage
Masthead Data has just released an update to its data lineage feature, allowing you to gain the most comprehensive insight into your data. Going forward, you’ll be able to see the connections between your data tables down to the smallest details of each table column. By highlighting related columns across different tables, Masthead enables you to understand how each column affects downstream data.
Â
With this update, you get:
While checking your data lineage, click the number of columns in a table in the Lineage tab.Â
You will visualize your table column dependencies and see how each of your table columns affects related columns across different tables.
14 Apr 2023
Integration with Looker
Masthead Data offers you complete control over the quality of your Looker’s data visualizations and dashboards. No one is immune to bad data, which can quickly spread and impact the quality of downstream tables that Looker dashboards rely on. By integrating Masthead with Looker, you can quickly identify reports and dashboards affected by data anomalies and easily visualize the root cause of the problem with data lineage.
Â
This update provides you with the following benefits:
Follow the instructions to create Looker configurations and allow Masthead to observe your Looker dashboards and real-time reports.
Visualize your table dependencies with data lineage to see how bad data spreads across your data tables and identify Looker reports and dashboards affected by the data anomaly.
30 Mar 2023
In-depth cloud cost tracking. Beta
Masthead Data introduces advanced functionality for managing your BigQuery fees. Normally, BigQuery only shows the total cost of using the cloud service, without any information on how individual pipelines and queries contribute to this fee. However, with our latest update, you can easily track the cost of each pipeline and query, giving you complete control over your pricing dynamics. Our new feature includes the following benefits:
When you onboard to Masthead, define your BigQuery pricing model.Â
Indicate the region where your data centers are located and see the price you pay for a terabyte of cloud resource or 100 slots of cloud compute power, depending on your commitment plan.
With our new feature, you will get a detailed overview of TB or slots consumed for running your pipeline, as well as detailed analytics on your cloud resource consumption and pricing dynamics. Here you can also check how additional database tools and services, such as dbt, contribute to your cloud cost.
With our latest feature, you’ll get a detailed breakdown of how much cloud resource you’re consuming or how many slots you’re using, and in-depth analytics on your consumption and pricing trends. You can even see how additional database tools and services, like dbt, are impacting your cloud cost.
If you want to review the pricing dynamics of individual queries, you can easily do that too. If you’re using an On-demand BigQuery pricing plan, Masthead will show you the query’s frequency and time, average memory usage in terabytes, average cost per terabyte, and total cost. For Flat-rate BigQuery subscribers, you’ll see an overview of the query’s frequency and time, the number of slots required to run it, average cost per slot, and total cost.
On top of that, you are able to check the query and its lineage of your pipeline.
With Masthead’s detailed analytics and breakdowns of your cloud resource consumption and pricing, you can stay on top of your cloud cost. By identifying which pipelines and queries are consuming the most resources and cost, you can make informed decisions and optimize your usage to pay less and save money.
26 Dec 2022
Lineage improvements
Effective data governance is critical for making informed and reliable decisions. As data engineers handle vast amounts of data at high speeds, it is crucial to have a clear understanding of the origin and transformation of the data. This is where data lineage comes in – it helps to ensure that the data being used is trustworthy, accurately transformed, and stored in the correct location. By tracking the lineage of data, organizations can ensure that strategic decisions are based on high-quality data.
We are excited to announce an update to our data lineage:
Masthead Lineage feature now includes visual highlighting to clearly identify tables that have been impacted by data anomalies or errors. Grey highlighting is also used to show downstream tables or views that may be affected by these errors.Â
In addition, we have carefully evaluated the color contrast of our Masthead UI and data lineage tools to ensure they meet accessibility standards outlined in the Web Content Accessibility Guidelines (WCAG). This includes considerations for individuals with color blindness, making our platform more inclusive for all users.
The Masthead Lineage now provides types of data in column-level lineage, allowing data engineers quickly see the types of data within each column.
This level of granularity is crucial for understanding the transformation and flow of data and its use downstream.
14 Nov 2022
Jira Integration
Masthead takes a solid step towards more efficient issue management. From now on, you can immediately find people associated with the detected data errors and collaborate to solve data issues faster. Jira integration enables:Â
How does this feature work?
When you review a particular error in a table (dataset, project), if applicable, you can see a service account that is associated with this error.
Use the “Push to Jira” button to automatically create a Jira ticket that can be populated to the Jira board of the appropriate team to resolve it faster. Â
The team will receive the notification about the new ticket. The ticket includes the problem summary, details, and a description or other required information. After that, the appropriate team can timely start analyzing and resolving the data issue without any back-and-forth messages needed for problem clarification. This allows you to speed up task delivery and distribution, as well as helps you process the problem in the shortest terms.
Once the Jira ticket is created at Masthead UI, you can check its number and status until it is resolved. So everyone on the team has a shared understanding if someone takes care of the data issue and its progress.Â
Find out how Masthead helps you deliver reliable data products at scale.
Copyright © 2023 Masthead
