mnemonic_cover

Masthead Data đť—‘ Mnemonic

Masthead Helps Mnemonic Cut Pipeline Compute Costs

About Mnemonic

Mnemonic is the first NFT intelligence data platform designed from the ground up to enable developers, enterprises, and brands to build new experiences that cater to bringing the next billion users to Web3. Trusted by companies including Coinbase, Salesforce, Polygon, and Into The Block, Mnemonic provides the most reliable and comprehensive NFT data and machine learning–powered insights possible to wallets, marketplaces, and brands using innovative Web3 technologies. 

About Masthead

Masthead Data is a data observability platform that helps data engineers and analysts pinpoint data anomalies and identify their root causes in real time. By ingesting only logs and metadata, Masthead provides non-intrusive data observability. Masthead doesn’t run SQL queries to detect data anomalies but processes cloud logs and metadata instead. Masthead Data also offers the ability to identify the compute cost for every pipeline ingesting data from BigQuery or any BigQuery data operation, including data modeling. This allows users to attribute compute costs to cloud solutions as efficiently as possible. 

How We Worked Together

The Challenge

Mnemonic indexes and processes billions of transactions and terabytes of data from blockchains per month and builds complex data and machine learning (ML) pipelines on top of the ingested data to generate actionable insights and make this data available to customers via its API. With some of its largest customers making up to a billion API calls each month, Mnemonic faces significant pressure on its data infrastructure. To handle this demand, Mnemonic relies on PostgreSQL, BigQuery, and Dataform to ensure data remains highly available and reliable in real time. However, under these conditions, observing all pipelines becomes a challenge. Accurately tracking CPU capacity slots consumed by regular queries to monitor BigQuery fees poses a difficulty for Mnemonic. Any bugs or unexpected spikes in CPU consumption can have a severe impact on customer fees, potentially reaching tens of thousands of dollars pers months.

Solution

Masthead Data was the perfect data observability solution for Mnemonic with its ability to track BigQuery fees. This empowers Mnemonic to closely monitor the costs associated with each pipeline and data modeling, granting them complete control over pricing dynamics. In just 24 hours after onboarding, Masthead detected inefficient pipelines that were consuming a significant load of compute power, causing great spikes in Mnemonic’s BigQuery costs. Masthead also helped our client identify numerous irrelevant queries each consuming more than $500 per month. 

Coupled with Masthead’s data lineage and real-time data anomaly detection, BigQuery cost tracking functionality equips Mnemonic with the ability to identify any factors influencing their cloud pricing, enabling them to take appropriate measures to minimize expenditures. Furthermore, Mnemonic achieves comprehensive observability of its data pipelines, enabling them to deliver the most accurate, up-to-date, and reliable data to customers. 

Additionally, Masthead is secure by design, operating solely based on cloud logs and metadata. Because Masthead’s non-intrusive approach poses no hazards to the company’s or customers’ data, the Mnemonic team deployed Masthead within 15 minutes and saw tangible results in less than 24 hours after onboarding. Also, Masthead doesn’t rely on running SQL queries to handle Mnemonic’s data, so it doesn’t consume BigQuery CPU capacity. As a result, Masthead doesn’t contribute to Mnemonic’s cloud costs, allowing the company to effectively manage their expenses.

Results

Masthead helped Mnemonic achieve data observability and reduce cloud compute costs in Google BigQuery by 20% within just 24 hours after the onboarding. With Masthead as their partner, Mnemonic now has the capability to meticulously track all factors influencing their cloud expenses and take appropriate actions to mitigate or control them. Thanks to Masthead, Mnemonic can maintain the reliability of their data without running costly SQL queries. 

” A 24-hour onboarding was enough for finding a couple of pesky queries that were consuming substantial compute cost in our project. Masthead helped us save 20% of our BigQuery costs in 24 hours after onboarding by pinpointing an irrelevant process” – Andrii Yasinetsky, CEO and Co-Founder at Mnemonic .