Supermetrics General Structure
Although it seems very simple to connect to data sources and push the data to Data Studio, Google Sheets, etc. it’s a pretty complicated task to achieve. Let’s run through the steps of creating such a tool like Supermetrics:
- Connecting to a data source – Each data source (e.g. Google Ads, Facebook Ads, HubSpot) has it’s own dimension, fields , logic, restrictions and authentication method. There’s no one way to connect to all the data sources and you first need to learn their API, understand the logic of queries what kind of data you can and cannot pull, what are the limits and quotas and how the results appear. This is a complicated task that can take months over months of trial and error until you get a decent result per your query. In addition, you need to handle all error and server errors your get from the data a source. As a tool, like Supermetrics, you cannot show the user a message like “Error 142, insufficient allocation blah blah“. You need to understand this error and show the user an alternative message, in human language. As an alternative, you can develop a logic that prevents the user from creating a query that can result in such error.
- Choosing data-target – So, you were able to connect to a YouTube channel, but you need to push the data somewhere. It can be Google Sheets, Excel, Google Data Studio, your own Database or whatever. Each of these “targets” has its own logic and methods. You need to decide how the data should look like. Is it a table? a chart? How Excel allow external tools to push data? Can I bring data to my MySQL database and what it requires? Now, the same way you worked your way connecting to data sources, you need to design the way you push the data in.
- Design a user interface – Now that you know how to pull data from a selected data source, and to push it somewhere, you want to give your end-user a way to create his own queries, so you create a query builder. This is where your end-user can choose to which data source he wishes to connect, what’s the date range, which metrics & Dimension he wants to see or adding filters. This is the “front-end” of your app.
- Keep it up-to-date – Even when everything works, most data sources change their API’s constantly. You need to be on top of that task 24/7. You should be recognized as a certified technology partner in some data vendors so you’ll get priority in the API announcements and features and handle any change in no time to support your user queries.
Now let’s run through some other things you need to know as an end-user:
Is It Secure
Most data fetchers like Supermetrics use the same concept. They do not ask you for your user/password for the data source, but instead, they use a common method called OAuth. If you wish to grant Supermetrics access to your Pinterest account, Supermetrics will request Pinterest to show its login page. This page is NOT located on Supermetrics servers, but on Pinterest servers. This login screen is secured and managed by Pinterest and not by Supermetrics which has no influence whatsoever on that screen. This is where you put your credentials, Pinterest processes them and then it shows you another message which tells you which permissions Supermetrics requires, e.g. access your email, manage your account, pull ads data, etc. Supermetrics and most of the data fetch tools, tend to request the minimum permissions. They do not change the data, only reading it. Once you permit it, Pinterest passes to Supermetrics a secret token, a code. With this token, Supermetrics make all the requests to fetch the data. Even if Supermetrics has a security breach and someone takes all these token, he cannot do anything with that since only Supermetircs can use it. If you change your Pinterest password, you sometimes need to re-authenticate Supermetrics. With the OAuth method, Pinterest can disconnect all users at once if it feels there’s a security breach.
There are some exceptions though. Some data sources, do not use OAuth at all, or any similar method. In order to fetch the data, Supermetrics needs you to supply your user and password. Yes, it’s not the standard way, to say the least, and although I’m sure Supermetrics and other vendors stores your credentials in a secured area, shit can happen. You’ll get a proper warning when connecting to a data source without the standard OAuth.
Authentication Issues
When you’re fetching data from YouTube using Supermetrics, Supermetrics or any other vendor for that matter, use a special token. A token is a key that they can fetch the data on your behalf. For many technical reasons, sometimes the data source (e.g. YouTube) can decline this token, or force the user to re-authenticate. Sometimes the authentication service fails to process this token and we get some wired errors over and over again. It happens a lot and usually, there’s nothing the vendor can do with but to nag the data source about this issue or bug.
Missing Dimensions & Metrics
The list of all possible dimensions and metrics is initially configured and approved by the data vendor to be accessed by 3rd party tools. Hence, Facebook can decide that a specific metric that’s available on your Facebook Ads interface, will not be part of their API and you cannot access it using Supermetircs or any other tool. There’s nothing we can do about it but to cross fingers and hope it will be in the next versions. Some 3rd party vendors might create their own fields to replace the missing one.
The Dimensions & Metrics Names Can Vary
Unfortunately, for some weird reason, some data vendors change the names of metrics/dimensions in their API, so they are different from the known names you can find in the original user interface. For example, you may see in Google Ads user interface the metric “% reach” but on their API it gets the name “Reach ratio” or something. Some of the 3rd party vendors, like Supermetrics, may “translate” it back to the original name so you’ll see the original names across Supermetrics UI, but it’s not the case all the time. When I’m looking for a specific metric and cannot find it, I usually add all the metrics with a similar name and see which shows the right data. Yes, it’s primitive.
Metrics & Dimension Can Be Deprecated For No Reason
Now that happens a lot. You’re using your favorite dimension/metrics and gets a message from your data fetching tool, like Supermetrics, that this field is deprecated soon, not to say, tomorrow and that you should update all of your reports. Now, sometimes, they say the name is changed, sometimes they say that you should use an alternative field and sometimes, well, it’s just gone with no explanation.
