Day by day Extraction of Azure Cloud Costs

…is it possible ??

Hello everybody, this story is dedicated to all those people that have encountered my same issue while facing Azure Cost Analysis Exports. So, I suppose you are already familiar with that feature on azure. However let’s make a brief recap to introduce the problem.

Azure Cost Analysis Exports

Azure cost analysis is a feature provided by Microsoft Azure that helps users monitor and optimize their spending on Azure resources. It provides insights into how much money is being spent on different services, and offers recommendations for reducing costs. Azure cost analysis enables users to track usage patterns, identify areas where cost savings can be made, and forecast future spending based on usage trends. The tool offers a range of reports and visualizations, making it easy to understand and manage Azure costs. Overall, Azure cost analysis is a valuable tool for businesses using Azure to manage their cloud spending and optimize their budgets.

Some of the most important features of Azure cost analysis include:

1. Cost Management Dashboard: This provides an overview of your Azure spending, usage patterns, and recommendations for cost optimization.
2. Cost analysis: Azure cost analysis allows you to analyze your costs in detail, including spending trends, usage by service, and spending by resource group.
3. Budgeting and forecasting: You can set budgets and forecast your spending based on usage patterns and trends.
4. Cost alerts: Azure cost analysis lets you set up alerts when spending exceeds certain thresholds or when you approach budget limits.
5. Cost optimization recommendations: The tool provides recommendations for cost optimization, such as resizing underutilized resources, turning off unused resources, and optimizing reserved instances.
6. Integration with other Azure services: Azure cost analysis integrates with other Azure services, such as Azure Advisor, to provide comprehensive recommendations for cost optimization.

Overall, Azure cost analysis provides a range of powerful features that help businesses monitor and optimize their Azure spending. Refer to documentation for more.

Azure Cost Analysis Export

But there is another feature it can really useful for who intend to customize analyse data through external tools. Me, personally, as a data scientist, it was something concerning me to export cost data and use python and pandas to explore and transform them. And there are a lot of occasion in which it can be useful. With Cost Export, users can export their cost and usage data in a variety of formats, including CSV, JSON, and Parquet. This feature allows users to create custom reports and analysis using their cost and usage data in conjunction with other business data.

N.B. Exported data will be sent to a destination container in an azure storage account, so you could in advance configure one.

But how to create an Export?

To either generate or view a data export, or to schedule an export, first select a scope within the Azure portal and then select “Cost analysis” from the menu. As an example, you could navigate to “Subscriptions”, choose a subscription from the available list, and then select “Cost analysis” from the menu. Once on the “Cost analysis” page, click on “Settings” at the top and then select “Exports” to proceed.

For any doubt whatch the video below and see the official guide.

Here’s a summary:

1. Click on the “Add” button and enter a name for your export.
2. Choose the type of cost data you want to export between:
   a) Actual cost (Usage and Purchases)
   b) Amortized cost (Usage and Purchases)
3. Select the type of export you want to create between:
   a) Daily export of month-to-date costs
   b) Weekly export of cost for the last seven days
   c) Monthly export of last month’s costs
   d) One-time export
4. Specify the subscription for your Azure storage account, then select a resource group or create a new one.
5. Select the storage account name or create a new one.
6. Select the location (Azure region) where you want to store your export data.
7. Specify the storage container and the directory path where you’d like the export file to be saved.

Let’s focus on the Exports Type !!

In order to understand the issue I encountered we have to analyze the types of export we can extract:

1. Daily export of month-to-date costs: This type of export provides a new file every day that contains the costs for the current month up to the day of the export. The data in the file is the aggregation of the previous daily exports.
2. Weekly export of cost for the last seven days: This type of export creates a weekly export of your costs for the past seven days from the selected start date of your export.
3. Monthly export of last month’s costs: This type of export provides you with an export of your last month’s costs compared to the current month that you create the export. Afterward, the schedule runs an export on the fifth day of every new month with your previous month’s costs.
4. One-time export: This type of export allows you to select a date range for the historical data you want to export to Azure blob storage. You can export a maximum of 90 days of historical costs from the day you choose. This export runs immediately and is available in your storage account within two hours.

Do you think we are missing something ????

No.

That’s all Microsoft Azure is offering as cost exportations. But does it meet our project requirements ?

Maybe yes ….or maye no.

In my case I was producing a FinOps Platform.

Everyday I need to collect cost data to move them on a owned database.

What was the first problem ?

Cost data in Azure need five dates in order to maturate completely. In fact, for example, if you use Daily export of month-to-date costs, the schedule will continue to send data every day for a specific month till the 5th day of the following month. Lucky, after a little analysis and research, I can conclude data after two days represents usually almost the totality of costs (99,99%).

So we can use the export of the current day as a valid cost for the date of two days ago. Obviously until the current day is not yet finished we can no take as good the cost of the day, while the cost of the previous day (yesterday let’s say) could represent a smaller or larger portion of the total cost, it usually depends by the time in which the schedule is extracting them.

I have to say anyone can manage this data as he want on the db, this story does not want to cover that argument.

So what was the main problem covered by this story?

ok, we got that by using Daily export of month-to-date is possible to obtain new data on daily basis about the whole month. But if for any reason, memory or whatever else …I want to obtain a daily export of the day cost.

In case the problem is to not overuse the storage account, an easy solution can be to customize some storage account in order to delete data time to time an an automation account runbook or a serverless function.

The code above works in order to clean the container of the cost export time to time. For each month the script removes all blobs except the last one. Personally I used it as an Azure Function with a time trigger. If you do not know how to create function see here.

The solution explained here below however is a little bit more sophisticated, but still use an Azure function.

Daily export of day-to-date

As you should have understood, the title of this paragraph refer to a “not-existing” type of export… but we are going to create it.

So, we need a custom export. First idea could be to use the One-time export …and that’s a good one.

But there is a problem.. in fact, this type of export only works one time… and it can not be scheduled.

The Cost Management APIs

With a quick research we can find Microsoft offers some Rest APIs to manage costs in Azure.

By the documentation we see there are three different groups of operations:

• Dimensions: allow you to obtain the supported dimensions for your usage data across multiple scopes
• Query: enable you to obtain aggregated cost and usage data based on the query you specify.
• Export: offers operations that allow you to schedule recurring exports of cost and usage data to blob storage.

Both the Query and the export APIs have the flexibility to filter, sort, and group the data by any available dimensions which can be accessed through the Dimensions API…

…or almost this is what you can infer by the documentation; but this is not the reality unfortunately. In fact, even if the Query API seems to be perfect for out necessity, the documentation is inconsistent and misleading; it can not be trusted and the API resulted to me (after various tricky attempts) not usable at all.

At the opposite, the Export API works good enough. It follows the documentation and it worsk correctly.

So let’s come back to the idea of the One-time export. With the API you are able to update an export and even to execute it. Everything seems to be so clear now.

We need a scheduled function that every day to update an existing One-time export changing the time extremes and then execute it.

In the Export API page the various possible operations are listed. It is clear that to run the export we need to call the execute operation. While in order to update an export we have to do a post call to the Create Or Update API, providing in the body not only the desired export settings, but even the eTag. In fact, to update an existing export (we do not want to create a new one) we have to specify the eTag of the export itself.

What is the eTag ?

It is a sort of identifier for the export, such that the API in this case could unequivocally identify the export you want to modify.

To know this eTag we have to make a further API call. The get operation allows to easily retrieve the eTag.

Summarizing our function need to make three API calls in total:

• a GET to the API here to retrieve the eTag
• a PUT call to the API here to update the existing export (providing the previously retrieve etag
• a POST call to execute the updated cost export

Final Solution is the Azure Function !

Following the three steps first identified, I decided to use python inside an Azure Function. The serverless function can be easily scheduled… it can run everyday a the same hour using the time trigger (see this tutorial).

Prerequisites

In order to develop this solution you need to do actions in advance:

1. Create a storage account and a container inside it; the cost data will be sent there.
2. Create an One-time export: it is not really important how you set it, when the function will run it will be update ad modified correctly.
3. Create a managed identity in Azure and give that permissions to manage cost.
4. Create a Function App that will host your function
5. Assign the managed identity to the Function App, this will garantee your function the access to the APIs.

The Function

The function follow the three calls already explained. To make the calls you need to authenticate using the managed identity credential and retrieve the barer token that will be an header in the API calls.

As you can see the function follow exactly the three steps previously indicated. Obviously it’s the most general one I could write, but anyone can adapt it to his own requirements.

This was the solution I adopted for my project. I hope it can be useful to someone! :)

So keep in touch and follow for more 🫠

BYE