Automated dashboard and Data Mining

This project was a solution to a repetitive problem in any business, the problem is the time required to collect data, analyze and while doing so working in deliveries, something every business needs. To me this repetitive work feel like a good opportunity to automate it with an high reward in time to the team.

For this problem I investigate multiple solutions, this 2 were the most liked, having some requirements for example, easy to use, highly customizable, and having the opportunity to start using it ASAP.

AirByte and Meltano

Both solutions Open Source, and I really liked how each one try to do it with a lot of thinking in the way of doing so.

I tested both, in that time it was a little more pulish the Hubspot integration in Airbyte, that was the principal reason we stay with Airbyte, and I wish both projects the best.

What is Airbyte?

Airbyte is an Extract Transform Load solution, it enables the creation of Data Pipelines. This data pipelines can have a lot of usages, in many cases enabling the analyses of data from multiple sources or having a historic data. Sometimes this data pipelines can have direct usage to actions, but since you define the interval when the data is extracted if you need something that need an immediate delivery is better going to another kind of data pipelines using webhooks for example.

Automate dashboards, reduce time to results

If you automate the results and you show them in a meaningful way you enable your team to find and optimize the solutions that they are doing. In our case was for real estate companies, if you can have dashboards reporting paid media results, how the sales team are evaluating the new leads, etc. you now have a team who can iterate over the data faster, find problems faster and reducing a lot of human time in reporting.

Then what?

Well you now have one source of truth, you don’t need to go each platform to see results, and you have a place to save the data and look to changes over time, in our case we simplified the data and model it in a galaxy data model, this model enable the possibility to create simplified queries looking for knowledge in the data.