Streaming Analytics Practice Project – Real-Time Sales Dashboard

I’ve created this practice project for learning Data Engineering. If you’d like to improve in this area I’d suggest trying to solve it.

Background

The sales team would like to have a real-time dashboard to display the most important numbers regarding the current sales opportunities and total revenue earned.

The team uses a CRM (Contact Management System) to store all their contacts and potential sales opportunities. Currently they manually create metrics in a spreadsheet once a day:

MetricValue
Number of open opportunities2
Number of qualified opportunities1
Number of opportunities won1
Number of opportunities lost3
Total revenue earned200.000 USD

The process of manually updating metrics is error prone, time consuming and slow; Therefore, they’d like to have a real-time dashboard that displays live data automatically.

Your objective is to create a data pipeline that extracts data from the CRM system about sales opportunities and delivers the data professionally to a dashboard.

Note that you can filter by client organization and by milestone as well.

Pipeline

  1. CRM
  2. Event Streaming Platform
  3. Data Warehouse/Data Lake
  4. Dashboard

CRM icon created by Freepik – Flaticon

Setup

  1. Capsule CRM
    1. Sign up for a free account
    2. Create a few contacts and opportunities for testing purposes
    3. Consult the documentation for the API
      1. Capsule API – Capsule API (capsulecrm.com)
      2. Suggestion: try using the API with Postman manually before trying to extract data in a pipeline
      3. Hint: You’ll going to need REST Hooks to get change notifications real time
  2. Message Stream
    1. Research possible solutions for Message/Event Streaming Platforms
      1. Apache Kafka
      2. Google Cloud Dataflow
      3. Azure EventHub
      4. Amazon Kinesis
    2. Sign up for or set up the selected Streaming solution
    3. Integrate Capsule CRM with the Event Stream
  3. Choose a Data Warehouse/Data Lake solution, and integrate it with the selected Event Stream
    1. Databricks Hint
      1. Use Structured Streaming or Delta Live Tables for processing the stream
  4. Choose a visualisation tool e.g. PowerBI or Tableau and create the dashboard shown above.