Skip to main content

Techie Talks Podcast - Architecting Data at Scale

Hi Deepak,

Thank you for agreeing to join me on the show. Your deep experience in data engineering and scaling complex systems is a topic our audience will find incredibly valuable.

This document outlines the planned flow for our 30-minute conversation. Please use it as a guide to prepare. The goal is a natural, insightful discussion, so please feel free to share personal stories and project examples.

Looking forward to our conversation! - Kuldeep

Part 1: The Data Engineer's Mindset (Approx. 8 mins)​

Kuldeep's Question: Welcome, Deepak! With over 8 years of experience, you've seen a lot. How would you describe the core role of a modern data engineer, and how has that role evolved over your career?

My Points:

  • Democratization of learning, system designs
  • Today GenAI is revolutioning the way we code at each level
  • Keep it simple
  • Architecture is important
  • Choice of data pipelines, event driven architectures, data warehouses (special mention to DuckDB).

Talking Points for You:

  • Moving beyond just building pipelines to designing scalable, efficient data ecosystems.
  • The importance of leading multi-disciplinary teams (devops, data science).
  • How you balance hands-on engineering with high-level system design.

Part 2: Deep Dive: Architecture for Speed & Scale (Approx. 12 mins)​

Kuldeep's Question: Your profile mentions using microservices, GraphQL, and Kafka to enhance development speed and flexibility. Can you walk us through a project where this combination was critical? What problem were you solving?

My Points:

  • 3 different technologies - Microservices, GraphQL and Kafka

  • It was for a startup, now a large IoT company, where you have to stream millions of data points each minute. There we need to move fast.

  • 10s of thousands of devices all over the country.

  • Druid as the data warehouse

  • Microservices are important

Talking Points for You:

  • Set the stage: Describe the business challenge or the system you needed to scale.
  • Explain how each technology played its part (e.g., Kafka for data streaming, microservices for independent deployment, GraphQL for efficient data fetching).
  • The "before and after" impact: How did this architecture improve things?

Kuldeep's Question: On the infrastructure side, you have expertise in tools like Elasticsearch, Prometheus, and Kubernetes. How do these tools support the data-intensive applications you build?

Talking Points for You:

  • The role of Kubernetes in managing and scaling your microservices.
  • How you use Elasticsearch for search and analytics on large datasets.
  • Why Prometheus is crucial for monitoring these complex, distributed systems to ensure reliability.
  • You cannot go blind, you need observability at each level
    • Tools like - SigNoz

Part 3: Leadership & Final Advice (Approx. 10 mins)​

Kuldeep's Question: You're passionate about mentoring and coaching. What's your approach to leading technical teams, especially when working with multiple stakeholders?

My points

  • Hands-on approach
  • Availability
  • Motivation
  • Letting them learn from their mistakes
  • Not spoon feeding them
  • Nudging them to the right direction
  • Extreme Ownership

Talking Points for You:

  • The importance of strong communication skills in translating technical concepts to business stakeholders.
  • How you foster a culture of learning and growth within your teams.
  • A key lesson you've learned about effective technical leadership.

Kuldeep's Question: Finally, what is one piece of advice for engineers who want to specialize in data engineering and eventually lead teams as you do?

My Points

  • Keep on learning, writing things, doing things and at last taking ownership.

Talking Points for You:

  • The need to understand the full data lifecycle, not just one tool.
  • Why developing business acumen is just as important as technical skill.
  • A final key takeaway for building a successful career in this field.