Part 1: Building a Strong Data Engineering Team on Databricks and Google Cloud

We'll also explain acronyms and terms so the whole picture is clear.

What "Good" Looks Like: The Target State

A target state is the picture of what your system, people, and processes should look like when everything is working well. It's like drawing the end goal before starting the journey.

1. Architecture (How the system is built)

When we talk about a target state, we need to start with the foundation: the architecture. Architecture defines how all the pieces of technology fit together, how data flows, and how security and governance are enforced. Without a strong foundation, even the best teams will struggle, because problems like inconsistent access, unreliable performance, or ballooning costs will slow everything down. By clearly defining the architectural components—such as Databricks Lakehouse, Google Cloud Platform, and Unity Catalog—we set the stage for a system that is both scalable and trustworthy. This section is included in the target state definition because it ensures that every decision about people, processes, and tools rests on a solid, secure, and future-proof technical base.

2. CI/CD (Continuous Integration / Continuous Delivery)

After defining the architecture, the next part of a solid target state is how changes are delivered safely and consistently—this is where CI/CD comes in. In any modern data platform, teams make constant updates: new pipelines, transformations, dashboards, and governance rules. Without a structured way to test and deploy these updates, every change risks breaking production or slowing down delivery. CI/CD (Continuous Integration and Continuous Delivery) gives you that safety net by automating testing, packaging, and deployment. By including this section in the target state, we highlight the importance of having reliable pipelines, repeatable processes, and secure automation tools. This ensures that the platform doesn't just work once—it keeps working as the system grows, new teams join, and the business demands faster, more frequent releases.

Tools to use:

• Databricks Asset Bundles (DABs): Containers that package code and environment settings together so what works in dev also works in prod.
• GitHub Actions or Cloud Build: Services that run pipelines, checking code quality, running tests, and deploying bundles.
• Terraform: A tool that treats infrastructure (servers, databases, permissions) as code.
• Workload Identity Federation (WIF): Connects GitHub pipelines to GCP securely, without passwords.
• Service Principals and OAuth: Robot accounts and token systems for secure automation.

3. Operations and Governance

The third pillar of the target state is operations and governance—the guardrails that keep the platform secure, cost-effective, and compliant. Even with the best architecture and CI/CD in place, things can quickly go off the rails if usage isn't monitored, permissions aren't enforced, or secrets aren't managed properly. Operations and governance provide visibility into how the system is used, protect against unnecessary spending, and safeguard sensitive information. By including this section in the target state, we make sure the platform isn't just powerful and efficient, but also controlled, auditable, and resilient enough to handle growth and regulatory demands without unpleasant surprises.

4. Process and Way of Working

The final piece of the target state is the process and way of working—how people actually collaborate to deliver value. Technology alone won't guarantee success; the way teams are structured and coordinated makes the difference between smooth delivery and constant fire drills. By combining the Scaled Agile Framework (SAFe) with concepts from Team Topologies, you create both alignment and flexibility. SAFe ensures large groups move in sync by defining clear leadership and coordination roles like the System Architect and Release Train Engineer (RTE). Team Topologies complements this by explaining how teams should be shaped—whether as stream-aligned squads that own a product end-to-end, a platform team that provides shared services, an enabling team that teaches new skills, or a complicated subsystem team that tackles specialized challenges. Including this section in the target state makes sure people know their responsibilities, how they interact with other teams, and how work flows from idea to production—removing ambiguity and accelerating delivery.

Org Topology: How Teams Should Be Organized

Once the target state is clear, the next step is to decide how teams should be organized to bring it to life. This is what we call the org topology—the structure of teams, their sizes, and their responsibilities. Good technology and processes will only succeed if the right people are in the right places, working in well-defined groups. By outlining the org topology, we show how platform specialists, business-aligned squads, short-term enablers, and experts on complex systems can all fit together. This section is included because it translates theory into practice: it explains who will run the platform, who will deliver business value, who will coach and support, and who will handle specialized challenges. A clear team structure removes overlaps, avoids gaps in responsibility, and makes sure every part of the system—technical and organizational—has an owner.

1. Platform (Lakehouse Platform) Team

At the heart of the org topology is the Platform (Lakehouse Platform) Team. This group acts as the backbone of the entire data environment. With about 5–7 people, the team's mission is to build and maintain the shared Databricks and GCP foundation that all other squads depend on. They manage critical pieces like Terraform code for infrastructure, CI/CD templates that standardize deployments, and security measures such as secret management and access controls. They also keep an eye on costs through dashboards and guardrails. We include this team in the design because, without it, every data squad would be forced to solve the same problems repeatedly—wasting time, duplicating effort, and risking inconsistency. The platform team ensures that best practices are baked in once and reused everywhere, giving product squads a solid, reliable base to build on.

2. Stream-Aligned Data Product Squads

The next key building block is the Stream-Aligned Data Product Squad. Each squad, usually 5–8 people, is focused on a single business domain such as finance, marketing, or operations. Their role is to handle the full flow of data for that area—ingesting raw data, transforming it into usable formats, and serving it to analysts or applications. A Product Owner manages the backlog, making sure the team is always working on the highest-value tasks for their business stakeholders. This structure is important because it keeps responsibility and ownership close to the business, rather than separating technology from real-world needs. By aligning squads to domains, you ensure that data products are not just technically correct, but also valuable, timely, and relevant to the people who rely on them.

3. Enabling Team

The Enabling Team plays a short-term but critical role in helping the organization adopt new practices. Usually made up of 2–3 people, this team's main job is to coach and guide the stream-aligned squads on specific skills—such as setting up testing frameworks, adopting CI/CD pipelines, or implementing data quality checks. They don't own long-term delivery; instead, they transfer knowledge and then step aside once the squads are self-sufficient. Including an enabling team in the org topology is important because it prevents delivery squads from getting stuck or slowed down while trying to learn new tools on their own. By accelerating adoption of best practices, enabling teams raise the maturity of the entire organization without creating permanent overhead.

4. Complicated Subsystem Team (Optional)

The Complicated Subsystem Team is an optional but highly valuable part of the organization when specialized challenges arise. This team takes on problems that require deep, rare expertise, such as building real-time streaming pipelines with tools like Google Pub/Sub or managing advanced change data capture (CDC) processes. These tasks are often too complex for stream-aligned squads to handle on top of their regular delivery work. By concentrating experts in one place, you ensure that tough technical issues are solved efficiently without distracting other teams from their core responsibilities. Including this team in the org topology gives the organization the flexibility to tackle specialized, high-stakes problems while allowing product squads to stay focused on delivering consistent business value.

SAFe Roles Recap

To tie the org topology together, it's important to recap the SAFe (Scaled Agile Framework) roles that provide structure and alignment across all teams. These roles ensure that while each squad has autonomy, the larger program moves forward in a coordinated way. The System Architect sets the long-term technical direction and enforces consistent standards, making sure the architecture runway is clear for future work. The Release Train Engineer (RTE) acts as the master facilitator, keeping multiple teams in sync, coordinating release schedules, and helping manage dependencies. The Product Owner (PO) drives business value by managing the backlog, defining features, and deciding what "done" means for the team. Finally, the Scrum Master serves as a coach, helping the team adopt agile practices, improve collaboration, and remove blockers that slow delivery. Together, these roles form the leadership layer that ensures teams stay aligned, productive, and focused on delivering outcomes that matter.

Why This Design Works

This organizational design works because it balances clarity, speed, and flexibility. Each team has clear responsibilities, so there's no confusion about who owns what—whether it's platform stability, business-specific pipelines, or specialized subsystems. The model also supports scalability: as demand grows, you can simply add more stream-aligned squads without redesigning the whole structure. At the same time, governance remains strong because the platform team enforces consistent standards across all squads. Business-aligned squads bring agility, delivering value faster since they stay close to stakeholder needs and own end-to-end delivery. Finally, the structure allows for flexibility—consultants or temporary enabling teams can be plugged in when rare expertise or extra capacity is needed, without disrupting the core organization. Together, these factors create a system that is both stable and adaptable, ensuring the data platform can grow and evolve alongside business priorities.

Practical Roadmap

It's one thing to define the target state and team structure, but the real challenge is knowing how to get there step by step. That's why we include a practical roadmap—to turn strategy into action. A roadmap breaks down the big vision into smaller, time-bound steps that can actually be delivered. It helps leaders prioritize what to do first, ensures teams don't get overwhelmed, and creates visible progress that builds confidence. By outlining milestones for the first 30, 60, and 90 days, as well as longer-term actions, we make sure the journey from concept to execution is structured and achievable. This section matters because without a clear sequence, even the best-designed target state can stall or lose momentum.

Conclusion

What's Next in the Series