A Risk-Managed Migration from Netezza to Databricks

Best practices that moved the needle

Four practices did the heavy lifting at CN.

Discovery before design. Before any conversion began, we inventoried the tables, pipelines, dependencies, and consumers, and we forced an explicit split between flows that were critical to operations and flows the business was willing to deprecate. This single step is what separates a controlled lift and shift from an unplanned rebuild, because it surfaces hidden complexity while it is still cheap to deal with.

A factory model with defined gates. Dozens of streams cannot move on heroics. We ran the program as a factory, with clear entry and exit criteria for every phase, so each stream cleared the same quality bar in the same way. Consistency, not effort, is what scales.

Automation with judgment held in reserve. A converter does the bulk of the work, and in a typical estimate the large majority of objects fall into the easy and medium buckets. The discipline is to keep human attention for the small tail of complex and very complex objects, where a careless automated conversion does the most damage.

Medallion architecture as the trust mechanism. Bronze mirrors the source exactly. Silver is where mappings, validations, and change data capture live. Gold serves the business. The hardest problems hide in the time dimension and the unique keys, so we resolved what every timestamp meant and which one identified the newest version of a record before we touched silver.

Managing the risks that actually matter

In a Netezza migration the dangerous risks are seldom the ones on the converter dashboard. They are the assumptions hidden in the appliance, the data the business trusts, and the handoffs between teams. We managed an entire register of these, and the most instructive are below. Each one carries the risk, the mitigation we ran, and the fallback when the mitigation was not enough. The full register, with every entry scored and tracked, accompanies this article as a separate working tool. Access the register here for your own use.

Undocumented mappings and business logic. The program ran cleanly into the transformation layer, then risked stalling because the rules behind trusted tables were never written down. We made confirmed mappings a hard entry criterion: silver work on a domain did not begin until its rules were validated by the customer's subject matter experts. When a gap appeared mid wave, we flagged the rework immediately, reprioritized so unblocked domains kept moving, and quarantined the disputed logic rather than guessing at it.

Constructs with no direct equivalent. Procedural code written in NZPLSQL, temporary tables, and column constraints such as UNIQUE and DEFAULT do not exist in the same form on Databricks. A literal conversion either fails or, worse, silently changes behavior. We refactored stored procedures into a combination of SQL and Python notebooks, replaced temporary tables with session scoped temporary views, and rebuilt constraints as explicit validation logic, establishing each pattern once and reusing it across waves. The genuinely hard cases were routed into a manual track so they never gated the automated flow.

Function level behavioral differences. Some code compiles and runs but produces different answers. A documented example is add_months, which handles leap years differently across the two platforms. That kind of difference passes a syntax check and fails a business audit. We maintained a catalog of known differences and added targeted tests for every function on it. When a difference surfaced in a parallel run, we corrected the pattern centrally and re-ran validation rather than patching one job at a time.

Transaction and rollback model differences. Teams assume Netezza concurrency semantics carry over. They do not. Delta defaults to a write serializable isolation level, under which a reader can briefly see a state the table history says never existed, and Netezza's ROLLBACK has no direct equivalent. We set isolation to serializable where concurrency correctness mattered and replaced rollback patterns with Delta versioning for audit and recovery. For the most sensitive pipelines, we ran them single threaded through cutover and lifted concurrency only once results were confirmed.

Data that does not reconcile at cutover. The program can reach go live only to have the business withhold sign off because no one can prove the numbers match. We treated validation as continuous rather than as a final event, running row count, column level, row level, and subset checks every cycle. When variance remained, the contingency was simple and firm: extend the parallel run and hold cutover until the difference was either zero or explained and accepted in writing.

Consumers stranded during transition. Decommissioning a source or cutting a pipeline before every downstream consumer has moved takes a critical report dark. We ran legacy and Databricks pipelines concurrently with temporary data flows so no consumer was ever without a source, and we staged the decommissioning of Netezza only after each consumer was confirmed cut over. The legacy path stayed warm and reversible until the final consumer was verified, which made a rollback a switch rather than a rebuild.

Two further risks deserve a mention even in summary, because they sink programs quietly: environments that arrive late, which we handled by requiring development and integration environments provisioned ahead of the start and routing any delay through formal change control so its schedule impact was visible rather than absorbed; and compute budgets sized against the appliance footprint rather than real access patterns, which we caught by validating budget against actual query behavior in discovery and escalating scope early instead of at the milestone.

The through line across every contingency is the same. The agile, wave based model gives a problem somewhere to live. A risk that surfaces inside a two week wave is a manageable event. The identical risk inside a big bang is a program failure.

Quality as a first class citizen

In a high volume, governed environment, the thing that lets a program move fast and still earn sign off is quality that is built into the pipeline rather than inspected at the end. This is where our accelerators earned their place. AutoDQ generates and enforces data quality rules inside the pipeline, so trust is continuous and decay is caught the moment it starts rather than when a downstream report breaks. i-QA generates test cases around datasets, which is what allows reconciliation to scale across thousands of objects without a proportional growth in manual effort. Embedded quality is not a luxury at this scale. It is the mechanism that makes confidence repeatable.