8 articles tagged Data Engineering.
A Data Pipeline references a connection, not a string — and that connection has a different ID in every workspace. Here's how the Connection Reference variable type swaps it per environment.
Deterministic hash surrogate keys or IDENTITY columns for your dimensions? A practitioner's pros and cons across load order, idempotency, join cost, and reprocessing.
Variable Libraries plus fabric-cicd's environment parameter give you a versioned, typed way to swap Key Vault names, storage accounts, and other per-environment constants automatically.
Copy pipeline via Virtual Network Data Gateway failing with LakehouseOperationFailed / InternalServerError at the _delta_log commit step? The fix is a single Azure subnet configuration change.
VACUUM keeps Delta tables healthy, but it can also break Change Data Feed replay windows. This walkthrough shows exactly when incremental pipelines fail and how to design a safe Fabric pattern with guardrails and fallback recovery.
Speed up your Fabric pipelines by running notebooks in parallel — without breaking your Spark cluster. Learn how High Concurrency Mode works and how to configure session tags.
When defining identity columns in Databricks tables, two common options for automatic identity value generation are GENERATED BY DEFAULT and GENERATED ALWAYS. This article explains the differences and when to use each.
Azure Data Factory can securely access credentials stored in Azure Key Vault. This article describes how to set up key vault secrets for Azure Data Factory linked services.