16 articles tagged Data Engineering.
Link to Fabric has two cost components: the Delta replica held in Dataverse database storage at $48/GB/month, and the Fabric capacity that consumes it. This covers both, with measured replica-to-source size ratios from a D365 F&O sandbox, current per-GB and per-SKU rates, the pipeline maintenance the premium buys away, and the levers that reduce consumption.
The Delta Lake (Parquet) export option in Azure Synapse Link for Dataverse is deprecated: closed to new customers from October 2026 and removed in December 2027. This covers the exact scope of the deprecation, what remains supported, and how to choose between Synapse Link CSV output and Link to Microsoft Fabric.
A Data Pipeline references a connection, not a string — and that connection has a different ID in every workspace. Here's how the Connection Reference variable type swaps it per environment.
Deterministic hash surrogate keys or IDENTITY columns for your dimensions? A practitioner's pros and cons across load order, idempotency, join cost, and reprocessing.
Variable Libraries plus fabric-cicd's environment parameter give you a versioned, typed way to swap Key Vault names, storage accounts, and other per-environment constants automatically.
Copy pipeline via Virtual Network Data Gateway failing with LakehouseOperationFailed / InternalServerError at the _delta_log commit step? The fix is a single Azure subnet configuration change.
Learn how to enable and leverage Change Data Feed (CDF) with Fabric Mirroring to build true incremental data pipelines and unlock near real-time data processing capabilities.
VACUUM keeps Delta tables healthy, but it can also break Change Data Feed replay windows. This walkthrough shows exactly when incremental pipelines fail and how to design a safe Fabric pattern with guardrails and fallback recovery.
Most lakehouse pipelines are still doing one expensive thing repeatedly: reprocessing entire tables, even when only a tiny fraction of rows has changed. This post explains the CDF-driven load pattern, why it works, where it can hurt, and how to implement it safely in Microsoft Fabric.
A technical look at how Fabric Mirrored Databases replicate Azure SQL Database into OneLake, what the replication pipeline actually does, and where the limits are.
How to keep your Spark jobs portable, safe for concurrency, and easier to orchestrate by using explicit ABFSS paths instead of attaching a default Lakehouse in Microsoft Fabric.
Speed up your Fabric pipelines by running notebooks in parallel — without breaking your Spark cluster. Learn how High Concurrency Mode works and how to configure session tags.
When defining identity columns in Databricks tables, two common options for automatic identity value generation are GENERATED BY DEFAULT and GENERATED ALWAYS. This article explains the differences and when to use each.
How to perform an in-place Synapse Spark pool upgrade for Synapse Link for Dataverse. Originally written for the 3.3 to 3.4 move — updated for Spark 3.5 and Delta Lake 3.0, since Spark 3.4 is now retired.
Microsoft Dynamics is a widely used product for implementing ERP and CRM solutions. A comparison of Export to Data Lake and Synapse Link for Dataverse for D365 Finance and Operations — updated with the confirmed retirement dates.
Azure Data Factory can securely access credentials stored in Azure Key Vault. This article describes how to set up key vault secrets for Azure Data Factory linked services.