Noufal E.

Databricks Lakeflow - Do we still need Azure Data Factory (ADF) in a Databricks world Traditionally, Azure Data Factory played the role of orchestrator, while Databricks focused on transformations. But the landscape is evolving fast. With advancements like Databricks Lakeflow Connect and Lakeflow Jobs, Databricks now supports ingestion, orchestration, event-driven pipelines and even complex control flows (all within a single platform). Microsoft is now focusing towards Fabric Data Factory. This means they are putting more effort into improving and adding new features there. The older tool, Azure Data Factory (ADF) is still working fine and will continue to be supported. But it is no longer the main place where new updates and innovations will happen. So what does this mean for modern data platforms? * If you are building a Databricks centric architecture, it is time to rethink ADF’s role. * Use ADF selectively and mainly for connectors not yet available in Databricks. * Let Databricks handle orchestration to unify pipeline logic, monitoring and CI/CD. Simplify your stack and avoid splitting responsibilities unless necessary. #Databricks #AzureDataFactory #DataEngineering #DataPlatform #DataArchitecture #Lakehouse #DataLakehouse #ModernDataStack #CloudData #BigData #Lakeflow #DataOrchestration #DataPipelines #ETL #ELT #MicrosoftFabric #Azure #CloudComputing #TechTrends #DataCommunity #Analytics #DigitalTransformation

52

2 Comments

Snowflake makes merge easy - MERGE ALL BY NAME When the target table and source must have the same number of columns and the same names for all of the columns, you can simplify MERGE operations by using MERGE ALL BY NAME. MERGE statements can update each column in the target table with the values of the column with the same name from the source table. MERGE statements can also insert rows from the source table into the target table based on column names when there is no match. These MERGE ALL BY NAME operations are supported even when the column order is different in the target and source tables. Example - Traditional way - MERGE INTO CUSTOMER_DIM AS tgt USING CUSTOMER_STG AS src ON tgt.NAME = src.NAME WHEN MATCHED THEN UPDATE SET tgt.CITY = src.CITY, tgt.COUNTRY = src.COUNTRY WHEN NOT MATCHED THEN INSERT (NAME, CITY, COUNTRY) VALUES (src.NAME, src.CITY, src.COUNTRY); New way - MERGE INTO CUSTOMER_DIM AS tgt USING CUSTOMER_STG AS src ON tgt.NAME = src.NAME WHEN MATCHED THEN UPDATE ALL BY NAME WHEN NOT MATCHED THEN INSERT ALL BY NAME; Note - if you merge contains and transformations avoid using this. Official snowflake document - https://lnkd.in/g4R3c_Rk Snowflake Snowflake Developers

2

🚀 Big Data Spotlight: Hadoop Distributed File System (HDFS) When dealing with petabytes of data, scalability and fault tolerance are key. That’s where HDFS comes in. 🔹 NameNode – manages metadata (directory, file info, block locations). 🔹 DataNodes – store the actual data blocks. 🔹 Replication – every block is replicated (usually 3x) across nodes for fault tolerance. 🔹 Client – reads/writes by communicating with the NameNode, while data transfer happens directly with DataNodes. ✅ Why HDFS matters? Handles large files efficiently. Provides high availability with replication. Powers ecosystems like Hive, Spark, Pig, and more. Foundation of big data analytics across industries (finance, healthcare, e-commerce, IoT). 📊 Whether you’re building real-time analytics or large-scale data lakes, HDFS remains a backbone for distributed storage systems. #BigData #HDFS #ApacheHadoop #DataEngineering #CloudComputing

1

Debugging Reveals Architecture Quality If every failure requires manual digging, the system is poorly designed. Senior engineers build observability into the architecture. #DataOps #Observability

7

True fact The biggest ops wins in data engineering are invisible. No outage. No incident call. No escalation. Pipelines keep running. Alerts stay quiet. Costs stay predictable. And demand is met because capacity was planned before it was needed. Most people notice ops only when something breaks. But real ops impact happens much earlier, every single day. If nothing happened in production this week, someone did a lot of things right. Curious how other teams handle this. What ops practice helped you manage scale or reduce incidents? #dataengineering

8

AWS glue pricing details for Normal and Flex Execution modes. ✅ Normal Execution  • Runs immediately on demand — your job starts as soon as you submit it.  • Uses standard on-demand DPUs, typically ~$𝟬.𝟰𝟰 𝗽𝗲𝗿 𝗗𝗣𝗨-𝗵𝗼𝘂𝗿 (varies by region).  • Best suited for 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗮𝗻𝗱 𝗱𝗮𝗶𝗹𝘆 𝗘𝗧𝗟 𝘄𝗶𝘁𝗵 𝘀𝘁𝗿𝗶𝗰𝘁 𝗦𝗟𝗔 𝗱𝗲𝗮𝗱𝗹𝗶𝗻𝗲𝘀.  • Guarantees no queuing — resources are provisioned right away.  • Simple to manage — no risk of waiting or unexpected delays.  • Ideal for time-critical jobs, like data freshness for dashboards or hourly feeds.  • Cost is purely DPUs × runtime × DPU-hour price — your Spark configs just affect how fast the job runs. ✅ Flex Execution  • Runs when spare capacity is available — may wait in a queue before starting.  • DPUs are ~𝟯𝟬–𝟳𝟬% 𝗰𝗵𝗲𝗮𝗽𝗲𝗿, 𝗮𝗿𝗼𝘂𝗻𝗱 $𝟬.𝟮𝟱–$𝟬.𝟯𝟬 𝗽𝗲𝗿 𝗗𝗣𝗨-𝗵𝗼𝘂𝗿.  • Same cluster specs — same DPUs, same vCPUs, same RAM, same performance when running.  • Perfect for 𝗻𝗼𝗻-𝘂𝗿𝗴𝗲𝗻𝘁 𝘄𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀: 𝗱𝗲𝘃/𝘁𝗲𝘀𝘁 𝗘𝗧𝗟, 𝗮𝗱 𝗵𝗼𝗰 𝗿𝘂𝗻𝘀, 𝗯𝗮𝗰𝗸𝗳𝗶𝗹𝗹𝘀, 𝗼𝗿 𝗯𝘂𝗹𝗸 𝗿𝗲𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴.  • Can dramatically cut costs for large or infrequent jobs.  • No code changes needed — you just choose Flex in the job config.  • Not suitable for time-sensitive pipelines — because there’s no guarantee when it will start.  • the Flex option is only available in Glue 3.0 and above. #aws #awsglue #etl #pricing #dataengineering

9

Databricks – Powering the Lakehouse for Scalable Data & AI 🔹 Key Benefits: ⚡ Unified platform for batch & streaming analytics with Delta Lake 📊 Built-in ML & AI support with MLflow integration 🔐 Enterprise-grade governance with Unity Catalog for fine-grained access control 🌐 Multi-cloud support across AWS, Azure, and GCP 📈 Seamless scaling from small workloads to petabyte-scale pipelines 🔗 End-to-end lineage, observability, and audit logging 🧩 Native integrations with Spark, Kafka, Airflow, dbt, Snowflake, and BI tools 🔹 How It Helps Teams: Accelerates data engineering & ML projects by unifying storage, compute, and governance Simplifies ETL/ELT with Auto Loader, Delta Live Tables, and optimized Delta Lake performance Improves reliability & compliance for PII/PHI with automated policies and lineage tracking Reduces operational overhead with autoscaling clusters & serverless SQL/ML runtimes Enables collaborative workflows across data engineers, analysts, and data scientists in one workspace 👉 With 11+ years in Data Engineering, I’ve seen first-hand how Databricks transforms modern data ecosystems. From real-time streaming pipelines with Spark & Kafka to governed enterprise data lakes on AWS and Azure, Databricks has been the backbone of delivering trusted, scalable, and high-performance analytics for business innovation. #Databricks #Lakehouse #DeltaLake #UnityCatalog #DataEngineering #BigData #AI #Cloud #Streaming

22

✨🚀 𝑨𝒑𝒂𝒄𝒉𝒆 𝑺𝒑𝒂𝒓𝒌 𝟒.𝟎.𝟏 — 𝑾𝒉𝒂𝒕’𝒔 𝑵𝒆𝒘 & 𝑾𝒉𝒚 𝑰𝒕 𝑴𝒂𝒕𝒕𝒆𝒓𝒔? I’m excited to share that Apache Spark 𝟒.𝟎.𝟏 has been released — bringing ⚙️ critical stability, correctness & performance improvements to the 4.0 branch. 🔗 Release notes: https://lnkd.in/gJUGTRWi 🔍 🆕 Major Improvements in Spark 4.0.1 🧩 Unlimited JSON size restored — No more JSON size limitations 🛡 Improved table creation logic — better handling of Thrift + Hive compatibility issues ⚡ Fixed race conditions — more stable execution under heavy parallelism 🐍 PySpark improvements — SparkContext logger no longer spams stdout 🧠 Better handling of complex types & state — especially nested StructTypes in Python 🔁 SQL UDF/TVF fixes — cyclic UDF detection, create/replace TVF improvements 📦 Upgraded dependencies — Kafka ➝ 𝟑.𝟗.𝟏, ORC format updates 🧪 More reliable internal build/test infrastructure 🎯 Why This Matters (for Data Engineers like us) ✔ Improved stability on large workloads (🛠 fewer random failures) ✔ Better support for complex schemas & nested data (🔍 ideal for Lakehouse & Iceberg setups) ✔ Cleaner debugging for PySpark (📉 reduced log noise) ✔ More seamless platform upgrades (📦 dependency alignment) ✔ Low-risk upgrade since it’s a maintenance release, not a major jump 🛠 What’s Next for Me 🔄 Running regression tests on key data pipelines 📊 Validating stability improvements on clickstream & large ETL workflows 📚 Updating internal docs on JSON size + schema evolution 👥 Briefing the team on UDF/TVF changes & dependency updates 💭 Have you upgraded yet or planning soon? Would love to hear your experience 👇 Let’s learn from each other! 🔥 Optimizing pipelines is fun, but optimizing stability is a superpower. #ApacheSpark #DataEngineering #Spark4 #PySpark #BigData #DataPlatform #TechLeadership #Lakehouse #depreparation #interview #dsa #bigdata #hadoop #emr #aws #gcp #azure #iceberg #apache

2