Big Data News and Headlines
Big Data News and Headlines
Big Data Pulse
EMAIL DIGESTS
Daily
Every morning
Weekly
Tuesday recap
EMAIL DIGESTS
Daily
Every morning
Weekly
Tuesday recap
Delta Lake’s Change Data Feed (CDF) lets engineers capture row‑level changes as soon as they occur, turning a Delta table into a built‑in change‑data‑capture engine. By enabling the table property delta.enableChangeDataFeed, only modified rows are read, eliminating costly full‑table scans for incremental ETL. The feed integrates seamlessly with Databricks Structured Streaming, supporting both continuous and micro‑batch pipelines, while also offering batch reads for audits or backfills. CDF’s metadata—including change type, version and timestamp—provides reliable, exactly‑once processing across schema evolution and concurrent writes.
Redpanda announced the general availability of its adaptable R1 streaming engine in Redpanda Streaming 26.1, a single‑modal platform that lets enterprises tailor performance, safety and cost at the topic level. The release integrates Cloud Topics, write caching, tiered storage, and Iceberg...
European policymakers are pushing digital sovereignty, yet the missing piece is data ownership. As AI models become commoditized, control over the underlying data is emerging as the true competitive moat. Countly, an open‑source, self‑hosted analytics platform, illustrates how firms can...
Wiliot announced a Built‑On partnership with Databricks, moving its Physical AI supply‑chain platform onto the Databricks lakehouse. The shift lets the company ingest and govern massive item‑level IoT Pixel data streams in a unified environment. By leveraging Databricks’ compute and...
NomadicML raised an $8.4 million seed round at a $50 million post‑money valuation to commercialize its vision‑language platform that auto‑annotates autonomous‑vehicle video. The tool transforms terabytes of archived footage into searchable, structured datasets, enabling rapid identification of rare edge‑case events for training...
Enterprise data warehouses are increasingly seen as costly, inflexible assets, prompting a shift toward open‑source alternatives. EDB Postgres AI introduces WarehousePG, a PostgreSQL‑based, petabyte‑scale MPP warehouse that promises up to 58% lower total cost of ownership while delivering predictable performance. The...
Profisee has launched the general‑availability of its 2026 R1 release, positioning the cloud‑native MDM platform as an AI leader. The update introduces a Model Context Protocol (MCP) Server that creates an open standard for linking master data to AI tools such...
SAP announced acquisition of Reltio, integrating its master data management platform into SAP Business Data Cloud. The move gives SAP control over the enterprise master data layer, especially for customers with mixed SAP and non‑SAP environments. By embedding Reltio, SAP...
CIBO Technologies has entered a three‑year strategic partnership with Ingredion to expand regenerative agriculture across its supply chain. The collaboration will use CIBO’s data and analytics platform, including AI and computer‑vision tools, to enroll and support farmers in Iowa and...
The article lists seven common data‑preparation challenges—poor profiling, missing or invalid values, name/address inconsistencies, cross‑system data mismatches, enrichment hurdles, and scaling issues—and offers practical ways to address each. It highlights that data preparation typically consumes the majority of effort in...
PostgreSQL relies on periodic vacuuming to reclaim space from dead tuples created by its MVCC architecture and to prevent transaction ID wraparound. The built‑in autovacuum daemon, enabled by default, triggers when dead rows exceed a threshold of 50 rows plus...
Palantir’s UK executive warned ministers against yielding to “ideologically motivated” campaigners as they consider invoking a break clause in the NHS’s £330 million (≈ $413 million) Federated Data Platform contract. The AI‑enabled platform is projected to generate £150 million (≈ $188 million) in benefits by 2030,...
Microsoft unveiled Database Hub and Fabric IQ at FabCon and SQLCon 2026, extending its Fabric SaaS analytics platform to unify roughly 20 data services under a single management plane. The new Database Hub adds AI‑driven, natural‑language exploration across Azure SQL, Cosmos DB, PostgreSQL and...
The Open Data Institute’s four‑month NDL‑Lite prototype scanned more than 100,000 public datasets from six UK sources, exposing pervasive labeling gaps, outdated records, and accessibility hurdles. Notably, a major Home Office crime dataset has not been refreshed since 2018, while...
Google Research released TimesFM 2.5, a decoder‑only time‑series foundation model with 200 million parameters, down from 500 million in version 2.0. The new model supports a 16 k context window, far exceeding the prior 2 048 limit, and adds an optional 30 million‑parameter quantile head for continuous...
Dodge Industrial’s reliance on a 20‑year‑old SAP BW system forced business users to build a shadow data warehouse in Power BI, which ultimately crashed the company’s entire tenant. The outage prompted Data & Analytics Manager Daniel Garrett to partner with Protiviti...
Within3 unveiled Dataverse, a unified real‑world data ecosystem that merges electronic health records, claims, and specialty analytics to sharpen pharmaceutical launch decisions. Avalere Health released a global framework to broaden genomic profiling in cancer care, while Emota’s report highlighted rising...
HYBE America announced Blake Foster as its new Head of Business Intelligence, a role designed to centralize data strategy across U.S. operations. The former Warner Music Group senior vice president will build the company’s analytics infrastructure and turn artist‑generated data...
Hammerspace unveiled a Unified Global Namespace that abstracts storage across on‑prem, AWS, Azure and OCI, letting data appear locally wherever compute runs. Its policy‑driven Objective‑Based Data Orchestration moves only the required blocks, eliminating heavyweight migrations for AI and GPU‑intensive workloads....
Apache Kafka’s pull‑based model excels for event‑driven microservices, but scaling consumer groups creates operational overhead, head‑of‑line blocking, and complex error handling. Large enterprises such as Wix and Uber have addressed these limits by deploying a centralized push‑based consumer proxy, achieving...
Lumen Technologies, a $12.4 billion telecom operator with a 500,000‑mile fiber network, faced fragmented inventory from decades of acquisitions, operating over 17 legacy systems and nearly 500 data sources. It built a unified data layer and AI‑driven digital twin, launching the...
The Mark Anthony Group (MAG) has moved from a traditional data warehouse to Snowflake Intelligence, turning its data platform into a generative business intelligence engine. By mandating Snowflake Secure Data Sharing in vendor RFPs, MAG streamlined real‑time data integration and...
dbt Labs unveiled a suite of eight AI agent skills that automate complex dbt tasks, including a migration from dbt Core 1.10 to Fusion that completed without human intervention. These skills distill hundreds of hours of community expertise into concise...
Toyota Motor Europe’s head of enterprise data and analytics, Thierry Martin, detailed how the company built a continent‑wide data mesh on Snowflake, launching over 100 data products in its internal marketplace. He described the broader tech stack—including Calibra, Dataiku, Qlik,...
The Forrester Wave™: Data Quality Solutions, Q1 2026 reveals a decisive shift toward AI‑driven automation, real‑time observability, and multimodal data handling. Vendors now embed generative and agentic AI to profile, classify, validate, and remediate data at scale, moving beyond traditional rule‑based...
Over 97% of companies worldwide have invested in big data, and analytics now yields an average return of $13 for every $1 spent, according to a Nucleus Research survey. The article argues that businesses that appear to win online without...
PostgreSQL’s upcoming 19 release introduces a native JSON output option for the COPY TO command, allowing users to stream query results as line‑delimited JSON objects (NDJSON). The feature supports the syntax COPY TO … WITH (FORMAT json) and includes a force_array...
Microsoft’s Africa government affairs director Akua Gyekye says the continent’s AI future hinges on effective data governance rather than just technology adoption. While 76 % of African nations now have data‑protection laws, fragmented policies and restrictive localisation impede cross‑border data flows....
Databricks’ high‑concurrency workloads can suffer performance loss when many jobs write to the same Delta tables. By optimizing table layout with partitions or liquid clustering, enabling row‑level concurrency, and automating file compaction, engineers maintain stable throughput. Disk caching and Delta’s...
Boston Children’s Hospital deployed Etiometry’s AI‑driven clinical intelligence platform to capture continuous high‑frequency physiologic data across its pediatric ICU. The system aggregates and visualizes signals in real time, giving clinicians a shared, longitudinal view of each patient’s trajectory. Early results...
Artificial intelligence is now integral to Digital Communications Governance and Archiving (DCGA) in financial services, automating the monitoring, summarising, and risk detection of employee communications across text, voice, video and AI‑generated content. Theta Lake showcases six real‑world use cases, from...
Reveal, Infragistics' embedded analytics platform, now lets enterprises embed conversational AI analytics directly into their applications. The solution transforms static dashboards into interactive, question‑answer experiences while enforcing existing data permissions. It also offers token‑based cost controls, giving software teams visibility...
Veritone announced a multi‑year agreement to migrate its core AI workloads, including aiWARE, Data Refinery, and Data Marketplace, to Oracle Cloud Infrastructure. The move aims to boost performance, security, and global scalability as the company tackles massive unstructured data volumes....
Telstra announced it will integrate the Apache Flink stream‑processing engine with its existing Kafka‑based event streaming platform, launching the project in the coming months. The pairing, delivered through Confluent’s managed services, aims to boost real‑time analytics across Telstra’s network observability...
Energy intelligence firm TGS has engaged Tape Ark to move roughly 40 petabytes of seismic and subsurface data into a hyperscale cloud environment. The migration leverages Tape Ark’s parallel ingest platform to accelerate high‑throughput transfer across multiple facilities. Once in the cloud, TGS...
Fivetran’s 2026 enterprise data infrastructure benchmark, based on a survey of 500 senior data leaders at firms with over 5,000 employees, reveals that fragile data pipelines are costing large organizations roughly $3 million in lost revenue each month. Nearly 97% of...
Snowflake announced a major upgrade to its Cortex Code AI coding agent, making it generally available inside Snowsight and adding native Windows support for the CLI. The update introduces Agent Teams, a coordination layer that lets multiple sub‑agents work in...
Patrick Gaskins explains how real‑time fleet data and predictive analytics are reshaping trucking operations. By giving dispatchers minute‑by‑minute visibility, carriers can match loads to trucks, cut empty miles, and lift loaded‑mile percentages. Integrated network‑wide platforms further align operations, sales, and...
Enterprises are racing to harness big data, with 99% of Fortune 1000 executives reporting active programs and 96% seeing success. The data landscape spans structured, semi‑structured and unstructured sources, generating roughly 2.5 quintillion bytes daily. Effective collection relies on ETL pipelines...
Snowflake announced a research preview of Project SnowWork, an autonomous AI platform embedded in its data cloud that lets business users trigger complex, multi‑step workflows with natural‑language prompts. The system deploys secure, data‑grounded AI agents that can query governed data,...
Lumi AI, founded in 2023, offers a natural‑language interface that plugs into ERP systems like SAP and Oracle, letting CPG and food‑retail teams query supply‑chain data instantly. The startup has secured $3.7 million in seed funding and counts Kroger, Growmark and...
Domo Inc. announced an AI agent builder that includes a library of enterprise data connectors powered by the Model Context Protocol. The platform lets users design conversational or goal‑oriented agents that can pull internal and external data, automate tasks, and...
Osmium Data Group warns that using US‑owned cloud providers for backups undermines European data‑sovereignty, even when the physical datacenter sits in the EU. The firm evaluated four source‑and‑destination scenarios, ranking a Europe‑owned source and datacenter as highest compliance, while a...
CData Software unveiled major upgrades to its CData Sync platform, adding native pipeline orchestration, an enhanced API 2.0, and enterprise‑grade change data capture (CDC) for IBM DB2 and SAP HANA. The solution now writes directly to open table formats such...
IBM completed an $11 billion acquisition of Confluent on March 17, 2026, adding the leading data‑streaming platform used by over 6,500 enterprises, including 40 % of the Fortune 500. IBM frames the deal as buying an AI‑focused data platform that delivers real‑time data to power...
Entrinsik Informer now offers insurance agencies an automated data‑quality layer that plugs into AMS360, surfacing missing fields, duplicate records, and inconsistent structures before reports are generated. The solution replaces manual data‑hunt routines with a continuous Data Report Card that highlights...
BMLL and Tradefeedr announced a partnership to create an AI‑ready analytics layer for equities and futures trading data, leveraging BMLL’s harmonised historical order‑book datasets. The collaboration will extend Tradefeedr’s existing FX analytics APIs to cover multi‑asset execution data, delivered through...
Airports are betting on data to drive efficiency, resilience and passenger experience, yet many still stumble on turning raw information into actionable insight. At the International Airport Summit in Berlin, senior leaders highlighted that reliable data, strong governance and clear...
Organizations that rely on opaque AI data sources expose themselves to integrity risks, compliance gaps, and trust deficits. Without auditable pipelines, security teams cannot verify data quality, leading to hallucinations and regulatory violations such as under the EU AI Act....
Singapore Cruise Centre (SCCPL) is entering the final stage of a five‑year digital transformation that centers on a real‑time data integration platform built on Solace’s event‑driven architecture. The platform unifies passenger, vessel, baggage, staff and resource data, enabling instant updates...
