Know What's Happening in Big Data
Know What's Happening in Big Data
Big Data Pulse
EMAIL DIGESTS
Daily
Every morning
Weekly
Tuesday recap
EMAIL DIGESTS
Daily
Every morning
Weekly
Tuesday recap
Minor Hotels builds global AI and data platform to personalize guest experiences
The hotel chain is creating a new AI‑driven data platform that links guest information across its 640‑plus properties and 12 brands. Powered by Google Cloud, Salesforce and OneTrust, the stack bypasses legacy systems to deliver real‑time, context‑aware personalization.
Also developing:
By the numbers: Dynatrace acquires Bindplane
FIATA and the Global Shippers Forum have introduced a signable version of their Data Governance Charter, converting previously voluntary principles into a binding framework for digital supply chains. The charter outlines mandatory standards on data ownership, permission controls, protection duties, breach reporting and fair platform practices. By requiring partners and technology providers to sign, FIATA embeds responsible data governance as a baseline condition for collaboration. The move aims to safeguard shippers, freight forwarders and other platform users from unauthorized or unfair commercial exploitation of their data.
Engineers often build custom Salesforce‑to‑warehouse pipelines, but frequent schema changes, API limits, and hidden failures turn maintenance into a monthly time sink. Snowflake’s OpenFlow connector automates schema detection and runs as a native, managed service within Snowflake, eliminating the need...
Immuta unveiled an Agentic Data Access module that lets autonomous AI agents retrieve enterprise data in real time while enforcing governance policies. The new capabilities treat agents as first‑class data users, applying least‑access privileges, zero standing privileges, and audit trails....
Elon Musk announced that SpaceX will launch data‑center satellites to power AI workloads from orbit, and he positioned the plan as a cornerstone of a confidential $75 billion IPO filing that could value SpaceX at $1.75‑$2 trillion. The proposal raises questions about...
SQL tip You ran a load job overnight. How do you know every record made it? Most people recount rows and hope the numbers match. There's a cleaner way. SELECT order_id FROM staging.orders EXCEPT SELECT order_id FROM production.orders; If this returns nothing, every order transferred successfully. If...
The Internal Revenue Service has launched a pilot of Palantir Technologies’ Selection and Analytic Platform (SNAP) to identify the highest‑value tax cheats. The move targets a $696 billion tax gap and follows more than $200 million in IRS contracts with Palantir since...
Data governance isn't cool or sexy. That's why nobody talks about it on the record. Meanwhile their AI projects keep failing. #DataGovernance #AI https://t.co/AAKL6A7DLM
The article details how a new JSON query‑and‑transform language built in Go slashes latency and Kubernetes expenses. A modest $400 token purchase unlocked roughly $500,000 in annual cost savings, illustrating a high‑return refactor. The author, once skeptical of vibe‑code, now...
At Databricks AI Days London 2026, executives highlighted how AI is reshaping enterprise data management by moving from slow, analyst‑driven reporting to instant, natural‑language queries. They emphasized the need for deterministic outputs to earn C‑suite trust and the rise of...
The episode explains how Apache Parquet’s hybrid columnar‑row format optimizes storage and query performance for large datasets. It contrasts row‑wise and pure columnar layouts, highlighting the inefficiencies of each, and then describes Parquet’s structure of row groups, column chunks, and...
China's Hukeda-2 satellite successfully completed its first in‑orbit refueling test on March 24, creating a flood of telemetry data that will be processed by big‑data platforms to improve satellite servicing and lifecycle management. The milestone highlights how massive data streams...
Kanzhun Limited announced Q2 2024 revenue of RMB 1.92 bn (≈$269 m), up 29% year‑over‑year, driven by a 25% rise in verified monthly active users and expanding AI recruitment services. The results underscore the firm’s accelerating foothold in China’s big‑data hiring market.
The article compares how Apache Flink and Kafka Streams manage state in real‑time stream processing. Flink treats state as a first‑class citizen, persisting snapshots to durable storage like S3 via periodic checkpoints. Kafka Streams materializes state changes in compacted Kafka...
A lot of founders have pinged me to ask what steps they would need to take to pull of what @jack has done restructuring @blocks this way. He answers it spot on 👇 "look at all the tools you're using. Look...
DOE’s Genesis Mission has produced SYNAPS‑I, an AI‑driven imaging platform that unifies neutron, X‑ray and microscopy data from more than 100 beamlines across seven national labs. The billion‑parameter foundation model can reconstruct ptychography scans in real time, turning 1.3 TB of...
The team migrated an on‑premises MongoDB golden source of reference data into a governed cloud pipeline using Kafka, Apache Iceberg, and Athena. They implemented a three‑layer architecture—Landing, Bronze, and Silver—to isolate raw ingestion, structural conversion, and consumer‑ready tables, each with...
RIP document extractors. Google just released LangExtract: Open-source. Free. Better than $100K enterprise tools. Here’s what it does: 🧵
The DBTA webinar highlighted that AI projects fail more often due to fragile data pipelines than model flaws. Speakers Kevin Hu and Jerod Johnson outlined how data engineering must evolve to support continuous, real‑time data, lineage, and repeatable outputs for...
Apploi migrated from Segment to RudderStack in just 30 days, cutting data‑pipeline costs by 35% and moving to a warehouse‑centric architecture built around Snowflake. The shift replaced MTU‑based pricing with event‑based fees, giving the company predictable expenses as event volume...
SQL tip Daily revenue is noisy. One bad Monday skews the whole picture. A 7-day moving average smooths it out. ROWS BETWEEN 6 PRECEDING AND CURRENT ROW tells SQL to look at today plus the 6 days before it. The result is a rolling...
Snowflake: Morningstar partnership, new CRO, rising AI workloads. Three signals, one direction. Enterprise data clouds aren't just storage. They're gravity wells for financial intelligence. The platform that controls data access controls the AI output.
The post outlines a production‑grade sessionization pipeline that turns raw event streams into actionable user sessions using Kafka Streams session windows, a Redis‑backed active‑session cache, and PostgreSQL for persistence. It highlights real‑time session tracking with sub‑millisecond lookups and a REST...
California State University has committed $17 million to license ChatGPT for all 23 campuses, covering 460,000 students and 63,000 faculty and staff. A university‑wide survey of 94,000 respondents shows high usage but stark disagreement over AI’s role, raising questions about data...
Data governance is fundamentally a design problem. And it's messy — because humans are messy. #DataGovernance #HumanCenteredDesign https://t.co/K5mFAET58s
The focus of AI safety is shifting from model‑centric controls to the data that fuels autonomous systems. Fragmented, outdated, or ungoverned data can cause unpredictable behavior, especially in regulated or customer‑facing contexts. Denodo’s virtual data‑fabric platform unifies disparate sources, enforces...
Data analytics and insight market projected to surpass $1.4 trillion by 2035, driven by a compound annual growth rate of up to 16.4%, far outpacing the advertising sector’s roughly 4% growth. Predictive analytics now represents over 40% of the market and...
Data platform teams often deliver technically complete stacks, yet consumer teams struggle because the operating interface is missing. The article argues that beyond schemas and APIs, platforms need explicit operational contracts, ownership models, adoption models, and communication patterns. It outlines...
Applied Computing, Wipro Limited and Databricks have formed a strategic partnership to deliver physics‑informed AI at scale for energy operators across the Middle East, India and Southeast Asia. The trio will combine Applied Computing’s Orbital platform, Wipro’s consulting expertise and...
Coleridge convened nearly 250 data experts, policymakers and technology leaders from 39 states at its 6th Annual National Conference in Arlington, VA, March 24‑27. The event’s “Data Beyond Borders” theme spotlighted inter‑agency data integration, AI‑ready datasets and secure enclaves, underscoring...
Kestra announced a $25 million Series A round led by RTP Global, bringing total funding to $36 million. The capital will back the launch of Kestra 2.0, a managed cloud offering, and a broader push into North America and Europe, underscoring the growing...
NomadicML announced an $8.4 million seed round at a $50 million post‑money valuation, led by TQ Ventures with participation from Pear VC and Jeff Dean. The funding will accelerate its AI‑driven platform that converts raw vehicle footage into structured, searchable data for...
SAP announced it will acquire master‑data specialist Reltio, with financial terms undisclosed. The deal integrates Reltio’s cloud‑based MDM platform into SAP’s Business Data Cloud, enhancing data governance for AI and analytics across both SAP and non‑SAP environments.
30+ years of proven patterns. Both still relevant. Inmon (1990): "A subject-oriented, integrated, time-variant, non-volatile collection for management decision-making." https://www.ssp.sh/brain/data-warehouse
Guillaume Delépine founded San Francisco‑based Longeye to use AI for sorting massive digital evidence, aiming to boost police case‑closure rates. The platform, now negotiating 20 contracts, ingests data such as phone records, emails and GPS to deliver searchable case summaries,...
Databricks announced an $850 million investment over three years to quadruple its London headquarters, grow its UK‑Ireland workforce to over 1,000 and train 100,000 data‑AI professionals. The move positions the company as a central hub for its Lakebase and Genie AI...
Delta Lake’s Change Data Feed (CDF) lets engineers capture row‑level changes as soon as they occur, turning a Delta table into a built‑in change‑data‑capture engine. By enabling the table property delta.enableChangeDataFeed, only modified rows are read, eliminating costly full‑table scans for...
AI is about to kill Tableau and PowerBI. Every dashboard can now be created in seconds with these Free Agents:
Redpanda announced the general availability of its adaptable R1 streaming engine in Redpanda Streaming 26.1, a single‑modal platform that lets enterprises tailor performance, safety and cost at the topic level. The release integrates Cloud Topics, write caching, tiered storage, and Iceberg...
SQL tip GROUP BY collapses your rows. Sometimes you need the ranking without losing the detail. That's what window functions do. PARTITION BY region restarts the ranking for each region. ORDER BY total_spend DESC puts the highest spender at rank 1. Every row stays intact....
Generative BI doesn’t accelerate everything. It compresses friction. But here’s the truth: It amplifies whatever it sits on top of. Strong foundation → intelligence at scale Weak foundation → chaos at scale That’s the inflection point. Article 2/4: https://t.co/E4BYVwebSq https://t.co/n2mvvezbl7
European policymakers are pushing digital sovereignty, yet the missing piece is data ownership. As AI models become commoditized, control over the underlying data is emerging as the true competitive moat. Countly, an open‑source, self‑hosted analytics platform, illustrates how firms can...
Domo is best known for its BI capabilities, but CIOs should take a fresh look at its no-code workflows, robust data integration capabilities, and emerging AI Catalyst for developing AI agents #cio #ai https://t.co/Rgkexj9XJP
It's April 1st and I have an announcement: @streamnativeio is a @apachekafka company now. Yes, the Pulsar people. We took Apache Kafka 4.2 and gave it a lakehouse foundation. Topics = Iceberg tables. 10x cheaper. Zero code changes. https://t.co/Waj3eO8BoZ
Oracle announced the Oracle AI Data Platform for U.S. federal agencies, a secure foundation that links generative AI models with agency data, applications, and workflows. The platform combines Oracle Cloud Infrastructure, Autonomous AI Database and Enterprise AI services to break...
In this episode, Dr. Bob Jarvie, Associate CMIO and Medical Director for Population Health Analytics at Corewell Health, explains why the health system built its own internal population health data platform instead of relying on external vendors. He highlights the...
New Hampshire officials have been quietly involved in the Northeast Public Health Collaborative since its inception, even though the state was omitted from the public announcement. Internal emails reveal the state’s continued participation in leadership calls and data‑sharing initiatives, highlighting...
I really suck at software demos, but hopefully that didn't diminish my new Revenue Intelligence feature I demo'd today. I may be able to methodically explain a P&L, but I realized today that I need training on software demos. Can...
Wiliot announced a Built‑On partnership with Databricks, moving its Physical AI supply‑chain platform onto the Databricks lakehouse. The shift lets the company ingest and govern massive item‑level IoT Pixel data streams in a unified environment. By leveraging Databricks’ compute and...
Reserve your spot at Friday's Coffee with Digital Trailblazer. Our topic this week: Redefining Data Governance: Is the Data Owner Role Obsolete in the AI Era? https://t.co/i7NcU4uICI #AI https://t.co/9hxa4qwsuS
NomadicML raised an $8.4 million seed round at a $50 million post‑money valuation to commercialize its vision‑language platform that auto‑annotates autonomous‑vehicle video. The tool transforms terabytes of archived footage into searchable, structured datasets, enabling rapid identification of rare edge‑case events for training...
