Future of DataFrames and Data Systems

A summary of Wes McKinney's talk — the creator of pandas and Apache Arrow — on where dataframes, interchange formats, and composable data systems are heading.

Overview

The video covers the future of dataframes and data systems, featuring Wes McKinney, the creator of pandas and Apache Arrow. Several key software projects are discussed, focusing on data interchange, analysis, and storage.

Key Software Projects

• Pandas: An open-source library providing data structures and data analysis tools for Python.
• Apache Arrow: An open-source project for in-memory data interchange.
• Ibis: An open-source project aiming to bridge the world of relational databases and data science tools.
• Spark: An open-source unified analytics engine for large-scale data processing.
• Presto: An open-source distributed SQL query engine.
• SQL: A structured query language.
• Iceberg: An open-source table format for big data.
• Parquet: An open-source columnar data file format.

Key Points from the Video

Apache Arrow is highlighted as a language-agnostic data interchange standard being used in composable data systems. Ibis is noted for allowing SQL queries to be run on various data sources.

One challenge of composable data systems is the absence of a unified data frame interface, but multiple projects are working towards standardizing data frames. Another challenge is managing unstructured data within these systems, with approaches like storing unstructured data separately from structured data being explored.

Software Projects Mentioned by Wes McKinney

• Apache Arrow: An open-source project for data interchange and in-memory processing.
• Ibis: A Python library providing a unified API for various SQL databases and data warehouses.
• Posit: A data science platform company offering a variety of tools and services.
• Pandas: A popular Python library for data analysis.
• DBT: A data transformation tool using SQL.
• Snowflake: A cloud-based data warehouse.
• Spark: A unified analytics engine for large-scale data processing.
• Modin: A library for scaling pandas workloads on Apache Spark or Ray.
• Iceberg: An open-source table format for data lakes.
• Parquet: A columnar data file format.
• Nimble: A new open-source file format designed for use with Arrow.
• Lance DB: A new storage project optimized for AI and vector search applications.

Data Technologies for Data Science

Here's a breakdown of the data technologies mentioned in the talk along with some additional insights:

Data Interchange and Processing

• Pandas: An open-source Python library providing data structures and data analysis tools.
• Apache Arrow: An open-source project for in-memory data interchange. Arrow is a language-agnostic standard that facilitates communication between different data systems.
• Parquet: An open-source columnar data file format for efficient storage and retrieval of data.
• Iceberg: An open-source table format for big data, designed for reliability and scalability in data lakes.
• Nimble: A new open-source file format designed for efficient use with Apache Arrow.

SQL and Data Warehouses

• SQL: A structured query language for interacting with relational databases.
• Ibis: An open-source project that allows you to write SQL queries that can be run on a variety of data sources, bridging the gap between relational databases and data science tools.
• DBT (Data Build Tool): A transformation tool that uses SQL to define and execute data transformations in a data warehouse.
• Snowflake: A cloud-based data warehouse platform.

Large-Scale Data Processing

• Spark: An open-source unified analytics engine for large-scale data processing.
• Modin: A library for scaling pandas workloads on Apache Spark or Ray, enabling distributed computing for pandas dataframes.

Other Technologies

• Posit: A data science platform company offering a variety of data science tools and services. (Not mentioned in the video, but included in the speaker's list.)
• Lance DB: A new storage project specifically optimized for AI and vector search applications. (Not mentioned in the video, but included in the speaker's list.)

Challenges and Future of Data Systems

The video highlights the challenges of composable data systems, including the lack of a single unified data frame interface and handling unstructured data. While there are ongoing efforts to standardize data frames, approaches for integrating unstructured data are still being explored.

The speaker emphasizes the importance of open standards for data lakes, such as Iceberg, to ensure interoperability and flexibility within the data ecosystem.

Where Things Stand Now

The talk's predictions have largely played out — the ecosystem has converged on shared, open building blocks. A quick lay of the land as of 2026:

• Arrow is the substrate. Apache Arrow is now the default in-memory standard. pandas 2.x ships optional PyArrow-backed dtypes, so the "language-agnostic interchange" idea is built into the most popular library itself.
• Polars went mainstream. The Rust-based, multi-threaded, lazy-evaluating dataframe is the go-to high-performance alternative to pandas for medium-to-large local workloads.
• DuckDB is everywhere. The in-process analytical engine — "SQLite for analytics" — pairs naturally with Arrow and Parquet and has become a default for local OLAP and quick data exploration.
• Iceberg won the table-format race. Open table formats consolidated around Apache Iceberg: Databricks acquired Tabular (2024) and Snowflake shipped its Polaris catalog, both pushing Iceberg interoperability across the lakehouse.
• Narwhals solved the "which dataframe?" problem. A lightweight compatibility layer now lets libraries support pandas, Polars, and others without hard-depending on any single one — directly addressing the "no unified dataframe interface" challenge raised in the talk.
• Ibis matured as a portable dataframe API that compiles a single expression to many SQL and dataframe backends.
• Unstructured + AI data drove growth in vector-native storage like LanceDB, alongside engines built on the composable stack (Arrow + DataFusion / Velox + Substrait).