# Table of Contents - [Narwhals](#narwhals) - [Series - Narwhals](#series-narwhals) - [DataFrame - Narwhals](#dataframe-narwhals) - [Complete example - Narwhals](#complete-example-narwhals) - [Conversion between libraries - Narwhals](#conversion-between-libraries-narwhals) --- # Narwhals Narwhals ======== \-![](assets/image.png) [![PyPI version](https://badge.fury.io/py/narwhals.svg)](https://badge.fury.io/py/narwhals) [![Downloads](https://static.pepy.tech/badge/narwhals/month)](https://pepy.tech/project/narwhals) [![Trusted publishing](https://img.shields.io/badge/Trusted_publishing-Provides_attestations-bright_green)](https://peps.python.org/pep-0740/) Extremely lightweight and extensible compatibility layer between dataframe libraries! * **Full API support**: cuDF, Modin, pandas, Polars, PyArrow * **Lazy-only support**: Dask * **Interchange-level support**: DuckDB, Ibis, Vaex, anything which implements the DataFrame Interchange Protocol Seamlessly support all, without depending on any! * ✅ **Just use** [a subset of **the Polars API**](api-reference/) , no need to learn anything new * ✅ **Zero dependencies**, Narwhals only uses what the user passes in so your library can stay lightweight * ✅ Separate **lazy** and eager APIs, use **expressions** * ✅ Support pandas' complicated type system and index, without either getting in the way * ✅ **100% branch coverage**, tested against pandas and Polars nightly builds * ✅ **Negligible overhead**, see [overhead](overhead/) * ✅ Let your IDE help you thanks to **full static typing**, see [`narwhals.typing`](api-reference/typing/#narwhals.typing) * ✅ **Perfect backwards compatibility policy**, see [stable api](backcompat/) for how to opt-in Who's this for? --------------- Anyone wishing to write a library/application/service which consumes dataframes, and wishing to make it completely dataframe-agnostic. Let's get started! Roadmap ------- See [roadmap discussion on GitHub](https://github.com/narwhals-dev/narwhals/discussions/1370) for an up-to-date plan of future work. Back to top --- # Series - Narwhals Series ====== In [dataframe](../dataframe/) , you learned how to write a dataframe-agnostic function. We only used DataFrame methods there - but what if we need to operate on its columns? Note that Polars does not have lazy columns. If you need to operate on columns as part of a dataframe operation, you should use expressions - but if you need to extract a single column, you need to ensure that you start with an eager `DataFrame`. To do that, you need to pass `eager_only=True` to `nw.from_native`. Example 1: filter based on a column's values -------------------------------------------- This can stay lazy, so we just use expressions: from/to\_native@narwhalify `import narwhals as nw from narwhals.typing import IntoFrameT def my_func(df: IntoFrameT) -> IntoFrameT: return nw.from_native(df).filter(nw.col("a") > 0).to_native()` `import narwhals as nw from narwhals.typing import FrameT @nw.narwhalify def my_func(df: FrameT) -> FrameT: return df.filter(nw.col("a") > 0)` and call it either on a eager or lazy dataframe: pandasPolars (eager)Polars (lazy)PyArrow `import pandas as pd df = pd.DataFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df))` `a b 1 1 5 2 3 -3` `import polars as pl df = pl.DataFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df))` `shape: (2, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪═════╡ │ 1 ┆ 5 │ │ 3 ┆ -3 │ └─────┴─────┘` `import polars as pl df = pl.LazyFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df).collect())` `shape: (2, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪═════╡ │ 1 ┆ 5 │ │ 3 ┆ -3 │ └─────┴─────┘` `import pyarrow as pa table = pa.table({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(table))` `pyarrow.Table a: int64 b: int64 ---- a: [[1,3]] b: [[5,-3]]` Example 2: multiply a column's values by a constant --------------------------------------------------- Let's write a dataframe-agnostic function which multiplies the values in column `'a'` by 2. This can also stay lazy, and can use expressions: from/to\_native@narwhalify `import narwhals as nw from narwhals.typing import IntoFrameT def my_func(df: IntoFrameT) -> IntoFrameT: return nw.from_native(df).with_columns(nw.col("a") * 2).to_native()` `import narwhals as nw from narwhals.typing import FrameT @nw.narwhalify def my_func(df: FrameT) -> FrameT: return df.with_columns(nw.col("a") * 2)` and call it either on a eager or lazy dataframe: pandasPolars (eager)Polars (lazy)PyArrow `import pandas as pd df = pd.DataFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df))` `a b 0 -2 3 1 2 5 2 6 -3` `import polars as pl df = pl.DataFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df))` `shape: (3, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪═════╡ │ -2 ┆ 3 │ │ 2 ┆ 5 │ │ 6 ┆ -3 │ └─────┴─────┘` `import polars as pl df = pl.LazyFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df).collect())` `shape: (3, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪═════╡ │ -2 ┆ 3 │ │ 2 ┆ 5 │ │ 6 ┆ -3 │ └─────┴─────┘` `import pyarrow as pa table = pa.table({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(table))` `pyarrow.Table a: int64 b: int64 ---- a: [[-2,2,6]] b: [[3,5,-3]]` Note that column `'a'` was overwritten. If we had wanted to add a new column called `'c'` containing column `'a'`'s values multiplied by 2, we could have used `Expr.alias`: `import narwhals as nw from narwhals.typing import FrameT @nw.narwhalify def my_func(df: FrameT) -> FrameT: return df.with_columns((nw.col("a") * 2).alias("c"))` pandasPolars (eager)Polars (lazy)PyArrow `import pandas as pd df = pd.DataFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df))` `a b c 0 -1 3 -2 1 1 5 2 2 3 -3 6` `import polars as pl df = pl.DataFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df))` `shape: (3, 3) ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ -1 ┆ 3 ┆ -2 │ │ 1 ┆ 5 ┆ 2 │ │ 3 ┆ -3 ┆ 6 │ └─────┴─────┴─────┘` `import polars as pl df = pl.LazyFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df).collect())` `shape: (3, 3) ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ -1 ┆ 3 ┆ -2 │ │ 1 ┆ 5 ┆ 2 │ │ 3 ┆ -3 ┆ 6 │ └─────┴─────┴─────┘` `import pyarrow as pa table = pa.table({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(table))` `pyarrow.Table a: int64 b: int64 c: int64 ---- a: [[-1,1,3]] b: [[3,5,-3]] c: [[-2,2,6]]` Example 3: finding the mean of a column as a scalar --------------------------------------------------- Now, we want to find the mean of column `'a'`, and we need it as a Python scalar. This means that computation cannot stay lazy - it must execute! Therefore, we'll pass `eager_only=True` to `nw.from_native` (or `nw.narwhalify`), and then, instead of using expressions, we'll extract a `Series`. from/to\_native@narwhalify `import narwhals as nw from narwhals.typing import IntoDataFrameT def my_func(df: IntoDataFrameT) -> float | None: return nw.from_native(df, eager_only=True)["a"].mean()` `import narwhals as nw from narwhals.typing import DataFrameT @nw.narwhalify(eager_only=True) def my_func(df: DataFrameT) -> float | None: return df["a"].mean()` Now we can call it on a eager dataframe only: pandasPolars (eager)PyArrow `import pandas as pd df = pd.DataFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df))` `1.0` `import polars as pl df = pl.DataFrame({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(df))` `1.0` `import pyarrow as pa table = pa.table({"a": [-1, 1, 3], "b": [3, 5, -3]}) print(my_func(table))` `1.0` Note that, even though the output of our function is not a dataframe nor a series, we can still use `narwhalify`. Back to top --- # DataFrame - Narwhals DataFrame ========= To write a dataframe-agnostic function, the steps you'll want to follow are: 1. Initialise a Narwhals DataFrame or LazyFrame by passing your dataframe to `nw.from_native`. All the calculations stay lazy if we start with a lazy dataframe - Narwhals will never automatically trigger computation without you asking it to. Note: if you need eager execution, make sure to pass `eager_only=True` to `nw.from_native`. 2. Express your logic using the subset of the Polars API supported by Narwhals. 3. If you need to return a dataframe to the user in its original library, call `nw.to_native`. Steps 1 and 3 are so common that we provide a utility `@nw.narwhalify` decorator, which allows you to only explicitly write step 2. Let's explore this with some simple examples. Example 1: descriptive statistics --------------------------------- Just like in Polars, we can pass expressions to `DataFrame.select` or `LazyFrame.select`. Make a Python file with the following content: from/to\_native@narwhalify `import narwhals as nw from narwhals.typing import IntoFrameT def func(df: IntoFrameT) -> IntoFrameT: return ( nw.from_native(df) .select( a_sum=nw.col("a").sum(), a_mean=nw.col("a").mean(), a_std=nw.col("a").std(), ) .to_native() )` `import narwhals as nw from narwhals.typing import FrameT @nw.narwhalify def func(df: FrameT) -> FrameT: return df.select( a_sum=nw.col("a").sum(), a_mean=nw.col("a").mean(), a_std=nw.col("a").std(), )` Let's try it out: pandasPolars (eager)Polars (lazy)PyArrow `import pandas as pd df = pd.DataFrame({"a": [1, 1, 2]}) print(func(df))` `a_sum a_mean a_std 0 4 1.333333 0.57735` `import polars as pl df = pl.DataFrame({"a": [1, 1, 2]}) print(func(df))` `shape: (1, 3) ┌───────┬──────────┬─────────┐ │ a_sum ┆ a_mean ┆ a_std │ │ --- ┆ --- ┆ --- │ │ i64 ┆ f64 ┆ f64 │ ╞═══════╪══════════╪═════════╡ │ 4 ┆ 1.333333 ┆ 0.57735 │ └───────┴──────────┴─────────┘` `import polars as pl df = pl.LazyFrame({"a": [1, 1, 2]}) print(func(df).collect())` `shape: (1, 3) ┌───────┬──────────┬─────────┐ │ a_sum ┆ a_mean ┆ a_std │ │ --- ┆ --- ┆ --- │ │ i64 ┆ f64 ┆ f64 │ ╞═══════╪══════════╪═════════╡ │ 4 ┆ 1.333333 ┆ 0.57735 │ └───────┴──────────┴─────────┘` `import pyarrow as pa table = pa.table({"a": [1, 1, 2]}) print(func(table))` `pyarrow.Table a_sum: int64 a_mean: double a_std: double ---- a_sum: [[4]] a_mean: [[1.3333333333333333]] a_std: [[0.5773502691896257]]` Example 2: group-by and mean ---------------------------- Just like in Polars, we can pass expressions to `GroupBy.agg`. Make a Python file with the following content: from/to\_native@narwhalify `import narwhals as nw from narwhals.typing import IntoFrameT def func(df: IntoFrameT) -> IntoFrameT: return ( nw.from_native(df).group_by("a").agg(nw.col("b").mean()).sort("a").to_native() )` `import narwhals as nw from narwhals.typing import FrameT @nw.narwhalify def func(df: FrameT) -> FrameT: return df.group_by("a").agg(nw.col("b").mean()).sort("a")` Let's try it out: pandasPolars (eager)Polars (lazy)PyArrow `import pandas as pd df = pd.DataFrame({"a": [1, 1, 2], "b": [4, 5, 6]}) print(func(df))` `a b 0 1 4.5 1 2 6.0` `import polars as pl df = pl.DataFrame({"a": [1, 1, 2], "b": [4, 5, 6]}) print(func(df))` `shape: (2, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞═════╪═════╡ │ 1 ┆ 4.5 │ │ 2 ┆ 6.0 │ └─────┴─────┘` `import polars as pl df = pl.LazyFrame({"a": [1, 1, 2], "b": [4, 5, 6]}) print(func(df).collect())` `shape: (2, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞═════╪═════╡ │ 1 ┆ 4.5 │ │ 2 ┆ 6.0 │ └─────┴─────┘` `import pyarrow as pa table = pa.table({"a": [1, 1, 2], "b": [4, 5, 6]}) print(func(table))` `pyarrow.Table a: int64 b: double ---- a: [[1,2]] b: [[4.5,6]]` Example 3: horizontal sum ------------------------- Expressions can be free-standing functions which accept other expressions as inputs. For example, we can compute a horizontal sum using `nw.sum_horizontal`. Make a Python file with the following content: from/to\_native@narwhalify `import narwhals as nw from narwhals.typing import IntoFrameT def func(df: IntoFrameT) -> IntoFrameT: return ( nw.from_native(df) .with_columns(a_plus_b=nw.sum_horizontal("a", "b")) .to_native() )` `import narwhals as nw from narwhals.typing import FrameT @nw.narwhalify def func(df: FrameT) -> FrameT: return df.with_columns(a_plus_b=nw.sum_horizontal("a", "b"))` Let's try it out: pandasPolars (eager)Polars (lazy)PyArrow `import pandas as pd df = pd.DataFrame({"a": [1, 1, 2], "b": [4, 5, 6]}) print(func(df))` `a b a_plus_b 0 1 4 5 1 1 5 6 2 2 6 8` `import polars as pl df = pl.DataFrame({"a": [1, 1, 2], "b": [4, 5, 6]}) print(func(df))` `shape: (3, 3) ┌─────┬─────┬──────────┐ │ a ┆ b ┆ a_plus_b │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪══════════╡ │ 1 ┆ 4 ┆ 5 │ │ 1 ┆ 5 ┆ 6 │ │ 2 ┆ 6 ┆ 8 │ └─────┴─────┴──────────┘` `import polars as pl df = pl.LazyFrame({"a": [1, 1, 2], "b": [4, 5, 6]}) print(func(df).collect())` `shape: (3, 3) ┌─────┬─────┬──────────┐ │ a ┆ b ┆ a_plus_b │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪══════════╡ │ 1 ┆ 4 ┆ 5 │ │ 1 ┆ 5 ┆ 6 │ │ 2 ┆ 6 ┆ 8 │ └─────┴─────┴──────────┘` `import pyarrow as pa table = pa.table({"a": [1, 1, 2], "b": [4, 5, 6]}) print(func(table))` `pyarrow.Table a: int64 b: int64 a_plus_b: int64 ---- a: [[1,1,2]] b: [[4,5,6]] a_plus_b: [[5,6,8]]` Example 4: multiple inputs -------------------------- `nw.narwhalify` can be used to decorate functions that take multiple inputs as well and return a non dataframe/series-like object. For example, let's compute how many rows are left in a dataframe after filtering it based on a series. Make a Python file with the following content: `import narwhals as nw from narwhals.typing import DataFrameT @nw.narwhalify(eager_only=True) def func(df: DataFrameT, s: nw.Series, col_name: str) -> int: return df.filter(nw.col(col_name).is_in(s)).shape[0]` We require `eager_only=True` here because lazyframe doesn't support `.shape`. Let's try it out: pandasPolars (eager)PyArrow `import pandas as pd df = pd.DataFrame({"a": [1, 1, 2, 2, 3], "b": [4, 5, 6, 7, 8]}) s = pd.Series([1, 3]) print(func(df, s.to_numpy(), "a"))` `3` `import polars as pl df = pl.DataFrame({"a": [1, 1, 2, 2, 3], "b": [4, 5, 6, 7, 8]}) s = pl.Series([1, 3]) print(func(df, s.to_numpy(), "a"))` `3` `import pyarrow as pa table = pa.table({"a": [1, 1, 2, 2, 3], "b": [4, 5, 6, 7, 8]}) a = pa.array([1, 3]) print(func(table, a.to_numpy(), "a"))` `3` Back to top --- # Complete example - Narwhals Complete example ================ We're going to write a dataframe-agnostic "Standard Scaler". This class will have `fit` and `transform` methods (like `scikit-learn` transformers), and will work agnostically for pandas and Polars. We'll need to write two methods: * `fit`: find the mean and standard deviation for each column from a given training set; * `transform`: scale a given dataset with the mean and standard deviations calculated during `fit`. Fit method ---------- Unlike the `transform` method, which we'll write below, `fit` cannot stay lazy, as we need to compute concrete values for the means and standard deviations. To be able to get `Series` out of our `DataFrame`, we'll pass `eager_only=True` to `nw.from_native`. This is because Polars doesn't have a concept of lazy `Series`, and so Narwhals doesn't either. We can specify that in the `@nw.narwhalify` decorator by setting `eager_only=True`, and the argument will be propagated to `nw.from_native`. from/to\_native@narwhalify `from typing import Self import narwhals as nw from narwhals.typing import IntoDataFrameT class StandardScaler: def fit(self: Self, df: IntoDataFrameT) -> Self: df_nw = nw.from_native(df, eager_only=True) self._means = {col: df_nw[col].mean() for col in df_nw.columns} self._std_devs = {col: df_nw[col].std() for col in df_nw.columns} self._columns = df_nw.columns return self` `from typing import Self import narwhals as nw from narwhals.typing import DataFrameT class StandardScaler: @nw.narwhalify(eager_only=True) def fit(self: Self, df: DataFrameT) -> Self: self._means = {col: df[col].mean() for col in df.columns} self._std_devs = {col: df[col].std() for col in df.columns} self._columns = df.columns return self` Transform method ---------------- We're going to take in a dataframe, and return a dataframe of the same type: from/to\_native@narwhalify `from typing import Self import narwhals as nw from narwhals.typing import IntoFrameT class StandardScaler: ... def transform(self: Self, df: IntoFrameT) -> IntoFrameT: df_nw = nw.from_native(df) return df_nw.with_columns( (nw.col(col) - self._means[col]) / self._std_devs[col] for col in self._columns ).to_native()` `from typing import Self import narwhals as nw from narwhals.typing import FrameT class StandardScaler: ... @nw.narwhalify def transform(self: Self, df: FrameT) -> FrameT: return df.with_columns( (nw.col(col) - self._means[col]) / self._std_devs[col] for col in self._columns )` Note that all the calculations here can stay lazy if the underlying library permits it, so we don't pass in any extra keyword-arguments such as `eager_only`, we just use the default `eager_only=False`. Putting it all together ----------------------- Here is our dataframe-agnostic standard scaler: from/to\_native@narwhalify `from typing import Self import narwhals as nw from narwhals.typing import IntoDataFrameT from narwhals.typing import IntoFrameT class StandardScaler: def fit(self: Self, df: IntoDataFrameT) -> Self: df_nw = nw.from_native(df, eager_only=True) self._means = {col: df_nw[col].mean() for col in df_nw.columns} self._std_devs = {col: df_nw[col].std() for col in df_nw.columns} self._columns = df_nw.columns return self def transform(self: Self, df: IntoFrameT) -> IntoFrameT: df_nw = nw.from_native(df) return df_nw.with_columns( (nw.col(col) - self._means[col]) / self._std_devs[col] for col in self._columns ).to_native()` `from typing import Self import narwhals as nw from narwhals.typing import DataFrameT from narwhals.typing import FrameT class StandardScaler: @nw.narwhalify(eager_only=True) def fit(self: Self, df: DataFrameT) -> Self: self._means = {col: df[col].mean() for col in df.columns} self._std_devs = {col: df[col].std() for col in df.columns} self._columns = df.columns return self @nw.narwhalify def transform(self: Self, df: FrameT) -> FrameT: return df.with_columns( (nw.col(col) - self._means[col]) / self._std_devs[col] for col in self._columns )` Next, let's try running it. Notice how, as `transform` doesn't use any eager-only features, so we can pass a Polars LazyFrame to it and have it stay lazy! pandasPolars `import pandas as pd df_train = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 7]}) df_test = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 7]}) scaler = StandardScaler() scaler.fit(df_train) print(scaler.transform(df_test))` `a b 0 -1.0 -0.872872 1 0.0 -0.218218 2 1.0 1.091089` `import polars as pl df_train = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 7]}) df_test = pl.LazyFrame({"a": [1, 2, 3], "b": [4, 5, 7]}) scaler = StandardScaler() scaler.fit(df_train) print(scaler.transform(df_test).collect())` `shape: (3, 2) ┌──────┬───────────┐ │ a ┆ b │ │ --- ┆ --- │ │ f64 ┆ f64 │ ╞══════╪═══════════╡ │ -1.0 ┆ -0.872872 │ │ 0.0 ┆ -0.218218 │ │ 1.0 ┆ 1.091089 │ └──────┴───────────┘` Back to top --- # Conversion between libraries - Narwhals Conversion between libraries ============================ Some library maintainers must apply complex dataframe operations, using methods and functions that may not (yet) be implemented in Narwhals. In such cases, Narwhals can still be highly beneficial, by allowing easy dataframe conversion. Dataframe X in, pandas out -------------------------- Imagine that you maintain a library with a function that operates on pandas dataframes to produce automated reports. You want to allow users to supply a dataframe in any format to that function (pandas, Polars, DuckDB, cuDF, Modin, etc.) without adding all those dependencies to your own project and without special-casing each input library's variation of `to_pandas` / `toPandas` / `to_pandas_df` / `df` ... One solution is to use Narwhals as a thin Dataframe ingestion layer, to convert user-supplied dataframe to the format that your library uses internally. Since Narwhals is zero-dependency, this is a much more lightweight solution than including all the dataframe libraries as dependencies, and easier to write than special casing each input library's `to_pandas` method (if it even exists!). To illustrate, we create dataframes in various formats: `import narwhals as nw from narwhals.typing import IntoDataFrame from typing import Any import duckdb import polars as pl import pandas as pd df_polars = pl.DataFrame( { "A": [1, 2, 3, 4, 5], "fruits": ["banana", "banana", "apple", "apple", "banana"], "B": [5, 4, 3, 2, 1], "cars": ["beetle", "audi", "beetle", "beetle", "beetle"], } ) df_pandas = df_polars.to_pandas() df_duckdb = duckdb.sql("SELECT * FROM df_polars")` Now, we define a function that can ingest any dataframe type supported by Narwhals, and convert it to a pandas DataFrame for internal use: `def df_to_pandas(df: IntoDataFrame) -> pd.DataFrame: return nw.from_native(df).to_pandas() print(df_to_pandas(df_polars))` `A fruits B cars 0 1 banana 5 beetle 1 2 banana 4 audi 2 3 apple 3 beetle 3 4 apple 2 beetle 4 5 banana 1 beetle` Dataframe X in, Polars out -------------------------- ### Via PyCapsule Interface Similarly, if your library uses Polars internally, you can convert any user-supplied dataframe which implements `__arrow_c_stream__`: `def df_to_polars(df_native: Any) -> pl.DataFrame: if hasattr(df_native, "__arrow_c_stream__"): return nw.from_arrow(df_native, native_namespace=pl).to_native() msg = ( f"Expected object which implements '__arrow_c_stream__' got: {type(df_native)}" ) raise TypeError(msg) print(df_to_polars(df_duckdb)) # You can only execute this line of code once.` `shape: (5, 4) ┌─────┬────────┬─────┬────────┐ │ A ┆ fruits ┆ B ┆ cars │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ i64 ┆ str │ ╞═════╪════════╪═════╪════════╡ │ 1 ┆ banana ┆ 5 ┆ beetle │ │ 2 ┆ banana ┆ 4 ┆ audi │ │ 3 ┆ apple ┆ 3 ┆ beetle │ │ 4 ┆ apple ┆ 2 ┆ beetle │ │ 5 ┆ banana ┆ 1 ┆ beetle │ └─────┴────────┴─────┴────────┘` It works to pass Polars to `native_namespace` here because Polars supports the [PyCapsule Interface](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html) for import. Note that the PyCapsule Interface makes no guarantee that you can call it repeatedly, so the approach above only works if you only expect to perform the conversion a single time on each input object. ### Via PyArrow If you need to ingest the same dataframe multiple times, then you may want to go via PyArrow instead. This may be less efficient than the PyCapsule approach above (and always requires PyArrow!), but is more forgiving: `def df_to_polars(df_native: IntoDataFrame) -> pl.DataFrame: df = nw.from_native(df_native).lazy().collect() return pl.DataFrame(nw.from_native(df, eager_only=True).to_arrow()) df_duckdb = duckdb.sql("SELECT * FROM df_polars") print(df_to_polars(df_duckdb)) # We can execute this... print(df_to_polars(df_duckdb)) # ...as many times as we like!` `shape: (5, 4) ┌─────┬────────┬─────┬────────┐ │ A ┆ fruits ┆ B ┆ cars │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ i64 ┆ str │ ╞═════╪════════╪═════╪════════╡ │ 1 ┆ banana ┆ 5 ┆ beetle │ │ 2 ┆ banana ┆ 4 ┆ audi │ │ 3 ┆ apple ┆ 3 ┆ beetle │ │ 4 ┆ apple ┆ 2 ┆ beetle │ │ 5 ┆ banana ┆ 1 ┆ beetle │ └─────┴────────┴─────┴────────┘ shape: (5, 4) ┌─────┬────────┬─────┬────────┐ │ A ┆ fruits ┆ B ┆ cars │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ i64 ┆ str │ ╞═════╪════════╪═════╪════════╡ │ 1 ┆ banana ┆ 5 ┆ beetle │ │ 2 ┆ banana ┆ 4 ┆ audi │ │ 3 ┆ apple ┆ 3 ┆ beetle │ │ 4 ┆ apple ┆ 2 ┆ beetle │ │ 5 ┆ banana ┆ 1 ┆ beetle │ └─────┴────────┴─────┴────────┘` Back to top ---