PROTOCOL_ARROW

PROTOCOL_ARROW supports serializing and deserializing DolphinDB tables without compression.

Prerequisites

• Install pyarrow 9.0.0 or later in Python environment.
• Installing the DolphinDB Arrow plugin

To use PROTOCOL_ARROW, we need to enable it in the DolphinDB session and DBConnectionPool objects by setting the protocol parameter to PROTOCOL_ARROW.

import dolphindb as ddb
import dolphindb.settings as keys

s = ddb.Session(protocol=keys.PROTOCOL_ARROW)
s.connect("localhost", 8848, "admin", "123456")

pool = ddb.DBConnectionPool("localhost", 8848, "admin", "123456", 10, protocol=keys.PROTOCOL_ARROW)

Polars is compatible with the Apache Arrow ecosystem and supports the Arrow data format. DolphinDB uses PROTOCOL_ARROW to exchange Arrow-format data with Polars, reducing intermediate format conversions and improving data transfer efficiency.

Example:

import uuid
import dolphindb as ddb
import dolphindb.settings as keys
import numpy as np
import polars as pl
# Prepare a Polars DataFrame with 1 million rows
df = pl.DataFrame(
    {
        "a": list(range(1_000_000)),
        "b": [uuid.uuid4().hex[:8] for _ in range(1_000_000)],
        "c": np.random.rand(1_000_000),
    }
)
# Establish a connection with the Arrow protocol enabled
# Once enabled, both data upload and download will use the Arrow format
conn = ddb.Session(protocol=keys.PROTOCOL_ARROW)
conn.connect("localhost", 8848, "admin", "123456")

# Convert a Polars DataFrame to an Arrow Table and upload it to DolphinDB.

pt = df.to_arrow()
conn.upload({"pt": pt})
# Data Download: DolphinDB -> Polars
# In PROTOCOL_ARROW mode, conn.run returns a pyarrow.Table object
res = conn.run("pt")
df2 = pl.from_arrow(res)