Orca

Overview of Orca

As multi-cluster deployments become the norm, enterprises are placing higher demands on streaming data products. The requirements for accessing stream data across clusters, performing complex computations, and managing operations are growing increasingly sophisticated. Traditional streaming architectures struggle to handle complex task dependencies, resource scheduling and high availability, gradually becoming a bottleneck to business development.

Current streaming architectures face several core challages:

  • Complex codes and high error rate

    Users must manually derive table schema and write code to handle parallel execution, cascading dependencies, and resource cleanup. This results in a complex development process that is prone to errors.

  • Exposure to low-level concepts increases cognitive load

    There is a lack of abstraction. For example, publishing a stream table requires the use of share keyword, and users need to understand underlying system details such as shared session, increasing the learning curve.

  • Complex deployment and maintenance

    Users must manually specify which physical node to deploy streaming task on. After a node restarts, the entire streaming framework must be rebuilt, as there is no built-in mechanism for automatic recovery.

  • Managing operation is under many limits

    Operations such as querying stream tables, writing data, and warming up engines must be executed on specific nodes, making unified management and automatic scheduling.

    Orca provides an abstraction and enhancement layer on top of DolphinDB existing streaming architecture. Its core goals include:

  • Computation abstraction

    Orca offers a declarative API that supports chainable programming style, significantly simplifying the construction of streaming computation pipelines. Users are freed from concerns such as schema inference, parallelization, cascading subscriptions, and resource cleanup.

  • Automatic scheduling

    Based on the defined stream graph, cluster topology and resource availability, Orca automatically handles task deployment and execution, eliminating the need for manual intervention.

  • High availability of streaming task

    With built-in Checkpoint mechanism, Orca can automatically recover the task to the latest snapshot state in the event of nodes failure or crash, ensuring no data is lost during processing.

System Architecture

Orca adopts a typical Master-Worker architecture, leveraging DolphinDB’s Distributed File System (DFS) to support automatic stream graph deployment, task scheduling, and fault recovery. The system components are clearly defined, and centralized state management ensures high availability for streaming computations.

Overall Architecture

The architecture of Orca is illustrated in Figure 1-1:

  • Users submit stream graph definitions by calling Orca APIs.
  • The Stream Master receives the logical stream graph, and generates the physical stream graph and scheduling plan based on the graph topology and current cluster resource status.
  • Stream Workers are responsible for building stream tables and engines, executing stream tasks, and performing Checkpoint operations.
  • All critical states—including stream graph structure, scheduling records, stream table locations, and Checkpoint metadata—are persisted to DFS tables.
  • Internally, heartbeat checks, state reporting, and the Barrier mechanism are used to ensure high availability and consistent execution of the stream graph.
Figure 1. Figure 1-1 Orca Architecture

Stream Master

The Stream Master runs on the controller node and is responsible for the following tasks:

  • Receiving user requests (e.g., submitting or deleting stream graphs, querying status).
  • Receiving status reports from Stream Workers.
  • Executing the stream graph state machine (from building → running → error recovery → destroyed).
  • Dispatching tasks to different nodes, managing parallelism and resource isolation.
  • Periodically triggering Checkpoints, coordinating all tasks within the stream graph to take snapshots.
  • Maintaining metadata and persisting it to DFS tables.

Stream Worker

The Stream Worker runs on data nodes or compute nodes and is responsible for the following tasks:

  • Receiving tasks dispatched by the Stream Master, building stream tables, engines, and the cascade and subscription topology;
  • Executing and monitoring streaming tasks (e.g., aggregation, state computation, metric generation);
  • Performing local state snapshots and uploading Checkpoints.

All data processing on the Stream Worker is performed entirely in memory. Except for public stream tables, no local persistence is performed. Engine states are preserved through the Checkpoint mechanism.

Roadmap

To further enhance Orca’s feature completeness, flexibility, and performance, we plan to optimize the system along the following directions:

Feature Enhancements

Cross-Cluster Capabilities: Support data streaming across DolphinDB clusters, including cross-cluster subscriptions, stream table access, and join operations, meeting integration needs in multi-cluster environments.

Enhanced State Management: Enable flexible control over computing tasks, such as pausing execution or resetting stream graph state, to improve operations and debugging efficiency.

Finer-Grained Parameter Tuning: Allow users to adjust parameters generated in the DStream API, such as subscribeTable and enableTableShareAndCachePurge, enabling more granular performance tuning.

DStream API Extensions: Expand API capabilities to allow explicit partition count configuration when defining stream tables and advanced usage like sourceByName within the same graph, improving flexibility in graph construction.

Optional High Availability Protocols: Introduce configurable stream data high availability protocols, allowing trade-offs between fault tolerance levels and latency, suitable for diverse business needs.

Optional Scheduling Algorithms: Provide flexible scheduling strategies that support custom trade-offs between throughput and latency, including the ability to integrate user-defined scheduling algorithms.

Performance Optimizations

Physical Graph Optimization: Improve join performance across stream tables and reduce data shuffle overhead, especially in cross-node scenarios.

Runtime Optimization: Replace certain subscription mechanisms with cascaded execution models to reduce latency and improve runtime efficiency.

Checkpoint Performance Improvements: Further optimize Barrier alignment and introduce incremental and asynchronous snapshot mechanisms to reduce Checkpoint overhead and shorten recovery time during fault tolerance.