Core Concepts

DolphinDB supports catalogs to organize various database objects, including distributed tables. Orca further extends the useage of catalogs by registering stream graphs, stream table, and engines into the catalog system. This allows users to access all types of objects in a unified manner.

To uniquely identify objects in the catalog, the current version adopts a three-part naming convention in the format <catalog>.<schema>.<name>, referred to as the Fully Qualified Name (FQN). When naming an object, users only need to specify the name, and Orca will automatically complete the FQN as follows:

• catalog is determined by the current catalog context.
• schema is set to one of orca_graph, orca_table, or orca_engine, depending on whether the object is a stream graph, stream table, or engine.

Orca provides a Declarative Stream API (DStream API) to simplify the construction of real-time stream graphs. This API abstracts away the complexities of low-level parallel scheduling, subscription logic, and resource cleanup, allowing users to focus on business logic.

Users do not need to manually derive table schemas, implement parallel engine logic, write subscription declarations, or manage task cleanup and destruction. The system automatically converts DStream definitions into corresponding DolphinDB function calls (e.g., streamTable, subscribeTable, createReactiveStateEngine) and packages them into executable stream tasks, which are then dispatched to appropriate nodes for execution.

Orca can convert stream graph defined by DStream API to function calls such as streamTable, subscribeTable, and createReactiveStateEngine, package them as stream tasks and send to physical nodes.

The DStream API consists of the following main categories:

• Node Definition APIs: e.g., source, buffer, sink, timeSeriesEngine, reactiveStateEngine
• Node Modifier APIs: e.g., setEngineName, parallelize, sync
• Edge Operation APIs: e.g., map, fork

For the complete list of interfaces, refer to the Declarative Stream API. You should think in terms of building a chain-like structure: continuously appending or modifying nodes at the end to progressively construct the full stream graph.

The computation pipeline constructed using the DStream API is abstracted as a Directed Acyclic Graph (DAG), referred to as a Stream Graph. In this graph, each node represents a stream table or an engine, and each edge denotes the data flow relationship between nodes—such as cascading or subscription.

Stream graphs are categorized into logical stream graphs and physical stream graphs:

• A logical stream graph is constructed step-by-step using the DStream API. It reflects the actual business logic, as illustrated in Figure 1-1.
• A physical stream graph is generated by the system upon submission of the logical graph via submit. During this process, the system automatically inserts private stream tables, applies scheduling optimizations, splits the graph into subgraphs, and generates parallel tasks. The resulting physical stream graph defines the concrete execution plan for streaming tasks, as shown in Figure 1-2.

To manage the lifecycle of stream graphs, Orca defines a series of states and transitions:

• building: The task has been scheduled. It is currently not yet dispatched for execution, or the system is performing cascading or subscription construction.
• running: The task has been constructed and is running normally.
• error: A recoverable error has occurred.
• failed: An unrecoverable error has occurred—typically due to a logical problem in the task itself. User intervention is required to modify the script, or an out-of-memory error occurs.
• destroying: The user has requested to destroy the stream graph. The system is currently in the process of destroying it.
• destroyed: The stream graph has been fully destroyed, and the state machine has terminated.

Orca supports two types of stream tables for data transmission and buffering:

• Private stream tables are non-persistent and used exclusively within a single stream graph. They are primarily intended for caching intermediate results, aligning parallelism across nodes, or supporting multiple downstream subscriptions. These tables are not accessible via SQL queries. All private stream tables are automatically destroyed when their associated stream graph is destroyed.
• Public stream tables are persistent and can be shared across multiple stream graphs. They are used for external data interaction, persistent output, or as streaming data sources. Public stream tables are directly queryable via SQL. When a public stream table is no longer subscribed to by any stream graph, it is automatically destroyed and its fully qualified name (FQN) is released.

When a user submits a logical stream graph, Orca performs the following processing steps:

• Cycle detection: The system checks for cycles using topological sorting. If a cycle is detected, an error is thrown.
• Insert stream tables: Private stream tables are added to buffer intermediate results. For example:
  • If an engine has multiple downstream nodes, stream tables are required because engines support only cascading output (not subscription).
  • If the parallelism levels of upstream and downstream nodes differ, a stream table is inserted to collect upstream data and redistribute it according to the downstream parallelism. This redistribution process is referred to as a shuffle.
• Optimization: Redundant private stream tables introduced during preprocessing are removed to simplify the graph.
• Subgraph partitioning: The entire stream graph is split into subgraphs with consistent parallelism. These subgraphs are constructed to be as long as possible, minimizing the number of tasks that need to be executed in parallel.
• Stream task generation: Within each subgraph, stream tasks are created based on the subgraph’s parallelism. A stream task is the basic unit of execution—it is dispatched to a single thread and consists of a sequential chain of engines and stream tables. Stream tasks communicate with each other by subscribing to upstream data. The total number of stream tasks determines the upper bound of thread usage in the system. Orca minimizes the number of tasks to maximize resource efficiency.
• Channel insertion: The system automatically inserts Channels to support Barrier alignment during Checkpoint operations (see Checkpoint mechanism).

As shown in Figure 1-3, the red box represents a subgraph and the blue box represents a stream task: