Stream Task Scheduling

To ensure proper task distribution and efficient execution, Orca provides a built-in distributed scheduling algorithm. It dynamically dispatches stream tasks based on node resource scoring, scheduling rule matching, and stream graph topology analysis. The system also supports error recovery rescheduling and compute group isolation to ensure both resource isolation and runtime stability.

Scheduling Principles

The scheduler follows these principles:

  • Load balancing: Preferentially dispatch tasks to nodes with more idle resources, avoiding CPU, memory, or disk overload.
  • Compute group isolation: Tasks within the same stream graph run within the same compute group.
  • Stream table placement constraints: Public stream tables must be deployed on data nodes.
  • Sibling task affinity: Tasks that share both upstream and downstream nodes are considered sibling tasks and are scheduled to the same node to reduce communication overhead.

Node Scoring Mechanism

Each candidate node is assigned a score based on its available resources to represent its scheduling priority. Different node types use different scoring metrics:

  • Compute nodes are scored based on their CPU usage and memory usage.
  • Data nodes are scored based on their CPU usage, memory usage, and disk usage.

Scheduling Process and Task Assignment

The scheduling process follows a greedy strategy. The overall workflow is:

  1. Sort tasks by complexity: The system estimates task complexity based on the number of operators in the stream task. More complex tasks are scheduled earlier.
  2. Score candidate nodes: For each task, all eligible nodes are scored according to the rules defined above.
  3. Select preferred nodes:
    • If the task contains public stream tables, only data nodes are considered.
    • If not, the scheduler selects from compute nodes within the same compute group.
    • If a node has already been assigned a sibling task, its score is increased.
  4. Assign the task: The task is dispatched to the highest-scoring node, and that node’s score is updated to reflect the new load, enabling adaptive scheduling for subsequent tasks.