pnodeRun

pnodeRun(function,[nodes],[addNodeAlias=true])

Call a local function on specified nodes in a cluster in parallel and then merge the results.

• If nodes is specified, the function is called on specified nodes.
• If nodes is not specified,
  • When pnodeRun is called on a compute node within a compute group, function is executed on all compute nodes within the group.
  • Otherwise it is executed on all data nodes and and compute nodes which are not included in any compute groups.

function is the local function to call. It must not be quoted. It can be a function with no parameters by definition, or a partial application that wraps the orginal function and its parameters to a function with no parameters. It can be a built-in function or a user-defined function.

nodes (optional) is a STRING scalar or vector indicating node alias(es). If not specified, the system will call the function on all active data nodes and compute nodes in the cluster.

addNodeAlias (optional) is a Boolean value specifying whether to add node aliases in results. The default value is true. Set to false if node results already contain aliases.

• If the called function returns a scalar, dictionary, or table, a table is returned.

• If the called function returns a vector, a matrix is returned.

• If the called function returns a dictionary, a table is returned.

• If the called function is a command (with no return value), null is returned.

• In other cases, a dictionary is returned.

Ex. 1 Call function getChunksMeta without specifying parameters

pnodeRun(getChunksMeta,,false);

Ex. 2 In the following example, the function sum and arguments 1..10 are wrapped into a partial application sum{1..10}.

pnodeRun(sum{1..10}, `nodeA`nodeB);

Ex. 3 pnodeRun is a very convenient tool for cluster management. For example, in a cluster of 4 nodes: "DFS_NODE1", "DFS_NODE2", "DFS_NODE3", and "DFS_NODE4", run the following script on each of the node:

def jobDemo(n){
    s = 0
    for (x in 1 : n) {
        s += sum(sin rand(1.0, 100000000)-0.5)
        print("iteration " + x + " " + s)
    }
    return s
};

submitJob("jobDemo1","job demo", jobDemo, 10);
submitJob("jobDemo2","job demo", jobDemo, 10);
submitJob("jobDemo3","job demo", jobDemo, 10);

To check the status of the most recent 2 completed batch jobs on each of the 4 nodes in the cluster:

pnodeRun(getRecentJobs{2});

pnodeRun(getRecentJobs{2}, `DFS_NODE3`DFS_NODE4);