org.kepler.spark.operator

Class Operator

java.lang.Object

org.kepler.spark.operator.Operator

Direct Known Subclasses:

CoGroupOperator, CrossOperator, DataSink, DataSource, JoinOperator, MapOperator, ReduceOperator

public abstract class Operator
extends java.lang.Object

A base class for operators in a Spark job. Derived classes need to provide implementations of execute().

Author:

Daniel Crawl

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	_configuration Configuration parameters.
protected org.apache.spark.api.java.JavaPairRDD<java.lang.Object,?>[]	_inputData The input data to this operator.
protected int	_numInstances The number of parallel instances for this operator.
protected java.lang.Object	_stub The function to execute.

Constructor Summary

Constructors

Constructor and Description
Operator(int numInputs, java.lang.Object stub, java.lang.String name) Create a new Operator.

Method Summary

Modifier and Type	Method and Description
protected org.apache.spark.api.java.JavaPairRDD<java.lang.Object,?>	_getOutput() Read the inputs and execute the operator to generate the output.
abstract org.apache.spark.api.java.JavaPairRDD<java.lang.Object,?>	execute() Execute the operator.
org.apache.hadoop.conf.Configuration	getParameters() Get the configuration parameters.
int	numInputs() Get the number of input operators.
void	setDegreeOfParallelism(int numInstances) Set the number of parallel instances to execute.
void	setInput(int index, Operator output) Set the input operator at an index.
void	setInput(Operator output) Set the (first) input operator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_numInstances

protected int _numInstances

The number of parallel instances for this operator.

_configuration

protected org.apache.hadoop.conf.Configuration _configuration

Configuration parameters.

_inputData

protected org.apache.spark.api.java.JavaPairRDD<java.lang.Object,?>[] _inputData

The input data to this operator.

_stub

protected java.lang.Object _stub

The function to execute.

Constructor Detail

Operator

public Operator(int numInputs,
                java.lang.Object stub,
                java.lang.String name)

Create a new Operator.

Parameters:

numInputs - the number of input operators

stub - the function to execute

name - the name of the operator

Method Detail

numInputs

public int numInputs()

Get the number of input operators.

getParameters

public org.apache.hadoop.conf.Configuration getParameters()

Get the configuration parameters.

setInput

public void setInput(Operator output)

Set the (first) input operator.

setInput

public void setInput(int index,
                     Operator output)

Set the input operator at an index.

setDegreeOfParallelism

public void setDegreeOfParallelism(int numInstances)

Set the number of parallel instances to execute.

execute

public abstract org.apache.spark.api.java.JavaPairRDD<java.lang.Object,?> execute()
                                                                           throws IllegalActionException

Execute the operator.

Throws:

IllegalActionException

_getOutput

protected org.apache.spark.api.java.JavaPairRDD<java.lang.Object,?> _getOutput()
                                                                        throws java.lang.Exception

Read the inputs and execute the operator to generate the output.

Throws:

java.lang.Exception