org.kepler.spark.stub

Class KeplerPairFlatMapFunction<T,K,V>

java.lang.Object

org.kepler.spark.stub.PairFlatMapFunctionBase<T,K,V>

org.kepler.spark.stub.KeplerPairFlatMapFunction<T,K,V>

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, org.apache.spark.api.java.function.PairFlatMapFunction<T,K,V>, ExecutionListener

Direct Known Subclasses:

KeplerCoGroupStub, KeplerCrossStub, KeplerMapStub, KeplerMatchStub, KeplerReduceStub

public abstract class KeplerPairFlatMapFunction<T,K,V>
extends PairFlatMapFunctionBase<T,K,V>
implements ExecutionListener

A Spark stub to run a Kepler workflow in a map. Derived classes need to implement the call() method.

Author:

Daniel Crawl

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.Exception	_keplerManagerException If the workflow is running in a separate thread (_runWorkflowLifecyclePerInput = false) and an exception is thrown, this field is set to that exception.
protected Manager	_manager Manager to execute the workflow.
protected boolean	_runWorkflowLifecyclePerInput If true, the entire workflow lifecycle is executed for each input.
protected StubSinkActor	_sinkActor A sink actor used for reading data from the workflow and writing it to Spark.
protected StubSourceActor	_sourceActor A source actor used for reading data from Spark and writing it the Kepler workflow.

Fields inherited from class org.kepler.spark.stub.PairFlatMapFunctionBase

_parameters

Constructor Summary

Constructors

Constructor and Description
KeplerPairFlatMapFunction(java.lang.String sourceActorName, java.lang.String sinkActorName)

Method Summary

Modifier and Type	Method and Description
protected void	_cleanup() Free resources.
protected void	_initialize() Configure the stub by loading the workflow and initialize the class fields.
void	executionError(Manager manager, java.lang.Throwable throwable) Report workflow execution errors to the log.
void	executionFinished(Manager manager) Do nothing.
void	managerStateChanged(Manager manager) Do nothing.

Methods inherited from class org.kepler.spark.stub.PairFlatMapFunctionBase

readExternal, setConfiguration, writeExternal

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.api.java.function.PairFlatMapFunction

call

Field Detail

_sourceActor

protected StubSourceActor _sourceActor

A source actor used for reading data from Spark and writing it the Kepler workflow.

_sinkActor

protected StubSinkActor _sinkActor

A sink actor used for reading data from the workflow and writing it to Spark.

_manager

protected Manager _manager

Manager to execute the workflow.

_runWorkflowLifecyclePerInput

protected boolean _runWorkflowLifecyclePerInput

If true, the entire workflow lifecycle is executed for each input.

_keplerManagerException

protected java.lang.Exception _keplerManagerException

If the workflow is running in a separate thread (_runWorkflowLifecyclePerInput = false) and an exception is thrown, this field is set to that exception.

Constructor Detail

KeplerPairFlatMapFunction

public KeplerPairFlatMapFunction(java.lang.String sourceActorName,
                                 java.lang.String sinkActorName)

Method Detail

_cleanup

protected void _cleanup()
                 throws java.lang.Exception

Free resources.

Throws:

java.lang.Exception

_initialize

protected void _initialize()
                    throws java.lang.Exception

Configure the stub by loading the workflow and initialize the class fields.

Throws:

java.lang.Exception

executionError

public void executionError(Manager manager,
                           java.lang.Throwable throwable)

Report workflow execution errors to the log.

Specified by:

executionError in interface ExecutionListener

Parameters:

manager - The manager controlling the execution.

throwable - The throwable to report.

executionFinished

public void executionFinished(Manager manager)

Do nothing.

Specified by:

executionFinished in interface ExecutionListener

Parameters:

manager - The manager controlling the execution.

managerStateChanged

public void managerStateChanged(Manager manager)

Do nothing.

Specified by:

managerStateChanged in interface ExecutionListener

Parameters:

manager - The manager controlling the execution.

See Also:

Manager.getState()