org.kepler.hadoop.director

Class HadoopEngine

java.lang.Object

org.kepler.ddp.director.DDPEngine

org.kepler.hadoop.director.HadoopEngine

All Implemented Interfaces:

java.lang.Cloneable

public class HadoopEngine
extends DDPEngine

An engine than runs workflows in Hadoop. This engine converts DDP pattern actors (Map, Reduce, Cross, CoGroup, and Match) and I/O actors (DDPDataSink and DDPDataSource) into a Hadoop job and runs it on the server.

NOTE: Only DDP pattern and I/O actors may be present in the workflow. Other actors must placed inside the composite pattern actors or in a different sub-workflow.

Author:

Jianwu Wang

Field Summary

Fields

Modifier and Type	Field and Description
boolean	_autoHDFSStage If true, input and output in local file system is automatically staged into or out from HDFS.
boolean	_overwriteOutput If true, the output directory will be deleted first.
Parameter	autoHDFSStage If true, input and output in local file system is automatically staged into or out from HDFS.
static java.lang.String	CORE_SITE
static java.lang.String	FS_NAME
static java.lang.String	HDFS_NAME
static java.lang.String	MAP_CHILD_ENV
static java.lang.String	MAPRED_SITE
static java.lang.String	MR_FRAMEWORK_NAME
static java.lang.String	MR_JOBTRACKER_ADDRESS
Parameter	overwriteOutput If true, the output directory will be deleted first.
static java.lang.String	REDUCE_CHILD_ENV
Parameter	removeTmpHDFSDir If true, the temporary dir created on HDFS during its execution will be removed after workflow execution.
static java.lang.String	RES_MANAGER_URL
static java.lang.String	YARN_RM_ADDRESS
static java.lang.String	YARN_SITE

Fields inherited from class org.kepler.ddp.director.DDPEngine

_additionalJars, _classLoader, _configDirStr, _CONNECT_TIMEOUT, _container, _degreeOfParallelism, _director, _displayRedirectDir, _engineName, _jobArgumentsMap, _jobDir, _jobLock, _sameJVM, _serverStartStopLock, _subWorkflows, _writeSubWorkflowsToFiles

Constructor Summary

Constructors

Constructor and Description
HadoopEngine(DDPDirector director)

Method Summary

Modifier and Type	Method and Description
protected void	_addParameters() Add parameters to the containing director.
protected void	_executeDDPJob() Execute the DDP job.
protected void	_removeParameters() Remove engine-specific parameters from the director.
protected java.util.Map<java.lang.String,Type>	_setPortTypes(DDPPatternActor actor) Set the port types inside a cloned pattern actor.
java.lang.Object	clone(Workspace workspace) Clone the HadoopEngine into the specified workspace.
boolean	postfire() Postfire the engine.
void	preinitialize() Initialize fields from parameters.
static void	shutdownServer() Shutdown the Hadoop server if one was started.
void	stop() Stop any running Hadoop jobs.
void	wrapup() Free resources.

Methods inherited from class org.kepler.ddp.director.DDPEngine

_checkConfigDir, _checkFilesBeforeStartingServer, _checkModel, _checkModelForNonDDPActors, _checkServer, _copyParameters, _createJobDirectory, _getJarList, _getParametersInHierarchy, _parseOutputFromStartingServer, attributeChanged, clone, closeAllEffigies, createEffigy, fire, getDisplayRedirectDir, getModel, getName, getServerTypes

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HDFS_NAME

public static final java.lang.String HDFS_NAME

See Also:

Constant Field Values

FS_NAME

public static final java.lang.String FS_NAME

See Also:

Constant Field Values

RES_MANAGER_URL

public static final java.lang.String RES_MANAGER_URL

See Also:

Constant Field Values

CORE_SITE

public static final java.lang.String CORE_SITE

See Also:

Constant Field Values

MAPRED_SITE

public static final java.lang.String MAPRED_SITE

See Also:

Constant Field Values

YARN_SITE

public static final java.lang.String YARN_SITE

See Also:

Constant Field Values

MAP_CHILD_ENV

public static final java.lang.String MAP_CHILD_ENV

See Also:

Constant Field Values

REDUCE_CHILD_ENV

public static final java.lang.String REDUCE_CHILD_ENV

See Also:

Constant Field Values

MR_JOBTRACKER_ADDRESS

public static final java.lang.String MR_JOBTRACKER_ADDRESS

See Also:

Constant Field Values

MR_FRAMEWORK_NAME

public static final java.lang.String MR_FRAMEWORK_NAME

See Also:

Constant Field Values

YARN_RM_ADDRESS

public static final java.lang.String YARN_RM_ADDRESS

See Also:

Constant Field Values

overwriteOutput

public Parameter overwriteOutput

If true, the output directory will be deleted first.

removeTmpHDFSDir

public Parameter removeTmpHDFSDir

If true, the temporary dir created on HDFS during its execution will be removed after workflow execution.

autoHDFSStage

public Parameter autoHDFSStage

If true, input and output in local file system is automatically staged into or out from HDFS.

_overwriteOutput

public boolean _overwriteOutput

If true, the output directory will be deleted first.

_autoHDFSStage

public boolean _autoHDFSStage

If true, input and output in local file system is automatically staged into or out from HDFS.

Constructor Detail

HadoopEngine

public HadoopEngine(DDPDirector director)
             throws IllegalActionException,
                    NameDuplicationException

Throws:

IllegalActionException

NameDuplicationException

Method Detail

clone

public java.lang.Object clone(Workspace workspace)
                       throws java.lang.CloneNotSupportedException

Clone the HadoopEngine into the specified workspace.

Overrides:

clone in class DDPEngine

Parameters:

workspace - The workspace for the cloned object.

Returns:

The new HadoopDirector.

Throws:

java.lang.CloneNotSupportedException - Not thrown in this base class

_executeDDPJob

protected void _executeDDPJob()
                       throws IllegalActionException

Description copied from class: DDPEngine

Execute the DDP job.

Specified by:

_executeDDPJob in class DDPEngine

Throws:

IllegalActionException

postfire

public boolean postfire()
                 throws IllegalActionException

Description copied from class: DDPEngine

Postfire the engine. In this base class, returns true.

Overrides:

postfire in class DDPEngine

Throws:

IllegalActionException

preinitialize

public void preinitialize()
                   throws IllegalActionException

Description copied from class: DDPEngine

Initialize fields from parameters.

Overrides:

preinitialize in class DDPEngine

Throws:

IllegalActionException

stop

public void stop()
          throws IllegalActionException

Stop any running Hadoop jobs.

Overrides:

stop in class DDPEngine

Throws:

IllegalActionException

shutdownServer

public static void shutdownServer()
                           throws IllegalActionException

Shutdown the Hadoop server if one was started.

Throws:

IllegalActionException

wrapup

public void wrapup()
            throws IllegalActionException

Free resources.

Overrides:

wrapup in class DDPEngine

Throws:

IllegalActionException

_addParameters

protected void _addParameters()
                       throws IllegalActionException,
                              NameDuplicationException

Add parameters to the containing director.

Overrides:

_addParameters in class DDPEngine

Throws:

IllegalActionException

NameDuplicationException

_removeParameters

protected void _removeParameters()
                          throws IllegalActionException,
                                 NameDuplicationException

Remove engine-specific parameters from the director.

Overrides:

_removeParameters in class DDPEngine

Throws:

IllegalActionException

NameDuplicationException

_setPortTypes

protected java.util.Map<java.lang.String,Type> _setPortTypes(DDPPatternActor actor)
                                                      throws IllegalActionException

Set the port types inside a cloned pattern actor. TODO: merge this with _setPortTypes() in the parent class.

Overrides:

_setPortTypes in class DDPEngine

Parameters:

actor - the cloned actor

Throws:

IllegalActionException