org.kepler.spark.actor

Class SparkConnection

java.lang.Object

ptolemy.kernel.util.NamedObj

ptolemy.kernel.util.Attribute

org.kepler.spark.actor.SparkConnection

All Implemented Interfaces:

java.lang.Cloneable, Changeable, Debuggable, DebugListener, Derivable, ModelErrorHandler, MoMLExportable, Moveable, Nameable

public class SparkConnection
extends Attribute

An attribute that contains connection information to a Spark server. The value specified in connectionName is used by Spark actors to refer to an instance of this attribute and use its Spark server.

Author:

Daniel Crawl

Nested Class Summary

Nested classes/interfaces inherited from class ptolemy.kernel.util.NamedObj

NamedObj.ContainedObjectsIterator

Field Summary

Fields

Modifier and Type	Field and Description
StringParameter	connectionName The name of the connection.
static java.lang.String	DEFAULT_DRIVER_MEMORY The default amount of memory for the Spark driver.
static int	DEFAULT_MASTER_PORT Default port for the master.
static int	DEFAULT_NUM_LOCAL_WORKERS The default number of local workers.
StringParameter	driverMemory The amount of memory to use in the Spark driver.
Parameter	numLocalWorkers The number of workers when running Spark locally.
StringParameter	serverName The host name of the Spark server.

Fields inherited from class ptolemy.kernel.util.NamedObj

_changeListeners, _changeLock, _changeRequests, _debugging, _debugListeners, _deferChangeRequests, _elementName, _isPersistent, _verbose, _workspace, ATTRIBUTES, CLASSNAME, COMPLETE, CONTENTS, DEEP, FULLNAME, LINKS

Constructor Summary

Constructors

Constructor and Description
SparkConnection(NamedObj container, java.lang.String name) Create a new SparkConnection with the specified name.

Method Summary

Modifier and Type	Method and Description
void	attributeChanged(Attribute attribute) React to a change in an attribute.
java.lang.String	getConnectionName() Get the name used for this attribute.
org.apache.spark.api.java.JavaSparkContext	getContext() Get the context using the parameters of this SparkConnection (serverName, driverMemory, numLocalWorkers, etc.).
static org.apache.spark.api.java.JavaSparkContext	getContext(java.lang.String masterNameAndPortStr, int numLocalWorkersVal, java.lang.String driverMemoryStr, java.lang.String connectionNameStr) Get the context for a specific master.
static org.apache.spark.api.java.JavaSparkContext	getDefaultContext() Get the default context.
static java.net.InetSocketAddress	getDefaultMasterSocketAddress() Get the default Spark master socket address.
static java.net.InetSocketAddress	getMasterAndPortAddress(java.lang.String masterHostAndPortStr) Parse a string containing the master host and port and return the socket address.
static java.lang.String	parseOutputFromStartingServer(java.io.InputStream input) Parse the output from starting the server using start-myspark.sh
static void	releaseContext()
static void	shutdown() Stop any contexts that were started.

Methods inherited from class ptolemy.kernel.util.Attribute

_checkContainer, _getContainedObject, _propagateExistence, clone, getContainer, moveDown, moveToFirst, moveToIndex, moveToLast, moveUp, setContainer, setName, updateContent

Methods inherited from class ptolemy.kernel.util.NamedObj

_addAttribute, _adjustOverride, _attachText, _cloneFixAttributeFields, _containedDecorators, _copyChangeRequestList, _debug, _debug, _debug, _debug, _debug, _description, _executeChangeRequests, _exportMoMLContents, _getIndentPrefix, _isMoMLSuppressed, _markContentsDerived, _notifyHierarchyListenersAfterChange, _notifyHierarchyListenersBeforeChange, _propagateValue, _removeAttribute, _splitName, _stripNumericSuffix, _validateSettables, addChangeListener, addDebugListener, addHierarchyListener, attributeDeleted, attributeList, attributeList, attributeTypeChanged, clone, containedObjectsIterator, decorators, deepContains, depthInHierarchy, description, description, event, executeChangeRequests, exportMoML, exportMoML, exportMoML, exportMoML, exportMoML, exportMoMLPlain, getAttribute, getAttribute, getAttributes, getChangeListeners, getClassName, getDecoratorAttribute, getDecoratorAttributes, getDerivedLevel, getDerivedList, getDisplayName, getElementName, getFullName, getModelErrorHandler, getName, getName, getPrototypeList, getSource, handleModelError, isDeferringChangeRequests, isOverridden, isPersistent, lazyContainedObjectsIterator, message, notifyOfNameChange, propagateExistence, propagateValue, propagateValues, removeAttribute, removeChangeListener, removeDebugListener, removeHierarchyListener, requestChange, setClassName, setDeferringChangeRequests, setDerivedLevel, setDisplayName, setModelErrorHandler, setPersistent, setSource, sortContainedObjects, toplevel, toString, uniqueName, validateSettables, workspace

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

connectionName

public StringParameter connectionName

The name of the connection.

serverName

public StringParameter serverName

The host name of the Spark server. Use "local" to run on the local machine.

numLocalWorkers

public Parameter numLocalWorkers

The number of workers when running Spark locally.

driverMemory

public StringParameter driverMemory

The amount of memory to use in the Spark driver.

DEFAULT_NUM_LOCAL_WORKERS

public static final int DEFAULT_NUM_LOCAL_WORKERS

The default number of local workers.

See Also:

Constant Field Values

DEFAULT_DRIVER_MEMORY

public static final java.lang.String DEFAULT_DRIVER_MEMORY

The default amount of memory for the Spark driver.

See Also:

Constant Field Values

DEFAULT_MASTER_PORT

public static final int DEFAULT_MASTER_PORT

Default port for the master.

See Also:

Constant Field Values

Constructor Detail

SparkConnection

public SparkConnection(NamedObj container,
                       java.lang.String name)
                throws IllegalActionException,
                       NameDuplicationException

Create a new SparkConnection with the specified name.

Throws:

IllegalActionException

NameDuplicationException

Method Detail

attributeChanged

public void attributeChanged(Attribute attribute)
                      throws IllegalActionException

Description copied from class: NamedObj

React to a change in an attribute. This method is called by a contained attribute when its value changes. In this base class, the method does nothing. In derived classes, this method may throw an exception, indicating that the new attribute value is invalid. It is up to the caller to restore the attribute to a valid value if an exception is thrown.

Overrides:

attributeChanged in class NamedObj

Parameters:

attribute - The attribute that changed.

Throws:

IllegalActionException - If the change is not acceptable to this container (not thrown in this base class).

getConnectionName

public java.lang.String getConnectionName()
                                   throws IllegalActionException

Get the name used for this attribute.

Throws:

IllegalActionException

getContext

public org.apache.spark.api.java.JavaSparkContext getContext()
                                                      throws IllegalActionException

Get the context using the parameters of this SparkConnection (serverName, driverMemory, numLocalWorkers, etc.).

Throws:

IllegalActionException

getContext

public static org.apache.spark.api.java.JavaSparkContext getContext(java.lang.String masterNameAndPortStr,
                                                                    int numLocalWorkersVal,
                                                                    java.lang.String driverMemoryStr,
                                                                    java.lang.String connectionNameStr)
                                                             throws IllegalActionException

Get the context for a specific master.

Parameters:

masterNameAndPortStr - The master host and port. If empty, then environment variables MASTER, SPARK_MASTER_IP, and SPARK_LOCAL_IP are checked for the master. If these are empty, then try to connect to a master running on localhost. If the connection fails, then a local master is used (runs in same JVM).

numLocalWorkersVal - The number of workers to use for a local master.

driverMemoryStr - The amount of memory in the driver.

connectionNameStr - The connecting name string.

Throws:

IllegalActionException

getDefaultContext

public static org.apache.spark.api.java.JavaSparkContext getDefaultContext()
                                                                    throws IllegalActionException

Get the default context.

Throws:

IllegalActionException

getDefaultMasterSocketAddress

public static java.net.InetSocketAddress getDefaultMasterSocketAddress()
                                                                throws IllegalActionException

Get the default Spark master socket address.

Throws:

IllegalActionException

getMasterAndPortAddress

public static java.net.InetSocketAddress getMasterAndPortAddress(java.lang.String masterHostAndPortStr)
                                                          throws IllegalActionException

Parse a string containing the master host and port and return the socket address.

Throws:

IllegalActionException

parseOutputFromStartingServer

public static java.lang.String parseOutputFromStartingServer(java.io.InputStream input)
                                                      throws java.io.IOException,
                                                             IllegalActionException

Parse the output from starting the server using start-myspark.sh

Returns:

The master URL if successful, otherwise null.

Throws:

java.io.IOException

IllegalActionException

releaseContext

public static void releaseContext()

shutdown

public static void shutdown()

Stop any contexts that were started.