org.kepler.bio.stratosphere.input

Class FASTAInputFormatBySize

java.lang.Object

eu.stratosphere.api.common.io.FileInputFormat<eu.stratosphere.types.Record>

eu.stratosphere.api.java.record.io.FileInputFormat

org.kepler.bio.stratosphere.input.FASTAInputFormatBySize

All Implemented Interfaces:

eu.stratosphere.api.common.io.InputFormat<eu.stratosphere.types.Record,eu.stratosphere.core.fs.FileInputSplit>, java.io.Serializable

public class FASTAInputFormatBySize
extends eu.stratosphere.api.java.record.io.FileInputFormat

This Input Format is to read fasta sequence file and generate key value pair records. The format of each key is : partitionNumber_(readPosIndex+start) and each value is a list of sequences. Parameter 'KEPLER::parameter::SequenceSizePerExecution' specify the maximum size of sequence data that will be put into each record. This Input Format can guarantee each sequence is read in a whole and each record won't have truncated sequences. It is built based on eu.stratosphere.pact.common.io.TextInputFormat class.

Author:

Jianwu Wang

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class eu.stratosphere.api.common.io.FileInputFormat

eu.stratosphere.api.common.io.FileInputFormat.AbstractConfigBuilder<T>, eu.stratosphere.api.common.io.FileInputFormat.ConfigBuilder, eu.stratosphere.api.common.io.FileInputFormat.FileBaseStatistics, eu.stratosphere.api.common.io.FileInputFormat.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DELIMITER_POSITION The configuration key to set position of the record delimiter. its value is null, prefix or suffix.
static java.lang.String	NUM_STATISTICS_SAMPLES The configuration key to set the number of samples to take for the statistics.
static java.lang.String	RECORD_DELIMITER The configuration key to set the record delimiter.
static java.lang.String	SEQ_SIZE_PER_EXE The configuration key to set the sequence data size (unite: MB).

Fields inherited from class eu.stratosphere.api.common.io.FileInputFormat

filePath, minSplitSize, numSplits, openTimeout, splitLength, splitStart, stream

Constructor Summary

Constructors

Constructor and Description
FASTAInputFormatBySize()

Method Summary

Modifier and Type	Method and Description
void	close() Closes the input by releasing all buffers and closing the file input stream.
void	configure(eu.stratosphere.configuration.Configuration parameters) Configures this input format by reading the path to the file from the configuration and the string that defines the record delimiter.
int	getBufferSize() Gets the size of the buffer internally used to parse record boundaries.
byte[]	getDelimiter() Gets the delimiter that defines the record boundaries.
eu.stratosphere.api.common.io.FileInputFormat.FileBaseStatistics	getStatistics(eu.stratosphere.api.common.io.statistics.BaseStatistics cachedStatistics)
boolean	nextRecord(eu.stratosphere.types.Record record)
void	open(eu.stratosphere.core.fs.FileInputSplit split) Opens the given input split.
boolean	reachedEnd() Checks whether the current split is at its end.
void	setBufferSize(int bufferSize) Sets the size of the buffer to be used to find record boundaries.

Methods inherited from class eu.stratosphere.api.common.io.FileInputFormat

configureFileFormat, createInputSplits, getFilePath, getFileStats, getInputSplitType, getMinSplitSize, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, setFilePath, setFilePath, setMinSplitSize, setNumSplits, setOpenTimeout, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

RECORD_DELIMITER

public static final java.lang.String RECORD_DELIMITER

The configuration key to set the record delimiter.

See Also:

Constant Field Values

NUM_STATISTICS_SAMPLES

public static final java.lang.String NUM_STATISTICS_SAMPLES

The configuration key to set the number of samples to take for the statistics.

See Also:

Constant Field Values

SEQ_SIZE_PER_EXE

public static final java.lang.String SEQ_SIZE_PER_EXE

The configuration key to set the sequence data size (unite: MB). It specifies the data size to be processed for each sub-workflow execution. Its default value is 1.

See Also:

Constant Field Values

DELIMITER_POSITION

public static final java.lang.String DELIMITER_POSITION

The configuration key to set position of the record delimiter. its value is null, prefix or suffix.

See Also:

Constant Field Values

Constructor Detail

FASTAInputFormatBySize

public FASTAInputFormatBySize()

Method Detail

getDelimiter

public byte[] getDelimiter()

Gets the delimiter that defines the record boundaries.

Returns:

The delimiter, as bytes.

setBufferSize

public void setBufferSize(int bufferSize)

Sets the size of the buffer to be used to find record boundaries. This method has only an effect, if it is called before the input format is opened.

Parameters:

bufferSize - The buffer size to use.

getBufferSize

public int getBufferSize()

Gets the size of the buffer internally used to parse record boundaries.

Returns:

The size of the parsing buffer.

configure

public void configure(eu.stratosphere.configuration.Configuration parameters)

Configures this input format by reading the path to the file from the configuration and the string that defines the record delimiter.

Specified by:

configure in interface eu.stratosphere.api.common.io.InputFormat<eu.stratosphere.types.Record,eu.stratosphere.core.fs.FileInputSplit>

Overrides:

configure in class eu.stratosphere.api.common.io.FileInputFormat<eu.stratosphere.types.Record>

Parameters:

parameters - The configuration object to read the parameters from.

getStatistics

public eu.stratosphere.api.common.io.FileInputFormat.FileBaseStatistics getStatistics(eu.stratosphere.api.common.io.statistics.BaseStatistics cachedStatistics)

Specified by:

getStatistics in interface eu.stratosphere.api.common.io.InputFormat<eu.stratosphere.types.Record,eu.stratosphere.core.fs.FileInputSplit>

Overrides:

getStatistics in class eu.stratosphere.api.common.io.FileInputFormat<eu.stratosphere.types.Record>

open

public void open(eu.stratosphere.core.fs.FileInputSplit split)
          throws java.io.IOException

Opens the given input split. This method opens the input stream to the specified file, allocates read buffers and positions the stream at the correct position, making sure that any partial record at the beginning is skipped.

Specified by:

open in interface eu.stratosphere.api.common.io.InputFormat<eu.stratosphere.types.Record,eu.stratosphere.core.fs.FileInputSplit>

Overrides:

open in class eu.stratosphere.api.common.io.FileInputFormat<eu.stratosphere.types.Record>

Parameters:

split - The input split to open.

Throws:

java.io.IOException

See Also:

eu.stratosphere.pact.common.io.FileInputFormat#open(eu.stratosphere.nephele.fs.FileInputSplit)

reachedEnd

public boolean reachedEnd()

Checks whether the current split is at its end.

Returns:

True, if the split is at its end, false otherwise.

nextRecord

public boolean nextRecord(eu.stratosphere.types.Record record)
                   throws java.io.IOException

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Closes the input by releasing all buffers and closing the file input stream.

Specified by:

close in interface eu.stratosphere.api.common.io.InputFormat<eu.stratosphere.types.Record,eu.stratosphere.core.fs.FileInputSplit>

Overrides:

close in class eu.stratosphere.api.common.io.FileInputFormat<eu.stratosphere.types.Record>

Throws:

java.io.IOException - Thrown, if the closing of the file stream causes an I/O error.