org.apache.sysml.runtime.matrix

Class CSVReblockMR

java.lang.Object

org.apache.sysml.runtime.matrix.CSVReblockMR

public class CSVReblockMR
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CSVReblockMR.AssignRowIDMRReturn
static class	CSVReblockMR.BlockRow
static class	CSVReblockMR.OffsetCount

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	hiddenFileFilter
static String	NUM_COLS_IN_MATRIX
static String	NUM_ROWS_IN_MATRIX
static String	ROWID_FILE_NAME
static String	SMALLEST_FILE_NAME_PER_INPUT

Method Summary

Modifier and Type	Method and Description
static String	findSmallestFile(org.apache.hadoop.mapred.JobConf job, String inputPath) Method to find the first (part)file in the order given by fs.listStatus() among all (part)files in inpathPath.
static CSVReblockMR.AssignRowIDMRReturn	runAssignRowIDMRJob(String[] inputs, InputInfo[] inputInfos, int[] brlens, int[] bclens, String reblockInstructions, int replication, String[] smallestFiles)
static JobReturn	runJob(MRJobInstruction inst, String[] inputs, InputInfo[] inputInfos, long[] rlens, long[] clens, int[] brlens, int[] bclens, String reblockInstructions, String otherInstructionsInReducer, int numReducers, int replication, byte[] resultIndexes, String[] outputs, OutputInfo[] outputInfos)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NUM_ROWS_IN_MATRIX

public static final String NUM_ROWS_IN_MATRIX

See Also:

Constant Field Values

NUM_COLS_IN_MATRIX

public static final String NUM_COLS_IN_MATRIX

See Also:

Constant Field Values

ROWID_FILE_NAME

public static final String ROWID_FILE_NAME

See Also:

Constant Field Values

SMALLEST_FILE_NAME_PER_INPUT

public static final String SMALLEST_FILE_NAME_PER_INPUT

See Also:

Constant Field Values

hiddenFileFilter

public static final org.apache.hadoop.fs.PathFilter hiddenFileFilter

Method Detail

findSmallestFile

public static String findSmallestFile(org.apache.hadoop.mapred.JobConf job,
                                      String inputPath)
                               throws FileNotFoundException,
                                      IOException

Method to find the first (part)file in the order given by fs.listStatus() among all (part)files in inpathPath.

Parameters:

job - job configuration

inputPath - path to input file

Returns:

file name

Throws:

IOException - if IOException occurs

FileNotFoundException - if FileNotFoundException occurs

runJob

public static JobReturn runJob(MRJobInstruction inst,
                               String[] inputs,
                               InputInfo[] inputInfos,
                               long[] rlens,
                               long[] clens,
                               int[] brlens,
                               int[] bclens,
                               String reblockInstructions,
                               String otherInstructionsInReducer,
                               int numReducers,
                               int replication,
                               byte[] resultIndexes,
                               String[] outputs,
                               OutputInfo[] outputInfos)
                        throws Exception

Throws:

Exception

runAssignRowIDMRJob

public static CSVReblockMR.AssignRowIDMRReturn runAssignRowIDMRJob(String[] inputs,
                                                                   InputInfo[] inputInfos,
                                                                   int[] brlens,
                                                                   int[] bclens,
                                                                   String reblockInstructions,
                                                                   int replication,
                                                                   String[] smallestFiles)
                                                            throws Exception

Throws:

Exception