Modifier and Type | Field and Description |
---|---|
protected MatrixCharacteristics |
MLMatrix.mc
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
MatrixCharacteristics |
MLOutput.getMatrixCharacteristics(String varName)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
MLOutput.getDF(org.apache.spark.sql.SparkSession sparkSession,
String varName,
MatrixCharacteristics mc)
Deprecated.
This methods improves the performance of MLPipeline wrappers.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
MLOutput.getDF(org.apache.spark.sql.SQLContext sqlContext,
String varName,
MatrixCharacteristics mc)
Deprecated.
This methods improves the performance of MLPipeline wrappers.
|
void |
MLContext.registerInput(String varName,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> rdd,
MatrixCharacteristics mc)
Deprecated.
|
void |
MLContext.registerInput(String varName,
MatrixBlock mb,
MatrixCharacteristics mc)
Deprecated.
|
Constructor and Description |
---|
MLMatrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
MatrixCharacteristics mc,
MLContext ml)
Deprecated.
|
Constructor and Description |
---|
MLOutput(Map<String,org.apache.spark.api.java.JavaPairRDD<?,?>> outputs,
Map<String,MatrixCharacteristics> outMetadata)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
MatrixCharacteristics |
MatrixMetadata.asMatrixCharacteristics()
Convert the matrix metadata to a MatrixCharacteristics object.
|
MatrixCharacteristics |
FrameMetadata.asMatrixCharacteristics()
Convert the frame metadata to a MatrixCharacteristics object.
|
MatrixCharacteristics |
BinaryBlockMatrix.getMatrixCharacteristics()
Obtain the SystemML binary-block matrix characteristics
|
MatrixCharacteristics |
BinaryBlockFrame.getMatrixCharacteristics()
Obtain the SystemML binary-block frame characteristics
|
Modifier and Type | Method and Description |
---|---|
void |
MatrixMetadata.setMatrixCharacteristics(MatrixCharacteristics matrixCharacteristics)
Set the MatrixMetadata fields based on a MatrixCharacteristics object.
|
void |
FrameMetadata.setMatrixCharacteristics(MatrixCharacteristics matrixCharacteristics)
Set the FrameMetadata fields based on a MatrixCharacteristics object.
|
Constructor and Description |
---|
BinaryBlockFrame(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> binaryBlocks,
MatrixCharacteristics matrixCharacteristics)
Create a BinaryBlockFrame, specifying the SystemML binary-block frame and
its metadata.
|
BinaryBlockMatrix(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks,
MatrixCharacteristics matrixCharacteristics)
Create a BinaryBlockMatrix, specifying the SystemML binary-block matrix
and its metadata.
|
FrameMetadata(FrameSchema frameSchema,
MatrixCharacteristics matrixCharacteristics)
Constructor to create a FrameMetadata object based on the frame schema
and a MatrixCharacteristics object.
|
FrameMetadata(MatrixCharacteristics matrixCharacteristics)
Constructor to create a FrameMetadata object based on a
MatrixCharacteristics object.
|
MatrixMetadata(MatrixCharacteristics matrixCharacteristics)
Constructor to create a MatrixMetadata object based on a
MatrixCharacteristics object.
|
Modifier and Type | Method and Description |
---|---|
void |
MRJobInstruction.printCompleteMRJobInstruction(MatrixCharacteristics[] resultStats) |
Modifier and Type | Method and Description |
---|---|
protected long |
BinarySPInstruction.getNumReplicas(MatrixCharacteristics mc1,
MatrixCharacteristics mc2,
boolean left) |
static MatrixBlock |
MatrixIndexingSPInstruction.inmemoryIndexing(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in1,
MatrixCharacteristics mcIn,
MatrixCharacteristics mcOut,
org.apache.sysml.runtime.util.IndexRange ixrange) |
static boolean |
MatrixIndexingSPInstruction.isMultiBlockLookup(org.apache.spark.api.java.JavaPairRDD<?,?> in,
MatrixCharacteristics mcIn,
MatrixCharacteristics mcOut,
org.apache.sysml.runtime.util.IndexRange ixrange)
Indicates if the given index range and input matrix exhibit the following properties:
(1) existing hash partitioner, (2) out-of-core input matrix (larger than aggregated memory),
(3) aligned indexing range (which does not required aggregation), and (4) the output fits
twice in memory (in order to collect the result).
|
static boolean |
MatrixIndexingSPInstruction.isSingleBlockLookup(MatrixCharacteristics mcIn,
org.apache.sysml.runtime.util.IndexRange ixrange)
Indicates if the given index range only covers a single blocks of the inputs matrix.
|
protected static boolean |
UaggOuterChainSPInstruction.preservesPartitioning(MatrixCharacteristics mcIn,
org.apache.sysml.runtime.functionobjects.IndexFunction ixfun) |
protected org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
CSVReblockSPInstruction.processFrameCSVReblockInstruction(org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext sec,
MatrixCharacteristics mcOut,
org.apache.sysml.parser.Expression.ValueType[] schema) |
protected org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
CSVReblockSPInstruction.processMatrixCSVReblockInstruction(org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext sec,
MatrixCharacteristics mcOut) |
void |
ParameterizedBuiltinSPInstruction.setOutputCharacteristicsForGroupedAgg(MatrixCharacteristics mc1,
MatrixCharacteristics mcOut,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> out) |
Constructor and Description |
---|
MergeWithShiftedBlocks(MatrixCharacteristics mc1,
MatrixCharacteristics mc2,
boolean cbind) |
ShiftMatrix(MatrixCharacteristics mc1,
MatrixCharacteristics mc2,
boolean cbind) |
Modifier and Type | Method and Description |
---|---|
static MatrixCharacteristics |
SparkUtils.computeMatrixCharacteristics(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> input)
Utility to compute dimensions and non-zeros in a given RDD of binary cells.
|
Modifier and Type | Method and Description |
---|---|
static org.apache.spark.api.java.JavaRDD<String> |
FrameRDDConverterUtils.binaryBlockToCsv(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> in,
MatrixCharacteristics mcIn,
CSVFileFormatProperties props,
boolean strict) |
static org.apache.spark.api.java.JavaRDD<String> |
RDDConverterUtils.binaryBlockToCsv(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
MatrixCharacteristics mcIn,
CSVFileFormatProperties props,
boolean strict) |
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
FrameRDDConverterUtils.binaryBlockToDataFrame(org.apache.spark.sql.SparkSession sparkSession,
org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> in,
MatrixCharacteristics mc,
org.apache.sysml.parser.Expression.ValueType[] schema) |
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
RDDConverterUtils.binaryBlockToDataFrame(org.apache.spark.sql.SparkSession sparkSession,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
MatrixCharacteristics mc,
boolean toVector) |
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
FrameRDDConverterUtils.binaryBlockToDataFrame(org.apache.spark.sql.SQLContext sqlContext,
org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> in,
MatrixCharacteristics mc,
org.apache.sysml.parser.Expression.ValueType[] schema)
Deprecated.
|
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
RDDConverterUtils.binaryBlockToDataFrame(org.apache.spark.sql.SQLContext sqlContext,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
MatrixCharacteristics mc,
boolean toVector)
Deprecated.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
FrameRDDConverterUtils.binaryBlockToMatrixBlock(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> input,
MatrixCharacteristics mcIn,
MatrixCharacteristics mcOut) |
static org.apache.spark.api.java.JavaRDD<String> |
FrameRDDConverterUtils.binaryBlockToTextCell(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> input,
MatrixCharacteristics mcIn) |
static org.apache.spark.api.java.JavaRDD<String> |
RDDConverterUtils.binaryBlockToTextCell(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
MatrixCharacteristics mc) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.binaryCellToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> input,
MatrixCharacteristics mcOut,
boolean outputEmptyBlocks) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtilsExt.coordinateMatrixToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input,
MatrixCharacteristics mcIn,
boolean outputEmptyBlocks)
Example usage:
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtilsExt.coordinateMatrixToBinaryBlock(org.apache.spark.SparkContext sc,
org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input,
MatrixCharacteristics mcIn,
boolean outputEmptyBlocks) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.csvToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> input,
MatrixCharacteristics mc,
boolean hasHeader,
String delim,
boolean fill,
double fillValue) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.csvToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> input,
MatrixCharacteristics mc,
org.apache.sysml.parser.Expression.ValueType[] schema,
boolean hasHeader,
String delim,
boolean fill,
double fillValue) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.csvToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaRDD<String> input,
MatrixCharacteristics mcOut,
boolean hasHeader,
String delim,
boolean fill,
double fillValue)
Example usage:
|
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.csvToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaRDD<String> input,
MatrixCharacteristics mcOut,
org.apache.sysml.parser.Expression.ValueType[] schema,
boolean hasHeader,
String delim,
boolean fill,
double fillValue) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.dataFrameToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
MatrixCharacteristics mc,
boolean containsID) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.dataFrameToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
MatrixCharacteristics mc,
boolean containsID,
boolean isVector) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.dataFrameToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
MatrixCharacteristics mc,
boolean containsID,
Pair<String[],org.apache.sysml.parser.Expression.ValueType[]> out) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
SparkUtils.getEmptyBlockRDD(org.apache.spark.api.java.JavaSparkContext sc,
MatrixCharacteristics mc)
Creates an RDD of empty blocks according to the given matrix characteristics.
|
static int |
SparkUtils.getNumPreferredPartitions(MatrixCharacteristics mc,
org.apache.spark.api.java.JavaPairRDD<?,?> in) |
static void |
RDDConverterUtils.libsvmToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
String pathIn,
String pathX,
String pathY,
MatrixCharacteristics mcOutX)
Converts a libsvm text input file into two binary block matrices for features
and labels, and saves these to the specified output files.
|
static org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,FrameBlock> |
FrameRDDConverterUtils.matrixBlockToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> input,
MatrixCharacteristics mcIn) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.matrixBlockToBinaryBlockLongIndex(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> input,
MatrixCharacteristics mcIn) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.textCellToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> input,
MatrixCharacteristics mcOut,
boolean outputEmptyBlocks) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.textCellToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> in,
MatrixCharacteristics mcOut,
org.apache.sysml.parser.Expression.ValueType[] schema) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.textCellToBinaryBlockLongIndex(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<Long,org.apache.hadoop.io.Text> input,
MatrixCharacteristics mc,
org.apache.sysml.parser.Expression.ValueType[] schema) |
Modifier and Type | Field and Description |
---|---|
protected MatrixCharacteristics |
MatrixDimensionsMetaData.matchar |
Modifier and Type | Method and Description |
---|---|
MatrixCharacteristics |
MatrixDimensionsMetaData.getMatrixCharacteristics() |
MatrixCharacteristics |
JobReturn.getMatrixCharacteristics(int i) |
Modifier and Type | Method and Description |
---|---|
static void |
MatrixCharacteristics.aggregateBinary(MatrixCharacteristics dim1,
MatrixCharacteristics dim2,
org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op,
MatrixCharacteristics dimOut) |
static void |
MatrixCharacteristics.aggregateUnary(MatrixCharacteristics dim,
org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixCharacteristics dimOut) |
static void |
MatrixCharacteristics.reorg(MatrixCharacteristics dim,
org.apache.sysml.runtime.matrix.operators.ReorgOperator op,
MatrixCharacteristics dimOut) |
void |
MatrixCharacteristics.set(MatrixCharacteristics that) |
void |
MatrixDimensionsMetaData.setMatrixCharacteristics(MatrixCharacteristics mc) |
Modifier and Type | Method and Description |
---|---|
static void |
MatrixCharacteristics.computeDimension(HashMap<Byte,MatrixCharacteristics> dims,
org.apache.sysml.runtime.instructions.mr.MRInstruction ins) |
Constructor and Description |
---|
JobReturn(MatrixCharacteristics[] sts,
boolean success) |
JobReturn(MatrixCharacteristics[] sts,
OutputInfo[] infos,
boolean success) |
JobReturn(MatrixCharacteristics mc,
long[] items,
int partition0,
long number0s,
boolean success) |
JobReturn(MatrixCharacteristics sts,
OutputInfo info,
boolean success) |
MatrixCharacteristics(MatrixCharacteristics that) |
MatrixDimensionsMetaData(MatrixCharacteristics mc) |
MatrixFormatMetaData(MatrixCharacteristics mc,
OutputInfo oinfo_,
InputInfo iinfo_) |
Constructor and Description |
---|
NumItemsByEachReducerMetaData(MatrixCharacteristics mc,
long[] nums,
int part0,
long num0) |
Copyright © 2017 The Apache Software Foundation. All rights reserved.