MLMatrix (SystemML 0.14.0 API)

java.lang.Object
- org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>
- - org.apache.sysml.api.MLMatrix

All Implemented Interfaces:

Serializable

Deprecated.
This will be removed in SystemML 1.0. Please migrate to MLContext
```
@Deprecated
public class MLMatrix
extends org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>
```
Experimental API: Might be discontinued in future release This class serves four purposes: 1. It allows SystemML to fit nicely in MLPipeline by reducing number of reblocks. 2. It allows users to easily read and write matrices without worrying too much about format, metadata and type of underlying RDDs. 3. It provides mechanism to convert to and from MLLib's BlockedMatrix format 4. It provides off-the-shelf library for Distributed Blocked Matrix and reduces learning curve for using SystemML. However, it is important to know that it is easy to abuse this off-the-shelf library and think it as replacement to writing DML, which it is not. It does not provide any optimization between calls. A simple example of the optimization that is conveniently skipped is: (t(m) %*% m)). Also, note that this library is not thread-safe. The operator precedence is not exactly same as DML (as the precedence is enforced by scala compiler), so please use appropriate brackets to enforce precedence. import org.apache.sysml.api.{MLContext, MLMatrix} val ml = new MLContext(sc) val mat1 = ml.read(sparkSession, "V_small.csv", "csv") val mat2 = ml.read(sparkSession, "W_small.mtx", "binary") val result = mat1.transpose() %*% mat2 result.write("Result_small.mtx", "text")

See Also:

Serialized Form

Field Summary

Fields
Modifier and Type Field and Description

protected MatrixCharacteristics mc
Deprecated.

protected MLContext ml
Deprecated.
- Fields inherited from class org.apache.spark.sql.Dataset
  org$apache$spark$sql$Dataset$$encoder

Fields
Modifier and Type	Field and Description
`protected MatrixCharacteristics`	`mc` Deprecated.
`protected MLContext`	`ml` Deprecated.

Constructor Summary

Constructors
Modifier	Constructor and Description
`protected`	`MLMatrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df, MatrixCharacteristics mc, MLContext ml)` Deprecated.
`protected`	`MLMatrix(org.apache.spark.sql.SparkSession sparkSession, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan, MLContext ml)` Deprecated.
`protected`	`MLMatrix(org.apache.spark.sql.SparkSession sparkSession, org.apache.spark.sql.execution.QueryExecution queryExecution, MLContext ml)` Deprecated.
`protected`	`MLMatrix(org.apache.spark.sql.SQLContext sqlContext, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan, MLContext ml)` Deprecated.
`protected`	`MLMatrix(org.apache.spark.sql.SQLContext sqlContext, org.apache.spark.sql.execution.QueryExecution queryExecution, MLContext ml)` Deprecated.

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`MLMatrix`	`$bang$eq(Double scalar)` Deprecated.
`MLMatrix`	`$bang$eq(MLMatrix that)` Deprecated.
`MLMatrix`	`$div(Double scalar)` Deprecated.
`MLMatrix`	`$div(MLMatrix that)` Deprecated.
`MLMatrix`	`$eq$eq(Double scalar)` Deprecated.
`MLMatrix`	`$eq$eq(MLMatrix that)` Deprecated.
`MLMatrix`	`$greater(Double scalar)` Deprecated.
`MLMatrix`	`$greater(MLMatrix that)` Deprecated.
`MLMatrix`	`$greater$eq(Double scalar)` Deprecated.
`MLMatrix`	`$greater$eq(MLMatrix that)` Deprecated.
`MLMatrix`	`$less(Double scalar)` Deprecated.
`MLMatrix`	`$less(MLMatrix that)` Deprecated.
`MLMatrix`	`$less$eq(Double scalar)` Deprecated.
`MLMatrix`	`$less$eq(MLMatrix that)` Deprecated.
`MLMatrix`	`$minus(Double scalar)` Deprecated.
`MLMatrix`	`$minus(MLMatrix that)` Deprecated.
`MLMatrix`	`$percent$div$percent(MLMatrix that)` Deprecated.
`MLMatrix`	`$percent$percent(MLMatrix that)` Deprecated.
`MLMatrix`	`$percent$times$percent(MLMatrix that)` Deprecated.
`MLMatrix`	`$plus(Double scalar)` Deprecated.
`MLMatrix`	`$plus(MLMatrix that)` Deprecated.
`MLMatrix`	`$times(Double scalar)` Deprecated.
`MLMatrix`	`$times(MLMatrix that)` Deprecated.
`MLMatrix`	`$up(MLMatrix that)` Deprecated.
`MLMatrix`	`add(Double scalar)` Deprecated.
`MLMatrix`	`add(MLMatrix that)` Deprecated.
`int`	`colsPerBlock()` Deprecated.
`MLMatrix`	`divide(Double scalar)` Deprecated.
`MLMatrix`	`divide(MLMatrix that)` Deprecated.
`MLMatrix`	`elementWiseMultiply(Double scalar)` Deprecated.
`MLMatrix`	`elementWiseMultiply(MLMatrix that)` Deprecated.
`MLMatrix`	`exp(MLMatrix that)` Deprecated.
`MLMatrix`	`integerDivision(MLMatrix that)` Deprecated.
`MLMatrix`	`minus(Double scalar)` Deprecated.
`MLMatrix`	`minus(MLMatrix that)` Deprecated.
`MLMatrix`	`modulus(MLMatrix that)` Deprecated.
`MLMatrix`	`multiply(MLMatrix that)` Deprecated.
`long`	`numCols()` Deprecated. Gets or computes the number of columns.
`long`	`numRows()` Deprecated. Gets or computes the number of rows.
`int`	`rowsPerBlock()` Deprecated.
`MLMatrix`	`transpose()` Deprecated.
`void`	`write(String filePath, String format)` Deprecated. Convenient method to write a MLMatrix.

Methods inherited from class org.apache.spark.sql.Dataset
agg, agg, agg, agg, agg, alias, alias, apply, as, as, as, cache, checkpoint, checkpoint, coalesce, col, collect, collectAsList, collectToPython, columns, count, createGlobalTempView, createOrReplaceTempView, createTempView, crossJoin, cube, cube, cube, cube, describe, describe, distinct, drop, drop, drop, drop, dropDuplicates, dropDuplicates, dropDuplicates, dropDuplicates, dropDuplicates, dtypes, except, explain, explain, explode, explode, exprEnc, filter, filter, filter, filter, first, flatMap, flatMap, foreach, foreach, foreachPartition, foreachPartition, groupBy, groupBy, groupBy, groupBy, groupByKey, groupByKey, head, head, inputFiles, intersect, isLocal, isStreaming, javaRDD, javaToPython, join, join, join, join, join, join, joinWith, joinWith, limit, logicalPlan, map, map, mapPartitions, mapPartitions, mapPartitionsInR, na, numericColumns, ofRows, orderBy, orderBy, orderBy, orderBy, org$apache$spark$sql$Dataset$$aggregatableColumns, org$apache$spark$sql$Dataset$$boundEnc, org$apache$spark$sql$Dataset$$classTag, org$apache$spark$sql$Dataset$$collect, org$apache$spark$sql$Dataset$$createTempViewCommand, org$apache$spark$sql$Dataset$$execute$1, org$apache$spark$sql$Dataset$$hasSideEffects$1, org$apache$spark$sql$Dataset$$rowFunction$1, org$apache$spark$sql$Dataset$$withPlan, persist, persist, printSchema, queryExecution, randomSplit, randomSplit, randomSplit, randomSplitAsList, rdd, reduce, reduce, registerTempTable, repartition, repartition, repartition, repartition, repartition, resolve, rollup, rollup, rollup, rollup, sample, sample, schema, select, select, select, select, select, select, select, select, select, selectExpr, selectExpr, selectUntyped, show, show, show, show, show, showString, showString$default$2, sort, sort, sort, sort, sortWithinPartitions, sortWithinPartitions, sortWithinPartitions, sortWithinPartitions, sparkSession, sqlContext, stat, storageLevel, take, takeAsList, toDF, toDF, toDF, toJavaRDD, toJSON, toLocalIterator, toPythonIterator, toString, transform, union, unionAll, unpersist, unpersist, where, where, withColumn, withColumn, withColumnRenamed, withNewExecutionId, withWatermark, write, writeStream

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

mc
```
protected MatrixCharacteristics mc
```
Deprecated.

ml
```
protected MLContext ml
```
Deprecated.

Constructor Detail

MLMatrix

protected MLMatrix(org.apache.spark.sql.SparkSession sparkSession,
                   org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan,
                   MLContext ml)

Deprecated.

MLMatrix

protected MLMatrix(org.apache.spark.sql.SQLContext sqlContext,
                   org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan,
                   MLContext ml)

Deprecated.

MLMatrix

protected MLMatrix(org.apache.spark.sql.SparkSession sparkSession,
                   org.apache.spark.sql.execution.QueryExecution queryExecution,
                   MLContext ml)

Deprecated.

MLMatrix

protected MLMatrix(org.apache.spark.sql.SQLContext sqlContext,
                   org.apache.spark.sql.execution.QueryExecution queryExecution,
                   MLContext ml)

Deprecated.

MLMatrix

protected MLMatrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
                   MatrixCharacteristics mc,
                   MLContext ml)
            throws DMLRuntimeException

Deprecated.

Throws:: DMLRuntimeException

Method Detail

write
```
public void write(String filePath,
                  String format)
           throws IOException,
                  DMLException
```
Deprecated.

Convenient method to write a MLMatrix.

Parameters:

filePath - the file path

format - the format

Throws:

IOException - if IOException occurs

DMLException - if DMLException occurs

numRows
```
public long numRows()
             throws IOException,
                    DMLException
```
Deprecated.

Gets or computes the number of rows.

Returns:

the number of rows

Throws:

IOException - if IOException occurs

DMLException - if DMLException occurs

numCols
```
public long numCols()
             throws IOException,
                    DMLException
```
Deprecated.

Gets or computes the number of columns.

Returns:

the number of columns

Throws:

IOException - if IOException occurs

DMLException - if DMLException occurs

rowsPerBlock
```
public int rowsPerBlock()
```
Deprecated.

colsPerBlock
```
public int colsPerBlock()
```
Deprecated.

$greater

public MLMatrix $greater(MLMatrix that)
                  throws IOException,
                         DMLException

Deprecated.

Throws:: IOException; DMLException

$less

public MLMatrix $less(MLMatrix that)
               throws IOException,
                      DMLException

Deprecated.

Throws:: IOException; DMLException

$greater$eq

public MLMatrix $greater$eq(MLMatrix that)
                     throws IOException,
                            DMLException

Deprecated.

Throws:: IOException; DMLException

$less$eq

public MLMatrix $less$eq(MLMatrix that)
                  throws IOException,
                         DMLException

Deprecated.

Throws:: IOException; DMLException

$eq$eq

public MLMatrix $eq$eq(MLMatrix that)
                throws IOException,
                       DMLException

Deprecated.

Throws:: IOException; DMLException

$bang$eq

public MLMatrix $bang$eq(MLMatrix that)
                  throws IOException,
                         DMLException

Deprecated.

Throws:: IOException; DMLException

$up

public MLMatrix $up(MLMatrix that)
             throws IOException,
                    DMLException

Deprecated.

Throws:: IOException; DMLException

exp

public MLMatrix exp(MLMatrix that)
             throws IOException,
                    DMLException

Deprecated.

Throws:: IOException; DMLException

$plus

public MLMatrix $plus(MLMatrix that)
               throws IOException,
                      DMLException

Deprecated.

Throws:: IOException; DMLException

add

public MLMatrix add(MLMatrix that)
             throws IOException,
                    DMLException

Deprecated.

Throws:: IOException; DMLException

$minus

public MLMatrix $minus(MLMatrix that)
                throws IOException,
                       DMLException

Deprecated.

Throws:: IOException; DMLException

minus

public MLMatrix minus(MLMatrix that)
               throws IOException,
                      DMLException

Deprecated.

Throws:: IOException; DMLException

$times

public MLMatrix $times(MLMatrix that)
                throws IOException,
                       DMLException

Deprecated.

Throws:: IOException; DMLException

elementWiseMultiply

public MLMatrix elementWiseMultiply(MLMatrix that)
                             throws IOException,
                                    DMLException

Deprecated.

Throws:: IOException; DMLException

$div

public MLMatrix $div(MLMatrix that)
              throws IOException,
                     DMLException

Deprecated.

Throws:: IOException; DMLException

divide

public MLMatrix divide(MLMatrix that)
                throws IOException,
                       DMLException

Deprecated.

Throws:: IOException; DMLException

$percent$div$percent

public MLMatrix $percent$div$percent(MLMatrix that)
                              throws IOException,
                                     DMLException

Deprecated.

Throws:: IOException; DMLException

integerDivision

public MLMatrix integerDivision(MLMatrix that)
                         throws IOException,
                                DMLException

Deprecated.

Throws:: IOException; DMLException

$percent$percent

public MLMatrix $percent$percent(MLMatrix that)
                          throws IOException,
                                 DMLException

Deprecated.

Throws:: IOException; DMLException

modulus

public MLMatrix modulus(MLMatrix that)
                 throws IOException,
                        DMLException

Deprecated.

Throws:: IOException; DMLException

$percent$times$percent

public MLMatrix $percent$times$percent(MLMatrix that)
                                throws IOException,
                                       DMLException

Deprecated.

Throws:: IOException; DMLException

multiply

public MLMatrix multiply(MLMatrix that)
                  throws IOException,
                         DMLException

Deprecated.

Throws:: IOException; DMLException

transpose

public MLMatrix transpose()
                   throws IOException,
                          DMLException

Deprecated.

Throws:: IOException; DMLException

$plus

public MLMatrix $plus(Double scalar)
               throws IOException,
                      DMLException

Deprecated.

Throws:: IOException; DMLException

add

public MLMatrix add(Double scalar)
             throws IOException,
                    DMLException

Deprecated.

Throws:: IOException; DMLException

$minus

public MLMatrix $minus(Double scalar)
                throws IOException,
                       DMLException

Deprecated.

Throws:: IOException; DMLException

minus

public MLMatrix minus(Double scalar)
               throws IOException,
                      DMLException

Deprecated.

Throws:: IOException; DMLException

$times

public MLMatrix $times(Double scalar)
                throws IOException,
                       DMLException

Deprecated.

Throws:: IOException; DMLException

elementWiseMultiply

public MLMatrix elementWiseMultiply(Double scalar)
                             throws IOException,
                                    DMLException

Deprecated.

Throws:: IOException; DMLException

$div

public MLMatrix $div(Double scalar)
              throws IOException,
                     DMLException

Deprecated.

Throws:: IOException; DMLException

divide

public MLMatrix divide(Double scalar)
                throws IOException,
                       DMLException

Deprecated.

Throws:: IOException; DMLException

$greater

public MLMatrix $greater(Double scalar)
                  throws IOException,
                         DMLException

Deprecated.

Throws:: IOException; DMLException

$less

public MLMatrix $less(Double scalar)
               throws IOException,
                      DMLException

Deprecated.

Throws:: IOException; DMLException

$greater$eq

public MLMatrix $greater$eq(Double scalar)
                     throws IOException,
                            DMLException

Deprecated.

Throws:: IOException; DMLException

$less$eq

public MLMatrix $less$eq(Double scalar)
                  throws IOException,
                         DMLException

Deprecated.

Throws:: IOException; DMLException

$eq$eq

public MLMatrix $eq$eq(Double scalar)
                throws IOException,
                       DMLException

Deprecated.

Throws:: IOException; DMLException

$bang$eq

public MLMatrix $bang$eq(Double scalar)
                  throws IOException,
                         DMLException

Deprecated.

Throws:: IOException; DMLException

Fields inherited from class org.apache.spark.sql.Dataset

Methods inherited from class org.apache.spark.sql.Dataset

Methods inherited from class java.lang.Object

mc

ml

MLMatrix

MLMatrix

MLMatrix

MLMatrix

MLMatrix

write

numRows

numCols

rowsPerBlock

colsPerBlock

$greater

$less

$greater$eq

$less$eq

$eq$eq

$bang$eq

$up

exp

$plus

add

$minus

minus

$times

elementWiseMultiply

$div

divide

$percent$div$percent

integerDivision

$percent$percent

modulus

$percent$times$percent

multiply

transpose

$plus

add

$minus

minus

$times

elementWiseMultiply

$div

divide

$greater

$less

$greater$eq

$less$eq

$eq$eq

$bang$eq