org.apache.sysml.api

Class MLContext

  • Deprecated. 
    This will be removed in SystemML 1.0. Please migrate to MLContext

    @Deprecated
public class MLContext
extends Object
    MLContext is useful for passing RDDs as input/output to SystemML. This API avoids the need to read/write from HDFS (which is another way to pass inputs to SystemML).

    Typical usage for MLContext is as follows: 

    
 scala> import org.apache.sysml.api.MLContext

    Create input DataFrame from CSV file and potentially perform some feature transformation 

    
 scala> val W = sparkSession.load("com.databricks.spark.csv", Map("path" -> "W.csv", "header" -> "false"))
 scala> val H = sparkSession.load("com.databricks.spark.csv", Map("path" -> "H.csv", "header" -> "false"))
 scala> val V = sparkSession.load("com.databricks.spark.csv", Map("path" -> "V.csv", "header" -> "false"))

    Create MLContext 

    
 scala> val ml = new MLContext(sc)

    Register input and output DataFrame/RDD Supported format:

    1. DataFrame
    2. CSV/Text (as JavaRDD<String> or JavaPairRDD<LongWritable, Text>)
    3. Binary blocked RDD (JavaPairRDD<MatrixIndexes,MatrixBlock>))
    Also overloaded to support metadata information such as format, rlen, clen, ... Please note the variable names given below in quotes correspond to the variables in DML script. These variables need to have corresponding read/write associated in DML script. Currently, only matrix variables are supported through registerInput/registerOutput interface. To pass scalar variables, use named/positional arguments (described later) or wrap them into matrix variable. 
    
 scala> ml.registerInput("V", V)
 scala> ml.registerInput("W", W)
 scala> ml.registerInput("H", H)
 scala> ml.registerOutput("H")
 scala> ml.registerOutput("W")

    Call script with default arguments: 

    
 scala> val outputs = ml.execute("GNMF.dml")

    Also supported: calling script with positional arguments (args) and named arguments (nargs): 

     
 scala> val args = Array("V.mtx", "W.mtx",  "H.mtx",  "2000", "1500",  "50",  "1",  "WOut.mtx",  "HOut.mtx")
 scala> val nargs = Map("maxIter"->"1", "V" -> "")
 scala> val outputs = ml.execute("GNMF.dml", args) # or ml.execute("GNMF_namedArgs.dml", nargs)

    To run the script again using different (or even same arguments), but using same registered input/outputs: 

     
 scala> val new_outputs = ml.execute("GNMF.dml", new_args)

    However, to register new input/outputs, you need to first reset MLContext 

     
 scala> ml.reset()
 scala> ml.registerInput("V", newV)

    Experimental API: To monitor performance (only supported for Spark 1.4.0 or higher), 

    
 scala> val ml = new MLContext(sc, true)

    If monitoring performance is enabled, 

     
 scala> print(ml.getMonitoringUtil().getExplainOutput())
 scala> ml.getMonitoringUtil().getRuntimeInfoInHTML("runtime.html")

    Note: The execute(...) methods does not support parallel calls from same or different MLContext. This is because current SystemML engine does not allow multiple invocation in same JVM. So, if you plan to create a system which potentially creates multiple MLContext, it is recommended to guard the execute(...) call using 

      
 synchronized(MLContext.class) { ml.execute(...); }

    • Constructor Detail

      • MLContext

        public MLContext(org.apache.spark.SparkContext sc)
          throws DMLRuntimeException
        Deprecated. 
        Create an associated MLContext for given spark session.
        Parameters:
        sc - SparkContext
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • MLContext

        public MLContext(org.apache.spark.api.java.JavaSparkContext sc)
          throws DMLRuntimeException
        Deprecated. 
        Create an associated MLContext for given spark session.
        Parameters:
        sc - JavaSparkContext
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

    • Method Detail

      • getSparkContext

        public org.apache.spark.SparkContext getSparkContext()
        Deprecated. 

      • setConfig

        public void setConfig(String paramName,
                      String paramVal)
        Deprecated. 
        Allow users to provide custom named-value configuration.
        Parameters:
        paramName - parameter name
        paramVal - parameter value

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)
                   throws DMLRuntimeException
        Deprecated. 
        Register DataFrame as input. DataFrame is assumed to be in row format and each cell can be converted into double through Double.parseDouble(cell.toString()). This is suitable for passing dense matrices. For sparse matrices, consider passing through text format (using JavaRDD<String>, format="text")

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        df - the DataFrame
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerFrameInput

        public void registerFrameInput(String varName,
                               org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)
                        throws DMLRuntimeException
        Deprecated. 
        Register DataFrame as input. DataFrame is assumed to be in row format and each cell can be converted into SystemML frame row. Each column could be of type, Double, Float, Long, Integer, String or Boolean.

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        df - the DataFrame
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
                          boolean containsID)
                   throws DMLRuntimeException
        Deprecated. 
        Register DataFrame as input. Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.
        Parameters:
        varName - variable name
        df - the DataFrame
        containsID - false if the DataFrame has an column ID which denotes the row ID.
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerFrameInput

        public void registerFrameInput(String varName,
                               org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
                               boolean containsID)
                        throws DMLRuntimeException
        Deprecated. 
        Register DataFrame as input. DataFrame is assumed to be in row format and each cell can be converted into SystemML frame row. Each column could be of type, Double, Float, Long, Integer, String or Boolean.

        Parameters:
        varName - variable name
        df - the DataFrame
        containsID - false if the DataFrame has an column ID which denotes the row ID.
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          MLMatrix df)
                   throws DMLRuntimeException
        Deprecated. 
        Experimental API. Not supported in Python MLContext API.
        Parameters:
        varName - variable name
        df - the DataFrame
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.api.java.JavaRDD<String> rdd,
                          String format,
                          boolean hasHeader,
                          String delim,
                          boolean fill,
                          double fillValue)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: Method for supplying csv file format properties, but without dimensions or nnz

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the RDD
        format - the format
        hasHeader - is there a header
        delim - the delimiter
        fill - if true, fill, otherwise don't fill
        fillValue - the fill value
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.rdd.RDD<String> rdd,
                          String format,
                          boolean hasHeader,
                          String delim,
                          boolean fill,
                          double fillValue)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: Method for supplying csv file format properties, but without dimensions or nnz

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the RDD
        format - the format
        hasHeader - is there a header
        delim - the delimiter
        fill - if true, fill, otherwise don't fill
        fillValue - the fill value
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.rdd.RDD<String> rdd,
                          String format,
                          boolean hasHeader,
                          String delim,
                          boolean fill,
                          double fillValue,
                          long rlen,
                          long clen,
                          long nnz)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: Method for supplying csv file format properties along with dimensions or nnz

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the RDD
        format - the format
        hasHeader - is there a header
        delim - the delimiter
        fill - if true, fill, otherwise don't fill
        fillValue - the fill value
        rlen - rows
        clen - columns
        nnz - non-zeros
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.api.java.JavaRDD<String> rdd,
                          String format,
                          boolean hasHeader,
                          String delim,
                          boolean fill,
                          double fillValue,
                          long rlen,
                          long clen,
                          long nnz)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: Method for supplying csv file format properties along with dimensions or nnz

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the JavaRDD
        format - the format
        hasHeader - is there a header
        delim - the delimiter
        fill - if true, fill, otherwise don't fill
        fillValue - the fill value
        rlen - rows
        clen - columns
        nnz - non-zeros
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.rdd.RDD<String> rdd,
                          String format)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: Convenience method without dimensions and nnz. It uses default file properties (example: delim, fill, ..)

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the RDD
        format - the format
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.api.java.JavaRDD<String> rdd,
                          String format)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: Convenience method without dimensions and nnz. It uses default file properties (example: delim, fill, ..)

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the JavaRDD
        format - the format
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.api.java.JavaRDD<String> rdd,
                          String format,
                          long rlen,
                          long clen)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: Convenience method with dimensions and but no nnz. It uses default file properties (example: delim, fill, ..)

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the JavaRDD
        format - the format
        rlen - rows
        clen - columns
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.rdd.RDD<String> rdd,
                          String format,
                          long rlen,
                          long clen)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: Convenience method with dimensions and but no nnz. It uses default file properties (example: delim, fill, ..)

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the RDD
        format - the format
        rlen - rows
        clen - columns
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.api.java.JavaRDD<String> rdd,
                          String format,
                          long rlen,
                          long clen,
                          long nnz)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: with dimensions and nnz. It uses default file properties (example: delim, fill, ..)

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the JavaRDD
        format - the format
        rlen - rows
        clen - columns
        nnz - non-zeros
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.rdd.RDD<String> rdd,
                          String format,
                          long rlen,
                          long clen,
                          long nnz)
                   throws DMLRuntimeException
        Deprecated. 
        Register CSV/Text as inputs: with dimensions and nnz. It uses default file properties (example: delim, fill, ..)

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the JavaRDD
        format - the format
        rlen - rows
        clen - columns
        nnz - non-zeros
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.api.java.JavaRDD<String> rddIn,
                          String format,
                          long rlen,
                          long clen,
                          FileFormatProperties props,
                          List<org.apache.sysml.parser.Expression.ValueType> schema)
                   throws DMLRuntimeException
        Deprecated. 
        Register Frame with CSV/Text as inputs: with dimensions. File properties (example: delim, fill, ..) can be specified through props else defaults will be used.

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rddIn - the JavaPairRDD
        format - the format
        rlen - rows
        clen - columns
        props - properties
        schema - List of column types
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> rdd,
                          long rlen,
                          long clen)
                   throws DMLRuntimeException
        Deprecated. 
        Register binary blocked RDD with given dimensions, default block sizes and no nnz

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the JavaPairRDD
        rlen - rows
        clen - columns
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> rdd,
                          long rlen,
                          long clen,
                          int brlen,
                          int bclen)
                   throws DMLRuntimeException
        Deprecated. 
        Register binary blocked RDD with given dimensions, given block sizes and no nnz

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the JavaPairRDD
        rlen - rows
        clen - columns
        brlen - block rows
        bclen - block columns
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerInput

        public void registerInput(String varName,
                          org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> rdd,
                          long rlen,
                          long clen,
                          int brlen,
                          int bclen,
                          long nnz)
                   throws DMLRuntimeException
        Deprecated. 
        Register binary blocked RDD with given dimensions, given block sizes and given nnz (preferred).

        Marks the variable in the DML script as input variable. Note that this expects a "varName = read(...)" statement in the DML script which through non-MLContext invocation would have been created by reading a HDFS file.

        Parameters:
        varName - variable name
        rdd - the JavaPairRDD
        rlen - rows
        clen - columns
        brlen - block rows
        bclen - block columns
        nnz - non-zeros
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • registerOutput

        public void registerOutput(String varName)
                    throws DMLRuntimeException
        Deprecated. 
        Marks the variable in the DML script as output variable. Note that this expects a "write(varName, ...)" statement in the DML script which through non-MLContext invocation would have written the matrix to HDFS.
        Parameters:
        varName - variable name
        Throws:
        DMLRuntimeException - if DMLRuntimeException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        Map<String,String> namedArgs,
                        boolean parsePyDML,
                        String configFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing named arguments using specified config file.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        namedArgs - named arguments
        parsePyDML - true if pydml, false otherwise
        configFilePath - path to config file
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        Map<String,String> namedArgs,
                        String configFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing named arguments using specified config file.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        namedArgs - named arguments
        configFilePath - path to config file
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        Map<String,String> namedArgs)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing named arguments with default configuration.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        namedArgs - named arguments
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        scala.collection.immutable.Map<String,String> namedArgs)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing named arguments.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        namedArgs - named arguments
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        Map<String,String> namedArgs,
                        boolean parsePyDML)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental: Execute PyDML script by passing named arguments if parsePyDML=true.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        namedArgs - named arguments
        parsePyDML - true if pydml, false otherwise
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        scala.collection.immutable.Map<String,String> namedArgs,
                        boolean parsePyDML)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental: Execute PyDML script by passing named arguments if parsePyDML=true.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        namedArgs - named arguments
        parsePyDML - true if pydml, false otherwise
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        String[] args,
                        String configFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing positional arguments using specified config file
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        args - arguments
        configFilePath - path to config file
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        ArrayList<String> args,
                        String configFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing positional arguments using specified config file This method is implemented for compatibility with Python MLContext. Java/Scala users should use 'MLOutput execute(String dmlScriptFilePath, String [] args, String configFilePath)' instead as equivalent scala collections (Seq/ArrayBuffer) is not implemented.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        args - arguments
        configFilePath - path to config file
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        String[] args)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing positional arguments using default configuration
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        args - arguments
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        ArrayList<String> args)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing positional arguments using default configuration. This method is implemented for compatibility with Python MLContext. Java/Scala users should use 'MLOutput execute(String dmlScriptFilePath, String [] args)' instead as equivalent scala collections (Seq/ArrayBuffer) is not implemented.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        args - arguments
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        ArrayList<String> args,
                        boolean parsePyDML)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental: Execute DML script by passing positional arguments if parsePyDML=true, using default configuration. This method is implemented for compatibility with Python MLContext. Java/Scala users should use 'MLOutput execute(String dmlScriptFilePath, String [] args, boolean parsePyDML)' instead as equivalent scala collections (Seq/ArrayBuffer) is not implemented.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        args - arguments
        parsePyDML - true if pydml, false otherwise
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        ArrayList<String> args,
                        boolean parsePyDML,
                        String configFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental: Execute DML script by passing positional arguments if parsePyDML=true, using specified config file. This method is implemented for compatibility with Python MLContext. Java/Scala users should use 'MLOutput execute(String dmlScriptFilePath, String [] args, boolean parsePyDML, String configFilePath)' instead as equivalent scala collections (Seq/ArrayBuffer) is not implemented.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        args - arguments
        parsePyDML - true if pydml, false otherwise
        configFilePath - path to config file
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        ArrayList<String> argsName,
                        ArrayList<String> argsValues,
                        String configFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing positional arguments using specified config file
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        argsName - argument names
        argsValues - argument values
        configFilePath - path to config file
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        ArrayList<String> argsName,
                        ArrayList<String> argsValues)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script by passing positional arguments using specified config file
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        argsName - argument names
        argsValues - argument values
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        String[] args,
                        boolean parsePyDML,
                        String configFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental: Execute DML script by passing positional arguments if parsePyDML=true, using specified config file.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        args - arguments
        parsePyDML - true if pydml, false otherwise
        configFilePath - path to config file
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        String[] args,
                        boolean parsePyDML)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental: Execute DML script by passing positional arguments if parsePyDML=true, using default configuration.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        args - arguments
        parsePyDML - true if pydml, false otherwise
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        String configFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script without any arguments using specified config path
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        configFilePath - path to config file
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Execute DML script without any arguments using default configuration.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        boolean parsePyDML,
                        String configFilePath)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental: Execute DML script without any arguments if parsePyDML=true, using specified config path.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        parsePyDML - true if pydml, false otherwise
        configFilePath - path to config file
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • execute

        public MLOutput execute(String dmlScriptFilePath,
                        boolean parsePyDML)
                 throws IOException,
                        DMLException,
                        org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental: Execute DML script without any arguments if parsePyDML=true, using default configuration.
        Parameters:
        dmlScriptFilePath - the dml script can be in local filesystem or in HDFS
        parsePyDML - true if pydml, false otherwise
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • reset

        public void reset()
           throws DMLRuntimeException
        Deprecated. 
        Call this method if you want to clear any RDDs set via registerInput, registerOutput. This is required if ml.execute(..) has been called earlier and you want to call a new DML script. Note: By default this doesnot clean up configuration set using setConfig method. To clean the configuration as along with registered input/outputs, please use reset(true);
        Throws:
        DMLRuntimeException - if DMLException occurs

      • executeScript

        public MLOutput executeScript(String dmlScript)
                       throws IOException,
                              DMLException
        Deprecated. 
        Execute a script stored in a string.
        Parameters:
        dmlScript - the script
        Returns:
        output as MLOutput
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • read

        public MLMatrix read(org.apache.spark.sql.SparkSession sparkSession,
                     String filePath,
                     String format)
              throws IOException,
                     DMLException,
                     org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental API: Might be discontinued in future release
        Parameters:
        sparkSession - the Spark Session
        filePath - the file path
        format - the format
        Returns:
        the MLMatrix
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs

      • read

        public MLMatrix read(org.apache.spark.sql.SQLContext sqlContext,
                     String filePath,
                     String format)
              throws IOException,
                     DMLException,
                     org.apache.sysml.parser.ParseException
        Deprecated. 
        Experimental API: Might be discontinued in future release
        Parameters:
        sqlContext - the SQL Context
        filePath - the file path
        format - the format
        Returns:
        the MLMatrix
        Throws:
        IOException - if IOException occurs
        DMLException - if DMLException occurs
        org.apache.sysml.parser.ParseException - if ParseException occurs
Skip navigation links

Copyright © 2017 The Apache Software Foundation. All rights reserved.