public class ScriptExecutor extends Object
execute(org.apache.sysml.api.mlcontext.Script)
method.
Script execution via the MLContext API typically consists of the following steps:
Modifications to these steps can be accomplished by subclassing ScriptExecutor. For example, the following code will turn off the global data flow optimization check by subclassing ScriptExecutor and overriding the globalDataFlowOptimization method.
ScriptExecutor scriptExecutor = new ScriptExecutor() {
// turn off global data flow optimization check
@Override
protected void globalDataFlowOptimization() {
return;
}
};
ml.execute(script, scriptExecutor);
For more information, please see the execute(org.apache.sysml.api.mlcontext.Script)
method.
Modifier and Type | Field and Description |
---|---|
protected org.apache.sysml.conf.DMLConfig |
config |
protected org.apache.sysml.parser.DMLProgram |
dmlProgram |
protected org.apache.sysml.parser.DMLTranslator |
dmlTranslator |
protected org.apache.sysml.runtime.controlprogram.context.ExecutionContext |
executionContext |
protected boolean |
explain |
protected MLContext.ExplainLevel |
explainLevel |
protected boolean |
init |
protected boolean |
maintainSymbolTable |
protected org.apache.sysml.runtime.controlprogram.Program |
runtimeProgram |
protected Script |
script |
protected boolean |
statistics |
protected int |
statisticsMaxHeavyHitters |
Constructor and Description |
---|
ScriptExecutor()
ScriptExecutor constructor.
|
ScriptExecutor(org.apache.sysml.conf.DMLConfig config)
ScriptExecutor constructor, where the configuration properties are passed
in.
|
Modifier and Type | Method and Description |
---|---|
protected void |
checkScriptHasTypeAndString()
Check that the Script object has a type (DML or PYDML) and a string
representing the content of the Script.
|
protected void |
cleanupAfterExecution()
Perform any necessary cleanup operations after program execution.
|
protected void |
cleanupRuntimeProgram()
If
maintainSymbolTable is true, delete all 'remove variable'
instructions so as to maintain the values in the symbol table, which are
useful when working interactively in an environment such as the Spark
Shell. |
protected void |
constructHops()
Construct DAGs of high-level operators (HOPs) for each block of
statements.
|
protected void |
constructLops()
Construct DAGs of low-level operators (LOPs) based on the DAGs of
high-level operators (HOPs).
|
protected void |
countCompiledMRJobsAndSparkInstructions()
Count the number of compiled MR Jobs/Spark Instructions in the runtime
program and set this value in the statistics.
|
protected void |
createAndInitializeExecutionContext()
Create an execution context and set its variables to be the symbol table
of the script.
|
MLResults |
execute(Script script)
Execute a DML or PYDML script.
|
protected void |
executeRuntimeProgram()
Execute the runtime program.
|
protected void |
generateRuntimeProgram()
Create runtime program.
|
org.apache.sysml.parser.DMLProgram |
getDmlProgram()
Obtain the program
|
org.apache.sysml.parser.DMLTranslator |
getDmlTranslator()
Obtain the translator
|
org.apache.sysml.runtime.controlprogram.context.ExecutionContext |
getExecutionContext()
Obtain the execution context
|
org.apache.sysml.runtime.controlprogram.Program |
getRuntimeProgram()
Obtain the runtime program
|
Script |
getScript()
Obtain the Script object associated with this ScriptExecutor
|
protected void |
globalDataFlowOptimization()
Optimize the program.
|
protected void |
initializeCachingAndScratchSpace()
Check security, create scratch space, cleanup working directories,
initialize caching, and reset statistics.
|
boolean |
isMaintainSymbolTable()
Obtain whether or not all values should be maintained in the symbol table
after execution.
|
protected void |
liveVariableAnalysis()
Liveness analysis is performed on the program, obtaining sets of live-in
and live-out variables by forward and backward passes over the program.
|
protected void |
parseScript()
Parse the script into an ANTLR parse tree, and convert this parse tree
into a SystemML program.
|
protected void |
restoreInputsInSymbolTable()
Restore the input variables in the symbol table after script execution.
|
protected void |
rewriteHops()
Apply static rewrites, perform intra-/inter-procedural analysis to
propagate size information into functions, apply dynamic rewrites, and
compute memory estimates for all HOPs.
|
protected void |
rewritePersistentReadsAndWrites()
Replace persistent reads and writes with transient reads and writes in
the symbol table.
|
void |
setConfig(org.apache.sysml.conf.DMLConfig config)
Set the SystemML configuration properties.
|
void |
setExplain(boolean explain)
Whether or not an explanation of the DML/PYDML program should be output
to standard output.
|
void |
setExplainLevel(MLContext.ExplainLevel explainLevel)
Set the level of program explanation that should be displayed if explain
is set to true.
|
void |
setInit(boolean init)
Whether or not to initialize the scratch_space, bufferpool, etc.
|
void |
setMaintainSymbolTable(boolean maintainSymbolTable)
Set whether or not all values should be maintained in the symbol table
after execution.
|
void |
setStatistics(boolean statistics)
Whether or not statistics about the DML/PYDML program should be output to
standard output.
|
void |
setStatisticsMaxHeavyHitters(int maxHeavyHitters) |
protected void |
setup(Script script)
Sets the script in the ScriptExecutor, checks that the script has a type
and string, sets the ScriptExecutor in the script, sets the script string
in the Spark Monitor, and globally sets the script type.
|
protected void |
showExplanation()
Output a description of the program to standard output.
|
protected void |
validateScript()
Semantically validate the program's expressions, statements, and
statement blocks in a single recursive pass over the program.
|
protected org.apache.sysml.conf.DMLConfig config
protected org.apache.sysml.parser.DMLProgram dmlProgram
protected org.apache.sysml.parser.DMLTranslator dmlTranslator
protected org.apache.sysml.runtime.controlprogram.Program runtimeProgram
protected org.apache.sysml.runtime.controlprogram.context.ExecutionContext executionContext
protected Script script
protected boolean init
protected boolean explain
protected boolean statistics
protected MLContext.ExplainLevel explainLevel
protected int statisticsMaxHeavyHitters
protected boolean maintainSymbolTable
public ScriptExecutor()
public ScriptExecutor(org.apache.sysml.conf.DMLConfig config)
config
- the configuration properties to use by the ScriptExecutorprotected void constructHops()
protected void rewriteHops()
protected void showExplanation()
protected void constructLops()
protected void generateRuntimeProgram()
protected void countCompiledMRJobsAndSparkInstructions()
protected void createAndInitializeExecutionContext()
public MLResults execute(Script script)
setup(Script)
parseScript()
liveVariableAnalysis()
validateScript()
constructHops()
rewriteHops()
rewritePersistentReadsAndWrites()
constructLops()
generateRuntimeProgram()
showExplanation()
globalDataFlowOptimization()
countCompiledMRJobsAndSparkInstructions()
initializeCachingAndScratchSpace()
cleanupRuntimeProgram()
createAndInitializeExecutionContext()
executeRuntimeProgram()
cleanupAfterExecution()
script
- the DML or PYDML script to executeprotected void setup(Script script)
script
- the DML or PYDML script to executeprotected void cleanupAfterExecution()
protected void restoreInputsInSymbolTable()
protected void cleanupRuntimeProgram()
maintainSymbolTable
is true, delete all 'remove variable'
instructions so as to maintain the values in the symbol table, which are
useful when working interactively in an environment such as the Spark
Shell. Otherwise, only delete 'remove variable' instructions for
registered outputs.protected void executeRuntimeProgram()
protected void initializeCachingAndScratchSpace()
protected void globalDataFlowOptimization()
protected void parseScript()
protected void rewritePersistentReadsAndWrites()
public void setConfig(org.apache.sysml.conf.DMLConfig config)
config
- The configuration propertiesprotected void liveVariableAnalysis()
protected void validateScript()
protected void checkScriptHasTypeAndString()
public org.apache.sysml.parser.DMLProgram getDmlProgram()
public org.apache.sysml.parser.DMLTranslator getDmlTranslator()
public org.apache.sysml.runtime.controlprogram.Program getRuntimeProgram()
public org.apache.sysml.runtime.controlprogram.context.ExecutionContext getExecutionContext()
public Script getScript()
public void setExplain(boolean explain)
explain
- true
if explanation should be output, false
otherwisepublic void setStatistics(boolean statistics)
statistics
- true
if statistics should be output, false
otherwisepublic void setStatisticsMaxHeavyHitters(int maxHeavyHitters)
public boolean isMaintainSymbolTable()
true
if all values should be maintained in the symbol
table, false
otherwisepublic void setMaintainSymbolTable(boolean maintainSymbolTable)
maintainSymbolTable
- true
if all values should be maintained in the symbol
table, false
otherwisepublic void setInit(boolean init)
init
- true
if should initialize, false
otherwisepublic void setExplainLevel(MLContext.ExplainLevel explainLevel)
explainLevel
- the level of program explanationCopyright © 2017 The Apache Software Foundation. All rights reserved.