public class ScriptExecutor extends Object
execute(org.apache.sysml.api.mlcontext.Script)
method.
Script execution via the MLContext API typically consists of the following steps:
Modifications to these steps can be accomplished by subclassing ScriptExecutor. For example, the following code will turn off the global data flow optimization check by subclassing ScriptExecutor and overriding the globalDataFlowOptimization method.
ScriptExecutor scriptExecutor = new ScriptExecutor() {
// turn off global data flow optimization check
@Override
protected void globalDataFlowOptimization() {
return;
}
};
ml.execute(script, scriptExecutor);
For more information, please see the execute(org.apache.sysml.api.mlcontext.Script)
method.
Modifier and Type | Field and Description |
---|---|
protected org.apache.sysml.conf.DMLConfig |
config |
protected org.apache.sysml.parser.DMLProgram |
dmlProgram |
protected org.apache.sysml.parser.DMLTranslator |
dmlTranslator |
protected org.apache.sysml.runtime.controlprogram.context.ExecutionContext |
executionContext |
protected MLContext.ExecutionType |
executionType |
protected boolean |
explain |
protected MLContext.ExplainLevel |
explainLevel |
protected boolean |
forceGPU |
protected boolean |
gpu |
protected boolean |
init |
protected boolean |
maintainSymbolTable |
protected boolean |
oldForceGPU |
protected boolean |
oldGPU |
protected boolean |
oldStatistics |
protected org.apache.sysml.runtime.controlprogram.Program |
runtimeProgram |
protected Script |
script |
protected boolean |
statistics |
protected int |
statisticsMaxHeavyHitters |
Constructor and Description |
---|
ScriptExecutor()
ScriptExecutor constructor.
|
ScriptExecutor(org.apache.sysml.conf.DMLConfig config)
ScriptExecutor constructor, where the configuration properties are passed
in.
|
Modifier and Type | Method and Description |
---|---|
protected void |
checkScriptHasTypeAndString()
Check that the Script object has a type (DML or PYDML) and a string
representing the content of the Script.
|
protected void |
cleanupAfterExecution()
Perform any necessary cleanup operations after program execution.
|
protected void |
cleanupRuntimeProgram()
If
maintainSymbolTable is true, delete all 'remove variable'
instructions so as to maintain the values in the symbol table, which are
useful when working interactively in an environment such as the Spark
Shell. |
void |
compile(Script script) |
void |
compile(Script script,
boolean performHOPRewrites)
Compile a DML or PYDML script.
|
protected void |
constructHops()
Construct DAGs of high-level operators (HOPs) for each block of
statements.
|
protected void |
constructLops()
Construct DAGs of low-level operators (LOPs) based on the DAGs of
high-level operators (HOPs).
|
protected void |
countCompiledMRJobsAndSparkInstructions()
Count the number of compiled MR Jobs/Spark Instructions in the runtime
program and set this value in the statistics.
|
protected void |
createAndInitializeExecutionContext()
Create an execution context and set its variables to be the symbol table
of the script.
|
MLResults |
execute(Script script)
Execute a DML or PYDML script.
|
protected void |
executeRuntimeProgram()
Execute the runtime program.
|
protected void |
generateRuntimeProgram()
Create runtime program.
|
org.apache.sysml.conf.DMLConfig |
getConfig()
Obtain the SystemML configuration properties.
|
org.apache.sysml.parser.DMLProgram |
getDmlProgram()
Obtain the program
|
org.apache.sysml.parser.DMLTranslator |
getDmlTranslator()
Obtain the translator
|
org.apache.sysml.runtime.controlprogram.context.ExecutionContext |
getExecutionContext()
Obtain the execution context
|
MLContext.ExecutionType |
getExecutionType()
Obtain the current execution environment.
|
org.apache.sysml.runtime.controlprogram.Program |
getRuntimeProgram()
Obtain the runtime program
|
Script |
getScript()
Obtain the Script object associated with this ScriptExecutor
|
protected void |
globalDataFlowOptimization()
Optimize the program.
|
protected void |
initializeCachingAndScratchSpace()
Check security, create scratch space, cleanup working directories,
initialize caching, and reset statistics.
|
boolean |
isMaintainSymbolTable()
Obtain whether or not all values should be maintained in the symbol table
after execution.
|
protected void |
liveVariableAnalysis()
Liveness analysis is performed on the program, obtaining sets of live-in
and live-out variables by forward and backward passes over the program.
|
protected void |
parseScript()
Parse the script into an ANTLR parse tree, and convert this parse tree
into a SystemML program.
|
protected void |
resetGlobalFlags()
Reset the global flags (for example: statistics, gpu, etc)
post-execution.
|
protected void |
restoreInputsInSymbolTable()
Restore the input variables in the symbol table after script execution.
|
protected void |
rewriteHops()
Apply static rewrites, perform intra-/inter-procedural analysis to
propagate size information into functions, apply dynamic rewrites, and
compute memory estimates for all HOPs.
|
protected void |
rewritePersistentReadsAndWrites()
Replace persistent reads and writes with transient reads and writes in
the symbol table.
|
void |
setConfig(org.apache.sysml.conf.DMLConfig config)
Set the SystemML configuration properties.
|
void |
setExecutionType(MLContext.ExecutionType executionType)
Set the execution environment.
|
void |
setExplain(boolean explain)
Whether or not an explanation of the DML/PYDML program should be output
to standard output.
|
void |
setExplainLevel(MLContext.ExplainLevel explainLevel)
Set the level of program explanation that should be displayed if explain
is set to true.
|
void |
setForceGPU(boolean enabled)
Whether or not to force GPU usage.
|
protected void |
setGlobalFlags()
Set the global flags (for example: statistics, gpu, etc).
|
void |
setGPU(boolean enabled)
Whether or not to enable GPU usage.
|
void |
setInit(boolean init)
Whether or not to initialize the scratch_space, bufferpool, etc.
|
void |
setMaintainSymbolTable(boolean maintainSymbolTable)
Set whether or not all values should be maintained in the symbol table
after execution.
|
void |
setStatistics(boolean statistics)
Whether or not statistics about the DML/PYDML program should be output to
standard output.
|
void |
setStatisticsMaxHeavyHitters(int maxHeavyHitters)
Set the maximum number of heavy hitters to display with statistics.
|
protected void |
setup(Script script)
Sets the script in the ScriptExecutor, checks that the script has a type
and string, sets the ScriptExecutor in the script, sets the script string
in the Spark Monitor, globally sets the script type, sets global flags,
and resets statistics if needed.
|
protected void |
showExplanation()
Output a description of the program to standard output.
|
protected void |
validateScript()
Semantically validate the program's expressions, statements, and
statement blocks in a single recursive pass over the program.
|
protected org.apache.sysml.conf.DMLConfig config
protected org.apache.sysml.parser.DMLProgram dmlProgram
protected org.apache.sysml.parser.DMLTranslator dmlTranslator
protected org.apache.sysml.runtime.controlprogram.Program runtimeProgram
protected org.apache.sysml.runtime.controlprogram.context.ExecutionContext executionContext
protected Script script
protected boolean init
protected boolean explain
protected boolean gpu
protected boolean oldGPU
protected boolean forceGPU
protected boolean oldForceGPU
protected boolean statistics
protected boolean oldStatistics
protected MLContext.ExplainLevel explainLevel
protected MLContext.ExecutionType executionType
protected int statisticsMaxHeavyHitters
protected boolean maintainSymbolTable
public ScriptExecutor()
public ScriptExecutor(org.apache.sysml.conf.DMLConfig config)
config
- the configuration properties to use by the ScriptExecutorprotected void constructHops()
protected void rewriteHops()
protected void showExplanation()
protected void constructLops()
protected void generateRuntimeProgram()
protected void countCompiledMRJobsAndSparkInstructions()
protected void createAndInitializeExecutionContext()
protected void setGlobalFlags()
protected void resetGlobalFlags()
public void compile(Script script)
public void compile(Script script, boolean performHOPRewrites)
setup(Script)
parseScript()
liveVariableAnalysis()
validateScript()
constructHops()
rewriteHops()
rewritePersistentReadsAndWrites()
constructLops()
generateRuntimeProgram()
showExplanation()
globalDataFlowOptimization()
countCompiledMRJobsAndSparkInstructions()
initializeCachingAndScratchSpace()
cleanupRuntimeProgram()
script
- the DML or PYDML script to compileperformHOPRewrites
- should perform static rewrites, perform intra-/inter-procedural analysis to propagate size information into functions and apply dynamic rewritespublic MLResults execute(Script script)
script
- the DML or PYDML script to executeprotected void setup(Script script)
script
- the DML or PYDML script to executeprotected void cleanupAfterExecution()
protected void restoreInputsInSymbolTable()
protected void cleanupRuntimeProgram()
maintainSymbolTable
is true, delete all 'remove variable'
instructions so as to maintain the values in the symbol table, which are
useful when working interactively in an environment such as the Spark
Shell. Otherwise, only delete 'remove variable' instructions for
registered outputs.protected void executeRuntimeProgram()
protected void initializeCachingAndScratchSpace()
protected void globalDataFlowOptimization()
protected void parseScript()
protected void rewritePersistentReadsAndWrites()
public void setConfig(org.apache.sysml.conf.DMLConfig config)
config
- The configuration propertiesprotected void liveVariableAnalysis()
protected void validateScript()
protected void checkScriptHasTypeAndString()
public org.apache.sysml.parser.DMLProgram getDmlProgram()
public org.apache.sysml.parser.DMLTranslator getDmlTranslator()
public org.apache.sysml.runtime.controlprogram.Program getRuntimeProgram()
public org.apache.sysml.runtime.controlprogram.context.ExecutionContext getExecutionContext()
public Script getScript()
public void setExplain(boolean explain)
explain
- true
if explanation should be output, false
otherwisepublic void setStatistics(boolean statistics)
statistics
- true
if statistics should be output, false
otherwisepublic void setStatisticsMaxHeavyHitters(int maxHeavyHitters)
maxHeavyHitters
- the maximum number of heavy hitterspublic boolean isMaintainSymbolTable()
true
if all values should be maintained in the symbol
table, false
otherwisepublic void setMaintainSymbolTable(boolean maintainSymbolTable)
maintainSymbolTable
- true
if all values should be maintained in the symbol
table, false
otherwisepublic void setInit(boolean init)
init
- true
if should initialize, false
otherwisepublic void setExplainLevel(MLContext.ExplainLevel explainLevel)
explainLevel
- the level of program explanationpublic void setGPU(boolean enabled)
enabled
- true
if enabled, false
otherwisepublic void setForceGPU(boolean enabled)
enabled
- true
if enabled, false
otherwisepublic org.apache.sysml.conf.DMLConfig getConfig()
public MLContext.ExecutionType getExecutionType()
public void setExecutionType(MLContext.ExecutionType executionType)
executionType
- the execution environmentCopyright © 2017 The Apache Software Foundation. All rights reserved.