org.apache.sysml.runtime.instructions.spark

Class MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuildFunction

java.lang.Object

org.apache.sysml.runtime.instructions.spark.MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuildFunction

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<scala.Tuple2<Long,FrameBlock>>,Integer,Object>

Enclosing class:

MultiReturnParameterizedBuiltinSPInstruction

public static class MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuildFunction
extends Object
implements org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<scala.Tuple2<Long,FrameBlock>>,Integer,Object>

This function pre-aggregates distinct values of recoded columns per partition (part of distributed recode map construction, used for recoding, binning and dummy coding). We operate directly over schema-specific objects to avoid unnecessary string conversion, as well as reduce memory overhead and shuffle.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
TransformEncodeBuildFunction(org.apache.sysml.runtime.transform.encode.Encoder encoder)

Method Summary

Modifier and Type	Method and Description
Iterator<scala.Tuple2<Integer,Object>>	call(Iterator<scala.Tuple2<Long,FrameBlock>> iter)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TransformEncodeBuildFunction

public TransformEncodeBuildFunction(org.apache.sysml.runtime.transform.encode.Encoder encoder)

Method Detail

call

public Iterator<scala.Tuple2<Integer,Object>> call(Iterator<scala.Tuple2<Long,FrameBlock>> iter)
                                            throws Exception

Specified by:

call in interface org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<scala.Tuple2<Long,FrameBlock>>,Integer,Object>

Throws:

Exception