@InterfaceAudience.Private public class MRToTezHelper extends Object
| Modifier and Type | Method and Description |
|---|---|
static void |
processMRSettings(org.apache.hadoop.conf.Configuration tezConf,
org.apache.hadoop.conf.Configuration mrConf)
Process the mapreduce configuration settings and
- copy as is the still required ones (like those used by FileInputFormat/FileOutputFormat)
- convert and set equivalent tez runtime settings
- handle compression related settings
|
static void |
setVertexConfig(org.apache.tez.dag.api.Vertex vertex,
boolean isMapVertex,
org.apache.hadoop.conf.Configuration conf)
Set config with Scope.Vertex in TezConfiguration on the vertex
|
static void |
translateMRSettingsForTezAM(org.apache.tez.dag.api.TezConfiguration dagAMConf) |
static void |
updateLocalResourcesForInputSplits(org.apache.hadoop.fs.FileSystem fs,
org.apache.tez.mapreduce.hadoop.InputSplitInfo inputSplitInfo,
Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
Exact copy of private method from from org.apache.tez.mapreduce.hadoop.MRInputHelpers
Update provided localResources collection with the required local
resources needed by MapReduce tasks with respect to Input splits.
|
static org.apache.tez.mapreduce.hadoop.InputSplitInfoDisk |
writeInputSplitInfoToDisk(org.apache.tez.mapreduce.hadoop.InputSplitInfoMem infoMem,
org.apache.hadoop.fs.Path inputSplitsDir,
org.apache.hadoop.mapred.JobConf jobConf,
org.apache.hadoop.fs.FileSystem fs)
Write input splits (job.split and job.splitmetainfo) to disk
|
public static void translateMRSettingsForTezAM(org.apache.tez.dag.api.TezConfiguration dagAMConf)
public static void setVertexConfig(org.apache.tez.dag.api.Vertex vertex,
boolean isMapVertex,
org.apache.hadoop.conf.Configuration conf)
vertex - Vertex on which config is to be setisMapVertex - Whether map or reduce vertex. i.e root or intermediate/leaf vertexconf - Config that contains the tez or equivalent mapreduce settings.public static void processMRSettings(org.apache.hadoop.conf.Configuration tezConf,
org.apache.hadoop.conf.Configuration mrConf)
tezConf - Configuration on which the mapreduce settings will have to be transferredmrConf - Configuration that contains mapreduce settingspublic static org.apache.tez.mapreduce.hadoop.InputSplitInfoDisk writeInputSplitInfoToDisk(org.apache.tez.mapreduce.hadoop.InputSplitInfoMem infoMem,
org.apache.hadoop.fs.Path inputSplitsDir,
org.apache.hadoop.mapred.JobConf jobConf,
org.apache.hadoop.fs.FileSystem fs)
throws IOException,
InterruptedException
IOExceptionInterruptedExceptionpublic static void updateLocalResourcesForInputSplits(org.apache.hadoop.fs.FileSystem fs,
org.apache.tez.mapreduce.hadoop.InputSplitInfo inputSplitInfo,
Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
throws IOException
fs - Filesystem instance to access status of splits related filesinputSplitInfo - Information on location of split fileslocalResources - LocalResources collection to be updatedIOExceptionCopyright © 2007-2017 The Apache Software Foundation