public class BinStorage extends FileInputLoadFunc implements StoreFuncInterface, LoadMetadata
| Modifier and Type | Field and Description |
|---|---|
protected long |
end |
| Constructor and Description |
|---|
BinStorage() |
BinStorage(String casterString) |
| Modifier and Type | Method and Description |
|---|---|
void |
checkSchema(ResourceSchema s)
Set the schema for data to be stored.
|
void |
cleanupOnFailure(String location,
org.apache.hadoop.mapreduce.Job job)
This method will be called by Pig if the job which contains this store
fails.
|
void |
cleanupOnSuccess(String location,
org.apache.hadoop.mapreduce.Job job)
This method will be called by Pig if the job which contains this store
is successful, and some cleanup of intermediate resources is required.
|
org.apache.hadoop.mapreduce.InputFormat |
getInputFormat()
This will be called during planning on the front end.
|
LoadCaster |
getLoadCaster()
This will be called on both the front end and the back
end during execution.
|
Tuple |
getNext()
Retrieves the next tuple to be processed.
|
org.apache.hadoop.mapreduce.OutputFormat |
getOutputFormat()
Return the OutputFormat associated with StoreFuncInterface.
|
String[] |
getPartitionKeys(String location,
org.apache.hadoop.mapreduce.Job job)
Find what columns are partition keys for this input.
|
ResourceSchema |
getSchema(String location,
org.apache.hadoop.mapreduce.Job job)
Get a schema for the data to be loaded.
|
ResourceStatistics |
getStatistics(String location,
org.apache.hadoop.mapreduce.Job job)
Get statistics about the data to be loaded.
|
int |
hashCode() |
void |
prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader,
PigSplit split)
Initializes LoadFunc for reading data.
|
void |
prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
Initialize StoreFuncInterface to write data.
|
void |
putNext(Tuple t)
Write a tuple to the data store.
|
String |
relToAbsPathForStoreLocation(String location,
org.apache.hadoop.fs.Path curDir)
This method is called by the Pig runtime in the front end to convert the
output location to an absolute path if the location is relative.
|
void |
setLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the loader the location of the object(s) being loaded.
|
void |
setPartitionFilter(Expression plan)
Set the filter for partitioning.
|
void |
setStoreFuncUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
StoreFuncInterface which it can use to store
information in the UDFContext which it needs to store between
various method invocations in the front end and back end. |
void |
setStoreLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the storer the location where the data needs to be stored.
|
byte[] |
toBytes(Boolean b) |
byte[] |
toBytes(DataBag bag) |
byte[] |
toBytes(org.joda.time.DateTime dt) |
byte[] |
toBytes(Double d) |
byte[] |
toBytes(Float f) |
byte[] |
toBytes(Integer i) |
byte[] |
toBytes(Long l) |
byte[] |
toBytes(Map<String,Object> m) |
byte[] |
toBytes(String s) |
byte[] |
toBytes(Tuple t) |
getSplitComparablegetAbsolutePath, getCacheFiles, getPathStrings, getShipFiles, join, relativeToAbsolutePath, setUDFContextSignature, warnpublic BinStorage()
public BinStorage(String casterString)
public Tuple getNext() throws IOException
LoadFuncgetNext in class LoadFuncIOException - if there is an exception while retrieving the next
tuplepublic void putNext(Tuple t) throws IOException
StoreFuncInterfaceputNext in interface StoreFuncInterfacet - the tuple to store.IOException - if an exception occurs during the writepublic byte[] toBytes(DataBag bag) throws IOException
IOExceptionpublic byte[] toBytes(String s) throws IOException
IOExceptionpublic byte[] toBytes(Double d) throws IOException
IOExceptionpublic byte[] toBytes(Float f) throws IOException
IOExceptionpublic byte[] toBytes(Integer i) throws IOException
IOExceptionpublic byte[] toBytes(Long l) throws IOException
IOExceptionpublic byte[] toBytes(Boolean b) throws IOException
IOExceptionpublic byte[] toBytes(org.joda.time.DateTime dt)
throws IOException
IOExceptionpublic byte[] toBytes(Map<String,Object> m) throws IOException
IOExceptionpublic byte[] toBytes(Tuple t) throws IOException
IOExceptionpublic org.apache.hadoop.mapreduce.InputFormat getInputFormat()
LoadFuncgetInputFormat in class LoadFuncpublic LoadCaster getLoadCaster() throws IOException
LoadFuncgetLoadCaster in class LoadFuncLoadCaster associated with this loader. Returning null
indicates that casts from byte array are not supported for this loader.
constructionIOException - if there is an exception during LoadCasterpublic void prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader,
PigSplit split)
LoadFuncprepareToRead in class LoadFuncreader - RecordReader to be used by this instance of the LoadFuncsplit - The input PigSplit to processpublic void setLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadFuncLoadFunc.relativeToAbsolutePath(String, Path). Implementations
should use this method to communicate the location (and any other information)
to its underlying InputFormat through the Job object.
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.setLocation in class LoadFunclocation - Location as returned by
LoadFunc.relativeToAbsolutePath(String, Path)job - the Job object
store or retrieve earlier stored information from the UDFContextIOException - if the location is not valid.public org.apache.hadoop.mapreduce.OutputFormat getOutputFormat()
StoreFuncInterfacegetOutputFormat in interface StoreFuncInterfaceOutputFormat associated with StoreFuncInterfacepublic void prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
StoreFuncInterfaceprepareToWrite in interface StoreFuncInterfacewriter - RecordWriter to use.public void setStoreLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreFuncInterfaceStoreFuncInterface here is the
return value of StoreFuncInterface.relToAbsPathForStoreLocation(String, Path)
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.
StoreFuncInterface.checkSchema(ResourceSchema) will be called before any call to
StoreFuncInterface.setStoreLocation(String, Job).setStoreLocation in interface StoreFuncInterfacelocation - Location returned by
StoreFuncInterface.relToAbsPathForStoreLocation(String, Path)job - The Job objectIOException - if the location is not valid.public void checkSchema(ResourceSchema s) throws IOException
StoreFuncInterfacecheckSchema in interface StoreFuncInterfaces - to be checkedIOException - if this schema is not acceptable. It should include
a detailed error message indicating what is wrong with the schema.public String relToAbsPathForStoreLocation(String location, org.apache.hadoop.fs.Path curDir) throws IOException
StoreFuncInterfaceLoadFunc.getAbsolutePath(java.lang.String, org.apache.hadoop.fs.Path) provides a default
implementation for hdfs and hadoop local file system and it can be used
to implement this method.relToAbsPathForStoreLocation in interface StoreFuncInterfacelocation - location as provided in the "store" statement of the scriptcurDir - the current working direction based on any "cd" statements
in the script before the "store" statement. If there are no "cd" statements
in the script, this would be the home directory -
/user/
IOException - if the conversion is not possiblepublic String[] getPartitionKeys(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadMetadatagetPartitionKeys in interface LoadMetadatalocation - Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)job - The Job object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration() and not to set/query
any runtime job information.IOException - if an exception occurs while retrieving partition keyspublic ResourceSchema getSchema(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadMetadatagetSchema in interface LoadMetadatalocation - Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)job - The Job object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration() and not to set/query
any runtime job information.IOException - if an exception occurs while determining the schemapublic ResourceStatistics getStatistics(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadMetadataLoadFunc, then LoadFunc.setLocation(String, org.apache.hadoop.mapreduce.Job)
is guaranteed to be called before this method.getStatistics in interface LoadMetadatalocation - Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)job - The Job object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration() and not to set/query
any runtime job information.IOException - if an exception occurs while retrieving statisticspublic void setPartitionFilter(Expression plan) throws IOException
LoadMetadataLoadMetadata.getPartitionKeys(String, Job), then this method is not
called by Pig runtime. This method is also not called by the Pig runtime
if there are no partition filter conditions.setPartitionFilter in interface LoadMetadataplan - that describes filter for partitioningIOException - if the filter is not compatible with the storage
mechanism or contains non-partition fields.public void setStoreFuncUDFContextSignature(String signature)
StoreFuncInterfaceStoreFuncInterface which it can use to store
information in the UDFContext which it needs to store between
various method invocations in the front end and back end. This is necessary
because in a Pig Latin script with multiple stores, the different
instances of store functions need to be able to find their (and only their)
data in the UDFContext object.setStoreFuncUDFContextSignature in interface StoreFuncInterfacesignature - a unique signature to identify this StoreFuncInterfacepublic void cleanupOnFailure(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreFuncInterfacecleanupOnFailure in interface StoreFuncInterfacelocation - Location returned by
StoreFuncInterface.relToAbsPathForStoreLocation(String, Path)job - The Job object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration() and not to set/query
any runtime job information.IOExceptionpublic void cleanupOnSuccess(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreFuncInterfacecleanupOnSuccess in interface StoreFuncInterfacelocation - Location returned by
StoreFuncInterface.relToAbsPathForStoreLocation(String, Path)job - The Job object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration() and not to set/query
any runtime job information.IOExceptionCopyright © 2007-2017 The Apache Software Foundation