Apache pig is a data flow high level script language from APACHE.It was developed in Yahoo and now it is the apache project for Hadoop.
Its language is called PIG Latin and the Grunt is the shell where PIG syntax is written and execute.
Pig as an abstraction on the top of Hadoop provides high level programming language designed for data processing converted into MapReduce and executed on Hadoop Clusters.
Pig is widely used Yahoo!, Twitter, Netflix, etc.In terms of map and reduce functions we require Java programmers.
Pig is a high-level language used by
Analysts,Data Scientists, Statisticians.
Features of Pig
- Join data sets
- Sort data sets
- Data types
- Group by
- User defined functions
Use Cases of Pig
Extract Transform Load (ETL):
Example: Processing large amounts of log data
clean bad entries, join with other data-sets
Research of “raw” information:
Example User Audit Logs
- Pig Latin
- Command based language
- Designed for data transformation and flow expression
- Execution Environment
- Environment in which Pig latin commands were executed
- Currently there is support for Local and Hadoop modes
- Pig compiler converts Pig Latin to MapReduce
- Compiler strives to optimize execution
- PIG is mainly used for ETL (Extract transform load) on Hadoop.
- It is a script base language. It is useful for those who is having no knowledge of java for map reduce.
- Also less no. of code in Pig Latinis written compare to javato execute map reduce jobs.
- Pig Latin helps to manipulate large and complex data sets.
- Pig via udf can be used to embed other languages (Jython, Python, Ruby, R etc.) With pig script.
- Pig can work with many data sources either structured and unstructured and then stores it into Hadoop cluster.
- Pig is used to clean bad entries in file. Pig is also useful to analyze Web Log data or Server Log data.