A coworker told me about this project today, and I thought I would share since it looks promising.
Packetpig is an open source project hosted on github by @packetloop that contains Hadoop InputFormats, Pig Loaders, Pig scripts and R scripts for processing and analyzing pcap data. It also has classes that allow you to stream packets from Hadoop to local snort and p0f processes so you can parallelize this type of packet processing.
Check it out:
- Packetpig - Open Source Big Data Security Analysis
- packetpig github project
- Single Node Installation and Deployment
–Jason
@jason_trost