FLUME-???: Support for using Netty and a replacement for the syslog sources using Netty

Review Request #1246 - Created Nov. 21, 2010 and updated

Bruce Mitchener
old-flume
flume
esammer, jon, phunt
This is 2 things rolled into 1 so that you can see how the Netty stuff is used.

Things to note:

 * I have one source and it can be configured to listen on TCP or UDP.
 * That source is using NIO.
 * This patch depends on a patch for QueueBasedSource that I haven't uploaded yet, so you can't run it right now.
 * The sources using NettySourceBase take a first parameter in URI form: tcp://localhost:12345/ or udp://hostname:port/ ... they currently bind to the hostname specified and I haven't cooked up a way to specify them binding to all available IPs / interfaces.
 * The syslog extraction should be pretty fast and does less allocation and data copying than before.
 * When the command line parsing stuff lands, I plan to supplement NettyConfiguration with that to allow configuring the other parameters that are there in key=value form.
 * In a future version of Netty, it supports additional things, like specifying what IPs can connect and so on, so there's good growth for the future here.
 * I plan on hooking up SSL support so that you can give a URI like ssl://localhost:12345/ and it would Just Work with no other changes.
 * I have another source that works with this same structure, but it implements a (really easy) binary protocol and supports all aspects of a Flume event. It is easy enough that someone can implement talking to it in C#, C, PHP, ActionScript, etc.  I'm currently arguing with myself as to where this code should live.
 * I plan to add support for a FlashPolicy file handler so that anything using NettySourceBase can easily enable that for having ActionScript / Flash talk to it.
 * Yeah, I need to add tests.
 * Yeah, I also need to add more doc comments and documentation.

Comments on style, structure, whatever are welcome ...
Minimal testing ... I've done:

bin/flume dump 'syslogSource("tcp://localhost:12345/")'

And then thrown good and bad input at it.
  1. In general, I like what I've read from the netty site, and this code looks pretty clean.  I'm going to defer to Phunt here -- here's got more experience wiht netty from converting zk to use it.  
  2. From the docs, I think close is more complicated.  Phunt?
  3. 
      
Loading...