FLUME-???: Support for using Netty and a replacement for the syslog sources using Netty
Review Request #1246 - Created Nov. 21, 2010 and updated
|esammer, jon, phunt|
This is 2 things rolled into 1 so that you can see how the Netty stuff is used. Things to note: * I have one source and it can be configured to listen on TCP or UDP. * That source is using NIO. * This patch depends on a patch for QueueBasedSource that I haven't uploaded yet, so you can't run it right now. * The sources using NettySourceBase take a first parameter in URI form: tcp://localhost:12345/ or udp://hostname:port/ ... they currently bind to the hostname specified and I haven't cooked up a way to specify them binding to all available IPs / interfaces. * The syslog extraction should be pretty fast and does less allocation and data copying than before. * When the command line parsing stuff lands, I plan to supplement NettyConfiguration with that to allow configuring the other parameters that are there in key=value form. * In a future version of Netty, it supports additional things, like specifying what IPs can connect and so on, so there's good growth for the future here. * I plan on hooking up SSL support so that you can give a URI like ssl://localhost:12345/ and it would Just Work with no other changes. * I have another source that works with this same structure, but it implements a (really easy) binary protocol and supports all aspects of a Flume event. It is easy enough that someone can implement talking to it in C#, C, PHP, ActionScript, etc. I'm currently arguing with myself as to where this code should live. * I plan to add support for a FlashPolicy file handler so that anything using NettySourceBase can easily enable that for having ActionScript / Flash talk to it. * Yeah, I need to add tests. * Yeah, I also need to add more doc comments and documentation. Comments on style, structure, whatever are welcome ...
Minimal testing ... I've done: bin/flume dump 'syslogSource("tcp://localhost:12345/")' And then thrown good and bad input at it.