FLUME-685: Flume requires restart after HBase goes down

Review Request #1847 - Created June 27, 2011 and updated

Information
Alex Baranau
old-flume
FLUME-685
Reviewers
flume
* added retry after put fails with IOException.
* did small refactoring along the way (related to the changes) extracted base class for hbase sinks with common functionality: open/close sink, writing with retries, etc.

  
Dani Abel Rayan
Jonathan Hsieh
Alex Baranau
Alex Baranau
Alex Baranau
Review request changed

Change Summary:

Updated diff based on https://review.cloudera.org/r/1900/. Fixed nit mentioned before

Diff:

Revision 2 (+14 -2)

Show changes

Jonathan Hsieh

   
Was curious what happens when interrupted, saw that hbase client converts into IOException.  (HConnectionManager.processBatchOfPuts in HBase 0.90.4) 

This is actually troublesome because we'll never be able to cleanup / reconfigure this logical node if the put gets into an error state! 

Unfortunately, this might require changing HBase.  

Also, using Flume's native recovery would suffer from the same problem.

Can you add this caveat to the docs?  (Also can you test to verify this?)
Dani Abel Rayan

   
A somewhat related bit: The more you want to get away with exceptions, the more you move closer to Asynchronous HBase Client. It is used in Flume-OpenTSDB sink too. Its benefits are here: http://groups.google.com/group/opentsdb/browse_thread/thread/c952c5b2a057ae3a?pli=1
Loading...