Review Board 1.6.3

FLUME-685: Flume requires restart after HBase goes down

Review Request #1847 - updated 2 years, 7 months ago

Alex Baranau Reviewers
flume
FLUME-685
None flume
* added retry after put fails with IOException.
* did small refactoring along the way (related to the changes) extracted base class for hbase sinks with common functionality: open/close sink, writing with retries, etc.

 
Review request changed
Updated 2 years, 8 months ago (August 22nd, 2011, 11:35 p.m.)
Updated diff based on https://review.cloudera.org/r/1900/. Fixed nit mentioned before
Posted 2 years, 8 months ago (August 25th, 2011, 3:33 a.m.)

   

  
Was curious what happens when interrupted, saw that hbase client converts into IOException.  (HConnectionManager.processBatchOfPuts in HBase 0.90.4) 

This is actually troublesome because we'll never be able to cleanup / reconfigure this logical node if the put gets into an error state! 

Unfortunately, this might require changing HBase.  

Also, using Flume's native recovery would suffer from the same problem.

Can you add this caveat to the docs?  (Also can you test to verify this?)
Posted 2 years, 7 months ago (August 30th, 2011, 5:58 p.m.)

   

  
A somewhat related bit: The more you want to get away with exceptions, the more you move closer to Asynchronous HBase Client. It is used in Flume-OpenTSDB sink too. Its benefits are here: http://groups.google.com/group/opentsdb/browse_thread/thread/c952c5b2a057ae3a?pli=1