Review Board 1.6.3

SplitTransaction.splitStoreFiles slows splits a lot

Review Request #1273 - updated 3 years, 4 months ago

Jean-Daniel Cryans Reviewers
None hbase svn
Patch that parallelizes the splitting of the files using ThreadPoolExecutor and Futures. The code is a bit ugly, but does the job really well as shown during cluster testing (which also uncovered HBASE-3318).

One new behavior this patch adds is that it's now possible to rollback a split because it took too long to split the files. I did some testing with a timeout of 5 secs on my cluster, even tho each machine did a few rollbacks the import went fine. The default is 30 seconds and isn't in hbase-default.xml as I don't think anyone would really want to change that.

Ship it!
Posted 3 years, 4 months ago (December 7th, 2010, 5:02 p.m.)
+1  Minor comment below.
Why not have an upper bound?  If 100 files thats 100 threads doing FS operations.  I bet if you had upper bound of 10 on the executorservice, it complete faster than an unbounded executorservice?
  1. I think we are already bounded by hbase.hstore.blockingStoreFiles
  2. That'll do.  +1 on commit.