Basic throttling in Flume,

Review Request #335 - Created July 19, 2010 and submitted

vibhor bhatt
HenryR, jon, phunt
Basic throttling patch, I haven't put the throttling limits required in the configuration file so that one could play with it on run-time in debug mode.
Tested it with 10 threads pumping asciisynth data into a file; the bucket variable in controller class never went below zero, and the total bandwidth consumed by the FlumeNode never went more than 5% above the limit (observed it through "iotop").
Review request changed


-This patch is to implement basic throttling in Flume. This new feature should be able to limit the number of bytes/sec being sent from all the source nodes to sink nodes on one single FlumeNode. This feature will help ensure that Flume does not exhaust all the bandwidth on one single machine.
+Basic throttling in Flume,




  1. Nice first shot.  We need to do alittle bit of work to make it stylistically ok, and there are a few questions about the implementation.
    Fix the spacing.  2 space indent is our convention.  Check the wiki here for a eclipse formatter config.  See this:
    "you can find profiles in Dropbox under cloudera/development/eclipse"
    Please comment public methods.  /** */ style.
    Not sure making (Throttle)Controller a subclass of thread makes sense.  This seems like something that should be encapsulated within it. Would prefer just a class/limited interface that hides the fact that you are using a separate thread for this.
  2. /** for javadoc comments  */ 
  3. grammar: delete push
  4. thrad
    decrements # of tokens in bucket
  5. till => until
    call => calls
  6. fix grammar
  7. Please give a more specific name to the class because controller means nothing to me.
    ThrottleController or Throttler sounds reasonable.
  8. why not static also? (its a constant right?)
    make a config property?
  10. currentBucketCapacity/Size/Tokens? or something like that
  11. /** comment */
    Also why return boolean? what does it mean for someone reading comments?
    Restriction on when this is run?  add Preconditions to call?
  12. synchronization issue?
  13. payloadHeaderSize?
  14. sp
    units might help.
  15. /** Comment please */
  16. Use Clock.sleep instead of Thread.sleep
  17. LOG error message
  18. comment saying that this doesn't block  until thread has finished.  It is just a signal.
    Why named halt?
  19. Use CountDownLatch?
  20. rename to spend/useTokens?
  21. LOG exception
  22. Config file?
    Also, don't need 'this' here.
  23. This thread is never going to be null!?
    Also, this will break if the node gets close and restarted.
  24. lower case variable names