HUE-8118 [core] Fine grain tracking of the memory usage

Review Request #12702 — Created March 22, 2018 and submitted

roohi
hue
HUE-7774-Autocomplete
HUE-8118
hue
enricoberti, jgauthier, johan, ranade, romain, weixia, yingc
commit a2a7569f7bdb796fef6c698125d7f4d6db4c4d73
Author: Roohi <roohisyeda@cloudera.com>
Date:   Thu Mar 22 14:34:41 2018 -0700

    HUE-8118 [core] Fine grain tracking of the memory usage

:100644 100644 679f9d1035... d2204f3291... M	desktop/core/src/desktop/log/access.py


  • 2
  • 0
  • 14
  • 0
  • 16
Description From Last Updated
Both are shown in mb in the log message But because the resource module returns Bytes for OSX I had ... roohi roohi
mb. mb is better as usage is usually above 100 mb roohi roohi
romain
  1. We probably want a main 'instrumentation' config flag, that will enable the tracking of it (we can reuse the same flag for the thrift/rest logging)

    1. So when the flag is on, only then append the duration and memory values to logging?

    2. Yes, the idea is to have on unique flag, than when turned on will add this tracking. This is similar to DEBUG mode but lighter weight. When DEBUG mode is on, we should automatically set this flag to ON too.

      The same flag will be used to trigger the RPC calls finer grain tracking, and the request profiler.

  2. desktop/core/src/desktop/log/access.py (Diff revision 1)
     
     
     

    Move to top?

  3. desktop/core/src/desktop/log/access.py (Diff revision 1)
     
     
     

    merge?

    Also, is using this way and not guppy (which we ship currently and has some examples in Hue already) better?

  4. desktop/core/src/desktop/log/access.py (Diff revision 1)
     
     

    Shouldn't we get a diff, not the total?
    (obviously other threads could bump the memory in the meantime though, but even a rough compare should help)

  5. 
      
roohi
romain
  1. Just a few nits

  2. Use the same description as in the conf.py?

  3. Also add to https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini?

  4. desktop/core/src/desktop/conf.py (Diff revision 2)
     
     

    To automatically set to true when the debug mode is on, use a dynamic_default?

  5. desktop/core/src/desktop/log/access.py (Diff revision 2)
     
     
     

    Move up below 'time' (official Python import grouped together)?

  6. desktop/core/src/desktop/log/access.py (Diff revision 2)
     
     

    Add a comment that this is a lightweight way to get the total number of memory and doing the diffing before/after request was too inconsistent so we display the last value?

  7. desktop/core/src/desktop/log/access.py (Diff revision 2)
     
     

    FYI: | is not like 'or' in Python

    Better use the dynamic_default of INSTRUMENTATION to be true if debug mode is on cf. above?

  8. desktop/core/src/desktop/log/access.py (Diff revision 2)
     
     

    Do we need the '+' if we don't diff anymore?

  9. 
      
roohi
romain
  1. 
      
  2. desktop/core/src/desktop/conf.py (Diff revision 3)
     
     

    return DJANGO_DEBUG_MODE.get()

  3. desktop/core/src/desktop/log/access.py (Diff revision 3)
     
     

    Make it kb for OSX? (that way the same for any os)

  4. desktop/core/src/desktop/log/access.py (Diff revision 3)
     
     

    It is mb or kb?

  5. desktop/core/src/desktop/settings.py (Diff revision 3)
     
     

    To move above logging? (alpha sorting)

  6. 
      
roohi
  1. 
      
  2. desktop/core/src/desktop/log/access.py (Diff revision 3)
     
     

    Both are shown in mb in the log message
    But because the resource module returns Bytes for OSX I had to do this https://github.com/scrapy/scrapy/issues/529

  3. desktop/core/src/desktop/log/access.py (Diff revision 3)
     
     

    mb. mb is better as usage is usually above 100 mb

  4. 
      
roohi
roohi
Review request changed

Status: Closed (submitted)

Loading...