HUE-8747 [editor] move fetch progress to django cache.

Review Request #13868 - Created April 18, 2019 and submitted

Jean-Francois Desjeans Gauthier
hue
master
HUE-8747
hue
jgauthier
commit c070b4f79cad6740b0a953a9bf13a6521f1e88bd
Author: jdesjean <jgauthier@cloudera.com>
Date:   Thu Apr 18 16:17:25 2019 -0700

    HUE-8747 [editor] Move fetch progress to django cache

:100644 100644 a0f661dc6d... 74d91e1066... M	desktop/libs/notebook/src/notebook/tasks.py


  • 1
  • 0
  • 0
  • 0
  • 1
Description From Last Updated
None will cache the value forever, if we don't use the pre-fork model (e.g. gevent), won't it leak? Also, might ... Romain Rigaux
  1. 
      
  2. None will cache the value forever, if we don't use the pre-fork model (e.g. gevent), won't it leak?

    Also, might be worth having a dedicated cache config for this at some point?

    In settings:

    Maybe eventually we have one for API server ('default'), one for the Workers and one for the query results?

    CACHES = {
    'default': {
    'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
    'LOCATION': 'unique-hue'
    }
    }

    1. I'm thinking of moving the results to django storage instead of cache.
      We can think of adding a timeout when we have cleanup cache. We don't so we should keep as long as needed.

    2. We can think of adding a timeout when we have cleanup task*. We don't so we should keep as long as needed.

    3. Looking at the django cache implementation, the timeout guarantee is really weak. It only removes a key from the cache if you try to access it after the timeout. It doesn't actually cleanup the cache on it's own.

    4. https://github.com/django/django/blob/12b7956fc3735101fcad597047b80b57efb5048a/django/core/cache/backends/filebased.py#L135

    5. We should expire cache results either by time or by size. The django cache does none of those things.

  3. 
      
Review request changed

Status: Closed (submitted)

Loading...