HUE-7248 [adls] Ability to browse data / content in ADLS

Review Request #11540 - Created Sept. 13, 2017 and submitted

Jean-Francois Desjeans Gauthier
hue
master
HUE-7248
hue
jgauthier, romain
commit 6b01cdacb9e9265ea954fcfdeb2703d2613423a1
Author: jdesjean <jgauthier@cloudera.com>
Date:   Wed Sep 13 10:00:02 2017 -0700

    HUE-7248 [adls] Ability to browse data / content in ADLS

:100644 100644 2df8558def... 72420c7b54... M	desktop/Makefile
:100644 100644 cf65a1705c... d92efcf2b1... M	desktop/core/src/desktop/lib/fsmanager.py
:100644 100644 7cff516906... 8258efe13c... M	desktop/core/src/desktop/models.py
:100644 100644 f29ebb892b... 77a6756bf2... M	desktop/core/src/desktop/static/desktop/js/apiHelper.js
:100644 100644 d9e85c7a21... 11ba2c855b... M	desktop/core/src/desktop/static/desktop/js/assist/assistHdfsEntry.js
:100644 100644 64f5bb0bef... 38f3111a34... M	desktop/core/src/desktop/static/desktop/js/ko.hue-bindings.js
:100644 100644 9b687e2bc3... 0e145b1ce6... M	desktop/core/src/desktop/templates/assist.mako
:000000 100644 0000000000... f0cd6694cd... A	desktop/libs/azure/Makefile
:000000 100644 0000000000... 4590f9ec46... A	desktop/libs/azure/babel.cfg
:000000 100644 0000000000... 5ca999a0ee... A	desktop/libs/azure/hueversion.py
:000000 100644 0000000000... 2cbfee33bb... A	desktop/libs/azure/setup.py
:000000 100644 0000000000... 6465e38c0d... A	desktop/libs/azure/src/azure/__init__.py
:000000 100644 0000000000... e028190bd1... A	desktop/libs/azure/src/azure/active_directory.py
:000000 100644 0000000000... 1854b36356... A	desktop/libs/azure/src/azure/adls/__init__.py
:000000 100644 0000000000... 22df043c61... A	desktop/libs/azure/src/azure/adls/webhdfs.py
:000000 100644 0000000000... 01c228b517... A	desktop/libs/azure/src/azure/client.py
:000000 100644 0000000000... 80665a6325... A	desktop/libs/azure/src/azure/conf.py
:000000 100644 0000000000... 5f56190f8e... A	desktop/libs/azure/src/azure/locale/en_US.pot
:100644 100644 5d0de1612c... 60e33e9db0... M	desktop/libs/hadoop/src/hadoop/core_site.py
:100644 100644 747c1e32fa... 4609eb028a... M	desktop/libs/hadoop/src/hadoop/fs/webhdfs.py


  • 0
  • 0
  • 36
  • 0
  • 36
Description From Last Updated
  1. Super nice!

    Some infos in the comments

  2. desktop/core/src/desktop/models.py (Diff revision 1)
     
     

    () not needed

  3. desktop/libs/azure/hueversion.py (Diff revision 1)
     
     

    FYI, this should be a symlink

    e.g.

    ls -l apps/oozie/hueversion.py -l
    lrwxrwxrwx 1 romain romain 13 Jun 8 14:21 apps/oozie/hueversion.py -> ../../VERSION

  4. desktop/libs/azure/src/azure/conf.py (Diff revision 1)
     
     

    How to create this one:

    https://github.com/cloudera/hue/blob/master/apps/filebrowser/src/filebrowser/settings.py#L25

    and prevent to be a default:

    https://github.com/cloudera/hue/blob/master/apps/useradmin/src/useradmin/models.py#L283

  5. def strip_normpath(): ?

  6. if self._strip_scheme:

  7. I see, might be needed here for headers. Are we sure we need it?

  8. 
      
  1. Globally excellent, just a set of nits!

  2. Cf. comment above about losing some of the safety checks on the existence of the path

  3. In the viewModel isTrashEnabled we lose these checks?

    1. I'm not sure I follow. Would you like me to move those capability checks to backend?

    2. Either we keep it in the backend or either we move it to the frontend (but we don't remove the checks)

  4. Possible to reuse Hdfs.abspath() or the S3 one https://github.com/cloudera/hue/blob/master/desktop/libs/aws/src/aws/s3/init.py#L81?

  5. "and len(pathSplit.scheme) > 0" no needed

  6. desktop/libs/azure/src/azure/__init__.py (Diff revision 2)
     
     
     
     
     
     
     
     
     
     
     
     

    Needed?

  7. isTokenExpired --> is_token_expired

  8. Hadoop --> Azure

  9. New line after?

  10. So this one is one trick with ADLS, we always need to provide the token?

    1. Yes every API call needs to have the token. It's not a session cookie.

  11. desktop/libs/azure/src/azure/client.py (Diff revision 2)
     
     

    New line below?

  12. desktop/libs/azure/src/azure/client.py (Diff revision 2)
     
     

    So here we cache the client and hence the token? (no pulled for each call)

    1. We "cache" the client, but this is not a client in the same sense as S3. Since we use REST HTTP calls, the connection is not persistent. The only thing we are really caching here is the authentication token.

  13. desktop/libs/azure/src/azure/client.py (Diff revision 2)
     
     
     

    Remove?

  14. desktop/libs/azure/src/azure/conf.py (Diff revision 2)
     
     

    Add below [aws] in

    https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini#L1396

    and

    https://github.com/cloudera/hue/blob/master/desktop/conf/pseudo-distributed.ini.tmpl

    ?

  15. Could we make sure that we also have the 9 languages/ po files (just need to be physically there, content inside does not matter as will be generated later)

    https://github.com/cloudera/hue/tree/master/desktop/libs/aws/src/aws/locale

    ?

    to tests:
    make locales

    (http://cloudera.github.io/hue/docs-4.0.0/sdk/sdk.html#internationalization)

  16. desktop/libs/hadoop/src/hadoop/fs/upload.py (Diff revision 2)
     
     
     
     
     

    Remove?

  17. if headers is None:

  18. 
      
  1. Last nits about the Python syntax, and then good to ship! Really nice!

  2. ts (trailing space)

  3. if splitRoot.scheme and not splitPath.scheme:

    More Pythonic

  4. apps/metastore/src/metastore/views.py (Diff revision 3)
     
     

    if split.netloc is None or not split.netloc:

  5. To add also in https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini

  6. This and _timeout not used?

  7. return path.startswith(self._scheme) if self._scheme else path == '/'

  8. scheme="",netloc="" --> scheme="", netloc=""

  9. if not split.path:

  10. if not self._is_remote:

    If checking for (False, '' or None) or just False

  11. ... or not split.netloc:

  12. 
      
Review request changed

Status: Closed (submitted)

Loading...