HUE-8978 [fb] First commit for Google Storage support.

Review Request #14256 — Created Aug. 26, 2019 and submitted

jgauthier
hue
master
HUE-8978
hue
jgauthier, johan, ranade, romain, weixia, yingc
commit 0ab33a4ebd891d41046defe1be4764344e1a4a93
Author: Jean-Francois Desjeans Gauthier <jf.desjeans.gauthier@gmail.com>
Date:   Mon Aug 26 17:04:29 2019 -0700

    HUE-8978 [fb] First commit for Google Storage support.
    
    Uses the boto library to operate on Google Storage. Only tested method
    is s3fs.listdir_stats via
    desktop.lib.fs.gc.tests:TestGCS.test_with_credentials. Other methods
    needs to be modified to support both S3 & GS.
    Only authentication that was testing was json_credentials provided for a
    service account.
    https://console.cloud.google.com/iam-admin/serviceaccounts
    
    Authentication is handled by the boto library, but pluggable via
    desktop.lib.fs.gc.client:OAuth2JsonServiceAccountClientAuth. We had to
    create our own authentication plugin, because the provided one in
    gcs_oauth2_boto_plugin.oauth2_plugin require fetching credentials via
    files and this won't work with IDBroker. Current IDBroker code is not
    tested.
    
    Instead of creating a desktop/libs which comes with a bunch of problems,
    added this lib in desktop.lib.fs.
    Authentication via IAM should also be supported, but not tested.

:100644 100644 31903e8093... 0c2516fa58... M	apps/filebrowser/src/filebrowser/settings.py
:100644 100644 002a81a87c... e6ec19e984... M	apps/useradmin/src/useradmin/models.py
:100644 100644 e6f2c00b4a... 07ae96597d... M	desktop/conf.dist/hue.ini
:100644 100644 690d93340c... 02d257417a... M	desktop/conf/pseudo-distributed.ini.tmpl
:100644 100644 0fc5fd008b... ad08802623... M	desktop/core/src/desktop/conf.py
:000000 100644 0000000000... f882a3ac7a... A	desktop/core/src/desktop/lib/fs/gc/__init__.py
:000000 100644 0000000000... df482c5156... A	desktop/core/src/desktop/lib/fs/gc/client.py
:000000 100644 0000000000... 52a305ccef... A	desktop/core/src/desktop/lib/fs/gc/tests.py
:100644 100644 9f4679d647... a3e875c10b... M	desktop/core/src/desktop/lib/fs/proxyfs_test.py
:100644 100644 921e1367eb... 26709d5773... M	desktop/core/src/desktop/lib/fsmanager.py
:100644 100644 5bd678dad6... 08ee792438... M	desktop/libs/aws/src/aws/s3/s3fs.py
:100644 100644 4ef5f8068c... 9b813b4b56... M	desktop/libs/aws/src/aws/s3/s3stat.py


  • 0
  • 0
  • 1
  • 1
  • 2
Description From Last Updated
romain
  1. Add to docs https://docs.gethue.com/latest/administrator/configuration/files/ and https://docs.gethue.com/latest/user/browsers/#gfs ? (gfs --> gs)

    1. It looks like they rebranded from Google Cloud Platform (gcp) to Google Cloud (gc) and the service is called Google Cloud Storage (gcs) (which is matched in the name of the library gcs_oauth2_boto_plugin). The scheme for the file system is 'gs'.
      Let's keep as is?

    2. Makes sense!

  2. 
      
jgauthier
Review request changed

Status: Closed (submitted)

Loading...