HUE-8726 [core] Switch file downloads to a separate process

Review Request #13662 - Created March 1, 2019 and updated - Latest diff uploaded

Jean-Francois Desjeans Gauthier
hue
master
HUE-8726
hue
jgauthier
commit b72ac4ea973d5e8ddee1b39f60270d908e2e5362
Author: jdesjean <jgauthier@cloudera.com>
Date:   Thu Feb 28 16:49:15 2019 -0800

    HUE-8726 [core] Switch file downloads to a separate process

:100644 100644 858fc2bba2... 113a018e54... M	apps/filebrowser/src/filebrowser/views.py
:100644 100644 8ea7aa9006... 57c6d6245d... M	desktop/core/src/desktop/lib/rest/resource.py
:100644 100644 c752ae1784... 6ad85ea945... M	desktop/core/src/desktop/lib/wsgiserver.py
:100644 100644 1e938b8d1d... 1ddab9ac50... M	desktop/core/src/desktop/settings.py
:100644 100644 3306b032c3... 3c687ae69f... M	desktop/libs/hadoop/src/hadoop/fs/webhdfs.py

This is a basic commit that needs some additional work.
All file fetches and downloads are executed on a separate process which does not block the main django process.
Tested that it works correctly to load Hue and download files from HDFS. Downloaded 5x 1GB files concurrently + browsing Hue seem to be working well.

What's needed:
Implement S3 & ADLS.
Full load test.
Cleanup threads & processes
Configure # of threads & processes via config file

Loading...