HUE-8726 [core] Switch file downloads to a separate process
Review Request #13662 - Created Feb. 28, 2019 and updated - Latest diff uploaded
|Jean-Francois Desjeans Gauthier|
commit b72ac4ea973d5e8ddee1b39f60270d908e2e5362 Author: jdesjean <email@example.com> Date: Thu Feb 28 16:49:15 2019 -0800 HUE-8726 [core] Switch file downloads to a separate process :100644 100644 858fc2bba2... 113a018e54... M apps/filebrowser/src/filebrowser/views.py :100644 100644 8ea7aa9006... 57c6d6245d... M desktop/core/src/desktop/lib/rest/resource.py :100644 100644 c752ae1784... 6ad85ea945... M desktop/core/src/desktop/lib/wsgiserver.py :100644 100644 1e938b8d1d... 1ddab9ac50... M desktop/core/src/desktop/settings.py :100644 100644 3306b032c3... 3c687ae69f... M desktop/libs/hadoop/src/hadoop/fs/webhdfs.py
This is a basic commit that needs some additional work.
All file fetches and downloads are executed on a separate process which does not block the main django process.
Tested that it works correctly to load Hue and download files from HDFS. Downloaded 5x 1GB files concurrently + browsing Hue seem to be working well.
Implement S3 & ADLS.
Full load test.
Cleanup threads & processes
Configure # of threads & processes via config file