HUE-1846 [core] Make HTTP client thread safe

Review Request #4038 — Created Jan. 16, 2014 and discarded

abec
old-hue-rw
HUE-1846
hue
enricoberti, romain
commit 36dd0c381627af7c054465f4194e6fefcab93654
Author: Abraham Elmahrek <abraham@elmahrek.com>
Date:   Fri Jan 3 16:57:38 2014 -0800

    HUE-1846 [core] Make HTTP client thread safe
    
    HttpClient's vary based on base_url and thread.
    Resources vary based on client base_url, relative path, and thread.
    Add proper re-entrant locking.

:100644 100644 ce23c82... ede32fb... M	apps/search/src/search/api.py
:100644 100644 d514bb9... 4cf1be4... M	apps/spark/src/spark/job_server_api.py
:100644 100644 5087ae0... a8a3e12... M	apps/sqoop/src/sqoop/client/base.py
:100644 100644 73d4a32... 9d425ac... M	desktop/core/src/desktop/lib/rest/http_client.py
:100644 100644 5cad3a3... a7fe074... M	desktop/core/src/desktop/lib/rest/resource.py
:100644 100644 4ca58f0... 4511df4... M	desktop/libs/hadoop/src/hadoop/fs/webhdfs.py
:100644 100644 d9a38d7... 7373f6f... M	desktop/libs/hadoop/src/hadoop/yarn/history_server_api.py
:100644 100644 b249481... 33e6a39... M	desktop/libs/hadoop/src/hadoop/yarn/mapreduce_api.py
:100644 100644 7f53f8b... b89b0ce... M	desktop/libs/hadoop/src/hadoop/yarn/node_manager_api.py
:100644 100644 6fd371f... 2139a11... M	desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py
:100644 100644 189b4f6... 5f922fe... M	desktop/libs/liboozie/src/liboozie/oozie_api.py
Tested manually on a non-kerberized cluster.
  • 3
  • 0
  • 0
  • 3
  • 6
Description From Last Updated
to doc? get_root() returns a client with a resource instance romain romain
ok so we have base_url + version as key so it will manage multiple versions romain romain
is it really needed? I think we should leave it failing and then fix it at the source as it ... romain romain
romain
  1. First pass!
    
    So the original whole problem was that HTTP client internal is using Resource which is not thread safe? Now we make sure that each client has its own Resource?
    1. The problem is that multiple cookie jars are being created since a new HttpClient were being created in some cases. Also, in other cases, the same cookie jar was being used and we hit race conditions where the cookie jar was being updated by 2 different threads. So then we must make the sessions some how globally available or provide constructors that return cached http clients. The caching/threading of resources is for simplicity when using the HttpClient. It doesn't actually need to be cached though as long as it's using a client that is part of the same thread. This seems hard to guarantee though since many of API classes (ie OozieApi) are singletons. Maybe we can construct resource classes instead of worrying about threading??
  2. to doc?
    
    get_root() returns a client with a resource instance
  3. won't deadlock as get_resource() does a   RESOURCE_LOCK.acquire() ?
    1. Re-entrant lock so that acquire can be called multiple times by the same thread as long as it calls release the same number of times.
  4. we son't setattr here?
    
    hum I see more now, but why not just merge both functions?
    1. Separated for simplicity really. Will comment.
  5. ok so we have base_url + version as key so it will manage multiple versions
    1. It will have multiple resources based on these 3 dimensions: base_url, rel_path, and thread.
  6. is it really needed?
    
    I think we should leave it failing and then fix it at the source as it is a big security risk
  7. 
      
romain
  1. 1. The problem is that multiple cookie jars are being created since a new HttpClient were being created in some cases. 
    2. Also, in other cases, the same cookie jar was being used and we hit race conditions where the cookie jar was being updated by 2 different threads.
    
    1. Is it really a problem?
    2. Let me think about this.
    
    Let's sync in person a bit!
  2. Hum, where is the simplicity?
    
    it is just one line:
    resource = resource_cls(client, rel_path)
    
    wit no need to comment
    
    
    1. Removed all the threading on resource.py.
  3. is it removed?
    1. No, still needed.
  4. 
      
abec
Review request changed

Status: Discarded

Loading...