[HUE-7667] Fix http client pool by correctly mounting Transport Adapter to a prefix.

Review Request #12068 - Created Nov. 22, 2017 and discarded

Prakash Ranade
hue
master
hue
enricoberti, jennykim, jgauthier, johan, krish, ranade, romain, subrata, weixia, yingc
commit d16fb420b09530cbd0fd2bd3365e558fb1f0f124
Author: Prakash Ranade <ranade@cloudera.com>
Date:   Wed Nov 22 20:42:27 2017 -0800

    [HUE-7667] Fix http client pool by correctly mounting Transport Adapter to a prefix.

:100644 100644 a3b333587e... 451e152802... M	desktop/core/src/desktop/lib/rest/http_client.py

Tested on cluster.

  • 0
  • 0
  • 1
  • 0
  • 1
Description From Last Updated
  1. 
      
  2. Use a standard Python lib if we need to get the domain name?

    e.g. urlparse.urlsplit
    https://stackoverflow.com/questions/9626535/get-domain-name-from-url

    (that way we avoid all our custom code)

    1. Domain name isn't actually enough, we need the entire URL through the port, IE:

      http://cconner168680-1.gce.cloudera.com:14000
      https://cconner168680-1.gce.cloudera.com:8088

      I couldn't find a way to get that easily with urlparse, if there is a good way, then we should use it?

    2. And something like

      from urlparse import urlparse
      parsed_uri = urlparse('http://cconner168680-1.gce.cloudera.com:14000/questions/1234567/blah-blah-blah-blah')
      print '%(scheme)s://%(netloc)s' % {'scheme': parsed_uri.scheme, 'netloc': parsed_uri.netloc}

      ?

  3. 
      
  1. I'm doing some thorough testing today, but in the patch I provided, I had:

    if CACHE_SESSION is None or url not in CACHE_SESSION.adapters:
    CACHE_SESSION_LOCK.acquire()
    try:
    if CACHE_SESSION is None or url not in CACHE_SESSION.adapters:
    CACHE_SESSION = requests.Session()

    Which was a change from the original http_client.py and it's different from your patch. When we initiate the sessions.py, it automatically mounts "https://" and "http://", so our stack only ends up with:

    http://domain:<firstserviceport>
    http://
    https://

    SO the first check, which is typically HTTPFS gets in the stack along with http:// and https://, giving us this stack:

    http://cconner168680-1.gce.cloudera.com:14000
    http://
    https://

    Then all requests for RM and Oozie match the http:// or https:// entries in the stack. That is why I added the check on url not in CACHE_SESSION.adapters, to make sure we get those included.

    So far in my testing today, I did these tests:

    • No patch for 5 minutes with 14 jmeter users

    Saw ~1000 401 errors between Hue and backend services

    • Tested with this patch for 5 minutes with 14 jmeter users

    Saw ~800 401 errors between Hue and backend services

    • Tested with this patch plus the if changes mentioned above for 5 minutes with 14 jmeter users

    Saw 10 401 errors between Hue and backend services which were the initial expected 401 errors.

    So I'm pretty confident that we need those if statements....

    1. Done testing, I tested with your patch and with your patch plus the above if changes and the difference is extremely drastic, ~1000 less 401 occurrences in 5 minutes. So we definitely need those changes.

  2. 
      
Review request changed

Status: Discarded

Loading...