HUE-7258 [jb] Add config check for Spark history server URL

Review Request #13430 - Created Oct. 8, 2018 and updated

Ying Chen
hue
master
HUE-7258
hue
enricoberti, jgauthier, johan, romain, weixia
commit c1179d6dbe5d3863c5fd53320557b16d458e6464 (HEAD -> pam_backend)
Author: Ying Chen <yingchen@cloudera.com>
Date:   Mon Oct 8 17:18:00 2018 -0700

    HUE-7258 [jb] Add config check for Spark history server URL

:100644 100644 b8d261fca2... b5725ff404... M    desktop/libs/hadoop/src/hadoop/conf.py
:100644 100644 acafd918bc... 779cd22835... M    desktop/libs/metadata/src/metadata/manager_client.py


  • 0
  • 0
  • 5
  • 1
  • 6
Description From Last Updated
  1. Nice one, just on question

  2. desktop/libs/hadoop/src/hadoop/conf.py (Diff revision 1)
     
     

    Never used bind_to, would it be the same as something like cluster.SPARK_HISTORY_SERVER_URL.get() ?

    1. Yes, it's the string of the URL.
  3. 
      
  1. 
      
  2. desktop/libs/hadoop/src/hadoop/conf.py (Diff revision 2)
     
     

    Is there a way to know that the Spark service is intalled?
    (to not show the warning if Spark is not there)

    1. CM API has rest API, http://example.com:7180/api/v19/clusters/Cluster%201/services/ for type of "SPARK_ON_YARN", but we also need to host of CM and the name of cluster.
      
      I tried cluster.SPARK_HISTORY_SERVER_URL.get(), but this will always return it default value "localhost:18088"
  3. 
      
  1. Nice!

    Main comment: If the Spark History server is not defined in the ini but we can retrieve it via CM, could we use it automatically?

  2. nit:
    if service_display_names:

    is enough (as an empty list would be false)

  3. 
      
  1. If the Spark History server is not defined in the ini but we can retrieve it via CM, could we use it automatically?

    1. e.g.
      https://github.com/cloudera/hue/blob/master/desktop/libs/metadata/src/metadata/conf.py#L79

  2. 
      
  1. 
      
  2. desktop/libs/metadata/src/metadata/manager_client.py (Diff revision 6)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    https://github.com/cloudera/hue/blob/master/desktop/libs/metadata/src/metadata/manager_client.py#L110

  3. 
      
Review request changed

Description:

~  

commit 97e8c9f8bc7a7607b543e0441d4ce82b4f54a0e7 (HEAD -> master)

  ~

commit c1179d6dbe5d3863c5fd53320557b16d458e6464 (HEAD -> pam_backend)

    Author: Ying Chen yingchen@cloudera.com
    Date: Mon Oct 8 17:18:00 2018 -0700

   
   
HUE-7258 [jb] Add config check for Spark history server URL
   
   

:100644 100644 b8d261fca2... b5725ff404... M desktop/libs/hadoop/src/hadoop/conf.py

~   :100644 100644 acafd918bc... 3ef3c6401d... M desktop/libs/metadata/src/metadata/manager_client.py

  ~ :100644 100644 acafd918bc... 779cd22835... M desktop/libs/metadata/src/metadata/manager_client.py

Diff:

Revision 7 (+112 -2)

Show changes

Loading...