HUE-7258 [jb] Add config check for Spark history server URL

Review Request #13430 - Created Oct. 9, 2018 and submitted

Ying Chen
hue
master
HUE-7258
hue
enricoberti, jgauthier, johan, romain, weixia
commit fbdc509c5556b0082c42af0e8c635b529f1d3fa9 (HEAD -> pam_backend)
Author: Ying Chen <yingchen@cloudera.com>
Date:   Mon Oct 8 17:18:00 2018 -0700

    HUE-7258 [jb] Add config check for Spark history server URL

:100644 100644 b8d261fca2... 2d1f2a138d... M    desktop/libs/hadoop/src/hadoop/conf.py
:100644 100644 acafd918bc... 65358e1d60... M    desktop/libs/metadata/src/metadata/manager_client.py


  • 0
  • 0
  • 7
  • 1
  • 8
Description From Last Updated
  1. Nice one, just on question

  2. desktop/libs/hadoop/src/hadoop/conf.py (Diff revision 1)
     
     

    Never used bind_to, would it be the same as something like cluster.SPARK_HISTORY_SERVER_URL.get() ?

    1. Yes, it's the string of the URL.
  3. 
      
  1. 
      
  2. desktop/libs/hadoop/src/hadoop/conf.py (Diff revision 2)
     
     

    Is there a way to know that the Spark service is intalled?
    (to not show the warning if Spark is not there)

    1. CM API has rest API, http://example.com:7180/api/v19/clusters/Cluster%201/services/ for type of "SPARK_ON_YARN", but we also need to host of CM and the name of cluster.
      
      I tried cluster.SPARK_HISTORY_SERVER_URL.get(), but this will always return it default value "localhost:18088"
  3. 
      
  1. Nice!

    Main comment: If the Spark History server is not defined in the ini but we can retrieve it via CM, could we use it automatically?

  2. nit:
    if service_display_names:

    is enough (as an empty list would be false)

  3. 
      
  1. If the Spark History server is not defined in the ini but we can retrieve it via CM, could we use it automatically?

    1. e.g.
      https://github.com/cloudera/hue/blob/master/desktop/libs/metadata/src/metadata/conf.py#L79

  2. 
      
  1. 
      
  2. desktop/libs/metadata/src/metadata/manager_client.py (Diff revision 6)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    https://github.com/cloudera/hue/blob/master/desktop/libs/metadata/src/metadata/manager_client.py#L110

  3. 
      
  1. 
      
  2. Rename

    has_spark_history_server --> get_spark_history_server_url

    ?

  3. There is the same in desktop/libs/hadoop/src/hadoop/conf.py?

  4. 
      
Review request changed

Status: Closed (submitted)

Loading...