- Download Diff

Summary:

HUE-7258 [jb] Add config check for Spark history server URL

Review Request #13430 - Created Oct. 8, 2018 and submitted Jan. 8, 2019, 12:51 p.m.

Information
Owner:	Ying Chen
Repository:	hue
Branch:	master
Bugs:	HUE-7258
Depends On:
Reviewers
Groups:	hue
People:	enricoberti, jgauthier, johan, romain, weixia

Description

commit fbdc509c5556b0082c42af0e8c635b529f1d3fa9 (HEAD -> pam_backend)
Author: Ying Chen <yingchen@cloudera.com>
Date:   Mon Oct 8 17:18:00 2018 -0700

    HUE-7258 [jb] Add config check for Spark history server URL

:100644 100644 b8d261fca2... 2d1f2a138d... M    desktop/libs/hadoop/src/hadoop/conf.py
:100644 100644 acafd918bc... 65358e1d60... M    desktop/libs/metadata/src/metadata/manager_client.py

Testing Done

Issues

0
0
7
1
All Issues: 8

Description	From	Last Updated
Never used bind_to, would it be the same as something like cluster.SPARK_HISTORY_SERVER_URL.get() ?	Romain Rigaux	Oct. 9, 2018, 2:08 p.m.
Is there a way to know that the Spark service is intalled? (to not show the warning if Spark is ...	Romain Rigaux	Nov. 6, 2018, 1:39 a.m.
nit: if service_display_names: is enough (as an empty list would be false)	Romain Rigaux	Nov. 13, 2018, 6:18 p.m.
nit: ditto	Romain Rigaux	Nov. 13, 2018, 6:18 p.m.
debug --> warn?	Romain Rigaux	Nov. 13, 2018, 6:18 p.m.
https://github.com/cloudera/hue/blob/master/desktop/libs/metadata/src/metadata/manager_client.py#L110	Romain Rigaux	Dec. 7, 2018, 12:20 p.m.
Rename has_spark_history_server --> get_spark_history_server_url ?	Romain Rigaux	Jan. 8, 2019, 12:33 p.m.
There is the same in desktop/libs/hadoop/src/hadoop/conf.py?	Romain Rigaux	Jan. 8, 2019, 12:33 p.m.

```
Nice one, just on question
```

desktop/libs/hadoop/src/hadoop/conf.py (Diff revision 1)

Show all issues

Never used bind_to, would it be the same as something like cluster.SPARK_HISTORY_SERVER_URL.get() ?

Ying Chen Oct. 9, 2018, 2:08 p.m.
```
Yes, it's the string of the URL.
```

Diff:

Revision 2 (+18)

Show changes

desktop/libs/hadoop/src/hadoop/conf.py

desktop/libs/hadoop/src/hadoop/conf.py (Diff revision 2)

Show all issues

Is there a way to know that the Spark service is intalled?
(to not show the warning if Spark is not there)

Ying Chen Oct. 10, 2018, 11:31 a.m.

CM API has rest API, http://example.com:7180/api/v19/clusters/Cluster%201/services/ for type of "SPARK_ON_YARN", but we also need to host of CM and the name of cluster.

I tried cluster.SPARK_HISTORY_SERVER_URL.get(), but this will always return it default value "localhost:18088"

Diff:

Revision 3 (+60 -1)

Show changes

	desktop/libs/hadoop/src/hadoop/conf.py
	desktop/libs/metadata/src/metadata/manager_client.py

Nice!

Main comment: If the Spark History server is not defined in the ini but we can retrieve it via CM, could we use it automatically?

desktop/libs/metadata/src/metadata/manager_client.py (Diff revision 3)

Show all issues

nit:

if service_display_names:

is enough (as an empty list would be false)

desktop/libs/metadata/src/metadata/manager_client.py (Diff revision 3)
Show all issues
```
nit: ditto
```
desktop/libs/metadata/src/metadata/manager_client.py (Diff revision 3)
Show all issues
```
debug --> warn?
```

Diff:

Revision 4 (+60 -1)

Show changes

	desktop/libs/hadoop/src/hadoop/conf.py
	desktop/libs/metadata/src/metadata/manager_client.py

Description:

~		commit ed9f43080264330c75f417191c47a353a7957f31 (HEAD -> master)
	~	commit c153f4f26ceb95112fb2bc9b26616977c4239a9c (HEAD -> master)
		Author: Ying Chen yingchen@cloudera.com
		Date: Mon Oct 8 17:18:00 2018 -0700

		HUE-7258 [jb] Add config check for Spark history server URL

~		:100644 100644 b8d261fca2... 65aeab0850... M desktop/libs/hadoop/src/hadoop/conf.py
	~	:100644 100644 b8d261fca2... 6505016071... M desktop/libs/hadoop/src/hadoop/conf.py
	+	:100644 100644 acafd918bc... 11ece0f743... M desktop/libs/metadata/src/metadata/manager_client.py

If the Spark History server is not defined in the ini but we can retrieve it via CM, could we use it automatically?

Romain Rigaux Nov. 27, 2018, 3:47 p.m.

e.g.
https://github.com/cloudera/hue/blob/master/desktop/libs/metadata/src/metadata/conf.py#L79

Diff:

Revision 5 (+112 -2)

Show changes

	desktop/libs/hadoop/src/hadoop/conf.py
	desktop/libs/metadata/src/metadata/manager_client.py

Description:

~		commit c153f4f26ceb95112fb2bc9b26616977c4239a9c (HEAD -> master)
	~	commit 97e8c9f8bc7a7607b543e0441d4ce82b4f54a0e7 (HEAD -> master)
		Author: Ying Chen yingchen@cloudera.com
		Date: Mon Oct 8 17:18:00 2018 -0700

		HUE-7258 [jb] Add config check for Spark history server URL

~		:100644 100644 b8d261fca2... 6505016071... M desktop/libs/hadoop/src/hadoop/conf.py
~		:100644 100644 acafd918bc... 11ece0f743... M desktop/libs/metadata/src/metadata/manager_client.py
	~	:100644 100644 b8d261fca2... b5725ff404... M desktop/libs/hadoop/src/hadoop/conf.py
	~	:100644 100644 acafd918bc... 3ef3c6401d... M desktop/libs/metadata/src/metadata/manager_client.py

Diff:

Revision 6 (+114 -2)

Show changes

	desktop/libs/hadoop/src/hadoop/conf.py
	desktop/libs/metadata/src/metadata/manager_client.py

desktop/libs/metadata/src/metadata/manager_client.py (Diff revision 6)

Show all issues

https://github.com/cloudera/hue/blob/master/desktop/libs/metadata/src/metadata/manager_client.py#L110

Description:

~		commit 97e8c9f8bc7a7607b543e0441d4ce82b4f54a0e7 (HEAD -> master)
	~	commit c1179d6dbe5d3863c5fd53320557b16d458e6464 (HEAD -> pam_backend)
		Author: Ying Chen yingchen@cloudera.com
		Date: Mon Oct 8 17:18:00 2018 -0700

		HUE-7258 [jb] Add config check for Spark history server URL

		:100644 100644 b8d261fca2... b5725ff404... M desktop/libs/hadoop/src/hadoop/conf.py
~		:100644 100644 acafd918bc... 3ef3c6401d... M desktop/libs/metadata/src/metadata/manager_client.py
	~	:100644 100644 acafd918bc... 779cd22835... M desktop/libs/metadata/src/metadata/manager_client.py

Diff:

Revision 7 (+112 -2)

Show changes

	desktop/libs/hadoop/src/hadoop/conf.py
	desktop/libs/metadata/src/metadata/manager_client.py

Ship it!

desktop/libs/metadata/src/metadata/manager_client.py (Diff revision 7)
Show all issues
```
Rename 
has_spark_history_server --> get_spark_history_server_url
?
```
desktop/libs/metadata/src/metadata/manager_client.py (Diff revision 7)
Show all issues
```
There is the same in desktop/libs/hadoop/src/hadoop/conf.py?
```

Description:

~		commit c1179d6dbe5d3863c5fd53320557b16d458e6464 (HEAD -> pam_backend)
	~	commit fbdc509c5556b0082c42af0e8c635b529f1d3fa9 (HEAD -> pam_backend)
		Author: Ying Chen yingchen@cloudera.com
		Date: Mon Oct 8 17:18:00 2018 -0700

		HUE-7258 [jb] Add config check for Spark history server URL

~		:100644 100644 b8d261fca2... b5725ff404... M desktop/libs/hadoop/src/hadoop/conf.py
~		:100644 100644 acafd918bc... 779cd22835... M desktop/libs/metadata/src/metadata/manager_client.py
	~	:100644 100644 b8d261fca2... 2d1f2a138d... M desktop/libs/hadoop/src/hadoop/conf.py
	~	:100644 100644 acafd918bc... 65358e1d60... M desktop/libs/metadata/src/metadata/manager_client.py

Diff:

Revision 8 (+108 -2)

Show changes

	desktop/libs/hadoop/src/hadoop/conf.py
	desktop/libs/metadata/src/metadata/manager_client.py

You have a pending review.

Review Board 3.0.3

HUE-7258 [jb] Add config check for Spark history server URL

Diff:

Diff:

Diff:

Description:

Diff:

Description:

Diff:

Description:

Diff:

Description:

Diff:

Status: Closed (submitted)