HUE-6659 [import] Support RDBMs as input via Sqoop

Review Request #10896 - Created June 8, 2017 and updated

Information
Prachi Poddar
hue
HUE-6659
91837de...
Reviewers
romain

dhcp-172-16-2-143:hue prachi.poddar$ git whatchanged -15
commit 687c4aba9e52131efa99437adf4898182cced1ee
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Mon Jul 24 11:51:27 2017 -0700

HUE-6659 [import] Cleaned code based on the code review. Added rdbms.py. Replaced test connection checkbox with button

:100644 100644 4c059b36f5... 2dfea9c063... M apps/oozie/src/oozie/templates/editor2/gen/workflow-sqoop.xml.mako
:100644 100644 d4b63c8bb3... 23e530c1b4... M desktop/conf.dist/hue.ini
:100644 100644 08ddda3733... a8b56959cc... M desktop/conf/pseudo-distributed.ini.tmpl
:100644 100644 f076329ddd... d7b99b4941... M desktop/libs/indexer/src/indexer/api3.py
:000000 100644 0000000000... 17eac54ad1... A desktop/libs/indexer/src/indexer/indexers/rdbms.py
:100644 000000 e574059ff2... 0000000000... D desktop/libs/indexer/src/indexer/rdbms_indexer.py
:100644 100644 bec0acca4e... 4c0f04d415... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 5daa6bffe0... 4a04d93367... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 aa9ff75739... ca8e4a1f7c... M desktop/libs/indexer/src/indexer/urls.py
:100644 100644 644044d263... e08129aa66... M desktop/libs/librdbms/src/librdbms/server/oracle_lib.py

commit 255a42cf62d280bae744cc352552283abaefe3e4
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Sun Jul 16 23:11:36 2017 -0700

HUE-6659 [import] added hive as a new option in Rdbms data import.

:100644 100644 758e2fde9d... f076329ddd... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 30c18f7e15... 5daa6bffe0... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 c51bf27be5... aa9ff75739... M desktop/libs/indexer/src/indexer/urls.py

commit 8e12fb18274b5d02ced7ea6cdb0b75e1fa4e6b3a
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Wed Jul 12 13:04:08 2017 -0700

HUE-6659 [import] added test connection front-end for custom db and output options for external databases

:100644 100644 2625651265... 758e2fde9d... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 5cb2bc75f9... bec0acca4e... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 ad8c316c60... 30c18f7e15... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 ee3f36f76e... c51bf27be5... M desktop/libs/indexer/src/indexer/urls.py

commit 55bffb4c3540e1bcea59395e0b30542c0a56a04f
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Tue Jul 11 15:49:38 2017 -0700

HUE-6659 [importer] Database name and Table name dropdown populated successfully for config mode

:100644 100644 7101159479... 2625651265... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 a5b79ba3f5... 5cb2bc75f9... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 f424eaea54... ad8c316c60... M desktop/libs/indexer/src/indexer/templates/importer.mako

commit 5564220755d2e4aef5b1758df617dee2889eb95d
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Sat Jun 24 19:53:10 2017 -0700

HUE-6659 [import] added custom and config database options

:100644 100644 d87bbc5a66... 7101159479... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 d84e48f9f1... e574059ff2... M desktop/libs/indexer/src/indexer/rdbms_indexer.py
:100644 100644 85a4e33f3f... a5b79ba3f5... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 e93c132366... f424eaea54... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 1bc826c81d... ee3f36f76e... M desktop/libs/indexer/src/indexer/urls.py

commit c4958756defa8e1fc74932f201a3aeb836e8c97b
Author: Romain Rigaux romain@cloudera.com
Date: Fri Jun 23 15:30:58 2017 -0700

HUE-6659 [importer] Quick clean-up to submit sqoop as a task

:100644 100644 d30edd9c20... d87bbc5a66... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 760c3c26e4... e93c132366... M desktop/libs/indexer/src/indexer/templates/importer.mako

commit 9aefe252ed09a7a9d2672ef2449313604b3b5e1f
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Sun Jun 11 19:13:24 2017 -0700

HUE-6659 [import] Support RDBMs as input via Sqoop

:100644 100644 ae6c2de88d... a9b53ed326... M apps/oozie/src/oozie/models2.py
:100644 100644 ccb4242c9c... d30edd9c20... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 ec0f517cff... a4036bcef9... M desktop/libs/indexer/src/indexer/conf.py
:000000 100644 0000000000... d84e48f9f1... A desktop/libs/indexer/src/indexer/rdbms_indexer.py
:000000 100644 0000000000... 85a4e33f3f... A desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 11b9ecedef... 760c3c26e4... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 76722d8b30... 02ef9f522a... M desktop/libs/notebook/src/notebook/connectors/base.py

commit a3ff7af21da91b035f927ba531849f5371af1224
Author: Romain Rigaux romain@cloudera.com
Date: Fri Jul 28 11:12:05 2017 -0700

HUE-5217 [backend] Resolve LGPL copyleft issue - Paramiko in boto 2.46.1

:100644 000000 1eb9db47d5... 0000000000... D desktop/core/ext-py/boto-2.46.1/boto/mashups/interactive.py

commit 7b15b90eb3d61249e03731dccd702cdfb2525cb4
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 17:26:49 2017 -0700

HUE-6968 [metadata] Gather column stats via Impala

:100644 100644 e3a936c3b3... 139469a596... M apps/beeswax/src/beeswax/server/dbms.py
:100644 100644 d34b433770... 66089ead57... M desktop/libs/metadata/src/metadata/optimizer_api.py

commit 9b715711e290702233c08a0afdcb0e77c21cd518
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 15:41:11 2017 -0700

HUE-6968 [metadata] Gather table stats via Impala

:100644 100644 82eb8aa1b6... 6a8a9bddd1... M apps/beeswax/src/beeswax/views.py
:100644 100644 75e098622e... d34b433770... M desktop/libs/metadata/src/metadata/optimizer_api.py

commit d4f453f6664256c56444303eb47c54597ba1e2ad
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:53:20 2017 -0700

HUE-6954 [indexer] Support different number of shards, replication at index creation

:100644 100644 ac39c6272e... ccb4242c9c... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 9779cfdccd... 5324a07482... M desktop/libs/indexer/src/indexer/solr_client.py
:100644 100644 2587fe9742... 11b9ecedef... M desktop/libs/indexer/src/indexer/templates/importer.mako

commit ff6fe4e70ecc99283adb94175f6dce114a711c76
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:46:55 2017 -0700

HUE-6974 [importer] Refresh Assist collection list after create table wizard success

When not using the batch indexer, we submit synchronously so we now it was created.

:100644 100644 4269d8bdc0... ac39c6272e... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 0947ee1f24... 2587fe9742... M desktop/libs/indexer/src/indexer/templates/importer.mako

commit e273d3347ad97cd4ed9084c21d9fa9c90a314573
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:38:43 2017 -0700

HUE-6973 [indexer] Open up the index page after creating a new index

:100644 100644 8ce1dce1bf... 4269d8bdc0... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 1ca429779b... 1bc826c81d... M desktop/libs/indexer/src/indexer/urls.py

commit 048b7b944d15cc0e3af094a3561379ceac3abf9f
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:24:22 2017 -0700

HUE-6972 [metadata] Sometimes the DB owner is shown as ()

:100644 100644 272a1991ad... 0f3f469181... M apps/metastore/src/metastore/templates/metastore.mako

commit ac41d006e5d1b7f8b12f6b8907430b8a976ec626
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:07:07 2017 -0700

HUE-6971 [jb] Schedule list is filtered by default on 7 days which hides most of them

:100644 100644 56d84589b8... 27547054e9... M apps/jobbrowser/src/jobbrowser/apis/schedule_api.py
:100644 100644 cc1e79017a... 2cf28fa571... M apps/jobbrowser/src/jobbrowser/templates/job_browser.mako

Yes. Working fine

Issues

  • 152
  • 14
  • 0
  • 166
Description From Last Updated
Could you revert those changes? Romain Rigaux Romain Rigaux
Could you remove all the changes here? (the local ini to modify in dev mode is desktop/conf/pseudo-distributed.ini) Romain Rigaux Romain Rigaux
Could you follow the same conventions as in the project? e.g. Licence file, CamelCase for Class names, import sorted in ... Romain Rigaux Romain Rigaux
Could you move the logic into a RdbmsIndexer class with guess_format/get_sample_data methods? Romain Rigaux Romain Rigaux
Move to around line 18? (as is higher dependency than the module) Romain Rigaux Romain Rigaux
This should probably return an empty dictionary for now. Romain Rigaux Romain Rigaux
Is snippet used? Wouldn't it be the name of the configured database in the hue.ini? Actually the constructor of RdbmsIndexer ... Romain Rigaux Romain Rigaux
Asserts should live in the tests class Romain Rigaux Romain Rigaux
len() should not be needed Romain Rigaux Romain Rigaux
Extra lines can be removed Romain Rigaux Romain Rigaux
One new line Romain Rigaux Romain Rigaux
Should be removed as not attributes Romain Rigaux Romain Rigaux
return {} Romain Rigaux Romain Rigaux
return {} Romain Rigaux Romain Rigaux
Grouped together Romain Rigaux Romain Rigaux
Known of those are used, so could be removed and just have a table_name variable? Romain Rigaux Romain Rigaux
This is probably not working as we are not defining any views yet. Instead, you should create a RdbmsIndexer() object ... Romain Rigaux Romain Rigaux
nits: Two newlines Romain Rigaux Romain Rigaux
connect_credentials --> user Romain Rigaux Romain Rigaux
self.user = user Romain Rigaux Romain Rigaux
self.interpreter --> {'type': 'mysql'} Romain Rigaux Romain Rigaux
Remove Romain Rigaux Romain Rigaux
sample = assist.get_sample_data(....) if sample['status'] == 0: retur sample api3.py elif file_format['inputFormat'] == 'dbms': sample = DbmsIndexer(request.user, 'mysql').get_sample_data( database=file_format['databaseName'], table=file_format['tableName']) ... Romain Rigaux Romain Rigaux
native Python nose (Hue lib) indexer Romain Rigaux Romain Rigaux
RdbmsIndexer(user, db_conf_name) Romain Rigaux Romain Rigaux
obj --> indexer get_sample_data() --> get_sample_data(database='hue', table='auth_users') Romain Rigaux Romain Rigaux
from django.contrib.auth.models import User client = make_logged_in_client() user = User.objects.get(username='test') Romain Rigaux Romain Rigaux
Add on top? !/usr/bin/env python Romain Rigaux Romain Rigaux
2 new lines above? Romain Rigaux Romain Rigaux
To move below desktop Romain Rigaux Romain Rigaux
2 new lines above Romain Rigaux Romain Rigaux
RdbmsIndexerTests --> TestRdbmsIndexer Romain Rigaux Romain Rigaux
'root' --> 'test' Romain Rigaux Romain Rigaux
Can remove this one for now, it will be a test in the API later Romain Rigaux Romain Rigaux
import logging from notebook.connectors.rdbms import Assist from librdbms.server import dbms Romain Rigaux Romain Rigaux
import json import logging from django.contrib.auth.models import User from nose.tools import assert_equal, assert_false, assert_not_equal, assert_true from desktop.lib.django_test_util import make_logged_in_client from ... Romain Rigaux Romain Rigaux
from indexer.rdbms_indexer import ... Romain Rigaux Romain Rigaux
To surround by 2 lines on top/above Romain Rigaux Romain Rigaux
server=file_format['rdbmsName'] Romain Rigaux Romain Rigaux
databaseName --> rdbmsDatabaseName ... no columns Romain Rigaux Romain Rigaux
Can revert Romain Rigaux Romain Rigaux
Rdbms --> External Database? Romain Rigaux Romain Rigaux
Can remove this line Romain Rigaux Romain Rigaux
self.rdbmsName = ko.observable(''); e.g. mysql4, oracle4, aka [[section]] name self.rdbmsTable = ko.observable(''); e.g. hue.desktop_auth_users self.rdbmsTableName = ko.computed(function() { return self.rdbmsTable().indexOf('.') ... Romain Rigaux Romain Rigaux
Remove? Romain Rigaux Romain Rigaux
Remove? Romain Rigaux Romain Rigaux
Could we add a feature flag to enable the Sqoop feature in https://github.com/cloudera/hue/blob/master/desktop/libs/indexer/src/indexer/conf.py#L69? Then we could raise a SkipTest when ... Romain Rigaux Romain Rigaux
Same Romain Rigaux Romain Rigaux
file --> 'file' Romain Rigaux Romain Rigaux
Remove () ? Romain Rigaux Romain Rigaux
self.subscribe(function(val) { if (val) { vm.createWizard.guessFormat(); } resizeElements(); }); Romain Rigaux Romain Rigaux
Not used? Romain Rigaux Romain Rigaux
To add to both ini templates cf. previouscomments Romain Rigaux Romain Rigaux
To disable with SkipTest cf previous comments Romain Rigaux Romain Rigaux
Could we move those into a rdbms.py in https://github.com/cloudera/hue/tree/master/desktop/libs/indexer/src/indexer/indexers ? This code is also 90% the same, could you combine ... Romain Rigaux Romain Rigaux
remove print Romain Rigaux Romain Rigaux
Is this used? Romain Rigaux Romain Rigaux
Combine with just this condition? elif source['inputFormat'] == 'rdbms': if destination['outputFormat'] in ('file', 'hive', 'hbase'): job_handle = ... Romain Rigaux Romain Rigaux
Could you remove all the prints and dead comments? Romain Rigaux Romain Rigaux
Move to rdbms.py? Then should be easy to add tests for the sqoop command generation. Romain Rigaux Romain Rigaux
Spaces between all the + Romain Rigaux Romain Rigaux
Usually it gets clearers with 'import-all-tables --connect jdbc:%(rdbmsName)s ....' % { 'rdbmsName': rdbmsName, .... } Romain Rigaux Romain Rigaux
Move those files to https://github.com/cloudera/hue/tree/master/desktop/libs/indexer/src/indexer/indexers ? (and drop _indexer from the name) Romain Rigaux Romain Rigaux
New line after #25? Romain Rigaux Romain Rigaux
New line above, no new line between both 'from indexer' statements? Romain Rigaux Romain Rigaux
Could we keep the new line? Romain Rigaux Romain Rigaux
ifnot: or ! createWizard.source.isAllTables() (no need of == false) Romain Rigaux Romain Rigaux
Could we keep all the previous new lines here and below? Romain Rigaux Romain Rigaux
MySQL Romain Rigaux Romain Rigaux
Is this used and looks like a duplication from Destination? Romain Rigaux Romain Rigaux
Spaces after if / else / { ? Is this code actually correct/needed? Romain Rigaux Romain Rigaux
Is this correct? (it basically override all the field correctness/output existing validations) Romain Rigaux Romain Rigaux
Could we move it to the other module cf previous comment? Romain Rigaux Romain Rigaux
Could we merge both cf previous comments? Romain Rigaux Romain Rigaux
Not used by anything? Romain Rigaux Romain Rigaux
Keep newline? Romain Rigaux Romain Rigaux
Keep newline? Romain Rigaux Romain Rigaux
Could you move this with the other connectors? https://github.com/cloudera/hue/tree/master/desktop/libs/indexer/src/indexer/indexers e.g.: rdbms.py Romain Rigaux Romain Rigaux
Cleaner, e.g.: @classmethod def setup_class(cls): if not ENABLE_SQOOP.get(): raise SkipTest Romain Rigaux Romain Rigaux
We should also raise a SkipTest if there if no 'mysql' DB is configured in librdms. Romain Rigaux Romain Rigaux
Could we keep all the new lines below like before? Romain Rigaux Romain Rigaux
Same about new lines Romain Rigaux Romain Rigaux
Is this actually used? Romain Rigaux Romain Rigaux
} space Romain Rigaux Romain Rigaux
Hbase --> HBase Romain Rigaux Romain Rigaux
sqoop1: { placeHolder: '${ _("Example: import --connect jdbc:hsqldb:file:db.hsqldb --table TT --target-dir hdfs://localhost:8020/user/foo -m 1") }', snippetImage: '${ static("sqoop/art/icon_sqoop_48.png") }' }, Romain Rigaux Romain Rigaux
Moved above the #Deprecated? Romain Rigaux Romain Rigaux
Would you also add the setting in https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini#L1255 https://github.com/cloudera/hue/blob/master/desktop/conf/pseudo-distributed.ini.tmpl#L1258 ? Romain Rigaux Romain Rigaux
You might want to build 'import-all-tables --connect jdbc:%(rdbmsName)s://%(rdbmsHost)s:%(rdbmsPort)s/%(rdbmsDatabaseName)s --username %(rdbmsUserName)s+ --password %(rdbmsPassword)s once. then add the different parameters in the ... Romain Rigaux Romain Rigaux
Actually this is gone, rever? Romain Rigaux Romain Rigaux
Have created a JIRA sub-task for this Prachi Poddar Prachi Poddar
Need more clarification on this. Prachi Poddar Prachi Poddar
Cannot do that as the statement syntax changes with each condition. Prachi Poddar Prachi Poddar
Remove? Romain Rigaux Romain Rigaux
--connect jdbc:%(rdbmsName)s://%(rdbmsHost)s:%(rdbmsPort)s/%(rdbmsDatabaseName)s --username %(rdbmsUserName)s --password %(rdbmsPassword)s Looks the same everywhere? Romain Rigaux Romain Rigaux
Also add in desktop/conf/pseudo*ini? Romain Rigaux Romain Rigaux
+statement+ --> %(statement)s Romain Rigaux Romain Rigaux
if not rdbms.get_query_server_config(server='mysql'): seems safer? Romain Rigaux Romain Rigaux
How come we get []? Romain Rigaux Romain Rigaux
Why commented? Romain Rigaux Romain Rigaux
Removed? Romain Rigaux Romain Rigaux
and in desktop/conf/pseudo-distributed.ini? Romain Rigaux Romain Rigaux
All this new css looks like a merge issue? Romain Rigaux Romain Rigaux
source_type --> sourceType 3 times? Romain Rigaux Romain Rigaux
ifnot: createWizard.source.inputFormat() == 'rdbms' && createWizard.source.isAllTables(): to give more context? Romain Rigaux Romain Rigaux
Spaces after commas, e.g. ['table', 'database'...] Romain Rigaux Romain Rigaux
Delete? Romain Rigaux Romain Rigaux
table', 'hive space Romain Rigaux Romain Rigaux
Between a % if ENABLE_SQOOP.get(): ? Romain Rigaux Romain Rigaux
if (['file', 'hive', 'hbase'].indexOf(format.value) != -1 && wizard.source.inputFormat() != 'rdbms') { return false; } ? Romain Rigaux Romain Rigaux
sourceType Romain Rigaux Romain Rigaux
Here too https://github.com/cloudera/hue/blob/master/desktop/conf/pseudo-distributed.ini.tmpl Romain Rigaux Romain Rigaux
Let's add some basic testing for this in https://issues.cloudera.org/browse/HUE-6966 Romain Rigaux Romain Rigaux
conf.HDFS_CLUSTERS['default'].FS_DEFAULTFS.get() --> request.fs.fs_defaultfs Romain Rigaux Romain Rigaux
'/test'/ to remove Romain Rigaux Romain Rigaux
files=[], Romain Rigaux Romain Rigaux
input-smal --> input-small Romain Rigaux Romain Rigaux
Add 'rdbms' prefix everywhere? Romain Rigaux Romain Rigaux
isConnection --> rdbmsCheckConnection ? Romain Rigaux Romain Rigaux
Add {'name': 'File', 'value': 'file'}, too? Romain Rigaux Romain Rigaux
files = [] if destination.get('sqoopJobLibPath'): files.append({"path": destination['sqoopJobLibPath'], "type": "jar"}) ... files=files ... As we don't always have a file. Romain Rigaux Romain Rigaux
on_success_url depends if the output is a file, a table or a hbase table For Hive on_success_url = reverse('metastore:describe_table', kwargs={'database': ... Romain Rigaux Romain Rigaux
also if 'the input is rdbms' (as we can create a Hive table from a file but should not see ... Romain Rigaux Romain Rigaux
After the main commit, can look at 'property-csv-hdfs-files' component Romain Rigaux Romain Rigaux
'File' in Sqoop if? Romain Rigaux Romain Rigaux
Hbase --> HBase Romain Rigaux Romain Rigaux
This is only used in the tests and can be removed? Romain Rigaux Romain Rigaux
outputFormat() == 'file' ? Romain Rigaux Romain Rigaux
For now we could remove 'hbase' and the custom logic? (we will need to add a dedicated name chooser for ... Romain Rigaux Romain Rigaux
Should probably remove this if and we manage the logic in isValidDestination. (and same we can remove hbase for now) Romain Rigaux Romain Rigaux
Remove keep here? .. Romain Rigaux Romain Rigaux
.. and then remove the if here? Romain Rigaux Romain Rigaux
So now that I think about it, 'table' and 'hive' are the same? Could we remove 'hive'? Romain Rigaux Romain Rigaux
Yes, it is present in desktop/conf/pseudo-distributed.ini also Prachi Poddar Prachi Poddar
Won't it become redundant. 'File' should be an option irrespective of whether sqoop is enable or not. That's why I ... Prachi Poddar Prachi Poddar
equal {"type": "csv"} ? Romain Rigaux Romain Rigaux
'rdbms' --> 'manual' ? Romain Rigaux Romain Rigaux
For now: wizard.source.inputFormat() == 'rdbms' --> wizard.source.inputFormat() != 'file' Romain Rigaux Romain Rigaux
Removed? Romain Rigaux Romain Rigaux
displayOnlyFolders=true ? uploadFolder no used? Romain Rigaux Romain Rigaux
true --> false Romain Rigaux Romain Rigaux
Until we officially test/release it: enable_sqoop=false Romain Rigaux Romain Rigaux
format_ = {'data': [], 'status': 1, 'message': ''} name = None Romain Rigaux Romain Rigaux
except Exception: --> except Exception, e: Romain Rigaux Romain Rigaux
LOG.error('Error accessing the database %s', Exception) ---> message = 'Error accessing the database %s' % name LOG.warn(message, e) format['message'] = ... Romain Rigaux Romain Rigaux
Are all the str() really needed? (it should not as we print them in a string) Romain Rigaux Romain Rigaux
-m 1 We should create a follow-up jira to support a configurable number of mappers Romain Rigaux Romain Rigaux
'/hbase/#HBase/' + destination_table_name Romain Rigaux Romain Rigaux
uploadFolder: true, remove? (not an option IIRC) Romain Rigaux Romain Rigaux
"Connection Failed." --> "${ _('Connection Failed: ') }" + resp.message Romain Rigaux Romain Rigaux
Double # before --> "## enable_sqoop=false" Romain Rigaux Romain Rigaux
(message) --> " _(message) " with a leading underscore _ Romain Rigaux Romain Rigaux
Romain Rigaux
Prachi Poddar
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Prachi Poddar
Romain Rigaux
Romain Rigaux
Prachi Poddar
Romain Rigaux
Romain Rigaux
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Prachi Poddar
Prachi Poddar
Enrico Berti
Prachi Poddar
Enrico Berti
Prachi Poddar
Romain Rigaux
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Romain Rigaux
Prachi Poddar
Romain Rigaux
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Review request changed

Description:

-  

NOTE: We use uploadFolder option. User can create the folder on the go. It is useful in scanario when no seperate folder is readily available to give as sqoop target-dir. Otherwise, user will have to go to file browser and create one which will be time consuming.

-  
-  

-  
   

dhcp-172-16-2-143:hue prachi.poddar$ git whatchanged -15

    commit 687c4aba9e52131efa99437adf4898182cced1ee
    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Mon Jul 24 11:51:27 2017 -0700

   
   
HUE-6659 [import] Cleaned code based on the code review. Added rdbms.py. Replaced test connection checkbox with button
   
   
   

:100644 100644 4c059b36f5... 2dfea9c063... M apps/oozie/src/oozie/templates/editor2/gen/workflow-sqoop.xml.mako

    :100644 100644 d4b63c8bb3... 23e530c1b4... M desktop/conf.dist/hue.ini
    :100644 100644 08ddda3733... a8b56959cc... M desktop/conf/pseudo-distributed.ini.tmpl
    :100644 100644 f076329ddd... d7b99b4941... M desktop/libs/indexer/src/indexer/api3.py
    :000000 100644 0000000000... 17eac54ad1... A desktop/libs/indexer/src/indexer/indexers/rdbms.py
    :100644 000000 e574059ff2... 0000000000... D desktop/libs/indexer/src/indexer/rdbms_indexer.py
    :100644 100644 bec0acca4e... 4c0f04d415... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 5daa6bffe0... 4a04d93367... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 aa9ff75739... ca8e4a1f7c... M desktop/libs/indexer/src/indexer/urls.py
    :100644 100644 644044d263... e08129aa66... M desktop/libs/librdbms/src/librdbms/server/oracle_lib.py

   
   

commit 255a42cf62d280bae744cc352552283abaefe3e4

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Sun Jul 16 23:11:36 2017 -0700

   
   
HUE-6659 [import] added hive as a new option in Rdbms data import.
   
   
   

:100644 100644 758e2fde9d... f076329ddd... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 30c18f7e15... 5daa6bffe0... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 c51bf27be5... aa9ff75739... M desktop/libs/indexer/src/indexer/urls.py

   
   

commit 8e12fb18274b5d02ced7ea6cdb0b75e1fa4e6b3a

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Wed Jul 12 13:04:08 2017 -0700

   
   
HUE-6659 [import] added test connection front-end for custom db and output options for external databases
   
   
   

:100644 100644 2625651265... 758e2fde9d... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 5cb2bc75f9... bec0acca4e... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 ad8c316c60... 30c18f7e15... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 ee3f36f76e... c51bf27be5... M desktop/libs/indexer/src/indexer/urls.py

   
   

commit 55bffb4c3540e1bcea59395e0b30542c0a56a04f

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Tue Jul 11 15:49:38 2017 -0700

   
   
HUE-6659 [importer] Database name and Table name dropdown populated successfully for config mode
   
   
   

:100644 100644 7101159479... 2625651265... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 a5b79ba3f5... 5cb2bc75f9... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 f424eaea54... ad8c316c60... M desktop/libs/indexer/src/indexer/templates/importer.mako

   
   

commit 5564220755d2e4aef5b1758df617dee2889eb95d

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Sat Jun 24 19:53:10 2017 -0700

   
   
HUE-6659 [import] added custom and config database options
   
   
   

:100644 100644 d87bbc5a66... 7101159479... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 d84e48f9f1... e574059ff2... M desktop/libs/indexer/src/indexer/rdbms_indexer.py
    :100644 100644 85a4e33f3f... a5b79ba3f5... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 e93c132366... f424eaea54... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 1bc826c81d... ee3f36f76e... M desktop/libs/indexer/src/indexer/urls.py

   
   

commit c4958756defa8e1fc74932f201a3aeb836e8c97b

    Author: Romain Rigaux romain@cloudera.com
    Date: Fri Jun 23 15:30:58 2017 -0700

   
   
HUE-6659 [importer] Quick clean-up to submit sqoop as a task
   
   
   

:100644 100644 d30edd9c20... d87bbc5a66... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 760c3c26e4... e93c132366... M desktop/libs/indexer/src/indexer/templates/importer.mako

   
   

commit 9aefe252ed09a7a9d2672ef2449313604b3b5e1f

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Sun Jun 11 19:13:24 2017 -0700

   
   
HUE-6659 [import] Support RDBMs as input via Sqoop
   
   
   

:100644 100644 ae6c2de88d... a9b53ed326... M apps/oozie/src/oozie/models2.py

    :100644 100644 ccb4242c9c... d30edd9c20... M desktop/libs/indexer/src/indexer/api3.py
    :100644 100644 ec0f517cff... a4036bcef9... M desktop/libs/indexer/src/indexer/conf.py
    :000000 100644 0000000000... d84e48f9f1... A desktop/libs/indexer/src/indexer/rdbms_indexer.py
    :000000 100644 0000000000... 85a4e33f3f... A desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 11b9ecedef... 760c3c26e4... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 76722d8b30... 02ef9f522a... M desktop/libs/notebook/src/notebook/connectors/base.py

   
   

commit a3ff7af21da91b035f927ba531849f5371af1224

    Author: Romain Rigaux romain@cloudera.com
    Date: Fri Jul 28 11:12:05 2017 -0700

   
   
HUE-5217 [backend] Resolve LGPL copyleft issue - Paramiko in boto 2.46.1
   
   
   

:100644 000000 1eb9db47d5... 0000000000... D desktop/core/ext-py/boto-2.46.1/boto/mashups/interactive.py

   
   

commit 7b15b90eb3d61249e03731dccd702cdfb2525cb4

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 17:26:49 2017 -0700

   
   
HUE-6968 [metadata] Gather column stats via Impala
   
   
   

:100644 100644 e3a936c3b3... 139469a596... M apps/beeswax/src/beeswax/server/dbms.py

    :100644 100644 d34b433770... 66089ead57... M desktop/libs/metadata/src/metadata/optimizer_api.py

   
   

commit 9b715711e290702233c08a0afdcb0e77c21cd518

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 15:41:11 2017 -0700

   
   
HUE-6968 [metadata] Gather table stats via Impala
   
   
   

:100644 100644 82eb8aa1b6... 6a8a9bddd1... M apps/beeswax/src/beeswax/views.py

    :100644 100644 75e098622e... d34b433770... M desktop/libs/metadata/src/metadata/optimizer_api.py

   
   

commit d4f453f6664256c56444303eb47c54597ba1e2ad

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:53:20 2017 -0700

   
   
HUE-6954 [indexer] Support different number of shards, replication at index creation
   
   
   

:100644 100644 ac39c6272e... ccb4242c9c... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 9779cfdccd... 5324a07482... M desktop/libs/indexer/src/indexer/solr_client.py
    :100644 100644 2587fe9742... 11b9ecedef... M desktop/libs/indexer/src/indexer/templates/importer.mako

   
   

commit ff6fe4e70ecc99283adb94175f6dce114a711c76

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:46:55 2017 -0700

   
   
HUE-6974 [importer] Refresh Assist collection list after create table wizard success
   
   
When not using the batch indexer, we submit synchronously so we now it was created.
   
   
   

:100644 100644 4269d8bdc0... ac39c6272e... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 0947ee1f24... 2587fe9742... M desktop/libs/indexer/src/indexer/templates/importer.mako

   
   

commit e273d3347ad97cd4ed9084c21d9fa9c90a314573

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:38:43 2017 -0700

   
   
HUE-6973 [indexer] Open up the index page after creating a new index
   
   
   

:100644 100644 8ce1dce1bf... 4269d8bdc0... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 1ca429779b... 1bc826c81d... M desktop/libs/indexer/src/indexer/urls.py

   
   

commit 048b7b944d15cc0e3af094a3561379ceac3abf9f

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:24:22 2017 -0700

   
   
HUE-6972 [metadata] Sometimes the DB owner is shown as ()
   
   
   

:100644 100644 272a1991ad... 0f3f469181... M apps/metastore/src/metastore/templates/metastore.mako

   
   

commit ac41d006e5d1b7f8b12f6b8907430b8a976ec626

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:07:07 2017 -0700

   
   
HUE-6971 [jb] Schedule list is filtered by default on 7 days which hides most of them
   
   
   

:100644 100644 56d84589b8... 27547054e9... M apps/jobbrowser/src/jobbrowser/apis/schedule_api.py

    :100644 100644 cc1e79017a... 2cf28fa571... M apps/jobbrowser/src/jobbrowser/templates/job_browser.mako

Commit:

-687c4aba9e52131efa99437adf4898182cced1ee
+91837dec3643f0967cfd0a0744c368fa5a189428

Diff:

Revision 31 (+549 -36)

Show changes

Romain Rigaux

2 nits and just go for:

git fetch origina
git rebase origin/master

git push origin HEAD:master

desktop/conf/pseudo-distributed.ini.tmpl (Diff revision 31)
 
 

Double # before

-->

"## enable_sqoop=false"

(message)

-->

" _(message) "

with a leading underscore _

Loading...