HUE-6659 [import] Support RDBMs as input via Sqoop

Review Request #10896 - Created June 8, 2017 and updated

Prachi Poddar
hue
HUE-6659
91837de...
romain

dhcp-172-16-2-143:hue prachi.poddar$ git whatchanged -15
commit 687c4aba9e52131efa99437adf4898182cced1ee
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Mon Jul 24 11:51:27 2017 -0700

HUE-6659 [import] Cleaned code based on the code review. Added rdbms.py. Replaced test connection checkbox with button

:100644 100644 4c059b36f5... 2dfea9c063... M apps/oozie/src/oozie/templates/editor2/gen/workflow-sqoop.xml.mako
:100644 100644 d4b63c8bb3... 23e530c1b4... M desktop/conf.dist/hue.ini
:100644 100644 08ddda3733... a8b56959cc... M desktop/conf/pseudo-distributed.ini.tmpl
:100644 100644 f076329ddd... d7b99b4941... M desktop/libs/indexer/src/indexer/api3.py
:000000 100644 0000000000... 17eac54ad1... A desktop/libs/indexer/src/indexer/indexers/rdbms.py
:100644 000000 e574059ff2... 0000000000... D desktop/libs/indexer/src/indexer/rdbms_indexer.py
:100644 100644 bec0acca4e... 4c0f04d415... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 5daa6bffe0... 4a04d93367... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 aa9ff75739... ca8e4a1f7c... M desktop/libs/indexer/src/indexer/urls.py
:100644 100644 644044d263... e08129aa66... M desktop/libs/librdbms/src/librdbms/server/oracle_lib.py

commit 255a42cf62d280bae744cc352552283abaefe3e4
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Sun Jul 16 23:11:36 2017 -0700

HUE-6659 [import] added hive as a new option in Rdbms data import.

:100644 100644 758e2fde9d... f076329ddd... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 30c18f7e15... 5daa6bffe0... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 c51bf27be5... aa9ff75739... M desktop/libs/indexer/src/indexer/urls.py

commit 8e12fb18274b5d02ced7ea6cdb0b75e1fa4e6b3a
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Wed Jul 12 13:04:08 2017 -0700

HUE-6659 [import] added test connection front-end for custom db and output options for external databases

:100644 100644 2625651265... 758e2fde9d... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 5cb2bc75f9... bec0acca4e... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 ad8c316c60... 30c18f7e15... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 ee3f36f76e... c51bf27be5... M desktop/libs/indexer/src/indexer/urls.py

commit 55bffb4c3540e1bcea59395e0b30542c0a56a04f
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Tue Jul 11 15:49:38 2017 -0700

HUE-6659 [importer] Database name and Table name dropdown populated successfully for config mode

:100644 100644 7101159479... 2625651265... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 a5b79ba3f5... 5cb2bc75f9... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 f424eaea54... ad8c316c60... M desktop/libs/indexer/src/indexer/templates/importer.mako

commit 5564220755d2e4aef5b1758df617dee2889eb95d
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Sat Jun 24 19:53:10 2017 -0700

HUE-6659 [import] added custom and config database options

:100644 100644 d87bbc5a66... 7101159479... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 d84e48f9f1... e574059ff2... M desktop/libs/indexer/src/indexer/rdbms_indexer.py
:100644 100644 85a4e33f3f... a5b79ba3f5... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 e93c132366... f424eaea54... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 1bc826c81d... ee3f36f76e... M desktop/libs/indexer/src/indexer/urls.py

commit c4958756defa8e1fc74932f201a3aeb836e8c97b
Author: Romain Rigaux romain@cloudera.com
Date: Fri Jun 23 15:30:58 2017 -0700

HUE-6659 [importer] Quick clean-up to submit sqoop as a task

:100644 100644 d30edd9c20... d87bbc5a66... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 760c3c26e4... e93c132366... M desktop/libs/indexer/src/indexer/templates/importer.mako

commit 9aefe252ed09a7a9d2672ef2449313604b3b5e1f
Author: Prachi Poddar prachi.poddar@cloudera.com
Date: Sun Jun 11 19:13:24 2017 -0700

HUE-6659 [import] Support RDBMs as input via Sqoop

:100644 100644 ae6c2de88d... a9b53ed326... M apps/oozie/src/oozie/models2.py
:100644 100644 ccb4242c9c... d30edd9c20... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 ec0f517cff... a4036bcef9... M desktop/libs/indexer/src/indexer/conf.py
:000000 100644 0000000000... d84e48f9f1... A desktop/libs/indexer/src/indexer/rdbms_indexer.py
:000000 100644 0000000000... 85a4e33f3f... A desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
:100644 100644 11b9ecedef... 760c3c26e4... M desktop/libs/indexer/src/indexer/templates/importer.mako
:100644 100644 76722d8b30... 02ef9f522a... M desktop/libs/notebook/src/notebook/connectors/base.py

commit a3ff7af21da91b035f927ba531849f5371af1224
Author: Romain Rigaux romain@cloudera.com
Date: Fri Jul 28 11:12:05 2017 -0700

HUE-5217 [backend] Resolve LGPL copyleft issue - Paramiko in boto 2.46.1

:100644 000000 1eb9db47d5... 0000000000... D desktop/core/ext-py/boto-2.46.1/boto/mashups/interactive.py

commit 7b15b90eb3d61249e03731dccd702cdfb2525cb4
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 17:26:49 2017 -0700

HUE-6968 [metadata] Gather column stats via Impala

:100644 100644 e3a936c3b3... 139469a596... M apps/beeswax/src/beeswax/server/dbms.py
:100644 100644 d34b433770... 66089ead57... M desktop/libs/metadata/src/metadata/optimizer_api.py

commit 9b715711e290702233c08a0afdcb0e77c21cd518
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 15:41:11 2017 -0700

HUE-6968 [metadata] Gather table stats via Impala

:100644 100644 82eb8aa1b6... 6a8a9bddd1... M apps/beeswax/src/beeswax/views.py
:100644 100644 75e098622e... d34b433770... M desktop/libs/metadata/src/metadata/optimizer_api.py

commit d4f453f6664256c56444303eb47c54597ba1e2ad
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:53:20 2017 -0700

HUE-6954 [indexer] Support different number of shards, replication at index creation

:100644 100644 ac39c6272e... ccb4242c9c... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 9779cfdccd... 5324a07482... M desktop/libs/indexer/src/indexer/solr_client.py
:100644 100644 2587fe9742... 11b9ecedef... M desktop/libs/indexer/src/indexer/templates/importer.mako

commit ff6fe4e70ecc99283adb94175f6dce114a711c76
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:46:55 2017 -0700

HUE-6974 [importer] Refresh Assist collection list after create table wizard success

When not using the batch indexer, we submit synchronously so we now it was created.

:100644 100644 4269d8bdc0... ac39c6272e... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 0947ee1f24... 2587fe9742... M desktop/libs/indexer/src/indexer/templates/importer.mako

commit e273d3347ad97cd4ed9084c21d9fa9c90a314573
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:38:43 2017 -0700

HUE-6973 [indexer] Open up the index page after creating a new index

:100644 100644 8ce1dce1bf... 4269d8bdc0... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 1ca429779b... 1bc826c81d... M desktop/libs/indexer/src/indexer/urls.py

commit 048b7b944d15cc0e3af094a3561379ceac3abf9f
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:24:22 2017 -0700

HUE-6972 [metadata] Sometimes the DB owner is shown as ()

:100644 100644 272a1991ad... 0f3f469181... M apps/metastore/src/metastore/templates/metastore.mako

commit ac41d006e5d1b7f8b12f6b8907430b8a976ec626
Author: Romain Rigaux romain@cloudera.com
Date: Thu Jul 27 14:07:07 2017 -0700

HUE-6971 [jb] Schedule list is filtered by default on 7 days which hides most of them

:100644 100644 56d84589b8... 27547054e9... M apps/jobbrowser/src/jobbrowser/apis/schedule_api.py
:100644 100644 cc1e79017a... 2cf28fa571... M apps/jobbrowser/src/jobbrowser/templates/job_browser.mako

Yes. Working fine

  • 152
  • 0
  • 14
  • 0
  • 166
Description From Last Updated
Could you revert those changes? Romain Rigaux
Could you remove all the changes here? (the local ini to modify in dev mode is desktop/conf/pseudo-distributed.ini) Romain Rigaux
Could you follow the same conventions as in the project? e.g. Licence file, CamelCase for Class names, import sorted in ... Romain Rigaux
Could you move the logic into a RdbmsIndexer class with guess_format/get_sample_data methods? Romain Rigaux
Move to around line 18? (as is higher dependency than the module) Romain Rigaux
This should probably return an empty dictionary for now. Romain Rigaux
Is snippet used? Wouldn't it be the name of the configured database in the hue.ini? Actually the constructor of RdbmsIndexer ... Romain Rigaux
Asserts should live in the tests class Romain Rigaux
len() should not be needed Romain Rigaux
Extra lines can be removed Romain Rigaux
One new line Romain Rigaux
Should be removed as not attributes Romain Rigaux
return {} Romain Rigaux
return {} Romain Rigaux
Grouped together Romain Rigaux
Known of those are used, so could be removed and just have a table_name variable? Romain Rigaux
This is probably not working as we are not defining any views yet. Instead, you should create a RdbmsIndexer() object ... Romain Rigaux
nits: Two newlines Romain Rigaux
connect_credentials --> user Romain Rigaux
self.user = user Romain Rigaux
self.interpreter --> {'type': 'mysql'} Romain Rigaux
Remove Romain Rigaux
sample = assist.get_sample_data(....) if sample['status'] == 0: retur sample api3.py elif file_format['inputFormat'] == 'dbms': sample = DbmsIndexer(request.user, 'mysql').get_sample_data( database=file_format['databaseName'], table=file_format['tableName']) ... Romain Rigaux
native Python nose (Hue lib) indexer Romain Rigaux
RdbmsIndexer(user, db_conf_name) Romain Rigaux
obj --> indexer get_sample_data() --> get_sample_data(database='hue', table='auth_users') Romain Rigaux
from django.contrib.auth.models import User client = make_logged_in_client() user = User.objects.get(username='test') Romain Rigaux
Add on top? !/usr/bin/env python Romain Rigaux
2 new lines above? Romain Rigaux
To move below desktop Romain Rigaux
2 new lines above Romain Rigaux
RdbmsIndexerTests --> TestRdbmsIndexer Romain Rigaux
'root' --> 'test' Romain Rigaux
Can remove this one for now, it will be a test in the API later Romain Rigaux
import logging from notebook.connectors.rdbms import Assist from librdbms.server import dbms Romain Rigaux
import json import logging from django.contrib.auth.models import User from nose.tools import assert_equal, assert_false, assert_not_equal, assert_true from desktop.lib.django_test_util import make_logged_in_client from ... Romain Rigaux
from indexer.rdbms_indexer import ... Romain Rigaux
To surround by 2 lines on top/above Romain Rigaux
server=file_format['rdbmsName'] Romain Rigaux
databaseName --> rdbmsDatabaseName ... no columns Romain Rigaux
Can revert Romain Rigaux
Rdbms --> External Database? Romain Rigaux
Can remove this line Romain Rigaux
self.rdbmsName = ko.observable(''); e.g. mysql4, oracle4, aka [[section]] name self.rdbmsTable = ko.observable(''); e.g. hue.desktop_auth_users self.rdbmsTableName = ko.computed(function() { return self.rdbmsTable().indexOf('.') ... Romain Rigaux
Remove? Romain Rigaux
Remove? Romain Rigaux
Could we add a feature flag to enable the Sqoop feature in https://github.com/cloudera/hue/blob/master/desktop/libs/indexer/src/indexer/conf.py#L69? Then we could raise a SkipTest when ... Romain Rigaux
Same Romain Rigaux
file --> 'file' Romain Rigaux
self.subscribe(function(val) { if (val) { vm.createWizard.guessFormat(); } resizeElements(); }); Romain Rigaux
Remove () ? Romain Rigaux
Not used? Romain Rigaux
To add to both ini templates cf. previouscomments Romain Rigaux
To disable with SkipTest cf previous comments Romain Rigaux
Could we move those into a rdbms.py in https://github.com/cloudera/hue/tree/master/desktop/libs/indexer/src/indexer/indexers ? This code is also 90% the same, could you combine ... Romain Rigaux
remove print Romain Rigaux
Is this used? Romain Rigaux
Combine with just this condition? elif source['inputFormat'] == 'rdbms': if destination['outputFormat'] in ('file', 'hive', 'hbase'): job_handle = ... Romain Rigaux
Could you remove all the prints and dead comments? Romain Rigaux
Move to rdbms.py? Then should be easy to add tests for the sqoop command generation. Romain Rigaux
Spaces between all the + Romain Rigaux
Usually it gets clearers with 'import-all-tables --connect jdbc:%(rdbmsName)s ....' % { 'rdbmsName': rdbmsName, .... } Romain Rigaux
Move those files to https://github.com/cloudera/hue/tree/master/desktop/libs/indexer/src/indexer/indexers ? (and drop _indexer from the name) Romain Rigaux
New line after #25? Romain Rigaux
New line above, no new line between both 'from indexer' statements? Romain Rigaux
Could we keep the new line? Romain Rigaux
ifnot: or ! createWizard.source.isAllTables() (no need of == false) Romain Rigaux
Could we keep all the previous new lines here and below? Romain Rigaux
MySQL Romain Rigaux
Is this used and looks like a duplication from Destination? Romain Rigaux
Spaces after if / else / { ? Is this code actually correct/needed? Romain Rigaux
Is this correct? (it basically override all the field correctness/output existing validations) Romain Rigaux
Could we move it to the other module cf previous comment? Romain Rigaux
Could we merge both cf previous comments? Romain Rigaux
Not used by anything? Romain Rigaux
Keep newline? Romain Rigaux
Keep newline? Romain Rigaux
Have created a JIRA sub-task for this Prachi Poddar
Could you move this with the other connectors? https://github.com/cloudera/hue/tree/master/desktop/libs/indexer/src/indexer/indexers e.g.: rdbms.py Romain Rigaux
Need more clarification on this. Prachi Poddar
Cleaner, e.g.: @classmethod def setup_class(cls): if not ENABLE_SQOOP.get(): raise SkipTest Romain Rigaux
We should also raise a SkipTest if there if no 'mysql' DB is configured in librdms. Romain Rigaux
Could we keep all the new lines below like before? Romain Rigaux
Same about new lines Romain Rigaux
Is this actually used? Romain Rigaux
} space Romain Rigaux
Hbase --> HBase Romain Rigaux
sqoop1: { placeHolder: '${ _("Example: import --connect jdbc:hsqldb:file:db.hsqldb --table TT --target-dir hdfs://localhost:8020/user/foo -m 1") }', snippetImage: '${ static("sqoop/art/icon_sqoop_48.png") }' }, Romain Rigaux
Moved above the #Deprecated? Romain Rigaux
Remove? Romain Rigaux
Would you also add the setting in https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini#L1255 https://github.com/cloudera/hue/blob/master/desktop/conf/pseudo-distributed.ini.tmpl#L1258 ? Romain Rigaux
You might want to build 'import-all-tables --connect jdbc:%(rdbmsName)s://%(rdbmsHost)s:%(rdbmsPort)s/%(rdbmsDatabaseName)s --username %(rdbmsUserName)s+ --password %(rdbmsPassword)s once. then add the different parameters in the ... Romain Rigaux
Cannot do that as the statement syntax changes with each condition. Prachi Poddar
--connect jdbc:%(rdbmsName)s://%(rdbmsHost)s:%(rdbmsPort)s/%(rdbmsDatabaseName)s --username %(rdbmsUserName)s --password %(rdbmsPassword)s Looks the same everywhere? Romain Rigaux
Actually this is gone, rever? Romain Rigaux
Also add in desktop/conf/pseudo*ini? Romain Rigaux
+statement+ --> %(statement)s Romain Rigaux
if not rdbms.get_query_server_config(server='mysql'): seems safer? Romain Rigaux
How come we get []? Romain Rigaux
Why commented? Romain Rigaux
Removed? Romain Rigaux
and in desktop/conf/pseudo-distributed.ini? Romain Rigaux
Yes, it is present in desktop/conf/pseudo-distributed.ini also Prachi Poddar
All this new css looks like a merge issue? Romain Rigaux
source_type --> sourceType 3 times? Romain Rigaux
ifnot: createWizard.source.inputFormat() == 'rdbms' && createWizard.source.isAllTables(): to give more context? Romain Rigaux
Spaces after commas, e.g. ['table', 'database'...] Romain Rigaux
Delete? Romain Rigaux
table', 'hive space Romain Rigaux
Between a % if ENABLE_SQOOP.get(): ? Romain Rigaux
if (['file', 'hive', 'hbase'].indexOf(format.value) != -1 && wizard.source.inputFormat() != 'rdbms') { return false; } ? Romain Rigaux
sourceType Romain Rigaux
Here too https://github.com/cloudera/hue/blob/master/desktop/conf/pseudo-distributed.ini.tmpl Romain Rigaux
Let's add some basic testing for this in https://issues.cloudera.org/browse/HUE-6966 Romain Rigaux
conf.HDFS_CLUSTERS['default'].FS_DEFAULTFS.get() --> request.fs.fs_defaultfs Romain Rigaux
'/test'/ to remove Romain Rigaux
files=[], Romain Rigaux
input-smal --> input-small Romain Rigaux
Add 'rdbms' prefix everywhere? Romain Rigaux
isConnection --> rdbmsCheckConnection ? Romain Rigaux
Add {'name': 'File', 'value': 'file'}, too? Romain Rigaux
Won't it become redundant. 'File' should be an option irrespective of whether sqoop is enable or not. That's why I ... Prachi Poddar
files = [] if destination.get('sqoopJobLibPath'): files.append({"path": destination['sqoopJobLibPath'], "type": "jar"}) ... files=files ... As we don't always have a file. Romain Rigaux
on_success_url depends if the output is a file, a table or a hbase table For Hive on_success_url = reverse('metastore:describe_table', kwargs={'database': ... Romain Rigaux
also if 'the input is rdbms' (as we can create a Hive table from a file but should not see ... Romain Rigaux
After the main commit, can look at 'property-csv-hdfs-files' component Romain Rigaux
'File' in Sqoop if? Romain Rigaux
Hbase --> HBase Romain Rigaux
This is only used in the tests and can be removed? Romain Rigaux
outputFormat() == 'file' ? Romain Rigaux
For now we could remove 'hbase' and the custom logic? (we will need to add a dedicated name chooser for ... Romain Rigaux
Should probably remove this if and we manage the logic in isValidDestination. (and same we can remove hbase for now) Romain Rigaux
Remove keep here? .. Romain Rigaux
.. and then remove the if here? Romain Rigaux
So now that I think about it, 'table' and 'hive' are the same? Could we remove 'hive'? Romain Rigaux
equal {"type": "csv"} ? Romain Rigaux
'rdbms' --> 'manual' ? Romain Rigaux
For now: wizard.source.inputFormat() == 'rdbms' --> wizard.source.inputFormat() != 'file' Romain Rigaux
Removed? Romain Rigaux
displayOnlyFolders=true ? uploadFolder no used? Romain Rigaux
true --> false Romain Rigaux
Until we officially test/release it: enable_sqoop=false Romain Rigaux
format_ = {'data': [], 'status': 1, 'message': ''} name = None Romain Rigaux
except Exception: --> except Exception, e: Romain Rigaux
LOG.error('Error accessing the database %s', Exception) ---> message = 'Error accessing the database %s' % name LOG.warn(message, e) format['message'] = ... Romain Rigaux
Are all the str() really needed? (it should not as we print them in a string) Romain Rigaux
-m 1 We should create a follow-up jira to support a configurable number of mappers Romain Rigaux
'/hbase/#HBase/' + destination_table_name Romain Rigaux
uploadFolder: true, remove? (not an option IIRC) Romain Rigaux
"Connection Failed." --> "${ _('Connection Failed: ') }" + resp.message Romain Rigaux
Double # before --> "## enable_sqoop=false" Romain Rigaux
(message) --> " _(message) " with a leading underscore _ Romain Rigaux
  1. 
      
  2. apps/useradmin/src/useradmin/urls.py (Diff revision 1)
     
     

    Could you revert those changes?

  3. desktop/conf.dist/hue.ini (Diff revision 1)
     
     

    Could you remove all the changes here?

    (the local ini to modify in dev mode is desktop/conf/pseudo-distributed.ini)

  4. Could you follow the same conventions as in the project?

    e.g.
    Licence file, CamelCase for Class names, import sorted in order and split by type, indentation of 2 , etc

    https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/tests.py#L44

  5. desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py (Diff revision 1)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    Could you move the logic into a RdbmsIndexer class with guess_format/get_sample_data methods?

  6. 
      
  1. Nice!

    Some comments inline.

    And afterwards we can see how to reuse RdbmsApi() properly to avoid the duplication.

  2. Move to around line 18?

    (as is higher dependency than the module)

  3. desktop/libs/indexer/src/indexer/rdbms_indexer.py (Diff revision 3)
     
     
     
     
     
     
     
     
     

    This should probably return an empty dictionary for now.

  4. Is snippet used?
    Wouldn't it be the name of the configured database in the hue.ini?

    Actually the constructor of RdbmsIndexer could take a dictionnary that contains all the parameters for connecting to the db (e.g. content of connect_credentials) and another one to have the name of the DB in the configuration.

  5. Asserts should live in the tests class

  6. len() should not be needed

  7. 
      
  1. 
      
  2. Extra lines can be removed

  3. Should be removed as not attributes

  4. Grouped together

  5. desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py (Diff revision 4)
     
     
     
     
     
     
     
     
     

    Known of those are used, so could be removed and just have a table_name variable?

  6. This is probably not working as we are not defining any views yet.

    Instead, you should create a RdbmsIndexer() object and call its methods here

  7. 
      
  1. 
      
  2. nits: Two newlines

  3. connect_credentials --> user

  4. self.user = user

  5. self.interpreter --> {'type': 'mysql'}

  6. sample = assist.get_sample_data(....)

    if sample['status'] == 0:
    retur sample

    api3.py

    elif file_format['inputFormat'] == 'dbms':
    sample = DbmsIndexer(request.user, 'mysql').get_sample_data( database=file_format['databaseName'], table=file_format['tableName'])

    format_ = {
        "sample": sample['rows'][:4],
        "columns": [
            Field(col.name, HiveFormat.FIELD_TYPE_TRANSLATE.get(col.type, 'string')).to_dict()
            for col in sample['headers']
        ]
    }
    

    {"sample": [["mark", "M", "NJ", "2011", "clerk", "21250", "2014-05-04T06:39:59Z", "2014-05-04 06:39:59.000000000"], ["john", "M", "NY", "2011", "engineer", "42500", "2014-06-04T06:39:59Z", "2014-06-04 06:39:59.000000000"], ["mary", "F", "CT", "2015", "manager", "87299", "2014-07-04T06:39:59Z", "2014-07-04 06:39:59.000000000"], ["alice", "F", "NJ", "2013", "dentist", "75000", "2014-08-04T06:39:59Z", "2014-08-04 06:39:59.000000000"], ["mike", "M", "NY", "2012", "sales", "59500", "2014-05-14T06:39:59Z", "2014-05-14 06:39:59.000000000"]], "columns": [{"operations": [], "comment": "", "nested": [], "name": "id", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "string", "showProperties": false, "scale": 0}, {"operations": [], "comment": "", "nested": [], "name": "gender_s", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "string", "showProperties": false, "scale": 0}, {"operations": [], "comment": "", "nested": [], "name": "loc_s", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "string", "showProperties": false, "scale": 0}, {"operations": [], "comment": "", "nested": [], "name": "year_i", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "long", "showProperties": false, "scale": 0}, {"operations": [], "comment": "", "nested": [], "name": "job_s", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "string", "showProperties": false, "scale": 0}, {"operations": [], "comment": "", "nested": [], "name": "salary_d", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "long", "showProperties": false, "scale": 0}, {"operations": [], "comment": "", "nested": [], "name": "date_dt", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "date", "showProperties": false, "scale": 0}, {"operations": [], "comment": "", "nested": [], "name": "date_ts", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "string", "showProperties": false, "scale": 0}]}

  7. native Python

    nose (Hue lib)

    indexer

  8. RdbmsIndexer(user, db_conf_name)

  9. obj --> indexer

    get_sample_data() --> get_sample_data(database='hue', table='auth_users')

  10. from django.contrib.auth.models import User

    client = make_logged_in_client()
    user = User.objects.get(username='test')

  11. 
      
  1. Nice!

    1 Some comments below

    2 Could you switch you local Hue to point to MySql? https://www.cloudera.com/documentation/enterprise/latest/topics/hue_dbs_mysql.html#hue_dbs_mysql

  2. Add on top?

    !/usr/bin/env python

  3. 2 new lines above?

  4. To move below desktop

  5. RdbmsIndexerTests --> TestRdbmsIndexer

  6. Can remove this one for now, it will be a test in the API later

  7. 
      
  1. 
      
  2. import logging

    from notebook.connectors.rdbms import Assist

    from librdbms.server import dbms

  3. import json
    import logging

    from django.contrib.auth.models import User
    from nose.tools import assert_equal, assert_false, assert_not_equal, assert_true

    from desktop.lib.django_test_util import make_logged_in_client

    from indexer.templates.rdbms_indexer import RdbmsIndexer

  4. from indexer.rdbms_indexer import ...

  5. To surround by 2 lines on top/above

  6. 
      
  1. 
      
  2. server=file_format['rdbmsName']

  3. databaseName --> rdbmsDatabaseName
    ...
    no columns

  4. Rdbms --> External Database?

  5. Can remove this line

  6. self.rdbmsName = ko.observable(''); e.g. mysql4, oracle4, aka [[section]] name

    self.rdbmsTable = ko.observable('');
    e.g. hue.desktop_auth_users

    self.rdbmsTableName = ko.computed(function() {
    return self.rdbmsTable().indexOf('.') > 0 ? self.table().split('.', 2)[1] : self.table();
    });
    self.rdbmsDatabaseName = ko.computed(function() {
    return self.rdbmsTable().indexOf('.') > 0 ? self.table().split('.', 2)[0] : 'default';

  7. 
      
  1. 
      
  2. dummy_file (Diff revision 9)
     
     

    Remove?

  3. 
      
  1. Nice!

  2. Could we add a feature flag to enable the Sqoop feature in https://github.com/cloudera/hue/blob/master/desktop/libs/indexer/src/indexer/conf.py#L69?

    Then we could raise a SkipTest when it is off here

  3. 
      
  1. https://github.com/cloudera/hue/blob/master/desktop/conf/pseudo-distributed.ini.tmpl
    https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini

  2. 
      
  1. 
      
  2. self.subscribe(function(val) {
    if (val) {
    vm.createWizard.guessFormat();
    }
    resizeElements();
    });

  3. 
      
  1. 
      
  2. desktop/libs/indexer/src/indexer/api3.py (Diff revision 11)
     
     

    Not used?

  3. desktop/libs/indexer/src/indexer/conf.py (Diff revision 11)
     
     

    To add to both ini templates cf. previouscomments

  4. To disable with SkipTest cf previous comments

  5. 
      
  1. Hi Prachi,
    I pushed a couple of fixes here: https://github.com/cloudera/hue/tree/HUE-6659_prachi so the page doesn't break anymore

    For the "getting list of db/tables" questions, you should expose the RdbmsIndexer class as API (like in api.py, api2, api3) so in that way you can then configure urls.py to have a URL to call to actually have the list. Then it's just a matter of using $.post('/new_rdbms_url_that_you_choose', {type: dbType, table: dbTable, ....}, function(){})

  2. 
      
  1. Those URLs on $.post are not valid, try to look on other mako files on how to render a valid URL in mako

  2. 
      
  1. In the description, could we get the list of changes?

    e.g.

    git whatchanged -4

    where 4 is the actualy number of commits

  2. desktop/libs/indexer/src/indexer/api3.py (Diff revision 16)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    Could we move those into a

    rdbms.py in https://github.com/cloudera/hue/tree/master/desktop/libs/indexer/src/indexer/indexers

    ?

    This code is also 90% the same, could you combine both methods together?

  3. desktop/libs/indexer/src/indexer/api3.py (Diff revision 16)
     
     

    remove print

  4. desktop/libs/indexer/src/indexer/api3.py (Diff revision 16)
     
     
     
     
     
     
     
     

    Is this used?

  5. desktop/libs/indexer/src/indexer/api3.py (Diff revision 16)
     
     
     
     
     
     
     

    Combine with just this condition?

    elif source['inputFormat'] == 'rdbms':
    if destination['outputFormat'] in ('file', 'hive', 'hbase'):
    job_handle = ...

  6. desktop/libs/indexer/src/indexer/api3.py (Diff revision 16)
     
     

    Could you remove all the prints and dead comments?

  7. desktop/libs/indexer/src/indexer/api3.py (Diff revision 16)
     
     

    Move to rdbms.py?

    Then should be easy to add tests for the sqoop command generation.

  8. desktop/libs/indexer/src/indexer/api3.py (Diff revision 16)
     
     

    Spaces between all the +

  9. desktop/libs/indexer/src/indexer/api3.py (Diff revision 16)
     
     

    Usually it gets clearers with

    'import-all-tables --connect jdbc:%(rdbmsName)s ....' % {
    'rdbmsName': rdbmsName,
    ....
    }

  10. Move those files to https://github.com/cloudera/hue/tree/master/desktop/libs/indexer/src/indexer/indexers ? (and drop _indexer from the name)

  11. New line after #25?

  12. New line above, no new line between both 'from indexer' statements?

  13. Could we keep the new line?

  14. ifnot:

    or

    ! createWizard.source.isAllTables()

    (no need of == false)

  15. desktop/libs/indexer/src/indexer/templates/importer.mako (Diff revision 16)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    Could we keep all the previous new lines here and below?

  16. desktop/libs/indexer/src/indexer/templates/importer.mako (Diff revision 16)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    Is this used and looks like a duplication from Destination?

  17. Spaces after if / else / { ?

    Is this code actually correct/needed?

  18. desktop/libs/indexer/src/indexer/templates/importer.mako (Diff revision 16)
     
     
     
     
     
     
     
     
     
     
     

    Is this correct?

    (it basically override all the field correctness/output existing validations)

  19. 
      
  1. Did you see all the previous comments in the previous review?

  2. desktop/libs/indexer/src/indexer/api3.py (Diff revision 17)
     
     

    Could we move it to the other module cf previous comment?

  3. desktop/libs/indexer/src/indexer/api3.py (Diff revision 17)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    Could we merge both cf previous comments?

  4. 
      
  1. 
      
  2. desktop/libs/indexer/src/indexer/api3.py (Diff revision 18)
     
     
     
     
     
     
     
     

    Not used by anything?

  3. desktop/libs/indexer/src/indexer/api3.py (Diff revision 18)
     
     

    Keep newline?

  4. desktop/libs/indexer/src/indexer/api3.py (Diff revision 18)
     
     

    Keep newline?

  5. Define

    format_ = {
    'data': []
    'status: -1
    }

    on top?

  6. Should probably remove the logging or at least remove the password logging, e.g.

    https://github.com/cloudera/hue/blob/master/apps/beeswax/src/beeswax/server/dbms.py#L111

  7. Could move line #71 just below here

  8. Here we always logs exceptions,

    e.g.

    except Exception:
    LOG.error('Error getting the database %s', e)

  9. Simpler and does not override list/dict builtins:

    format_['data'] = [{'name: element, 'value': element} for element in data]

  10. desktop/libs/indexer/src/indexer/indexers/rdbms.py (Diff revision 18)
     
     
     
     
     
     
     
     
     

    Let's remove unused comments

  11. '/user/admin/mysql-connector-java.jar' should be moved into a UI parameters

  12. Could you move this with the other connectors?
    https://github.com/cloudera/hue/tree/master/desktop/libs/indexer/src/indexer/indexers

    e.g.:
    rdbms.py

  13. Cleaner, e.g.:

    @classmethod
    def setup_class(cls):
    if not ENABLE_SQOOP.get():
    raise SkipTest

  14. We should also raise a SkipTest if there if no 'mysql' DB is configured in librdms.

  15. Could we keep all the new lines below like before?

  16. Same about new lines

  17. desktop/libs/indexer/src/indexer/templates/importer.mako (Diff revision 18)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    Is this actually used?

  18. desktop/libs/indexer/src/indexer/templates/importer.mako (Diff revision 18)
     
     
     
     
     
     

    sqoop1: {
    placeHolder: '${ _("Example: import --connect jdbc:hsqldb:file:db.hsqldb --table TT --target-dir hdfs://localhost:8020/user/foo -m 1") }',
    snippetImage: '${ static("sqoop/art/icon_sqoop_48.png") }'
    },

  19. desktop/libs/indexer/src/indexer/urls.py (Diff revision 18)
     
     
     
     

    Moved above the #Deprecated?

  20. 
      
  1. Almost!

  2. desktop/libs/indexer/src/indexer/api3.py (Diff revision 19)
     
     
     
     
     
     
     
     

    Remove?

  3. desktop/libs/indexer/src/indexer/api3.py (Diff revision 19)
     
     

    Remove new line?

  4. desktop/libs/indexer/src/indexer/conf.py (Diff revision 19)
     
     

    Would you also add the setting in

    https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini#L1255
    https://github.com/cloudera/hue/blob/master/desktop/conf/pseudo-distributed.ini.tmpl#L1258
    ?

  5. FYI, usually simpler

    if not source['rdbmsDatabaseName'] or not source['rdbmsDatabaseName']:

  6. --warehoude-dir ?

  7. You might want to build

    'import-all-tables --connect jdbc:%(rdbmsName)s://%(rdbmsHost)s:%(rdbmsPort)s/%(rdbmsDatabaseName)s --username %(rdbmsUserName)s+ --password %(rdbmsPassword)s

    once.

    then add the different parameters in the ifs

  8. Maybe extend objects?

    https://github.com/cloudera/hue/blob/master/desktop/libs/metadata/src/metadata/optimizer_client_tests.py#L43

    If not looks good

  9. Actually this is gone, rever?

  10. 
      
  1. 
      
  2. Have created a JIRA sub-task for this

  3. Need more clarification on this.

  4. Cannot do that as the statement syntax changes with each condition.

  5. 
      
  1. Bunch of other comments from last review?

  2. desktop/libs/indexer/src/indexer/api3.py (Diff revision 19)
     
     
     
     
     
     
     
     

    Remove?

  3. --connect jdbc:%(rdbmsName)s://%(rdbmsHost)s:%(rdbmsPort)s/%(rdbmsDatabaseName)s --username %(rdbmsUserName)s --password %(rdbmsPassword)s

    Looks the same everywhere?

  4. 
      
  1. Almost, a few more comments.

    Also could we have it with a latest rebase?

  2. desktop/conf.dist/hue.ini (Diff revision 20)
     
     

    Also add in desktop/conf/pseudo*ini?

  3. +statement+ --> %(statement)s

  4. if not rdbms.get_query_server_config(server='mysql'):

    seems safer?

  5. How come we get []?

  6. desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py (Diff revision 20)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    Why commented?

  7. 
      
  1. 
      
  2. desktop/conf.dist/hue.ini (Diff revision 21)
     
     

    and in desktop/conf/pseudo-distributed.ini?

  3. desktop/libs/indexer/src/indexer/templates/importer.mako (Diff revision 21)
     
     
     
     
     
     
     
     
     
     

    All this new css looks like a merge issue?

  4. desktop/libs/indexer/src/indexer/templates/importer.mako (Diff revision 21)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    source_type --> sourceType 3 times?

  5. ifnot: createWizard.source.inputFormat() == 'rdbms' && createWizard.source.isAllTables():

    to give more context?

  6. Spaces after commas, e.g. ['table', 'database'...]

  7. table', 'hive

    space

  8. Between a

    % if ENABLE_SQOOP.get():

    ?

  9. desktop/libs/indexer/src/indexer/templates/importer.mako (Diff revision 21)
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    if (['file', 'hive', 'hbase'].indexOf(format.value) != -1 && wizard.source.inputFormat() != 'rdbms') {
    return false;
    }

    ?

  10. 
      
  1. Almost!

  2. desktop/conf.dist/hue.ini (Diff revision 22)
     
     

    Here too https://github.com/cloudera/hue/blob/master/desktop/conf/pseudo-distributed.ini.tmpl

  3. Let's add some basic testing for this in https://issues.cloudera.org/browse/HUE-6966

  4. conf.HDFS_CLUSTERS['default'].FS_DEFAULTFS.get()

    -->

    request.fs.fs_defaultfs

  5. '/test'/ to remove

  6. input-smal --> input-small

  7. Add 'rdbms' prefix everywhere?

  8. isConnection --> rdbmsCheckConnection

    ?

  9. Add

    {'name': 'File', 'value': 'file'},

    too?

  10. 
      
  1. 
      
  2. files = []

    if destination.get('sqoopJobLibPath'):
    files.append({"path": destination['sqoopJobLibPath'], "type": "jar"})

    ...
    files=files
    ...

    As we don't always have a file.

  3. on_success_url depends if the output is a file, a table or a hbase table

    For Hive
    on_success_url = reverse('metastore:describe_table', kwargs={'database': database, 'table': table_name})

    For HBase, something like
    /hbase/#HBase/<table_name>

  4. also if 'the input is rdbms'

    (as we can create a Hive table from a file but should not see it there)

  5. After the main commit, can look at
    'property-csv-hdfs-files'
    component

  6. 'File' in Sqoop if?

  7. 
      
  1. 
      
  2. desktop/libs/indexer/src/indexer/indexers/rdbms.py (Diff revision 24)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    This is only used in the tests and can be removed?

  3. outputFormat() == 'file'

    ?

  4. For now we could remove 'hbase' and the custom logic?

    (we will need to add a dedicated name chooser for HBase anyway later)

  5. Should probably remove this if and we manage the logic in isValidDestination.

    (and same we can remove hbase for now)

  6. Remove keep here?
    ..

  7. ..
    and then remove the if here?

  8. So now that I think about it, 'table' and 'hive' are the same?

    Could we remove 'hive'?

  9. 
      
  1. 
      
  2. desktop/conf.dist/hue.ini (Diff revision 21)
     
     

    Yes, it is present in desktop/conf/pseudo-distributed.ini also

  3. Won't it become redundant. 'File' should be an option irrespective of whether sqoop is enable or not. That's why I kept it outside ENABLE_SQOOP.get() block

  4. 
      
  1. 
      
  2. equal {"type": "csv"} ?

  3. 'rdbms' --> 'manual'
    ?

  4. For now:

    wizard.source.inputFormat() == 'rdbms'

    -->

    wizard.source.inputFormat() != 'file'

  5. 
      
  1. Ship It!
  2. 
      
  1. Nice, very almost there!

    Could you add the 'git whatchanged -N' in the 'description' of the review?

  2. displayOnlyFolders=true
    ?

    uploadFolder no used?

  3. 
      
  1. 
      
  2. desktop/conf.dist/hue.ini (Diff revision 30)
     
     

    true --> false

  3. desktop/conf/pseudo-distributed.ini.tmpl (Diff revision 30)
     
     

    Until we officially test/release it:

    enable_sqoop=false

  4. format_ = {'data': [], 'status': 1, 'message': ''}
    name = None

  5. except Exception: --> except Exception, e:

  6. LOG.error('Error accessing the database %s', Exception)

    --->

    message = 'Error accessing the database %s' % name
    LOG.warn(message, e)
    format['message'] = (message)

  7. Are all the str() really needed?

    (it should not as we print them in a string)

  8. -m 1

    We should create a follow-up jira to support a configurable number of mappers

  9. '/hbase/#HBase/' + destination_table_name

  10. uploadFolder: true, remove? (not an option IIRC)

  11. "Connection Failed." --> "${ _('Connection Failed: ') }" + resp.message

  12. 
      
Review request changed

Description:

-  

NOTE: We use uploadFolder option. User can create the folder on the go. It is useful in scanario when no seperate folder is readily available to give as sqoop target-dir. Otherwise, user will have to go to file browser and create one which will be time consuming.

-  
-  

-  
   

dhcp-172-16-2-143:hue prachi.poddar$ git whatchanged -15

    commit 687c4aba9e52131efa99437adf4898182cced1ee
    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Mon Jul 24 11:51:27 2017 -0700

   
   
HUE-6659 [import] Cleaned code based on the code review. Added rdbms.py. Replaced test connection checkbox with button
   
   

:100644 100644 4c059b36f5... 2dfea9c063... M apps/oozie/src/oozie/templates/editor2/gen/workflow-sqoop.xml.mako

    :100644 100644 d4b63c8bb3... 23e530c1b4... M desktop/conf.dist/hue.ini
    :100644 100644 08ddda3733... a8b56959cc... M desktop/conf/pseudo-distributed.ini.tmpl
    :100644 100644 f076329ddd... d7b99b4941... M desktop/libs/indexer/src/indexer/api3.py
    :000000 100644 0000000000... 17eac54ad1... A desktop/libs/indexer/src/indexer/indexers/rdbms.py
    :100644 000000 e574059ff2... 0000000000... D desktop/libs/indexer/src/indexer/rdbms_indexer.py
    :100644 100644 bec0acca4e... 4c0f04d415... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 5daa6bffe0... 4a04d93367... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 aa9ff75739... ca8e4a1f7c... M desktop/libs/indexer/src/indexer/urls.py
    :100644 100644 644044d263... e08129aa66... M desktop/libs/librdbms/src/librdbms/server/oracle_lib.py

   
   

commit 255a42cf62d280bae744cc352552283abaefe3e4

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Sun Jul 16 23:11:36 2017 -0700

   
   
HUE-6659 [import] added hive as a new option in Rdbms data import.
   
   

:100644 100644 758e2fde9d... f076329ddd... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 30c18f7e15... 5daa6bffe0... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 c51bf27be5... aa9ff75739... M desktop/libs/indexer/src/indexer/urls.py

   
   

commit 8e12fb18274b5d02ced7ea6cdb0b75e1fa4e6b3a

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Wed Jul 12 13:04:08 2017 -0700

   
   
HUE-6659 [import] added test connection front-end for custom db and output options for external databases
   
   

:100644 100644 2625651265... 758e2fde9d... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 5cb2bc75f9... bec0acca4e... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 ad8c316c60... 30c18f7e15... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 ee3f36f76e... c51bf27be5... M desktop/libs/indexer/src/indexer/urls.py

   
   

commit 55bffb4c3540e1bcea59395e0b30542c0a56a04f

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Tue Jul 11 15:49:38 2017 -0700

   
   
HUE-6659 [importer] Database name and Table name dropdown populated successfully for config mode
   
   

:100644 100644 7101159479... 2625651265... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 a5b79ba3f5... 5cb2bc75f9... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 f424eaea54... ad8c316c60... M desktop/libs/indexer/src/indexer/templates/importer.mako

   
   

commit 5564220755d2e4aef5b1758df617dee2889eb95d

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Sat Jun 24 19:53:10 2017 -0700

   
   
HUE-6659 [import] added custom and config database options
   
   

:100644 100644 d87bbc5a66... 7101159479... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 d84e48f9f1... e574059ff2... M desktop/libs/indexer/src/indexer/rdbms_indexer.py
    :100644 100644 85a4e33f3f... a5b79ba3f5... M desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 e93c132366... f424eaea54... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 1bc826c81d... ee3f36f76e... M desktop/libs/indexer/src/indexer/urls.py

   
   

commit c4958756defa8e1fc74932f201a3aeb836e8c97b

    Author: Romain Rigaux romain@cloudera.com
    Date: Fri Jun 23 15:30:58 2017 -0700

   
   
HUE-6659 [importer] Quick clean-up to submit sqoop as a task
   
   

:100644 100644 d30edd9c20... d87bbc5a66... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 760c3c26e4... e93c132366... M desktop/libs/indexer/src/indexer/templates/importer.mako

   
   

commit 9aefe252ed09a7a9d2672ef2449313604b3b5e1f

    Author: Prachi Poddar prachi.poddar@cloudera.com
    Date: Sun Jun 11 19:13:24 2017 -0700

   
   
HUE-6659 [import] Support RDBMs as input via Sqoop
   
   

:100644 100644 ae6c2de88d... a9b53ed326... M apps/oozie/src/oozie/models2.py

    :100644 100644 ccb4242c9c... d30edd9c20... M desktop/libs/indexer/src/indexer/api3.py
    :100644 100644 ec0f517cff... a4036bcef9... M desktop/libs/indexer/src/indexer/conf.py
    :000000 100644 0000000000... d84e48f9f1... A desktop/libs/indexer/src/indexer/rdbms_indexer.py
    :000000 100644 0000000000... 85a4e33f3f... A desktop/libs/indexer/src/indexer/rdbms_indexer_tests.py
    :100644 100644 11b9ecedef... 760c3c26e4... M desktop/libs/indexer/src/indexer/templates/importer.mako
    :100644 100644 76722d8b30... 02ef9f522a... M desktop/libs/notebook/src/notebook/connectors/base.py

   
   

commit a3ff7af21da91b035f927ba531849f5371af1224

    Author: Romain Rigaux romain@cloudera.com
    Date: Fri Jul 28 11:12:05 2017 -0700

   
   
HUE-5217 [backend] Resolve LGPL copyleft issue - Paramiko in boto 2.46.1
   
   

:100644 000000 1eb9db47d5... 0000000000... D desktop/core/ext-py/boto-2.46.1/boto/mashups/interactive.py

   
   

commit 7b15b90eb3d61249e03731dccd702cdfb2525cb4

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 17:26:49 2017 -0700

   
   
HUE-6968 [metadata] Gather column stats via Impala
   
   

:100644 100644 e3a936c3b3... 139469a596... M apps/beeswax/src/beeswax/server/dbms.py

    :100644 100644 d34b433770... 66089ead57... M desktop/libs/metadata/src/metadata/optimizer_api.py

   
   

commit 9b715711e290702233c08a0afdcb0e77c21cd518

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 15:41:11 2017 -0700

   
   
HUE-6968 [metadata] Gather table stats via Impala
   
   

:100644 100644 82eb8aa1b6... 6a8a9bddd1... M apps/beeswax/src/beeswax/views.py

    :100644 100644 75e098622e... d34b433770... M desktop/libs/metadata/src/metadata/optimizer_api.py

   
   

commit d4f453f6664256c56444303eb47c54597ba1e2ad

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:53:20 2017 -0700

   
   
HUE-6954 [indexer] Support different number of shards, replication at index creation
   
   

:100644 100644 ac39c6272e... ccb4242c9c... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 9779cfdccd... 5324a07482... M desktop/libs/indexer/src/indexer/solr_client.py
    :100644 100644 2587fe9742... 11b9ecedef... M desktop/libs/indexer/src/indexer/templates/importer.mako

   
   

commit ff6fe4e70ecc99283adb94175f6dce114a711c76

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:46:55 2017 -0700

   
   
HUE-6974 [importer] Refresh Assist collection list after create table wizard success
   
   
When not using the batch indexer, we submit synchronously so we now it was created.
   
   

:100644 100644 4269d8bdc0... ac39c6272e... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 0947ee1f24... 2587fe9742... M desktop/libs/indexer/src/indexer/templates/importer.mako

   
   

commit e273d3347ad97cd4ed9084c21d9fa9c90a314573

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:38:43 2017 -0700

   
   
HUE-6973 [indexer] Open up the index page after creating a new index
   
   

:100644 100644 8ce1dce1bf... 4269d8bdc0... M desktop/libs/indexer/src/indexer/api3.py

    :100644 100644 1ca429779b... 1bc826c81d... M desktop/libs/indexer/src/indexer/urls.py

   
   

commit 048b7b944d15cc0e3af094a3561379ceac3abf9f

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:24:22 2017 -0700

   
   
HUE-6972 [metadata] Sometimes the DB owner is shown as ()
   
   

:100644 100644 272a1991ad... 0f3f469181... M apps/metastore/src/metastore/templates/metastore.mako

   
   

commit ac41d006e5d1b7f8b12f6b8907430b8a976ec626

    Author: Romain Rigaux romain@cloudera.com
    Date: Thu Jul 27 14:07:07 2017 -0700

   
   
HUE-6971 [jb] Schedule list is filtered by default on 7 days which hides most of them
   
   

:100644 100644 56d84589b8... 27547054e9... M apps/jobbrowser/src/jobbrowser/apis/schedule_api.py

    :100644 100644 cc1e79017a... 2cf28fa571... M apps/jobbrowser/src/jobbrowser/templates/job_browser.mako

Commit:

-687c4aba9e52131efa99437adf4898182cced1ee
+91837dec3643f0967cfd0a0744c368fa5a189428

Diff:

Revision 31 (+549 -36)

Show changes

  1. 2 nits and just go for:

    git fetch origina
    git rebase origin/master

    git push origin HEAD:master

  2. desktop/conf/pseudo-distributed.ini.tmpl (Diff revision 31)
     
     

    Double # before

    -->

    "## enable_sqoop=false"

  3. (message)

    -->

    " _(message) "

    with a leading underscore _

  4. 
      
Loading...