HUE-6659 [import] Support RDBMs as input via Sqoop

Review Request #10896 - Created June 8, 2017 and updated

Information
Prachi Poddar
hue
HUE-6659
5aeb63f...
Reviewers
romain

Still not clear on the following points:
1. importer.mako: Line 754 "indexFile" method is called on click event. But could not find the defination of this method.
2. api3.py: Line 456 What should be the value of the statement?

Not yet.

Issues

  • 54
  • 0
  • 0
  • 54
Description From Last Updated
Could you revert those changes? Romain Rigaux Romain Rigaux
Could you remove all the changes here? (the local ini to modify in dev mode is desktop/conf/pseudo-distributed.ini) Romain Rigaux Romain Rigaux
Could you follow the same conventions as in the project? e.g. Licence file, CamelCase for Class names, import sorted in ... Romain Rigaux Romain Rigaux
Could you move the logic into a RdbmsIndexer class with guess_format/get_sample_data methods? Romain Rigaux Romain Rigaux
Move to around line 18? (as is higher dependency than the module) Romain Rigaux Romain Rigaux
This should probably return an empty dictionary for now. Romain Rigaux Romain Rigaux
Is snippet used? Wouldn't it be the name of the configured database in the hue.ini? Actually the constructor of RdbmsIndexer ... Romain Rigaux Romain Rigaux
Asserts should live in the tests class Romain Rigaux Romain Rigaux
len() should not be needed Romain Rigaux Romain Rigaux
Extra lines can be removed Romain Rigaux Romain Rigaux
One new line Romain Rigaux Romain Rigaux
Should be removed as not attributes Romain Rigaux Romain Rigaux
return {} Romain Rigaux Romain Rigaux
return {} Romain Rigaux Romain Rigaux
Grouped together Romain Rigaux Romain Rigaux
Known of those are used, so could be removed and just have a table_name variable? Romain Rigaux Romain Rigaux
This is probably not working as we are not defining any views yet. Instead, you should create a RdbmsIndexer() object ... Romain Rigaux Romain Rigaux
nits: Two newlines Romain Rigaux Romain Rigaux
connect_credentials --> user Romain Rigaux Romain Rigaux
self.user = user Romain Rigaux Romain Rigaux
self.interpreter --> {'type': 'mysql'} Romain Rigaux Romain Rigaux
Remove Romain Rigaux Romain Rigaux
sample = assist.get_sample_data(....) if sample['status'] == 0: retur sample api3.py elif file_format['inputFormat'] == 'dbms': sample = DbmsIndexer(request.user, 'mysql').get_sample_data( database=file_format['databaseName'], table=file_format['tableName']) ... Romain Rigaux Romain Rigaux
native Python nose (Hue lib) indexer Romain Rigaux Romain Rigaux
RdbmsIndexer(user, db_conf_name) Romain Rigaux Romain Rigaux
obj --> indexer get_sample_data() --> get_sample_data(database='hue', table='auth_users') Romain Rigaux Romain Rigaux
from django.contrib.auth.models import User client = make_logged_in_client() user = User.objects.get(username='test') Romain Rigaux Romain Rigaux
Add on top? !/usr/bin/env python Romain Rigaux Romain Rigaux
2 new lines above? Romain Rigaux Romain Rigaux
To move below desktop Romain Rigaux Romain Rigaux
2 new lines above Romain Rigaux Romain Rigaux
RdbmsIndexerTests --> TestRdbmsIndexer Romain Rigaux Romain Rigaux
'root' --> 'test' Romain Rigaux Romain Rigaux
Can remove this one for now, it will be a test in the API later Romain Rigaux Romain Rigaux
import logging from notebook.connectors.rdbms import Assist from librdbms.server import dbms Romain Rigaux Romain Rigaux
import json import logging from django.contrib.auth.models import User from nose.tools import assert_equal, assert_false, assert_not_equal, assert_true from desktop.lib.django_test_util import make_logged_in_client from ... Romain Rigaux Romain Rigaux
from indexer.rdbms_indexer import ... Romain Rigaux Romain Rigaux
To surround by 2 lines on top/above Romain Rigaux Romain Rigaux
server=file_format['rdbmsName'] Romain Rigaux Romain Rigaux
databaseName --> rdbmsDatabaseName ... no columns Romain Rigaux Romain Rigaux
Can revert Romain Rigaux Romain Rigaux
Rdbms --> External Database? Romain Rigaux Romain Rigaux
Can remove this line Romain Rigaux Romain Rigaux
self.rdbmsName = ko.observable(''); e.g. mysql4, oracle4, aka [[section]] name self.rdbmsTable = ko.observable(''); e.g. hue.desktop_auth_users self.rdbmsTableName = ko.computed(function() { return self.rdbmsTable().indexOf('.') ... Romain Rigaux Romain Rigaux
Remove? Romain Rigaux Romain Rigaux
Remove? Romain Rigaux Romain Rigaux
Could we add a feature flag to enable the Sqoop feature in https://github.com/cloudera/hue/blob/master/desktop/libs/indexer/src/indexer/conf.py#L69? Then we could raise a SkipTest when ... Romain Rigaux Romain Rigaux
Same Romain Rigaux Romain Rigaux
file --> 'file' Romain Rigaux Romain Rigaux
Remove () ? Romain Rigaux Romain Rigaux
self.subscribe(function(val) { if (val) { vm.createWizard.guessFormat(); } resizeElements(); }); Romain Rigaux Romain Rigaux
Not used? Romain Rigaux Romain Rigaux
To add to both ini templates cf. previouscomments Romain Rigaux Romain Rigaux
To disable with SkipTest cf previous comments Romain Rigaux Romain Rigaux
Romain Rigaux
Prachi Poddar
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Prachi Poddar
Romain Rigaux
Romain Rigaux
Prachi Poddar
Romain Rigaux
Romain Rigaux
Romain Rigaux
Prachi Poddar
Romain Rigaux
Prachi Poddar
Review request changed

Description:

~  

A naive version of task 1 is complete.

~   Dropdown with "File" option is populated when we click on Next button from "External Database" option.

  ~

Still not clear on the following points:

  ~ 1. importer.mako: Line 754 "indexFile" method is called on click event. But could not find the defination of this method.
  + 2. api3.py: Line 456 What should be the value of the statement?

Testing Done:

~  

Yes. Works fine.

  ~

Not yet.

-   Just that when I inspect element -> Network, guess_field_type occurs multiple times with few errors. Not sure why.

Commit:

-b440af3b5438e601fccf55e5215777532aa3f19e
+5aeb63fc887c08da8ba84751e515e04135222a79

Diff:

Revision 12 (+237 -10)

Show changes

Loading...