HUE-4266 [indexer] more operations for scalable indexer

Review Request #7898 - Created July 5, 2016 and submitted

Aaron Peddle
hue
solr-smart-indexer
HUE-4266
hue
enricoberti, jennykim, johan, krish, romain, weixia

commit f44ab683de8de4d354cfaf61907355b76f9365ab
Author: Aaron Peddle <aaron.peddle@cloudera.com>
Date: Tue Jul 5 15:54:04 2016 -0700

[HTML_REMOVED] white space cleanup

:100644 100644 5fc3e90... 9248b5b... M desktop/libs/indexer/src/data/oozie_workspace/find_replace_operation.conf
:100644 100644 6cb1840... 775ff1f... M desktop/libs/indexer/src/data/oozie_workspace/geo_ip_operation.conf
:100644 100644 bcd151a... f9f7bfc... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 a716955... cc7a684... M desktop/libs/indexer/src/indexer/smart_indexer.py
:100644 100644 44f9836... 4b967a5... M desktop/libs/indexer/src/indexer/templates/indexer.mako

commit 0ebb80fa6ebd98405c34f954fbf404e1370e5fa6
Author: Aaron Peddle <aaron.peddle@cloudera.com>
Date: Tue Jul 5 15:33:02 2016 -0700

[HTML_REMOVED] cleanup

:100644 100644 f6e188f... 568ffa0... M desktop/libs/indexer/src/data/oozie_workspace/convert_date_operation.conf
:100644 100644 70211e0... 21a96dd... M desktop/libs/indexer/src/data/oozie_workspace/extract_uri_components_operation.conf
:100644 100644 9230201... 5fc3e90... M desktop/libs/indexer/src/data/oozie_workspace/find_replace_operation.conf
:100644 100644 510daf5... 6cb1840... M desktop/libs/indexer/src/data/oozie_workspace/geo_ip_operation.conf
:100644 100644 02338f1... 2e4d83e... M desktop/libs/indexer/src/data/oozie_workspace/grok_operation.conf
:100644 100644 92d59b1... 21f7846... M desktop/libs/indexer/src/data/oozie_workspace/split_operation.conf
:100644 100644 7e84abe... 1505e8b... M desktop/libs/indexer/src/data/oozie_workspace/translate_operation.conf
:100644 100644 7fd42ea... bcd151a... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 c346e23... a716955... M desktop/libs/indexer/src/indexer/smart_indexer.py

commit 091e4531eeadd9e9ece929c2b57a2571aca324b2
Author: Aaron Peddle <aaron.peddle@cloudera.com>
Date: Tue Jul 5 15:31:40 2016 -0700

[HTML_REMOVED] more indexer operations

:100644 100644 58366d5... 9f6874f... M apps/oozie/src/oozie/models2.py
:000000 100644 0000000... 70211e0... A desktop/libs/indexer/src/data/oozie_workspace/extract_uri_components_operation.conf
:000000 100644 0000000... 9230201... A desktop/libs/indexer/src/data/oozie_workspace/find_replace_operation.conf
:000000 100644 0000000... 510daf5... A desktop/libs/indexer/src/data/oozie_workspace/geo_ip_operation.conf
:100644 100644 984060c... 55ebbe1... M desktop/libs/indexer/src/data/oozie_workspace/morphline_template.conf
:000000 100644 0000000... 7e84abe... A desktop/libs/indexer/src/data/oozie_workspace/translate_operation.conf
:100644 100644 fba813e... 7fd42ea... M desktop/libs/indexer/src/indexer/api3.py
:100644 100644 509de20... c346e23... M desktop/libs/indexer/src/indexer/smart_indexer.py
:100644 100644 c12680f... 44f9836... M desktop/libs/indexer/src/indexer/templates/indexer.mako



  • 1
  • 0
  • 10
  • 0
  • 11
Description From Last Updated
Do we have a todo about how to get this? And a link to this in gdrive? Will show you ... Romain Rigaux
  1. Nice!
    Squash everything in one commit maybe?

    1. And keep the commit name as: HUE-4266 [indexer] more operations for scalable indexer

  2. Not an issue now, but we need to make sure that we add the GeoLite2 CC4 license information in all our documentation then (docs + github + website i think?)

  3. 
      
  1. Nice one!

  2. desktop/libs/indexer/src/indexer/templates/indexer.mako (Diff revision 1)
     
     
     
     
     
     
     
     
     

    Unused? If not, Argument.prototype.getInstance

    The name getInstance makes me think it'll always return the same instance. Perhaps createArgument?

  3. Consider a simple function instead, var createMappingArgument = function...

    And above: return createMappingArgument(arg)

  4. 'self' is a bit of a dangerous variable name as we always use self = this. Better with 'argument' or something.

  5. desktop/libs/indexer/src/indexer/templates/indexer.mako (Diff revision 1)
     
     
     
     
     
     
     
     
     
     
     
     

    You could use this inside Argument

    ...
    self.value = createArgumentValue(arg);

    And get rid of MappingArgument

  6. Not that clear what 'self' is here

  7. Better to move this up

  8. 
      
  1. Could you squash the commits together and make the title more explicit?

    e.g.

    HUE-4266 [indexer] Add geo, host, grok, split operations

  2. desktop/libs/indexer/src/data/oozie_workspace/morphline_template.conf (Diff revision 5)
     
     
     
     
     
     
     
     
     
     

    This should be optional?

  3. Do we have a todo about how to get this?
    And a link to this in gdrive?

    Will show you how to document those in the hue.inis too

  4. spaces around the + ? (to put some "air")

  5. should we rename it

    isAvailableName ?

  6. 
      
Review request changed

Status: Closed (submitted)

Loading...