[filebrowser] read parquet in filebrowser

Review Request #4310 — Created April 14, 2014 and submitted

abec
old-hue-rw
HUE-2077
hue
enricoberti, romain
commit bb7945e580bc69b118e851bfe7d997760d841c60
Author: Abraham Elmahrek <abraham@elmahrek.com>
Date:   Mon Apr 14 09:56:35 2014 -0700

    [filebrowser] read parquet in filebrowser

:100644 100644 b7893d8... 8e5f72b... M	apps/filebrowser/src/filebrowser/views.py

commit 0bbee8522286b6093bc168c8f82d81a3506107c3
Author: Abraham Elmahrek <abraham@elmahrek.com>
Date:   Mon Apr 14 11:29:36 2014 -0700

    [core] update parquet-python to use file objects

:100644 100644 f082aee... f9635d2... M	desktop/core/ext-py/parquet-python/parquet/__init__.py

commit accc03724a3c01c3ba743439434820613282eac2
Author: Abraham Elmahrek <abraham@elmahrek.com>
Date:   Mon Apr 14 09:40:55 2014 -0700

    [core] add parquet-python library

:000000 100644 0000000... 4947287... A	desktop/core/ext-py/parquet-python/LICENSE
:000000 100644 0000000... cf21c0b... A	desktop/core/ext-py/parquet-python/README.md
:000000 100644 0000000... f082aee... A	desktop/core/ext-py/parquet-python/parquet/__init__.py
:000000 100644 0000000... 4b94659... A	desktop/core/ext-py/parquet-python/parquet/__main__.py
:000000 100644 0000000... ab807a2... A	desktop/core/ext-py/parquet-python/parquet/bitstring.py
:000000 100644 0000000... a236326... A	desktop/core/ext-py/parquet-python/parquet/constants.py
:000000 100644 0000000... 0eaad73... A	desktop/core/ext-py/parquet-python/parquet/encoding.py
:000000 100644 0000000... 88fec11... A	desktop/core/ext-py/parquet-python/parquet/schema.py
:000000 100644 0000000... e07d644... A	desktop/core/ext-py/parquet-python/parquet/ttypes.py
:000000 100644 0000000... 0a0c1de... A	desktop/core/ext-py/parquet-python/setup.py
:000000 100755 0000000... 5bbf0d5... A	desktop/core/ext-py/parquet-python/test-data/gzip-nation.impala.parquet
:000000 100644 0000000... ee71b02... A	desktop/core/ext-py/parquet-python/test-data/nation.csv
:000000 100755 0000000... 5008ac0... A	desktop/core/ext-py/parquet-python/test-data/nation.dict.parquet
:000000 100755 0000000... bc61f97... A	desktop/core/ext-py/parquet-python/test-data/nation.impala.parquet
:000000 100755 0000000... dd236ec... A	desktop/core/ext-py/parquet-python/test-data/nation.plain.parquet
:000000 100755 0000000... 6714403... A	desktop/core/ext-py/parquet-python/test-data/snappy-nation.impala.parquet
:000000 100644 0000000... 538e1de... A	desktop/core/ext-py/parquet-python/test/test_encoding.py
:000000 100644 0000000... c4deb8a... A	desktop/core/ext-py/parquet-python/test/test_read_support.py

commit 8fda7cd651fe2651c11e8d2bc6e64e7e31b4975c
Author: Abraham Elmahrek <abraham@elmahrek.com>
Date:   Mon Apr 14 09:45:59 2014 -0700

    [core] make snappy a first class citizen

:100644 100644 1f69ce1... a6f4aff... M	README.rst
:100644 100644 24ada56... b7893d8... M	apps/filebrowser/src/filebrowser/views.py
:100644 100644 4df93fa... 4fd9d2a... M	apps/filebrowser/src/filebrowser/views_test.py

commit 9493d2aef72918059591f7dfe635cfbb5a05f52e
Author: Abraham Elmahrek <abraham@elmahrek.com>
Date:   Mon Apr 14 09:57:55 2014 -0700

    [core] Add python-snappy library

:000000 100644 0000000... 08e6a64... A	desktop/core/ext-py/python-snappy-0.5/AUTHORS
:000000 100644 0000000... 7560c68... A	desktop/core/ext-py/python-snappy-0.5/MANIFEST.in
:000000 100644 0000000... db9baab... A	desktop/core/ext-py/python-snappy-0.5/PKG-INFO
:000000 100644 0000000... ee5b979... A	desktop/core/ext-py/python-snappy-0.5/README.rst
:000000 100644 0000000... 9a607b0... A	desktop/core/ext-py/python-snappy-0.5/crc32c.c
:000000 100644 0000000... 3534daa... A	desktop/core/ext-py/python-snappy-0.5/crc32c.h
:000000 100644 0000000... 7351d41... A	desktop/core/ext-py/python-snappy-0.5/setup.py
:000000 100644 0000000... d0555fa... A	desktop/core/ext-py/python-snappy-0.5/snappy.py
:000000 100644 0000000... ff52e79... A	desktop/core/ext-py/python-snappy-0.5/snappymodule.cc
:000000 100644 0000000... d9a1c17... A	desktop/core/ext-py/python-snappy-0.5/test_snappy.py
Was able to view parquet file.
Loading file attachments...

  • 0
  • 0
  • 0
  • 2
  • 2
Description From Last Updated
romain
  1. Tried a big file? ;)
    Could we upper case all the commit titles to be consistent?
  2. README.rst (Diff revision 1)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
    parquet too?
    1. Actually, parquet-python reads directly from disk. No dependency on a C library.
  3. I guess parquet is splittable so no max size
    1. Bingo :). Parquet actually flattens and orients by column. Column chunks are divided into pages. These pages are compressed.
  4. 
      
abec
abec
Review request changed

Status: Closed (submitted)

Loading...