HUE-698 hell app does not work with a secured cluster

Review Request #2072 — Created May 7, 2012 and submitted

romain
old-hue-rw
HUE-698
hue
bcwalrus
commit d097d968f2a0c56d7c2eb074d5d0857bcebf033d
Author: Romain Rigaux <romain@cloudera.com>
Date:   Mon May 7 18:20:30 2012 -0700

    HUE-698 Shell app does not work with a secured cluster

:100644 100644 d0c002e... 87cbc6b... M	apps/shell/src/shell/shellmanager.py
:100644 100644 4a2c786... ff32cf2... M	desktop/core/ext-py/urllib2_kerberos-0.1.6/urllib2_kerberos.py
:100644 100644 e81b06f... 2549567... M	desktop/libs/hadoop/credentials-merger/src/main/java/com/cloudera/hue/CredentialsMerger.java
:100644 100644 8764189... 9509127... M	desktop/libs/hadoop/src/hadoop/job_tracker.py
First pass.

Unit tests are passing.

The whole problem was to merge the JT and NN tokens together into a file and give this file to the new shell.
Adding a unit test for this would be nice but it might not be simple.


I spent time trying to setup the Pig shell (unrelated problems probably) and I am not there yet but at least it gets the NN ticket.


2012-05-07 18:17:40,243 [Thread-10] INFO org.apache.hadoop.hdfs.DFSClient - Created HDFS_DELEGATION_TOKEN token 81 for hdfs on 172.29.112.225:8020
2012-05-07 18:17:40,243 [Thread-10] INFO org.apache.hadoop.mapreduce.security.TokenCache - Got dt for hdfs://centos56-17.ent.cloudera.com:8020;uri=172.29.112.225:8020;t.service=172.29.112.225:8020

2012-05-07 18:17:41,005 [Thread-10] INFO org.apache.hadoop.hdfs.DFSClient - Created HDFS_DELEGATION_TOKEN token 82 for hdfs on 172.29.112.225:8020
2012-05-07 18:17:41,005 [Thread-10] INFO org.apache.hadoop.mapreduce.security.TokenCache - Got dt for hdfs://centos56-17.ent.cloudera.com:8020;uri=172.29.112.225:8020;t.service=172.29.112.225:8020
2012-05-07 18:17:41,011 [Thread-10] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2012-05-07 18:17:41,011 [Thread-10] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1

2012-05-07 18:17:41,056 [Thread-10] WARN org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library is available
2012-05-07 18:17:41,057 [Thread-10] INFO org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
2012-05-07 18:17:41,057 [Thread-10] INFO org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library loaded
2012-05-07 18:17:41,062 [Thread-10] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1

2012-05-07 18:17:41,174 [Thread-10] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2012-05-07 18:17:41,196 [Thread-10] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2012-05-07 18:17:41,197 [Thread-10] WARN org.apache.hadoop.conf.Configuration - mapred.job.name is deprecated. Instead, use mapreduce.job.name
2012-05-07 18:17:41,197 [Thread-10] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2012-05-07 18:17:41,197 [Thread-10] WARN org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
2012-05-07 18:17:41,198 [Thread-10] WARN org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2012-05-07 18:17:41,198 [Thread-10] WARN org.apache.hadoop.conf.Configuration - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir

2012-05-07 18:17:42,036 [Thread-10] INFO org.apache.hadoop.mapred.ResourceMgrDelegate - Submitted application application_1336404116533_0001 to ResourceManager at centos56-17.ent.cloudera.com/172.29.112.225:8032

2012-05-07 18:17:42,356 [Thread-10] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://centos56-17.ent.cloudera.com:8088/proxy/application_1336404116533_0001/

2012-05-07 18:17:42,472 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete

2012-05-07 18:17:42,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1336404116533_0001
2012-05-07 18:17:42,975 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://blah:50030/jobdetails.jsp?jobid=job_1336404116533_0001

2012-05-07 18:17:47,479 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1336404116533_0001 has failed! Stop running all dependent jobs
2012-05-07 18:17:47,479 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete

2012-05-07 18:17:47,543 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2012-05-07 18:17:47,544 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.23.1-cdh4.0.0-SNAPSHOT 0.9.2-cdh4.0.0-SNAPSHOT hdfs 2012-05-07 18:17:36 2012-05-07 18:17:47 UNKNOWN

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_1336404116533_0001 a MAP_ONLY Message: Job failed! hdfs://centos56-17.ent.cloudera.com:8020/tmp/temp1166092263/tmp570738767,

Input(s):
Failed to read data from "hdfs://centos56-17.ent.cloudera.com:8020/user/hdfs/txt"

Output(s):
Failed to produce result in "hdfs://centos56-17.ent.cloudera.com:8020/tmp/temp1166092263/tmp570738767"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1336404116533_0001


2012-05-07 18:17:47,545 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2012-05-07 18:17:47,552 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias a
Details at logfile: /dev/null
grunt> 
  • 0
  • 0
  • 3
  • 0
  • 3
Description From Last Updated
bcwalrus
  1. 
      
  2. apps/shell/src/shell/shellmanager.py (Diff revision 1)
     
     
    Would remove the "as CONF" part. Another 4 characters is worth the readability.
  3. To emulate rpartition(), it should be:
        host_parts = host.rsplit(':', 1)
        domain = host_parts[0]
    
    You don't need the host_parts[-1], because rsplit() returns a list of size 1 when the separator is not found.
    1. I see, I updated it
    1. It is good practice yes.
  4. 
      
romain
romain
Review request changed

Status: Closed (submitted)

Loading...