HUE-706 [beeswax] Adding a resource to a query fails

Review Request #2044 — Created April 25, 2012 and submitted April 26, 2012, 3:57 a.m.

Information
Owner:	romain
Repository:	old-hue-rw
Branch:
Bugs:	HUE-706
Depends On:
Reviewers
Groups:	hue
People:	bcwalrus

Description

commit e532308871235ed1c7d874e48e15b46c512ca801
Author: Romain Rigaux <romain@cloudera.com>
Date:   Wed Apr 25 14:05:43 2012 -0700

    HUE-706 [beeswax] Adding a resource to a query fails

:100644 100644 5c25645... da1bd24... M	apps/beeswax/src/beeswax/views.py

Testing Done

This is fixing the Hue bug but then it hits on another bug (maybe Hive related). Any idea about this?

The udf file is not copied to the local task directory (or not aliased/symlinked correctly) or added to its PATH.


Example:

hdfs://localhost:8020/user/romain/square.py exist



File: /user/romain/square.py
Query: SELECT TRANSFORM (salary) USING 'square.py' AS b FROM rr

Fails with 
ERROR security.UserGroupInformation: PriviledgedActionException as:romain (auth:SIMPLE) cause:BeeswaxException(message:Driver returned: 9.  Errors: converting to local hdfs://localhost:8020/user/romain/square.py
Cannot run program "square.py": java.io.IOException: error=2, No such file or directory



File: /user/romain/square.py
Query: SELECT TRANSFORM (salary) USING '/tmp/romain/hive_resources/square.py' AS b FROM rr

Works




Error logs

12/04/25 13:56:10 INFO exec.HiveHistory: Hive history file=/tmp/romain/hive_job_log_romain_201204251356_817588712.txt
converting to local hdfs://localhost:8020/user/romain/square.py
12/04/25 13:56:10 INFO SessionState: converting to local hdfs://localhost:8020/user/romain/square.py
12/04/25 13:56:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library
Added resource: /tmp/romain/hive_resources/square.py
12/04/25 13:56:10 INFO SessionState: Added resource: /tmp/romain/hive_resources/square.py
12/04/25 13:56:10 INFO ql.Driver: <PERFLOG method=compile>
12/04/25 13:56:10 INFO parse.ParseDriver: Parsing command: SELECT TRANSFORM (salary) USING 'square.py' AS b FROM rr
12/04/25 13:56:10 INFO parse.ParseDriver: Parse Completed
12/04/25 13:56:10 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
12/04/25 13:56:10 INFO parse.SemanticAnalyzer: Completed phase 1 of Semantic Analysis
12/04/25 13:56:10 INFO parse.SemanticAnalyzer: Get metadata for source tables
12/04/25 13:56:10 INFO metastore.HiveMetaStore: 1: get_table : db=default tbl=rr
12/04/25 13:56:10 INFO metastore.HiveMetaStore: 1: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
12/04/25 13:56:10 INFO metastore.ObjectStore: ObjectStore, initialize called
12/04/25 13:56:10 INFO metastore.ObjectStore: Initialized ObjectStore
12/04/25 13:56:10 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MColumnDescriptor [Table : CDS, InheritanceStrategy : new-table]
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MSerDeInfo [Table : SERDES, InheritanceStrategy : new-table]
12/04/25 13:56:10 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MStorageDescriptor [Table : SDS, InheritanceStrategy : new-table]
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MTable [Table : TBLS, InheritanceStrategy : new-table]
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MSerDeInfo.parameters [Table : SERDE_PARAMS]
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MTable.parameters [Table : TABLE_PARAMS]
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MTable.partitionKeys [Table : PARTITION_KEYS]
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MStorageDescriptor.bucketCols [Table : BUCKETING_COLS]
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MStorageDescriptor.parameters [Table : SD_PARAMS]
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MStorageDescriptor.sortCols [Table : SORT_COLS]
12/04/25 13:56:10 INFO DataNucleus.Persistence: Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MColumnDescriptor.cols [Table : COLUMNS_V2]
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table SERDES
12/04/25 13:56:10 INFO Datastore.Schema: Validating 0 foreign key(s) for table SERDES
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 index(es) for table SERDES
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 unique key(s) for table TBLS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 foreign key(s) for table TBLS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 4 index(es) for table TBLS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table SDS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 foreign key(s) for table SDS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 3 index(es) for table SDS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table CDS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 0 foreign key(s) for table CDS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 index(es) for table CDS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table PARTITION_KEYS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 foreign key(s) for table PARTITION_KEYS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 index(es) for table PARTITION_KEYS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table COLUMNS_V2
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 foreign key(s) for table COLUMNS_V2
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 index(es) for table COLUMNS_V2
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table BUCKETING_COLS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 foreign key(s) for table BUCKETING_COLS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 index(es) for table BUCKETING_COLS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table SORT_COLS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 foreign key(s) for table SORT_COLS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 index(es) for table SORT_COLS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table SERDE_PARAMS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 foreign key(s) for table SERDE_PARAMS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 index(es) for table SERDE_PARAMS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table TABLE_PARAMS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 foreign key(s) for table TABLE_PARAMS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 index(es) for table TABLE_PARAMS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 unique key(s) for table SD_PARAMS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 1 foreign key(s) for table SD_PARAMS
12/04/25 13:56:10 INFO Datastore.Schema: Validating 2 index(es) for table SD_PARAMS
12/04/25 13:56:10 INFO DataNucleus.MetaData: Listener found initialisation for persistable class org.apache.hadoop.hive.metastore.model.MColumnDescriptor
12/04/25 13:56:10 INFO DataNucleus.MetaData: Listener found initialisation for persistable class org.apache.hadoop.hive.metastore.model.MSerDeInfo
12/04/25 13:56:10 INFO DataNucleus.MetaData: Listener found initialisation for persistable class org.apache.hadoop.hive.metastore.model.MStorageDescriptor
12/04/25 13:56:10 INFO DataNucleus.MetaData: Listener found initialisation for persistable class org.apache.hadoop.hive.metastore.model.MTable
12/04/25 13:56:10 INFO DataNucleus.MetaData: Listener found initialisation for persistable class org.apache.hadoop.hive.metastore.model.MFieldSchema
12/04/25 13:56:10 INFO hive.log: DDL: struct rr { string code, string description, i32 total_emp, i32 salary}
12/04/25 13:56:10 INFO parse.SemanticAnalyzer: Get metadata for subqueries
12/04/25 13:56:10 INFO parse.SemanticAnalyzer: Get metadata for destination tables
12/04/25 13:56:10 INFO parse.SemanticAnalyzer: Completed getting MetaData in Semantic Analysis
12/04/25 13:56:10 INFO hive.log: DDL: struct rr { string code, string description, i32 total_emp, i32 salary}
12/04/25 13:56:10 WARN parse.SemanticAnalyzer: Common Gby keys:null
12/04/25 13:56:10 INFO ppd.OpProcFactory: Processing for FS(3)
12/04/25 13:56:10 INFO ppd.OpProcFactory: Processing for SCR(2)
12/04/25 13:56:10 INFO ppd.OpProcFactory: Processing for SEL(1)
12/04/25 13:56:10 INFO ppd.OpProcFactory: Processing for TS(0)
12/04/25 13:56:10 INFO hive.log: DDL: struct rr { string code, string description, i32 total_emp, i32 salary}
12/04/25 13:56:10 INFO hive.log: DDL: struct rr { string code, string description, i32 total_emp, i32 salary}
12/04/25 13:56:10 INFO hive.log: DDL: struct rr { string code, string description, i32 total_emp, i32 salary}
12/04/25 13:56:10 INFO physical.MetadataOnlyOptimizer: Looking for table scans where optimization is applicable
12/04/25 13:56:10 INFO physical.MetadataOnlyOptimizer: Found 0 metadata only table scans
12/04/25 13:56:10 INFO parse.SemanticAnalyzer: Completed plan generation
12/04/25 13:56:10 INFO ql.Driver: Semantic Analysis Completed
12/04/25 13:56:10 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:b, type:string, comment:null)], properties:null)
12/04/25 13:56:10 INFO ql.Driver: </PERFLOG method=compile start=1335387370186 end=1335387370817 duration=631>
Hive history file=/tmp/romain/hive_job_log_romain_201204251356_1028327477.txt
12/04/25 13:56:10 INFO exec.HiveHistory: Hive history file=/tmp/romain/hive_job_log_romain_201204251356_1028327477.txt
12/04/25 13:56:10 INFO ql.Driver: <PERFLOG method=Driver.execute>
12/04/25 13:56:10 INFO ql.Driver: Starting command: SELECT TRANSFORM (salary) USING 'square.py' AS b FROM rr
Total MapReduce jobs = 1
12/04/25 13:56:10 INFO ql.Driver: Total MapReduce jobs = 1
Launching Job 1 out of 1
12/04/25 13:56:10 INFO ql.Driver: Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
12/04/25 13:56:10 INFO exec.Task: Number of reduce tasks is set to 0 since there's no reduce operator
12/04/25 13:56:10 INFO exec.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
12/04/25 13:56:10 INFO exec.ExecDriver: adding libjars: file:///usr/lib/hive/lib/hive-builtins-0.8.1-cdh4b2-SNAPSHOT.jar
12/04/25 13:56:10 INFO exec.ExecDriver: Processing alias rr
12/04/25 13:56:10 INFO exec.ExecDriver: Adding input file hdfs://localhost:8020/user/hive/warehouse/rr
12/04/25 13:56:10 INFO exec.Utilities: Content Summary not cached for hdfs://localhost:8020/user/hive/warehouse/rr
12/04/25 13:56:11 INFO exec.ExecDriver: Making Temp Directory: hdfs://localhost:8020/tmp/hive-beeswax-romain/hive_2012-04-25_13-56-10_200_1143237335457991203/-ext-10001
12/04/25 13:56:11 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/25 13:56:11 WARN snappy.LoadSnappy: Snappy native library is available
12/04/25 13:56:11 INFO snappy.LoadSnappy: Snappy native library loaded
12/04/25 13:56:11 INFO io.CombineHiveInputFormat: CombineHiveInputSplit creating pool for hdfs://localhost:8020/user/hive/warehouse/rr; using filter path hdfs://localhost:8020/user/hive/warehouse/rr
12/04/25 13:56:11 INFO mapred.FileInputFormat: Total input paths to process : 1
12/04/25 13:56:11 INFO io.CombineHiveInputFormat: number of splits 1
Starting Job = job_201204221336_0028, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201204221336_0028
12/04/25 13:56:11 INFO exec.Task: Starting Job = job_201204221336_0028, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201204221336_0028
Kill Command = null/bin/hadoop job  -Dmapred.job.tracker=localhost:8021 -kill job_201204221336_0028
12/04/25 13:56:11 INFO exec.Task: Kill Command = null/bin/hadoop job  -Dmapred.job.tracker=localhost:8021 -kill job_201204221336_0028
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
12/04/25 13:56:13 INFO exec.Task: Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
12/04/25 13:56:13 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2012-04-25 13:56:13,479 Stage-1 map = 0%,  reduce = 0%
12/04/25 13:56:13 INFO exec.Task: 2012-04-25 13:56:13,479 Stage-1 map = 0%,  reduce = 0%
12/04/25 13:56:35 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2012-04-25 13:56:35,630 Stage-1 map = 100%,  reduce = 100%
12/04/25 13:56:35 INFO exec.Task: 2012-04-25 13:56:35,630 Stage-1 map = 100%,  reduce = 100%
12/04/25 13:56:35 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
Ended Job = job_201204221336_0028 with errors
12/04/25 13:56:35 ERROR exec.Task: Ended Job = job_201204221336_0028 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
12/04/25 13:56:35 ERROR ql.Driver: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
12/04/25 13:56:35 INFO ql.Driver: </PERFLOG method=Driver.execute start=1335387370857 end=1335387395709 duration=24852>
MapReduce Jobs Launched: 
12/04/25 13:56:35 INFO ql.Driver: MapReduce Jobs Launched: 
Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
12/04/25 13:56:35 INFO ql.Driver: Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
12/04/25 13:56:35 INFO ql.Driver: Total MapReduce CPU Time Spent: 0 msec
12/04/25 13:56:35 ERROR beeswax.BeeswaxServiceImpl: Exception while processing query
BeeswaxException(message:Driver returned: 9.  Errors: converting to local hdfs://localhost:8020/user/romain/square.py
Added resource: /tmp/romain/hive_resources/square.py
Hive history file=/tmp/romain/hive_job_log_romain_201204251356_1028327477.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201204221336_0028, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201204221336_0028
Kill Command = null/bin/hadoop job  -Dmapred.job.tracker=localhost:8021 -kill job_201204221336_0028
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2012-04-25 13:56:13,479 Stage-1 map = 0%,  reduce = 0%
2012-04-25 13:56:35,630 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201204221336_0028 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched: 
Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
, log_context:7de99cb7-c82b-4401-97b4-293d059c7763, handle:QueryHandle(id:7de99cb7-c82b-4401-97b4-293d059c7763, log_context:7de99cb7-c82b-4401-97b4-293d059c7763))
	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:318)
	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:523)
	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:512)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:337)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1159)
	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:512)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
12/04/25 13:56:36 ERROR security.UserGroupInformation: PriviledgedActionException as:romain (auth:SIMPLE) cause:BeeswaxException(message:Driver returned: 9.  Errors: converting to local hdfs://localhost:8020/user/romain/square.py
Added resource: /tmp/romain/hive_resources/square.py
Hive history file=/tmp/romain/hive_job_log_romain_201204251356_1028327477.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201204221336_0028, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201204221336_0028
Kill Command = null/bin/hadoop job  -Dmapred.job.tracker=localhost:8021 -kill job_201204221336_0028
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2012-04-25 13:56:13,479 Stage-1 map = 0%,  reduce = 0%
2012-04-25 13:56:35,630 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201204221336_0028 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched: 
Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
, log_context:7de99cb7-c82b-4401-97b4-293d059c7763, handle:QueryHandle(id:7de99cb7-c82b-4401-97b4-293d059c7763, log_context:7de99cb7-c82b-4401-97b4-293d059c7763))
12/04/25 13:56:36 ERROR beeswax.BeeswaxServiceImpl: Caught BeeswaxException
BeeswaxException(message:Driver returned: 9.  Errors: converting to local hdfs://localhost:8020/user/romain/square.py
Added resource: /tmp/romain/hive_resources/square.py
Hive history file=/tmp/romain/hive_job_log_romain_201204251356_1028327477.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201204221336_0028, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201204221336_0028
Kill Command = null/bin/hadoop job  -Dmapred.job.tracker=localhost:8021 -kill job_201204221336_0028
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2012-04-25 13:56:13,479 Stage-1 map = 0%,  reduce = 0%
2012-04-25 13:56:35,630 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201204221336_0028 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched: 
Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
, log_context:7de99cb7-c82b-4401-97b4-293d059c7763, handle:QueryHandle(id:7de99cb7-c82b-4401-97b4-293d059c7763, log_context:7de99cb7-c82b-4401-97b4-293d059c7763))
	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:318)
	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:523)
	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:512)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:337)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1159)
	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:512)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)




MR will fail with:
2012-04-25 12:11:27,620 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"code":"00-0000","description":"All Occupations","total_emp":134354250,"salary":40690}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
	at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
	at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Cannot initialize ScriptOperator
	at org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:320)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:531)
	... 9 more
Caused by: java.io.IOException: Cannot run program "square.py": java.io.IOException: error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
	at org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:279)
	... 18 more
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
	at java.lang.ProcessImpl.start(ProcessImpl.java:65)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
	... 19 more

Issues

0
0
0
1
All Issues: 1

Description	From	Last Updated
Would use request.fs.join(..., ...).	bcwalrus	April 26, 2012, 3:09 a.m.

Ship it!

Let's track the "fail to convert to local" issue in a jira. Is this on cdh3 or cdh4?

bcwalrus April 25, 2012, 12:21 p.m.
```
I mean to say mr1 or mr2.
```

romain April 26, 2012, 3:21 a.m.

It is with mr1.

Do you mean a Hive Jira?

apps/beeswax/src/beeswax/views.py (Diff revision 1)

Show all issues

Would use request.fs.join(..., ...).

romain April 26, 2012, 3:21 a.m.

Actually it is not working as the file path is absolute. A better solution should be to get the fullpath from the form at some point.


join (p, q) -- function of module posixpath
Join the paths p and q intelligently: If q is an absolute path, the return value is q. Otherwise, the concatenation of p and q is returned, with a slash ('/') inserted unless p is empty or ends in a slash.

>>> import posixpath
>>> posixpath.join('hdfs://localhost:8020', '/user/romain/square.py')
'/user/romain/square.py'
>>> posixpath.join('hdfs://localhost:8020/', '/user/romain/square.py')
'/user/romain/square.py'
>>> posixpath.join('hdfs://localhost:8020/', 'user/romain/square.py')
'hdfs://localhost:8020/user/romain/square.py'
>>> posixpath.join('hdfs://localhost:8020', 'user/romain/square.py')
'hdfs://localhost:8020/user/romain/square.py'

You have a pending review.

Review Board 3.0.18

HUE-706 [beeswax] Adding a resource to a query fails

Status: Closed (submitted)