hadoop - Running Hive through Oozie -

i'm facing issues while running hive through oozie. in oozie console, i'm getting following error:

2013-05-03 04:48:24,248  warn hiveactionexecutor:542 - user[ambari_qa] group[-] token[]  app[hive-wf] job[0000013-130502155316029-oozie-oozi-w]  action[0000013-130502155316029-  oozie-oozi-w@hive-node] launcher exception:    org.apache.hadoop.hive.ql.metadata.hiveexception: java.lang.classnotfoundexception:     org.apache.hcatalog.security.hdfsauthorizationprovider java.lang.runtimeexception: org.apache.hadoop.hive.ql.metadata.hiveexception:  java.lang.classnotfoundexception: org.apache.hcatalog.security.hdfsauthorizationprovider @ org.apache.hadoop.hive.ql.session.sessionstate.start(sessionstate.java:293) @ org.apache.hadoop.hive.cli.clidriver.run(clidriver.java:669) @ org.apache.hadoop.hive.cli.clidriver.main(clidriver.java:613) @ org.apache.oozie.action.hadoop.hivemain.runhive(hivemain.java:303) @ org.apache.oozie.action.hadoop.hivemain.run(hivemain.java:280) @ org.apache.oozie.action.hadoop.launchermain.run(launchermain.java:37) @ org.apache.oozie.action.hadoop.hivemain.main(hivemain.java:55) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:25) @ java.lang.reflect.method.invoke(method.java:597) @ org.apache.oozie.action.hadoop.launchermapper.map(launchermapper.java:467) @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:50) @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:436) @ org.apache.hadoop.mapred.maptask.run(maptask.java:372) @ org.apache.hadoop.mapred.child$4.run(child.java:255) @ java.security.accesscontroller.doprivileged(native method) @ javax.security.auth.subject.doas(subject.java:396) @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1178) @ org.apache.hadoop.mapred.child.main(child.java:249) caused by: org.apache.hadoop.hive.ql.metadata.hiveexception: java.lang.classnotfoundexception: org.apache.hcatalog.security.hdfsauthorizationprovider @ org.apache.hadoop.hive.ql.metadata.hiveutils.getauthorizeprovidermanager(hiveutils.java:342) @ org.apache.hadoop.hive.ql.session.sessionstate.start(sessionstate.java:287) ... 19 more caused by: java.lang.classnotfoundexception: org.apache.hcatalog.security.hdfsauthorizationprovider @ java.net.urlclassloader$1.run(urlclassloader.java:202) @ java.security.accesscontroller.doprivileged(native method) @ java.net.urlclassloader.findclass(urlclassloader.java:190) @ java.lang.classloader.loadclass(classloader.java:306) @ sun.misc.launcher$appclassloader.loadclass(launcher.java:301) @ java.lang.classloader.loadclass(classloader.java:247) @ java.lang.class.forname0(native method) @ java.lang.class.forname(class.java:247) @ org.apache.hadoop.hive.ql.metadata.hiveutils.getauthorizeprovidermanager(hiveutils.java:335) ... 20 more 

i have mentioned system classpath in job.properties file. here job properties file:

namenode=hdfs://ip-10-0-0-92:8020 jobtracker=ip-10-0-0-93:50300 queuename=default wferoot=wfe  oozie.use.system.libpath=true oozie.libpath=/user/oozie/share/lib/hive  oozie.wf.application.path=${namenode}/user/${user.name}/${wferoot}/hive-oozie 

and here workflow.xml file:

<action name="hive-node">     <hive xmlns="uri:oozie:hive-action:0.2">      <job-tracker>${jobtracker}</job-tracker>         <name-node>${namenode}</name-node>      <prepare>             <delete path="${namenode}/user/${wf:user()}/${wferoot}/output-data/hive"/>             <mkdir path="${namenode}/user/${wf:user()}/${wferoot}/output-data"/>         </prepare>      <job-xml>hive-site.xml</job-xml>         <configuration>             <property>                 <name>mapred.job.queue.name</name>                 <value>${queuename}</value>             </property>       <property>                 <name>oozie.log.hive.level</name>                 <value>debug</value>             </property>             <property>                 <name>oozie.hive.defaults</name>                 <value>hive-default.xml</value>             </property>         </configuration>         <script>script.q</script>     </hive>     <ok to="end"/>     <error to="fail"/> </action>  <kill name="fail">     <message>hive failed, error message[${wf:errormessage(wf:lasterrornode())}]</message> </kill> <end name="end"/> 

i have copied hive-site.xml & hive-default.xml files hdfs.

any idea whats going on here?

i got same error when did not have jars hcatalog in oozie sharelib folder. line in error output gives away.

java.lang.runtimeexception: org.apache.hadoop.hive.ql.metadata.hiveexception:  java.lang.classnotfoundexception: org.apache.hcatalog.security.hdfsauthorizationprovider 

to verfiy, run

hadoop fs -ls /user/oozie/share/lib/hive | grep hcatalog 

you should expect see both hcatalog-core-0.5.0 , hcatalog-server-extensions-0.5.0 (the version might different depending on how distribution packages it).

the solution run following. need run hdfs (or whichever user cluster's root user) running sudo su hdfs if dfs.permissions = true on cluster.

cd /usr/lib/hcatalog/share/hcatalog ls hadoop fs -put hcatalog-core-0.5.0.jar /user/oozie/share/lib/hive hadoop fs -put hcatalog-server-extensions-0.5.0.jar /user/oozie/share/lib/hive hadoop fs -chmod 777 /user/oozie/share/lib/hive/hcatalog-core-0.5.0.jar hadoop fs -chmod 777 /user/oozie/share/lib/hive/hcatalog-server-extensions-0.5.0.jar 

again, exact file names , file paths vary hadoop distribution. i'm suggesting 777 permissions (read, write, , execute owner, group, , other users) because permissive , guarantee through bug. if have security concerns, different permissions might required. can @ other files in folder reasonable default.

this problem can occur when setting or upgrading new hadoop cluster if distributions upgrade tools/documentation miss step.