hadoop - Pattern match input files for Amazon Elastic MapReduce -


i trying run mapreduce streaming job takes input files directories in s3 bucket match given pattern. pattern bucket-name/[date]/product/logs/[hour]/[logfilename]. example log in while bucket-name/2013-05-02/product/logs/05/log123456789.

i can job work passing hour portion of file name wildcard. example: bucket-name/2013-05-02/product/logs/*/. picks each log file each hour, , passes them individually mappers.

the problem comes try make date wildcard, example: bucket-name/*/product/logs/*/. when this, job gets created no tasks created , fails. error printed in syslog.

2013-05-02 08:03:41,549 error org.apache.hadoop.streaming.streamjob (main): job not successful. error: job initialization failed: java.lang.outofmemoryerror: java heap space     @ java.util.regex.matcher.<init>(matcher.java:207)     @ java.util.regex.pattern.matcher(pattern.java:888)     @ org.apache.hadoop.conf.configuration.substitutevars(configuration.java:378)     @ org.apache.hadoop.conf.configuration.get(configuration.java:418)     @ org.apache.hadoop.conf.configuration.getlong(configuration.java:523)     @ org.apache.hadoop.mapred.skipbadrecords.getmappermaxskiprecords(skipbadrecords.java:247)     @ org.apache.hadoop.mapred.taskinprogress.<init>(taskinprogress.java:146)     @ org.apache.hadoop.mapred.jobinprogress.inittasks(jobinprogress.java:722)     @ org.apache.hadoop.mapred.jobtracker.initjob(jobtracker.java:4238)     @ org.apache.hadoop.mapred.eagertaskinitializationlistener$initjob.run(eagertaskinitializationlistener.java:79)     @ java.util.concurrent.threadpoolexecutor$worker.runtask(threadpoolexecutor.java:886)     @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:908)     @ java.lang.thread.run(thread.java:662)  2013-05-02 08:03:41,549 info org.apache.hadoop.streaming.streamjob (main): killjob...

on further testing, looks multiple wildcard syntax works expected in command line client. had trouble getting work @ first, before realizing requiring ruby 1.8.7 meant requires exactly ruby 1.8.7, , nothing later.


Comments

Post a Comment

Popular posts from this blog

linux - Does gcc have any options to add version info in ELF binary file? -

i mean whether gcc can insert source code version infor elf binary section or similar. not want change source file, add info gcc option in makefile. you can emit version info text file, turn text file object file statically link executable. the first step simple have write code: script or write version info in format plain text file. write makefile rule produce version.o version.txt, using objcopy . you'll have object file 2 useful symbols defined in it: beginning , end of textual version info. add generated object executable, , you'll able access version 2 ways: running strings on binary, or writing code in application print version string (you'll need declare start , end symbols variables in header file).
Read more

javascript - Clean way to programmatically use CSS transitions from JS? -

as title implies, there proper way set initial css properties (or class) , tell browser transition these value? for example ( fiddle ): var el = document.queryselector('div'), st = el.style; st.opacity = 0; st.transition = 'opacity 2s'; st.opacity = 1; this not animate opacity of element in chrome 29/firefox 23. because ( source ): [...] you’ll find if apply both sets of properties, 1 after other, browser tries optimize property changes, ignoring initial properties , preventing transition. behind scenes, browsers batch property changes before painting which, while speeding rendering, can have adverse affects. the solution force redraw between applying 2 sets of properties. simple method of doing accessing dom element’s offsetheight property [...] in fact, hack work in current chrome/firefox versions. updated code ( fiddle - click run after opening fiddle run animation again): var el = document.queryselector('div'...
Read more

android - send complex objects as post php java -

is possible send contact numbers via post php class inside eclipse ? or post limited primitive types , have find work arounds send them string ? i propose: send them json in post request, befor start, have @ gson project , keep in mind set content application -> json. luck
Read more