InputSplit customization in Hadoop -


i understand in hadoop, large input file splits small files , gets processed in different nodes map functions. got know can customize inputsplits. know if following type of customization possible inputsplit:

i have large input file coming in hadoop, want subset of file, i.e. set of lines in file go along every input split. mean data chunks of large file should contain these set of lines, irrespective of whatever way file split.

to make question more clear, if need compare part of input file (say a) rest of file content, in case inputsplits going map function need have a part comparison. kindly guide me on this.

theoretically possible split big file (a, b, c, d, ...) splits (a, b), (a, c), (a, d), .... you'd have write lot of custom classes purpose. filesplit, extends inputsplit, says split file begins @ position start , has fixed length. actual access file done recordreader, i.e. linerecordreader. have implement code, read not actual split, header (part a) of file well.

i'd argue, approach you're looking unpractical. reason record reader accesses positions (start, start+length) data-locality. big file, parts a , z on 2 different nodes.

depending on size of part a, better idea storing common part in distributedcache. in way access common data in each of mappers in efficient way. refer javadoc , http://developer.yahoo.com/hadoop/tutorial/module5.html#auxdata further information.


Comments

Post a Comment

Popular posts from this blog

linux - Does gcc have any options to add version info in ELF binary file? -

i mean whether gcc can insert source code version infor elf binary section or similar. not want change source file, add info gcc option in makefile. you can emit version info text file, turn text file object file statically link executable. the first step simple have write code: script or write version info in format plain text file. write makefile rule produce version.o version.txt, using objcopy . you'll have object file 2 useful symbols defined in it: beginning , end of textual version info. add generated object executable, , you'll able access version 2 ways: running strings on binary, or writing code in application print version string (you'll need declare start , end symbols variables in header file).
Read more

android - send complex objects as post php java -

is possible send contact numbers via post php class inside eclipse ? or post limited primitive types , have find work arounds send them string ? i propose: send them json in post request, befor start, have @ gson project , keep in mind set content application -> json. luck
Read more

charts - What graph/dashboard product is facebook using in Dashboard: PUE & WUE -

i love facebook dashboard: pue & wue. beautifull can point out if there open source or commercial product offer similar style? here link https://www.facebook.com/forestcitydatacenter?sk=app_288655784601722&app_data so read through 29,888 lines of javascript ya.. biggest js file i've ever seen. looks make graph using variety of tools, including tween.js . it'd fascinating chat whoever made it. code wheel graph object starts @ line 27,266 text var graphcomponent = class.extend({ . can find source @ https://www.fbpuewue.com/assets/application-3a3af20f7fa3591229c79442ae3d1c26.js . sorry if dissapoints it's not awesome plugin can use :/ share pain. let me know if have questions :)
Read more