unix - grep -f alternative for huge files -


grep -f -f file1  file2 

file1 90 mb (2.5 million lines, 1 word per line)

file2 45 gb

that command doesn't produce whatsoever, no matter how long leave running. clearly, beyond grep's scope.

it seems grep can't handle many queries -f option. however, following command does produce desired result:

head file1  >  file3 grep -f -f file3   file2 

i have doubts whether sed or awk appropriate alternatives either, given file sizes.

i @ loss alternatives... please help. worth learn sql commands? easy? can point me in right direction?

try using lc_all=c . turns searching pattern utf-8 ascii speeds 140 time original speed. have 26g file take me around 12 hours down couple of minutes. source: grepping huge file (80gb) way speed up?

so is:

lc_all=c fgrep "pattern" <input >output 

Comments

Popular posts from this blog

linux - Does gcc have any options to add version info in ELF binary file? -

android - send complex objects as post php java -

charts - What graph/dashboard product is facebook using in Dashboard: PUE & WUE -