Loading multiple CSV files into MySQL -


i working on metrics project team. have load several different reports central repository , create tables , reports off of data.

the data sources are:

  1. csv files
  2. pdfs
  3. ad-hoc/manual data.

i playing talend , mysql. little confused how load csv files. should have collection of directories , 1 or more scheduled tasks load files?

another thought write custom file processor load file based on naming convention. thoughts?

"pdf" complicated. pdf... "ad-hoc/manual data" needs more details.
if focus on csv , question related guys if i'm right, i'd writing app calls sp in mysql db, handing on full path csv (and additional data, such table's "user friendly name" if needed - or other meta-data you'd store) executes import using mysql load data.
reason is, there can many rules in "business logic" after csv imported, , it's easier maintain app according changing business requirements, changing db behavior time, and, if goes terribly wrong db safe , "import manager app" fails - don't have store neither nor csvs on same system db is.
dbs, relational dbs storing data, , retrieving data rapidly based on 'set theory', not taking care of how data gets system.

think these questions before start implementing anything:

  • what happens csv after processed? can deleted? should moved e.g. "processed" folder? should remain/stay intact?
  • if should stay , was, should know processed file? (set "ready archive" flag, instance? touch "last modified" date , set 1950.01.01? add property file?
  • what should if csv import fails (e.g. invalid data in file, or null value shouldn't have nulls)? display error? mark csv unusable? send e-mail? move "processing_failed" folder?
  • what if file count grows huge in input folder?
  • how can change import/process/etc if business logic changes, or csv format changes?

and on. think through options have , decide.

i hope answered question ;)


Comments

Post a Comment

Popular posts from this blog

linux - Does gcc have any options to add version info in ELF binary file? -

i mean whether gcc can insert source code version infor elf binary section or similar. not want change source file, add info gcc option in makefile. you can emit version info text file, turn text file object file statically link executable. the first step simple have write code: script or write version info in format plain text file. write makefile rule produce version.o version.txt, using objcopy . you'll have object file 2 useful symbols defined in it: beginning , end of textual version info. add generated object executable, , you'll able access version 2 ways: running strings on binary, or writing code in application print version string (you'll need declare start , end symbols variables in header file).
Read more

android - send complex objects as post php java -

is possible send contact numbers via post php class inside eclipse ? or post limited primitive types , have find work arounds send them string ? i propose: send them json in post request, befor start, have @ gson project , keep in mind set content application -> json. luck
Read more

charts - What graph/dashboard product is facebook using in Dashboard: PUE & WUE -

i love facebook dashboard: pue & wue. beautifull can point out if there open source or commercial product offer similar style? here link https://www.facebook.com/forestcitydatacenter?sk=app_288655784601722&app_data so read through 29,888 lines of javascript ya.. biggest js file i've ever seen. looks make graph using variety of tools, including tween.js . it'd fascinating chat whoever made it. code wheel graph object starts @ line 27,266 text var graphcomponent = class.extend({ . can find source @ https://www.fbpuewue.com/assets/application-3a3af20f7fa3591229c79442ae3d1c26.js . sorry if dissapoints it's not awesome plugin can use :/ share pain. let me know if have questions :)
Read more