logging - Log delay in Amazon S3 -

i have hosted in amazon s3, , need log files calculate statistics "get", "put", "list" operations in objects.

and i've observed log files organized weirdly. don't know when log appear(not immediatly, @ least 20 minutes after operation) , how many lines of logs contained in 1 log file.

after that, need download these log files , analyse them. can't figure out how this.

can help? thanks.

what describe (log files being made available delays , being in unpredictable order) declared aws behaviour expect. nature of distributed system, aws s3 using provide s3 service, same request may served each time different server - have seen 5 different ip addresses being provided publishing.

so solution is: accept delay, see delay experience , add time , learn living total delay (i expect 30 60 minutes, statistics tell more).

if need log records ordered, have either sort them yourself, or search log processing solutions - have seen applications being offered purpose.

in case, need log file short delay, have make logs , means, have write , run frontend, gives access files on s3 , @ same time keeps logging needed.

i run such solution, users user name , password , url of frontend. send request, evaluate, if provide proper credentials , if allowed see given resource, , if so, create few minutes valid temporary url resource , redirect request that.

but such fronted costs money (you have run frontend somewhere) , less robust, accessing directly aws s3.

good luck, lulu.