ruby on rails - Converting string to float or decimal sometimes results in long precision -


i have string want convert decimal.
eg.

tax_rate = "0.07" tax_rate.to_d 

in cases converts 0.07 converts 0.07000000000000001. why?

if helps, when @ log of when insert value db, relevant part:

["tax_rate", #<bigdecimal:7f9b6221d7c0,'0.7000000000 000001e-1',18(45)>] 

hopefully makes sense someone.

disclaimer:
have feeling going ask why i'm doing this.
i've created simple settings model, user can update global settings.
setting model has name, value , var_type columns.
value column string. use helper method retrieve value in appropriate format depending on value of var_type column.

i cannot explain why there chance can tell how avoid having kind of trouble when dealing numbers: use rationals.

here documentation: http://ruby-doc.org/core-1.9.3/rational.html

as stated, rational give exact number want , avoid rounding errors.

from doc:

10.times.inject(0){|t,| t + 0.1}              #=> 0.9999999999999999 10.times.inject(0){|t,| t + rational('0.1')}  #=> (1/1) 

let me know if solves problem. : )


Comments

Popular posts from this blog

linux - Does gcc have any options to add version info in ELF binary file? -

android - send complex objects as post php java -

charts - What graph/dashboard product is facebook using in Dashboard: PUE & WUE -