ruby on rails - Converting string to float or decimal sometimes results in long precision -
i have string want convert decimal.
eg.
tax_rate = "0.07" tax_rate.to_d
in cases converts 0.07
converts 0.07000000000000001
. why?
if helps, when @ log of when insert value db, relevant part:
["tax_rate", #<bigdecimal:7f9b6221d7c0,'0.7000000000 000001e-1',18(45)>]
hopefully makes sense someone.
disclaimer:
have feeling going ask why i'm doing this.
i've created simple settings model, user can update global settings.
setting
model has name
, value
, var_type
columns.
value
column string. use helper method retrieve value in appropriate format depending on value of var_type
column.
i cannot explain why there chance can tell how avoid having kind of trouble when dealing numbers: use rationals.
here documentation: http://ruby-doc.org/core-1.9.3/rational.html
as stated, rational give exact number want , avoid rounding errors.
from doc:
10.times.inject(0){|t,| t + 0.1} #=> 0.9999999999999999 10.times.inject(0){|t,| t + rational('0.1')} #=> (1/1)
let me know if solves problem. : )
Comments
Post a Comment