If you know how to convert Story Points to Days or Hours, why estimate in one way and convert them into other way? You could directly estimate in time units, isn’t it?
Story points are rough estimates and are error prone whereas time estimates are precise, so the translation doesn’t work. A 5 point story could be smaller than a 3 pointer and vice versa due to chance of error at product backlog level.
Story points are relative estimates and represent size of the work to complete without considering who is going to work on it where as time estimates are generally attached to who is working on it.
Relative estimation in Story points could be given by a technical expert and actual schedule will be derived only when the development team is in place. If you got a slower team, the schedule is going to be longer and faster team completes the work sooner. Once the points are translated into time units, that is generally taken as fixed. For example a 100 point project can take 3 months or 6 months depending on the team where as 100 days (conversion is 1 point = 1 day) work is expected to be completed in 100 days.
Same team working on same product backlog in different organisations could take different amount of time due to various factors in the organisation. So, collecting the empirical data during the execution is more accurate than predicting all those factors at the time of estimation. Story Points separate these aspects ie. size of the work and how long it takes to complete the work.