I use the FRED database very intensively for economic studies. Unfortunately, in the past I had to observe regularly that the data history of the database is NOT updated, only the current data, BUT NOT the entire history of the data series.
It is normal with economic data that not only the current data is calculated but also the whole history of the data series is adjusted back.
It is therefore absolutely essential that the entire data history of a data series is updated!!!
I would therefore like to ask the Optuma team to implement structures to ensure that the FRED database in Optuma has the same data as the original FRED database as on their website (https://fred.stlouisfed.org/).
I understand that the FRED database is a monster but without the right data the FRED database in Optuma makes no sense!
Since some economic data is released every day, it would be necessary to update Optuma’s entire FRED database on a daily basis, including of course the data history and not just the current data.
I have the same problem with FRED data so I wrote some code to query FRED DB and download updated series data daily. I created a directory I use as a data source.
I find it useful to merge various series into new data. Example is average cost per house times number if houses sold provides house value per month.
If there is enough interest I can turn it into a user friendly app at some cost.
I am not sure whether you have noticed my post about the data history issue of the Optuma FRED database (see above) therefore I send you this post to bring this topic again to your attention.
To sum up the Optuma FRED database issue: The problem is that the data history of the Optuma FRED database is NOT updated, only the current data, BUT NOT the entire history of the data series are updated. Since most of the economic data series are adjusted back it is absolutely necessary that the entire data history is updated when a new data release is published.
If you calculate e.g. the percent change from a year ago of the CPI and the history in the Optuma FRED database is not the same as the FRED database history on their web site you get of course the wrong inflation calculation.
This is really problematic when everyone on the street knows the right values and you have the wrong values in Optuma since the Optuma FRED database history is not update only the current value is updated and of course correct. The credibility in your work vanishes in a second.
This one is interesting as we did not realise that FRED would restate economic data. What we are working on is determining a way to get a list of which data sets need to be updated so we can handle it automatically.
Do you have a couple of examples that you have found so that we can look into it?
thank your for your replay. Concerning the FRED database I have to say that most if not any economic data series are adjusted back or restated if new data is published. Interest rates e.g. are of course not restated.
I have attached my watchlist with the FRED database series I am currently use. All data series, except the ones under the “Headers” “FED Balance Sheet & US Corporate Bond Yields” and “Commodities” are ALWAYS restated when new data is published.
I am here again. I did some thought about how to solve the “restate” issue since there are many many data series in the FRED database that are restated.
Perhaps it would be a good idea that you update your Optuma FRED database every day so the Optuma FRED database has the same data history as the FRED database on their web site and each time an Optuma user opens a watchlist or chart that includes a FRED data series the WHOLE data series file is downloaded from the Optuma FRED database and replaces the old FRED data series file that is already stored on the user’s computer.
A part of the issue, as far as I am able to understand it, is the “old” FRED data series file that is already stored on the user’s computer. That data file has to be replaced and not only updated with the lastest data. The data history is the problem not the update with the current data.
I check 3300 series I watch each day. The problem is FRED throttles requests so I pause a random period of time between each call. Still, at points I will begin to get null data back and need to pause to 30 seconds before FRED accepts requests again. It takes around 25 minutes to find those series I watch that have been updated.
You may be able to work with them to get a list of daily/weekly updates or for them to remove throttling in middle of night.