Can data mining predict the movement of S&P 500 ?

Poll

Can data mining predict the movement of S&P 500 ? [209 votes total] Most of the time (18) 9% Occasionally, under some conditions (76) 36% Only in research, but not in real life (34) 16% No, it cannot successfully predict S&P (66) 32% Don't know (15) 7%

Comments

• Julia Lin, Aug 22, 2002
  Historical performances don't support future performance prediction. That's why stock prices are not predictable.
  
  
• Evgenii, Aug 12, 2002
  I developed a program for Stock Market Forecasting based on approach:
  Kovalerchuk B., Vityaev E. Data Mining in Finance: Advances in Relational and Hybrid methods. Kluwer Academic Publishers, 2000, p.308.
  http://www.math.nsc.ru/LBRT/logic/viyaev
  
  During some years I trained the program on S&P500
  emini, using data from CME. Results are presented on my exclusive website on "Stock Market Forecasting". The address of this site may be send by request. The Annual Gross Gain was more then 150%.
  
  
• Roger, Aug 10, 2002
  "Historical performance is no guarantee of future returns"
  
  
• Christian, Aug 10, 2002,
  interesting how many people expect from NN models than classical methods better result but in my humble opinion the problem is more theoretic & the question of effecitivity of information-diffusion in markets ?
  
  
• Fredrick Nowatzke, Aug 9, 2002
  Aaron,
  I'd be very interested to know some rough annual Return On Margin (ROM) numbers for your S&P model, along with any others you care to share. My own work using a high end NN as the primary model development tool returns about 300% annual ROM for the S&P.
  
  
• Aaron J. Owens, Aug 9, 2002
  A colleague and I have beeen using proprietary data mining techniques to predict the short-term (day to day) movements of futures for the S&P 500 and more than a dozen related commodity, stock, and bond markets. The key issue is not "How accurately you can predict up versus down days?" but rather "Can you make predictions and use them with a trading strategy to make money with minimal risk?".
  
  As with any data mining problem, the effective solution involves having the right data base, formulating the problem correctly, and using the right data mining tool. The first two issues require domain knowledge best provided by a successful hedge fund trader. The latter is crucial: I have tried many out-of-the-box data mining algorithms (PLS, neural nets, various decision trees, ...) and found that generally they cannot predict the futures markets significantly better than chance. Our proprietary method, specifically tuned to this kind of problem, does.
  
  
• Vladimir Miheev, Aug 9, 2002
  The knowledge discovery activities on S&P 500 should have two main directions:
  1) Model should predict not S&P 500 itself, but the behavior of real successful expert.
  2) Solutions for brokerage house. Model ought to generate unifying hypothesis on S&P 500 based on stock market gamblers move, since the actions of the client is well known to the broker.
  
  
• Martijn Wiertz, Aug 9, 2002
  The condition is probably, as with all data mining activities, to get the right data to describe what influences the things you want to predict.
  (ties in nicely with the previous poll on methodologies by the way, avoiding diving into data headfirst without first giving some thought on the business issue at hand)
  
  And as John Elder mentions: you don't have to get it right 100% of the time, as long as you can improve it enough to make you money ;-)
  
  
• Fredrick Nowatzke, Aug 8, 2002
  In my research and consulting for investors, the S&P 500 Index future (CME) is one of the best major financial instruments for predictive modeling, at least in the short term.
  
  Annual return on margin results are typically in a range of 300% to 400% for a trading strategy based on detecting short term moves of 2 or 3 days. Other commodities such as the Euro, Yen, T-Bills and most of the agricultural futures are much harder to model effectively. I do not have a good theory that explains this difference.
  Effective model development for the S&P is a rather complex undertaking and requires the development of multiple overlayed models, dozens of carefully selected independent variables and the use of high end data mining software.
  
  
• John Elder, Aug 8, 2002
  It's tough, but doable. Building financial investment models is the main work Elder Research, Inc. does. We've found that markets, or their directions, can be predicted enough better than randomly to make it worthwhile.
  
  
• Editor, Aug 7, 2002
  Predicting Stock Market
  I frequently see references for various data mining programs that can successfully predict stock market indices such as S&P 500, but it is hard to judge their success, especially given recent volatility. What is your opinion (and experience) -- can data mining methods predict S&P 500 or other stock market indicators?