This Data Visualization is the First Step for Effective Feature Selection
Understanding the most important features to use is crucial for developing a model that performs well. Knowing which features to consider requires experimentation, and proper visualization of your data can help clarify your initial selections. The scatter pairplot is a great place to start.
Image by Benjamin O. Tayo.
The scatter pairplot is a visualization of pairwise relationships in a dataset and is the first step for effective feature selection. It provides a qualitative analysis of the pairwise correlation between features and is a powerful tool for feature selection and dimensionality reduction. For an introduction of the pairplot using the seaborn package, see this link: https://seaborn.pydata.org/generated/seaborn.pairplot.html
In this article, we will analyze a portfolio of stocks to examine the ones that are strongly correlated to the overall market. The portfolio contains 22 stocks (see Table 1) from different sectors such as Healthcare, Real Estate, Consumer Discretionary, Energy, Industrials, Telecommunication Services, Information Technology, Consumer Staples, and Financials.
|AAL||American Airlines||EDIT||Editas Medicine||UAL||United Airlines|
|AAPL||Apple||HPP||Hudson Pacific Properties||WEN||Wendy|
|ABT||Abbott Laboratories||JNJ||Johnson & Johnson||WFC||Wells Fargo|
|BXP||Boston Properties||MRO||Marathon Oil Corporation||XOM||Exxon Mobile|
|CCL||Carnival Corporation||PFE||Pfizer||SP500||Stock Market Index|
|DAL||Delta Airlines||SLG||SL Green Realty|
Table 1. Portfolio of 22 stocks from diverse sectors.
Our goal is to answer the question: which stocks in the portfolio correlate strongly with the stock market? We will use the S&P 500 index as a measure of the total stock market. We will assume a threshold correlation coefficient of 70% for a stock to be considered to be strongly correlated to the S&P 500.
Data Collection and Processing
Raw data were obtained from the yahoo finance website: https://finance.yahoo.com/
The historical data for each stock has information on daily open price, high price, low price, and closing price. The CSV file was downloaded for each stock, and then the column “close” was extracted and combined to create the dataset, which can be found here: portfolio.csv
Generate Scatter Pairplot
Calculate Covariance Matrix
The scatter pairplot is the first step, which provides a qualitative analysis of pairwise correlations between features. To quantify the degree of correlation, the covariance matrix has to be computed.
Compressed Output Showing Pairplots and Correlation Coefficients
Since we are only interested in the correlations between the 22 stocks in the portfolio with the S&P 500, Figure 1 below shows the final output from our analysis.
Figure 1. Scatter pairplots and correlation coefficients between portfolio stocks and the S&P 500.
Figure 1 shows that out of the 22 stocks, 8 have a correlation coefficient less than 70%. Interestingly, except for WEN stock, all the other stocks have a positive correlation with the S&P 500 index.
The full covariance matrix is shown in Figure 2.
Figure 2. Covariance matrix visualization.
In summary, we’ve shown how the scatter pairplot can be used as a first step for feature selection. Other advanced methods for feature selection and dimensionality reduction include the following: PCA (Principal Component Analysis) and LDA (Linear Discriminant Analysis).
- A simple static visualization can often be the best approach
- How to create stunning visualizations using python from scratch
- Creating Good Meaningful Plots: Some Principles