tag:blogger.com,1999:blog-8497841368693123869.post1512299526226073774..comments2017-06-23T08:27:35.668-04:00Comments on explain my data: Should you apply PCA to your data?Alex Rubinsteynhttps://plus.google.com/113490018825870321190noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-8497841368693123869.post-82561658724215053412017-06-22T18:32:25.763-04:002017-06-22T18:32:25.763-04:00Hi Sergey,
Wonderful article! I'm curious abo...Hi Sergey,<br /><br />Wonderful article! I'm curious about your accuracy estimation. Is this prediction accuracy based on a held-out test set or cross-validation, and if not, what do you think the impact of PCA and other dimensionality reduction techniques would be in this case? It seems to me that multicollinearity in the predictors should make the model less stable on repeated subsamples of the data (or at generalizing to new data), which could be one motivation for PCA outside of dimensionality reduction, and I wonder if the added noise introduced by PCA may have some positive attributes in improving stability of the model, but I have not run any simulations to test these hypotheses so I'd love to know your thoughts before I delve further into the problem.Charlie Redmonhttp://www.blogger.com/profile/15877179312477296447noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-19036189721003008892017-05-25T11:36:59.990-04:002017-05-25T11:36:59.990-04:00Hi Sergey,
Nice work. It seems pre-processing bef...Hi Sergey,<br /><br />Nice work. It seems pre-processing before PCA helps in obtaining the useful max variance dimensions. I am curious about one thing - what happens to the performance comparison if you extract n/2 dimensions (n is the # of original dimensions) for each pre-processing case? It might actually be useful to plot out the performance curves for each of these cases, while mentioning the 99% variance caps, to get the full picture. It might be so that ZCA+PCA is 'steadier' in choosing dimensions (# of dimensions extracted is consistently higher) and might perform worse if restricted to lesser amount of dimensions.Sushrut Thorathttp://www.blogger.com/profile/01748915279070088297noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-13362286542986250132017-04-07T02:03:04.793-04:002017-04-07T02:03:04.793-04:00Depending on the academic institution a student re...Depending on the academic institution a student researcher is associated with, graduate level research can be defined in a variety of ways. Some institutions have rigorous guidelines you must abide by while other academic entities have more relaxed rules.<a href="http://www.qualitativeresearchcritique.com/" rel="nofollow">useful site</a><br /><br />James Kateronhttp://www.blogger.com/profile/16941590433247270275noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-47825344287454717192016-08-06T06:49:45.897-04:002016-08-06T06:49:45.897-04:00when did uase pca in data and what kind of data s...when did uase pca in data and what kind of data set using pacKannan Khttp://www.blogger.com/profile/09278648098125151506noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-33041417064344492132015-11-09T05:02:11.216-05:002015-11-09T05:02:11.216-05:00The stats within statistics report gives you more ...The stats within statistics report gives you more understand then you almost becomes more able to produce your ideas by yourself, so looking forward to influence more the ideas within this. <a href="http://www.statisticaldataanalysis.net/faqs-on-statistical-data-analysis/" rel="nofollow">statistical analysis with missing data</a>Erma Casiashttp://www.blogger.com/profile/05435675622017105709noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-85537348592995910592015-01-02T01:52:15.825-05:002015-01-02T01:52:15.825-05:00ZCA is usually used as normalization. Rotation do...ZCA is usually used as normalization. Rotation does affect the RF. By the way, Your blog is really informative. Thanks for sharing.Visit <a href="http://great-college-paper.com/" rel="nofollow">college paper</a> for best papers.Terrance Doughertyhttp://www.blogger.com/profile/13867150602609408683noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-9273203251665826122014-05-14T20:57:17.478-04:002014-05-14T20:57:17.478-04:00The results for PCA on 0-to-1 normalized data also...The results for PCA on 0-to-1 normalized data also looks encouraging and I am wondering why it wasn't mentioned. It achieves much better dimensionality reduction with relatively comparable accuracy rates (to ZCA+PCA).Nataraj Dhttp://www.blogger.com/profile/11619337301117758554noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-66446643585985849302012-08-31T12:44:03.636-04:002012-08-31T12:44:03.636-04:00If you have a good predictor in your dataset and a...If you have a good predictor in your dataset and another variable, which is highly correlated to the good predictor, both will be projected to the same dimension and noise is added to the good predictor. ItÂ´s like blurring the "good" variable<br />I generated a toy dataset which illustrates that problem (btw. my post was inspired by yours):<br />http://machine-master.blogspot.de/2012/08/pca-or-polluting-your-clever-analysis.htmlChristoph Molnarhttp://www.blogger.com/profile/10027428521484723878noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-2570546843374560462012-07-24T02:38:04.315-04:002012-07-24T02:38:04.315-04:00I hate how the answer with these things always see...I hate how the answer with these things always seems to be "do it all the ways then validate"...<br /><br />work, work, work :-)Johnhttp://www.blogger.com/profile/18312224023541798450noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-78993939193343849342012-07-12T18:32:35.006-04:002012-07-12T18:32:35.006-04:00hi amy! see the Addendum table at the bottom of t...hi amy! see the Addendum table at the bottom of the post =)Sergey Feldmanhttp://www.blogger.com/profile/11369485324420056756noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-40777972097211975342012-07-12T17:55:36.874-04:002012-07-12T17:55:36.874-04:00What was the dimensionality of those various datas...What was the dimensionality of those various datasets? I wonder if PCA is more helpful if your number of dimensions is large with respect to the size of your dataset??Amyhttp://www.blogger.com/profile/11573392015410961658noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-35351233003103737832012-07-12T12:28:28.652-04:002012-07-12T12:28:28.652-04:00an unregularized linear classifier will not be aff...an unregularized linear classifier will not be affected by rotations and scaling. however! when using regularization, the same regularization parameter may yield better or worse results. if you cross-validate thoroughly, i think you should be able to get nearly identical performance.<br /><br />just for fun though, i tried a multi-class regularized ridge regression classifier before and after ZCA (rotation + scaling only). the results are very close, even though i used the same default regularization parameter.Sergey Feldmanhttp://www.blogger.com/profile/11369485324420056756noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-91490202658456996672012-07-12T05:51:13.161-04:002012-07-12T05:51:13.161-04:00Piotr: On the other hand, PCA might combine severa...Piotr: On the other hand, PCA might combine several noisy redundant features into a single axis, which could potentially be beneficial. I don't think it's possible to say what effect PCA will have without reference to particular data. <br /><br />Sergey: It *seems* (very dangerous word) that linear classifiers should be affected by rotations in the feature space differently that axis-aligned thresholders (aka, decision trees). Any chance you'll try the same experiments with a linear SVM?Alex Rubinsteynhttp://www.blogger.com/profile/04049149415019007603noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-40906926558969294752012-07-11T21:15:27.194-04:002012-07-11T21:15:27.194-04:00It is well known that PCA can remove the data that...It is well known that PCA can remove the data that contains the features which are essential for classification. PCA dimensionality reduction maintains what is common in data and not what differentiates them.Piotr Gawronhttp://www.blogger.com/profile/16231129942286199238noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-71117380296893874152012-07-11T17:41:50.056-04:002012-07-11T17:41:50.056-04:00maverick: it's kind of a mess! i can post ind...maverick: it's kind of a mess! i can post individual functions or datasets, if there's something in particular you're interested in.<br /><br />brooks: good question. i'll run some experiments later & make an addendum to the post.Sergey Feldmanhttp://www.blogger.com/profile/11369485324420056756noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-53469423266706890452012-07-11T15:54:28.218-04:002012-07-11T15:54:28.218-04:00You reduce the dimension using PCA by keeping only...You reduce the dimension using PCA by keeping only as many eigenvectors as needed to explain 99% of the variance -- what's the dimensionality, then, of the transformed data? How much lower is it than dimensionality of the greedy forward feature selection in your last post?Brooks Paigehttp://www.blogger.com/profile/02776916984778878708noreply@blogger.comtag:blogger.com,1999:blog-8497841368693123869.post-63049337498818293142012-07-11T15:41:12.940-04:002012-07-11T15:41:12.940-04:00would you mind posting your code?would you mind posting your code?Maverickhttp://www.blogger.com/profile/17817335403323735440noreply@blogger.com