https://ask.ghassem.com/tag/data-cleaning Powered by Question2Answer https://ask.ghassem.com/1042/how-to-analyse-imbalanced-categorical-colum-in-dataset Hello,<br /> <br /> I have a dataset with a categorical column that contains three categories. One of the categories represents 98% of the data, while the remaining 2% are distributed between the other two categories, with a few (maybe around 50) in each. It is worth mentioning that the output for these 50 rows is the same, which suggests that these data points may be important.<br /> <br /> However, the data is obviously imbalanced, and I am unable to perform any analysis. Should I drop the entire column, or perform a chi-square test on the data as-is? Data Science https://ask.ghassem.com/1042/how-to-analyse-imbalanced-categorical-colum-in-dataset Sat, 24 Jun 2023 17:55:23 +0000 https://ask.ghassem.com/1006/know-which-encoder-convert-categorical-variables-numerical So say I have a column with categorical data like different styles of temperature: &#039;Lukewarm&#039;, &#039;Hot&#039;, &#039;Scalding&#039;, &#039;Cold&#039;, &#039;Frostbite&#039;,... etc.<br /> <br /> <br /> I know that we can use pd.get_dummies to convert the column to numerical data within the dataframe, but I also know that there are other &#039;converters&#039; (not sure if that&#039;s the correct terminology) that we can use, i.e. OneHotEncoder from Sk-learn (like I could use the pipeline module to make a nice pipeline and feed my dataframe through the pipeline to also get my categorical data encoded to numerical).<br /> <br /> <br /> How do I know which to use? Does it matter? If it does matter, when does it matter the most (i.e. what types of problems? When there are lots of categorical variables, or few?) If anyone can give me any pointers on this type of stuff I&#039;d greatly appreciate it. Exploratory Data Analysis https://ask.ghassem.com/1006/know-which-encoder-convert-categorical-variables-numerical Mon, 29 Nov 2021 04:09:06 +0000 https://ask.ghassem.com/775/how-to-filter-a-dataframe <p>Consider the Pandas DataDrame&nbsp;<code>df</code>&nbsp;below. Filter it appropriately so that it outputs the shown results.</p> <pre class="prettyprint lang-python" data-pbcklang="python" data-pbcktabsize="4"> gh owner language repo stars 0 pandas-dev python pandas 17800 1 tidyverse R dplyr 2800 2 tidyverse R ggplot2 3500 3 has2k1 python plotnine 1450</pre> <h2>Expected Output</h2> <pre class="prettyprint lang-" data-pbcklang="" data-pbcktabsize=""> gh owner language repo stars 0 pandas-dev python pandas 17800</pre> Python Interview Questions https://ask.ghassem.com/775/how-to-filter-a-dataframe Wed, 25 Dec 2019 05:56:14 +0000 https://ask.ghassem.com/430/what-are-basic-steps-for-treating-missing-values Exploratory Data Analysis https://ask.ghassem.com/430/what-are-basic-steps-for-treating-missing-values Fri, 19 Oct 2018 04:08:48 +0000 https://ask.ghassem.com/427/what-are-the-general-steps-in-data-cleaning General https://ask.ghassem.com/427/what-are-the-general-steps-in-data-cleaning Fri, 19 Oct 2018 03:54:38 +0000