Ask Ghassem - Recent questions tagged statistics

Individual and group relative strength in a fixed pool of players: How to approach the problem?

Tue, 29 Oct 2019 20:00:28 +0000

I apologize in advance if my question sounds too basic to be worthy of anyone's time, but statistics are not part of my curriculum.

I am developing a proof of concept of a web application modeling the contribution of individual soccer player with respect to the different teams they've played with throughout their career. In particular, I am looking into a way of ranking both individuals and groups of players as follows::

teammates relative strength: the best/worst combinations of players when playing in the same team in the same matches;
opponents relative strength: the best/worst combinations of players when playing in opposite teams in the same matches, i.e. which tuples of teammates are the best/worst against which;

I must admit I don't quite know how to approach the problem (as I said I have no formal education in statistics or data science). I would be very grateful if anyone could give me some directions. How should I frame this particular problem and what resources in statistics or machine learning (if indeed this is a task fit for machine learning, perhaps I am mistaken on this) would be appropriate to tackle it?

I am eager to learn, so both practical examples or theoretical references (book chapters, online articles, etc) would be very welcome.

Thanks in advance!

Using aggregate data to generate observation-level data statistically sound?

Tue, 11 Jun 2019 22:04:01 +0000

Context: In the realm of Paid Search Marketing. Current reporting does not provide event level data only aggregate totals with different segments. Want to compare distributions/test statistical significance of A/B test results. Did not want to assume that data followed normal distribution or know STDEV for data so came with this approach.

My Question: I am going to use the average "CPA" or "CTR" for a date range, and generate an observation for each conversion based off the average for a time range. Is this statistically sound way if I want to generate raw data? Would I have wonky distributions because of the multiple averages? Just want a gutcheck if I'm completely off base.

My Aggregate data looks like below:

Day	Cost	Acquisition	CPA or CTR
1	40	2	$20
2	75	3	$25

Observation data I generate looks like below:

Day	Acquisition
1	$20
1	$20
2	$25
2	$25
2	$25

I really appreciate your help with this question! An important project to me at work.

Looking for guidance on whether I have the necessary data to answer a Regression question

Sun, 24 Mar 2019 23:40:09 +0000

Hi everyone.

I'm currently working on my final project for a Data Science degree and after a month of literature review, exploratory analysis and model testing, I'm not sure if the questions I set out to answer are suitable for the data I have.

This is a very broad question I'm asking here, as it's more guidance than anything else, so if this is not the place to ask, I would appreciate it if you could redirect me to the right place.

You can find the data sets and code on my github here. The code is messy but working; I've only picked up programming last year.

The data

Indoor Air Quality data recorded hourly through 4 sensors (Kitchen, Bedroom, Living Room, Bathroom) for 7 days in a house for a total of 3 houses. For 6 of those days, each sensor was in a different room and on the last one, all sensors were together so we could see how spread apart their signals were and account for that). So in here I have 9 continuous variables: Temperature, Relative Humidity, CO, CO2, TVOC, PM2.5, NO2, Ozone and Air Pressure.

I then got 3 manually-filled questionnaires on Occupant Activity, one for each house, such as "Door open/closed", "Window open/closed", "Heating On/off", "Frying", "Boiling", "Hoovering", "Mopping", etc. Now, these logs were missing a lot of data.

These questionnaires were a mess and a lot of the missing values had to be imputed. This data is reported in binary format such as "Did Activity X occur at hour Y? - Yes(1)/No(1).

With this project I've chosen to predict a sensor data variable (in this case CO2), based on activities.

Models

Just to have a feel for the data, I ran a Linear Regression, Decision Tree and Random Forest model with a choice of only Occupant Activity predictors and both Occupant Activity and other sensor variables as predictors on individual rooms of each house and the results are just atrocious in every case. Cross-validation shows the model's performance to be all over the place and looking at features for statistical significance gives me different significant features in every room of every house, it's like I'm playing feature roulette. Problem with some features such as Mopping, Frying, Boiling, Hoovering is that there will be a lot of "0"s in comparison to "1"s due to the nature of the feature, so one or two "1"s in the wrong place is enough to give a misguided correlation.

As you can tell and see from this, I'm still a Data Scientist in-training here, having only done a few models in the past and rather new-ish to programming (1 year experience).

What I'm looking for

I suppose that more than anything, I'm asking for guidance on whether pursuing this as a Regression problem is feasible or not.

I'm very short on time but if this won't work, I can look into alternatives. For instance, Air Pollutants have safety thresholds. I could create a class feature on whether the value is over the threshold or not and turn it into a classification problem or even a cluster one to identify the room based on activities and air pollutants..

Bottom-line is that I have a 12,500 word paper to deliver in a month, I've been at this for month already with nothing to show for, so I'm hoping someone with more experience under their belt could see if I'm chasing a dead end. Any help in the form of guidance would be so very much appreciated, I've ran out of ideas here.

Thanks,

Tom

How do I know when it is appropriate to use stratified sampling?

Tue, 19 Feb 2019 18:52:48 +0000

What is the easiest way to distinguish whether to use a z value test or a t value test?

Tue, 18 Dec 2018 04:33:27 +0000

nonpooled independent samples t-interval method

Fri, 07 Dec 2018 23:06:04 +0000

Computing the Confidence Interval for a Difference Between Two Means

Fri, 07 Dec 2018 23:02:45 +0000

How to find the strength of a P-value against a null hypothesis?

Fri, 07 Dec 2018 22:31:33 +0000

What is degree of Freedom while calculating confidence interval?

Fri, 07 Dec 2018 22:27:27 +0000

How do you know when to use T-Distribution instead of regular Confidence Interval?

Wed, 05 Dec 2018 22:50:48 +0000

What is the T Distribution?

Fri, 30 Nov 2018 21:22:02 +0000

Probability for the data items within mean +- 1, +-2, +-3 standard deviation?

Fri, 30 Nov 2018 21:15:35 +0000

What is 'Degrees of Freedom'?

Thu, 29 Nov 2018 18:49:06 +0000

What is 'Degrees of Freedom'?

How do I calculate a and z a/2?

Thu, 29 Nov 2018 18:19:23 +0000

What is the rule that determines percentages with the mean +- 1,2 or 3 standard deviations?

Sat, 17 Nov 2018 19:03:35 +0000

What will be probability for the data items within mean +- 1, +-2, +-3 standard deviation?

Wed, 14 Nov 2018 15:10:15 +0000

How much percentage of data items are consumed within mean +-1 , +-2, +-3 standard deviation?

Wed, 14 Nov 2018 15:04:52 +0000

What is a test statistic in hypothesis testing and how does it relate to the p-value?

Tue, 13 Nov 2018 00:51:11 +0000

What is sampling error?

Tue, 13 Nov 2018 00:47:51 +0000

What is P-value in Statistical Testing?

Mon, 12 Nov 2018 13:41:39 +0000

What is P-value in Statistical testing?

What is a sampling distribution of sample means?

Mon, 12 Nov 2018 13:38:30 +0000

What is a sampling distribution of sample means?

In the first week of November 2003.... What is the expected number of homicides per week in the GTA?

Fri, 02 Nov 2018 19:34:40 +0000

This is from question one from lecture 4B's exercises: "In the first week of November 2003, five unrelated homicides were reported in the GTA. The GTA usually has about 78 homicides per year. What is the expected number of homicides per week in the GTA?"

What is summary statistics?

Thu, 01 Nov 2018 19:45:16 +0000

What is the difference between univariate and multivariate analysis?

Tue, 30 Oct 2018 11:39:08 +0000

What is the purpose of randomization in statistics?

Tue, 30 Oct 2018 11:35:52 +0000

In statistics, whats the difference between contingency and frequency tables?

Tue, 30 Oct 2018 11:19:48 +0000

Which scenarios among the following are a valid reason to use regularization?

Sat, 27 Oct 2018 17:31:43 +0000

A. To drop the least useful variables of a model

B. To reduce over-fitting

C. To reduce the bias of a model

D. To decrease p-value

How do you tell a permutation problem from a combination problem?

Mon, 15 Oct 2018 15:34:13 +0000

Given the questions:

1. How many different groups of 4 students can be made from a class of 40?

and

2. How many different lists of 4 students can be made from a class of 40?

How do you determine which requires a permutation and which requires a combination?

What is the use of the Poisson Distribution?

Sat, 13 Oct 2018 15:37:29 +0000

Also, when would one use this distribution ?

How do we determine if two events are independent

Sat, 13 Oct 2018 15:23:39 +0000

In the context of probability, what is the "event space"

Sat, 13 Oct 2018 15:10:28 +0000

What is the "Three-Standard-Deviations Rule"

Sat, 13 Oct 2018 14:57:53 +0000

This rule is also known as the "68–95–99.7 rule". What does this rule stand for and how does it help me?

How do histograms help us understand a data set?

Sat, 13 Oct 2018 14:42:11 +0000

What is a discrete random variable?

Sat, 13 Oct 2018 01:23:24 +0000

What is the difference between permutation and combination?

Sat, 13 Oct 2018 01:19:28 +0000

What is the difference between Qualitative and Quantitative data?

Sat, 13 Oct 2018 01:15:59 +0000

If x is the number that comes up when you roll a 20 sided die. What is the expected value of x?

Fri, 12 Oct 2018 22:43:05 +0000

If x is the number that comes up when you roll a 20 sided die. What is the expected value of x?

What is the probability of getting a King of Hearts?

Fri, 12 Oct 2018 22:31:45 +0000

What is the probability of picking a King of Hearts in a standard 52 card deck?

Define measures of center (Median and Mode) ?

Fri, 12 Oct 2018 22:28:59 +0000

Define the 2 measures of centers.

What is "Random Sampling"?

Fri, 12 Oct 2018 22:26:06 +0000

Define random sampling

What is binomial experiment and when do we use it?

Fri, 12 Oct 2018 22:25:59 +0000

When should we use permutation and combination?

Fri, 12 Oct 2018 22:16:45 +0000

Calculate IQR(Inter Quartile Range) of {18, 24, 19, 16, 21}?

Fri, 12 Oct 2018 22:07:58 +0000

What do population, parameter, census and sample mean in statistics?

Fri, 12 Oct 2018 21:50:34 +0000

Question for Discrete random variable

Fri, 12 Oct 2018 21:00:40 +0000

An investor has a 0.60 probability of making a 20000 dollar profit and a 0.40 probability of suffering a 25000 dollar loss. What is the expected value?

Sample question of permutation and combination.

Fri, 12 Oct 2018 20:47:03 +0000

You can bring along 2 friends! you have 5 friends who want to come along. How many different groups of friends could you take with you?

{1.75, 1.63, 1.55, 1.92, 1.81, 1.79, 1.81}. Determine the mean, the median, and the 20th percentile?

Fri, 12 Oct 2018 19:42:32 +0000

What is difference between discrete numerical variable and continuous numerical variable?

Fri, 12 Oct 2018 19:25:11 +0000

What is the mean and standard deviation of a standard normal distribution?

Fri, 12 Oct 2018 01:39:12 +0000

How to calculate normal distribution?

Sat, 01 Sep 2018 20:20:01 +0000