Ask Ghassem - Recent questions in General

Battery data projects

Wed, 02 Mar 2022 18:11:57 +0000

Where can I find projects related to battery data?

How can you build dynamic pricing model with data only from rigid pricing?

Fri, 21 Jan 2022 06:44:31 +0000

I want to build a dynamic pricing model which means if product is too expansive for a client and there is a risk that we might loose a client we lower the price for them but if client doesn't care that much about the price we might increase price a little.

All the articles I've seen describe some kind of A/B testing for the pricing and then create a model.

I want to build a model only on the existing rigid pricing data. So I have prices offered to customers and I know who bought the product and who went to other company.

How can I do the increasing price part?

Text Mining, Artificial Neural Networks, Speech Processing, Cloud Computing in DS? Essential for a good Data Scientist ?

Wed, 27 Oct 2021 19:15:16 +0000

Classification of data object might be incorrect

Mon, 25 Oct 2021 15:26:46 +0000

I am learning a new Salesforce product (Evergage) for the company I work for. In the program's documentation they have listed a set of data objects as an example. It appears to me that the classification might be incorrect. Their system makes a division between 'catalog objects' and 'profile objects' and the example they have given is a banking institution. They classified Customer Credit Card as a profile object and Credit Card Level as a catalog object. Seems to me that it should be the other way i.e Customer Credit Card = catalog object and Credit Card Level = profile object. Maybe I am not reading the context correctly?

here is a link to an image with the complete classification: https://drive.google.com/file/d/1nG4aX4Ty_NoHxm04AQo1Ow61m3MZ3pXm/view?usp=sharing

Can Data Science solve this problem?

Sun, 24 Oct 2021 15:43:11 +0000

So, I live in Brazil, and I have a task for college that I don't know what data science method to use, if at all, to solve it. My idea is the following: We Brazilians have Real (BRL) as currency, and we of course have the dollar quotation value to see "how many Reais a dollar is worth". What I wanted to do was to make a research and see whether the Country News have any influence over this price. So for example, if Bolsonaro, our president, says some dumb stuff, the dollar got up in price, and vice versa. What I wanted to do was collect all dollar values and variance over a set time interval, and try and get webscraping to get the news over some economy sites. Here's my question then: How can I correlate the news with the dollar variance over a set time? Can data science do that? How do I preprocess this, if at all? Do I need to use bag-of-words? At least I heard so... Please help and thank you for reading.

Searching for movie dataset containing movie synopses/plots?

Thu, 27 May 2021 09:57:31 +0000

Hello
To build a hybrid recommendation system, I used the movielens 1M dataset, for the collaborative filtering part. Now, I'm looking for a database/dataset that contains descriptions/summaries/details/synopses/plots of movies for the content-based recommendation.
Is there someone who could help me and tell me where I can find a such dataset?
thank you in advance.

My GloVe word embeddings contain sentiment?

Sun, 03 Jan 2021 14:09:37 +0000

I've been researching sentiment analysis with word embeddings. I read papers that state that word embeddings ignore sentiment information of the words in the text. One paper states that among the top 10 words that are semantically similar, around 30 percent of words have opposite polarity e.g. happy - sad.

So, I computed word embeddings on my dataset (Amazon reviews) with the GloVe algorithm in R. Then, I looked at the most similar words with cosine similarity and I found that actually every word is sentimentally similar. (E.g. beautiful - lovely - gorgeous - pretty - nice - love). Therefore, I was wondering how this is possible since I expected the opposite from reading several papers. What could be the reason for my findings?

Two of the many papers I read:

Yu, L. C., Wang, J., Lai, K. R. & Zhang, X. (2017). Refining Word Embeddings Using Intensity Scores for Sentiment Analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(3), 671-681.
Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T. & Qin, B. (2014). Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 1: Long Papers, 1555-1565

How to remove unwanted Jupyter notebook kernels?

Fri, 30 Oct 2020 17:15:17 +0000

Whener I run Jupyter notebook there are some kernels that do not exist on system and generate errors. How can I remove them?

How can this data be structured for mongodb

Sun, 19 Jul 2020 19:08:50 +0000

https://prnt.sc/tkr2g7 Hello I have a PFE about determining risks of pedestrians, and I have to make a simulator to generate data with something related to this, this is my first time working on this. I would like to know, the structure of data, I will be working with mangodb, so I would love to see an example on JSON

What are the differences among Data Science, Artificial Intelligence and Machine Learning?

Thu, 05 Mar 2020 03:02:31 +0000

What are the differences among Data Science, Artificial Intelligence and Machine Learning?

How to convert Jupyter Notebook or a webpage to PDF using Chrome?

Thu, 27 Feb 2020 18:39:19 +0000

Please show us how to print to pdf in Google Colab or any other webpages in Google Chrome?

How to share a Jupyter Notebook document on Google Colab?

Thu, 27 Feb 2020 16:17:13 +0000

How can I give access to others to view, comment or edit my Jupyter Notebook on Google Colab?

What are the most common data types in data science?

Wed, 19 Feb 2020 17:28:54 +0000

What are the main data types?

How to install Matplotlib

Sun, 02 Feb 2020 21:30:41 +0000

Hi Everyone,

How to install Matplotlib in Python IDLE. 3.7.4

I'm a student in Data Science and I want to do The Birthday Paradox and it requires me to import matplotlib.pyplot

I'm using MAC and in the terminal, I tried to write:

(base) MacBook-Pro:~ nasir$ sudo apt install python3-matplotlib

Password: **************

I then received the below error:

Unable to locate an executable at "/Library/Java/JavaVirtualMachines/jdk-11.0.1.jdk/Contents/Home/bin/apt" (-1)

(base) MacBook-Pro:~ nasir$

ideas and opinion on what kind of analyses needs to be done

Fri, 26 Jul 2019 18:38:55 +0000

Hello Guys,

I have been given a task where I need to analyse the impact of all the products that were discontinued last year on the customers and sales? I work for a retail company which has 100s of stores in North America and my analyses needs to be "Qualitatively better than the regular BI".

I quote that because thats what I was asked to do. I am struggling to come up with ideas. What kind of hypothesis should I test. Or any Bayesian analyses that can be done. Any ideas within the realms of data science and machine learning is welcome.

I generally do my analyses using Jupyter Notebook and my skill set consist of SQL, Python including ML libraries like sklearn

What are the most important Python libraries for data science?

Mon, 08 Jul 2019 04:42:28 +0000

How to reshape in pandas dataframe?

Fri, 05 Apr 2019 13:41:30 +0000

Dataframe looks like below

I have dataframe like above. which I want to a~t reshape (a~t, 1)

I want to reshape dataframe like below ( b~t column is go to under the a column)

날짜 역번호 역명 구분 a

2018-01-01 150 서울역 승차 379

2018-01-01 150 서울역 승차 287

2018-01-01 150 서울역 승차 371

2018-01-01 150 서울역 승차 876

2018-01-01 150 서울역 승차 965

....

2008-01-01 152 종각 승차 2920

2008-01-01 152 종각 승차 2290

2008-01-01 152 종각 승차 802

2008-01-01 152 종각 승차 1559

like df = df.reshape(len(data2)*a~t, 1)

i tried pd.melt but It does not work well.

df2 = pd.melt(df, id_vars=["날짜", "역번호", "역명", "구분"], value_name="t")

is remove b ~ t but i want insert b~t behind a

dataset is https://drive.google.com/file/d/1Upb5PgymkPB5TXuta_sg6SijwzUuEkfl/view?usp=sharing

Is digital marketing and marketing internships worth it for a data science student?

Thu, 04 Apr 2019 22:58:54 +0000

it's about data science career

How to open Jupyter notebook files on Windows or Mac without web browser?

Sat, 16 Mar 2019 23:36:01 +0000

Passing variable length sentences to Tensorflow LSTM

Mon, 11 Feb 2019 05:06:27 +0000

I have a tensorflow LSTM model for predicting the sentiment. I build the model with the maximum sequence length 150. (Maximum number of words) While making predictions, i have written the code as below:

batchSize = 32
maxSeqLength = 150

def getSentenceMatrix(sentence):
    arr = np.zeros([batchSize, maxSeqLength])
    sentenceMatrix = np.zeros([batchSize,maxSeqLength], dtype='int32')
    cleanedSentence = cleanSentences(sentence)
    cleanedSentence = ' '.join(cleanedSentence.split()[:150])
    split = cleanedSentence.split()
    for indexCounter,word in enumerate(split):
        try:
            sentenceMatrix[0,indexCounter] = wordsList.index(word)
        except ValueError:
            sentenceMatrix[0,indexCounter] = 399999 #Vector for unkown words
    return sentenceMatrix

input_text = "example data"
inputMatrix = getSentenceMatrix(input_text)

In the code i'm truncating my input text to 150 words and ignoring remaining data.Due to this my predictions are wrong.

cleanedSentence = ' '.join(cleanedSentence.split()[:150])

I know that if we have lesser length than sequence length we can pad with zero's. What we need to do if we have more length. Can you suggest me the best way to do this. Thanks in advance.

How do you copy a list without updating the original list?

Tue, 30 Oct 2018 11:04:03 +0000

What are the general steps in data cleaning?

Fri, 19 Oct 2018 03:54:38 +0000

How should I reduce file size of a word document when I attach high resolution pictures?

Fri, 12 Oct 2018 06:17:56 +0000

I need to submit an assignment in Word Document format, and the size of file is so big after attaching the photos taken by my cell-phone, what should I do?

How to use Ask Data Science! ?

Wed, 03 Oct 2018 07:01:30 +0000

This post is for providing the best practices for using Ask Data Science!

How to upload images on Ask Data Science! website?

Sun, 16 Sep 2018 21:02:27 +0000

I know I can attach pictures on Ask Data Science! (https://askdatascience.com), but I cannot upload. How can I upload image somewhere else and attach it here?

How to use LinkedIn Learning (formerly Lynda.com) for free?

Sun, 16 Sep 2018 20:32:55 +0000

I heard some people use Lynda.com resources for free. Could someone let me know how does it work?