Enhancing Yioop's Recommendation System with Hash2Vec by creating context for words seen by users to improve scores of threads with similar words that would otherwise not get recommended because of a low tf-idf score.

Hey Anirudh,

Thanks for this patch! I have a new student, Parth Patel, who is going to extend it and get a new patch ready for Yioop.

Best, Chris

Hi Professor,

I have uploaded a new patch with the refactored code of Anirudh's Hash2Vec approach.

Thanks, Parth

Hi Professor,

Uploading a new patch with the few changes we discussed.

Thanks, Parth

Hey Parth,

Thanks for your patch! I applied it but then had to back it out because it had numerous issues. Below are my comments. When you have time make a new patch and upload it. In the note you add with your uploaded patch please indicate how you addressed these issues.

Best, Chris

(1) You should as part of the documentation to the recommendation job explain clearly somewhere how the two approaches to computing recommendations are computed rather than just say hash2vec approach. This clarity will help eliminate, unnecessary/bogus stuff in your next patch.

(2) Your patch creates a table USER_ITEM_SIMILARITY_HASH2VEC which is never used.

(3) You should avoid the phrase HASH2VEC in table names USER_TERM_WEIGHTS_HASH2VEC, HASH2VEC_TERM_SIMILARITY. There currently is no table named TERM_SIMILARITY so why not just call it that? Also, if the hash2vec method is being used can we get away with just having the table USER_TERM_WEIGHTS as the first thing you do is copy all of USER_TERM_WEIGHTS into USER_TERM_WEIGHTS_HASH2VEC?

(3) Why isn't there a table ITEM__TERM_WEIGHTS for the has2vec context?

(4) Do not use the vague notion word as part of variable names. Use term instead. So WORD_SPLIT_PATTERN should be TERM_SPLIT_PATTERN.

(5) See how terms are split in PhraseParser and use its mechanism rather make a new one .

(6) In initializeHash2Vec() the SQL code AND LOWER(DESCRIPTION) NOT LIKE '%-0700%' seems very fragile and not likely to work in general.

(7) Write null not NULL

(8) strtolower is not multi-byte safe so will garble non-English text. Use mb_strtolower.

(9) Code like if (C\RECOMMENDATION_TYPE == 1) { is mysterious. Nobody will know what 1 means. Make a constant for each possible recommendation type.

(10) The start of generateVectors' code is excessively complicated for what it is doing. For example, six lines of code is used to create a fixed (always same) array $context_distance_vector whose only use is to compute a variable $standard_deviation. Try to just use a closed formula for the fixed standard deviation in question. This would also eliminated the need for the function calculateStandardDeviation.

(11) The wordToId code seems fragile/method name doesn't adequately indicate the method's side-effects.

(12) Yioop has a LinearAlgebra class which contains dot products, etc which should be used rather than to rewrite these methods.

(13) normalizeVectors computes an L1 norm. Is that what was intended or did you want an L2 norm?

Hi Professor,

I am uploading a new patch, that contains the code for recommendation job updated to replace old recommendation mechanism for Groups and Threads, add recommendation support for Wiki Resources.

Note: Before apply this patch please apply the patch for this issue - https://www.seekquarry.com/mantis/view.php?id=252

recommendation.patch

Hi Professor,

Uploading updated patch to address the patch update for issue - https://www.seekquarry.com/mantis/view.php?id=252.

recommendation-2.patch

Hi Professor,

Uploading the patch with the changes we discussed today.

recommendation-3.patch

Hi Parth,

Thanks for you patch. It has been applied and pushed to the git repository. The old recommendation tables had indexes on some of their columns to speed up querying. The two new tables have no indexes. Can look into if it makes sense to add indexes on some columns, and if so, make a patch with the appropriate CREATE INDEX commands.

Best, Chris

HI Parth,

I tried testing your code out on the Yioop site. Postgres does not have a blob type. I fixed that by checking and using BYTEA instead. However, currently your code runs outs of memory after it tries to compute Item Embeddings. You seem to be trying to read in the whole RECOMMENDATION_ITEM_EMBEDDING into memory. I doubt that will work. Can you fix your code under the assumption that many of the database tables will be too large to fit in memory?

Chris

Hi Professor,

I am reading the entire table data in order to preserve the embeddings for items not calculated during the current run.

To avoid the memory issues due of reading all the values from DB, can we instead of deleting all the item embeddings only delete those existing item beddings which are updated in current run using IN operator on ID and providing list containing IDs of updated item embeddings?

That sounds like it could work.

Best, Chris

Okay will try it and update the patch once done.

Thanks, Parth

Hi Professor,

Uploading a patch that tries to fix the out of memory. Please let me know if the fix worked for Yioop site or not.

Best, Parth

recommendation_fix.patch

Applied and pushed that patch. It still fails with out of memory errors. For reference if dies at line 890 in computeWikiTermEmbeddings.

Chris

Thanks for the update. Should we meet to discuss more about it?

Best, Parth

I am on Skype now if you want to do a short chat.

Hi Professor,

I have created a patch with the method we discussed regarding using LRU cache and pack-unpack method. Also created indices on GROUP_ITEM_RECOMMENDATION and GROUP_RESOURCE_RECOMMENDATION tables, did not create index for other table because they have primary keys defined on the columns which are used for lookup.

I tested this code by duplicating 3 times the resources created by ManyUserExperiment which totals to around 450 resources. The max memory usage reported while running recommendation job was 12616104. Not sure about the unit of this number. Also while writing pack binary strings to DB for items was created DB insert errors so had to unpack it to array and serialize for inserting.

recommendation_fix_cache.patch

Hi Parth,

I get a lot of warnings where I try to run your patch such as:

Warning: pack(): 1 arguments unused in /Applications/MAMP/htdocs/git/yioop/src/library/media_jobs/RecommendationJob.php on line 384 PHP Warning: Undefined array key 0 in /Applications/MAMP/htdocs/git/yioop/src/library/media_jobs/RecommendationJob.php on line 378

Switch d to E for the pack format. It looked like you had sqlite3 and mysql specific SQL code, but Yioop uses Postgres. Is there a way to avoid this, or can you push it into the datasource manager? Have you tested the recommendations after the change? When I looked at the embeddings after this patch, I was getting just blank strings. I also wasn't sure on the issue of needing to do serialize, but I was worried that these might be connected. Finally, I couldn't understand what the intent of the line: $embedding = serialize(unpack("E$this->size", $embedding)); was. It seems to me that in the database it should be a packed string. If you are worried about putting binary data, i don't think it should cause a problem, but you could try base64 encoding if it was.

Hi Professor,

I tested the patch on fresh copy of Yioop and it worked without any issue for my machine having Sqlite3 DB.

Will switch d to E for pack formatting. For $embedding = serialize(unpack("E$this->size", $embedding)); - I had to do this because directly writing binary string to DB for item_embeddings was giving DB errors, not exactly sure why because the same thing for term embeddings worked without any error. I will try to revise the patch and upload it once done.

Its got to work cross platform to be considered working. The warnings I get are on a fresh copy of Yioop. Did you make sure to turn on error reporting in the configure activity so you can see all errors? Did you check that it is making non-zero embeddings after the change? (What I am getting). I would switch to base64 for storage into DB rather than serialize if this is really necessary.

Hi Professor,

I just turned on the errors reporting and now I see the warnings you mentioned. Also I am getting the zero embeddings. Working again on the pack and unpack thing.

Hi Professor,

When you tested last time, where all the embedding values zero or it had values at few index?

I just checked, a few were non zero.

One reason could be fresh copy of Yioop does not have proper thread data, which could lead to zero embedding vectors for terms. I created 10 threads with around 50 terms in each and I could see more non-zero values in term embeddings.

Let me know if you think this could be the reason and I will create a patch and upload here.

It probably is the reason, go ahead and create a new patch and I'll try it out. Just make sure it gets rid of the warnings.

Best, Chris

Yeah I got rid of all PHP warnings. Did you face any DB errors when you tested?

Best, Parth

I haven't pushed it to the Postgres site (YIoop), because I was worried about the other issues first. You should avoid having Sqlite and Mysql specific code though.

Hi Professor,

Uploading the revised patch. Used base64 encoding - decoding for writing to DB. Also tried to use general SQL code, but not 100% sure if something is not supported by Postgres.

Best, Parth

recommendation_fix_cache-2.patch

This patch seems not to give warnings which is good. It seems to be taking a while to process recommendation, during which time it is not giving any log messages. Can you do a git pull and see how I've added a couple L\crawlTimeoutLog messages in a nested for situation which might run more than 30 seconds. Can you add L\crawlTimeoutLog's in other places where this might happen? I am still waiting for my current test to complete.

Best,

Chris

Sure adding it. Are you testing on fresh copy of Yioop or live site?

Best, Parth

It took much longer for it to die. But it still gives an out of memory error at line 903.

Testing on pollett.org, which is a smaller live site.

Oh no, so now what else can we try?

Maybe we just reduce the size of the LRU cache to 500 and see if it works.

Yeah we can try reducing cache size. Other thing is we can limit the number of resources to process in a single run just like group items?

Yeah, we should do that too.

Okay will upload new patch with these changes.

Hi Professor,

Created patch with below changes 1. Reduced cache size to 500 2. Added limit of 200 resources to process at a time 3. Added timeout logs

Best, Parth

recommendation_fix_cache-3.patch

So it doesn't crash now! It is giving databased locked errors when it does it first set of inserts into a sqlite db. I will probably investigate tomorrow.

Hi Professor,

Uploading a patch with changes to RECOMMENDATION_TERM_EMBEDDING table where ID column is now VARCHAR(16) to store canonicalTerm as id instead of integer id.

canonical_term.patch

Thanks Parth, Anirudh, for your work on this issue. A version that doesn't give errors for SQLITE, MYSQL, and Postgres commited with 9cde4c6ecc76e. The last issue with database locking in Sqlite resolved by processing group items in batches. Thanks Parth, for the switch to using canonicalTerm for term_id's.

Attachments:

my.patch

hash2vec.patch

hash2vec-2.patch