Sunday, November 6, 2016

Generative Adversarial Networks

No one  can explain it better than OpenAI

Recommendation systems

The 3 types
  • Item-Item Content Filtering: “If you liked this item, you might also like …”
  • Item-Item Collaborative Filtering: “Customers who liked this item also liked …”
  • User-Item Collaborative Filtering: “Customers who are similar to you also liked …”
There are other trivial domain dependent (like most-popular / trending today). They use global data (or global-data per category) and do not distinguish between items

Item-Item Content Filtering: If you liked this item, you might also like …”
How: Find similarity between items. Finding similarity is an art of itself, but essentially it's a two steps procedure: cut-into-(good)-features and calculate-distance-using-features. Lately deep-learning is doing both together.
Example: Pandora processes a song into it's features (rock/classic, fast/slow) and finds similar songs.
For books, you can look into book-genre + author-name
For fashion photos, you can (but it's hard) use deep-learning to find similar dress/shoe style

Item-Item Collaborative Filtering: “Customers who liked this item also liked …”
Example: Amazon (at least the first years)
How: Use a lot of user data. Good for companies with big user data and not for new-comers.
Market-basket based: If you have purchase-history you just count at the most common items purchased together (for cat-food:  cat-toy=50% , cat-litter=40%, dog-food=0.3% etc). Drawback of this approach is that we use one purchase items and not the whole history of the user, and that we know he bought  it, but don't know if he liked it in the end or returned it.
Rating based: If you also have enough user-rating(stars), you can look at how much users ranked what, for example: those giving harry-potter 1 '5 stars' also gave harry-potter 2 '5 stars' average, but gave dan-brown '1 star' , 

User-Item Collaborative Filtering: “Customers who are similar to you also liked …”
Example: Netflix challenge
Find similarity between users according to the rating they gave to the same movies, but do that after normalization (some users love-everything with mean score of 4 of 5, some are "haters" with 2 of 5 mean average, so the first giving 3, is like the hater giving 1).


Thursday, August 11, 2016

statefarm - retrospect

Reviewing the best team competition results.

1. How to train each single-model 

1.1 Syntesize/Augment to generate huge amount of data
  • Synthesize 5M new images: create 5M images, by combining left and right (almost) half of images from the same class.  This is so good, it was abet o train from-scratch google net V3 to 0.15 . ref
  • Synthesize images by combing images from the test-set (As in this competition, they all used the same video)

1.2 If not possible, use pre-trained. The stronger, the better.
Resnet-152 > VGG-19 > VGG-16 > googlenet 

1.3 Use semi-supervised learning
"dark-knowledge" - let an ensample predict on the test-set, take most cofident. 6-12K images, don't use too many

Some numbers to compare... to compare from the same team/model.googlenet V3 
0.31 pre-trained , augmented (flip/rotate)
0.26 pretrained, augmented + "dark-knowledge"/semi-supervised
0.15 from scratch: but 5M synthesized images(!)

2. How to run a single model

If test-data data can be clusered use this fact (in this competition, yes, it was):
  • hack-the-input and get 3rd place . as the input was sequence of images, use NN for better training and test.  (resnet 0.27->0.18)
  • Other-approach is to run all images on VGG, take mid-layer output and cluster it (1000 clusters) and use the cluster mean result

All images or part of it?
  • Most ran the image as a whole (with/out clustering)
  • R-CNN (tuned differently from object-detection) helped  VGG 0.22>0.175

3. How to choose models for en-sample

  • Try to use different models, trained differently. For example, one VGG and another Resnet. one augmented, the other not...
  • X-fold is common, but basic.

4. How to combine models

  • use scipy minimize function and create a custom geometric average function to minimize logloss of all models.

Statefarm - experiment 3 - VGG16 finetune

VGG:   [conv(x2/3)->max-pool]xfew-times Then classifier head (4096->4096->1000)

Finetune VGG16

I saw one approach , with great results(!) where the the whole model was loaded, and only the last softmax layer was changed from the original(1000) to the new target (10).
In that case finetuning was done on ALL the model together, with slow learning-rate (sgd 1e-4)

I will use another approach:
We will replace the whole classifier head (4096->4096->1000).
1. [optional to save time later]  Load the model without the last dense part. Run it once on all the images and save to disk all the intermediate output  (512x7x7) per image of the train/validate/test info.  for reference, 10K files should take 1.9 GB of disk space.

2. Create alternative classifier. I used a small one due to a bit weak machine.   256->10
    model = Sequential()  
    model.add(Dense(256, activation='relu'))
    model.add(Dense(10, activation='softmax'))

Train it. I used few optimizers
SGD(lr=1e-3, momentum=0.9, nesterov=True)

SGD(lr=1e-3, momentum=0.9, nesterov=False)- BESTSaved model to disk vgg_head_only1_out_0_epoc20

SGD(lr=1e-4, momentum=0.9, nesterov=True)

SGD(lr=1e-4, momentum=0.9, nesterov=False)