Monday, January 11, 2016

Priority based decision of event

This is a domain-specific problem, not a general technology post, so read is as a computer-science riddle...

The problem

Match-making site between 2 people (like a dating site) or 2+ people activity (like a community basketball game of 5 on 5).
  • Each person can have a list of properties and preferences for activities, for example: age-group, distance from home, activity-type.
  • A person can choose multiple overlapping activities with priorities between them.
  • The selection should be bi-directional and we should make sure that there will not be "sterlie" selections where the other person did not even had a chance to choose him back.
Few other requirements:
  • There will be a rating system for people, after enough activities were made.


One approach will be to try and cluster the data before any user-query is given, so that later queries will be answered faster. Many search engines use this approach.
The issue here is to define the clustering criteria.  It is clear that we can cluster by location (NYC is in different cluster than San Francisco.
We may also cluster in NYC by age  (a year for cluster).
And by degrees (none , 1 or 2 and above).
So now in NYC there are 40 clusters (20-60 age group) x 3 degree-types.
Although it can reduce the query time, a query on NYC women, any-degrees, age 25-35 will still yield a lot of data. (8m people in NYC area. This can yield 8m(2x4)=1m. Considrable reduction but still a big cluster...


Storage: Each (1M) user has document/description of up to 1K, of which only small amount can be used for filtering. For a total of 1GB of storage.
Calculation: We assume the DB sort-by field is not enough, and that we need a smarter score function (like this)

[V1] Let`s first start with the trivial but but slow solution.

  • On Init, we will load all the N persons into memory, and check match between them (N^2), and will save a mapping of  person1-person2 = 50%.  person1-person3=95%, etc. we will have N^2/2 such mapping.
  • On a change in person preferences, we will recheck all his matches (N)
  • On a new person, we will check all his matches (N).

Assuming there are 1M persons, the match matrix can grow to be quite big. (1Mx1M)
In addition we need to save the history choices of every user (let`s say 1K for each) and filter them out, so they will not appear again. The issue here is that when we sort the results, for an old user, we will always filter out the first few thousand results.

[V2] Let`s try to refine this. As we know that only people with close-location proximity and age proximity are relevant, we first filter out big mismatches (using database-query)

On-demand, but cache query results
On Init, we do nothing
On change in preferences, we do the N/100 query, sort, and cache the results which will serve our paging result.
On new-person we do another query, and add the recent results to any cached result, as a secondary list.

To illustrate:
PersonA have a cached query  [10,000 sorted results, of them we choose the first 5,000] and a list of people already ranked by the user [the last 320 people he saw]
New PersonZ arrives and creates a query. Person C just changed it`s location to Far-away
PersonA now will also have a new-candidate [personF score 54] to combine during the next pagination, and [-MINUS- personF score ]

The first query (each time a user changes it`s preferences) requires ranking N/100. but it is cached.
The bigger issue, is how to save the updates to the cache. If a new user arrives, we may need to update 1,000-10,000 options.  No DB can simply persist this load. It may be solved by having the second-table in-memory with no-persistence.

[V3] Only on-demand, no-caching
Starting at a certain hour, only the logged-in users are the subset to query from. This will further considerably reduce the user set. Also use ElasticSearch to do the calculation in-server-farm
see function_score there.
There will be no caching  (except maybe client side cache of the next page of results, 5-10 tops).
Problem:  as we do re-ranking each time, of everything, it must be super fast, and it is hard to imagine how we can achieve it when needing to rank N/100 results.

[V4] On the first call, do a query on N/100 and save result and time of query.
On the second call, query only new data which changed preferences and merge to create a new snapshot.
Problem: snapshot change can be add-new-person.  update-up-score

BackEnd as a service (BaaS)

Standing on the shoulders of giants

Building software is becoming simpler and simpler.
Server-logic abstractions in the form of web-services (like google-apis, cloud databases).
Choosing the right level of abstraction will mean developing a website/app very fast, with most of the scaling problem already solved and instead of having 'opertaions' team, just use a credit card.

Why wasn't it done till now? why BaaS is becoming so popular today?
Cause of Mobile. When you had web, the developer already had his web-server and the skill-set to configure it to his needs.  iOS/Android applications are hosted in iOS/Google-play store, so there is no web-server, and their developers skill set is typically do not include web. Hence the need rose and many try to solve it.

A lot of the mobile apps, have common requirements from the BaaS:
Single-player games require: user-login, high-score, cloud save-game (upload file)
Social apps require: user-login, chat, uploading text and files,
All need analytics

  1. User Login abstraction
    1. Allow using identity from other site, like facebook/google.
    2. Allow email+password pair, including activation and password recovery
    3. Create the relevant user-tables and track their frequency of visiting 
  2. Analytics
  3. Push-notifications
  4. Database
    1. Usually non-SQL
    2. ** There are numerous hosted databases. MongolLab/Composeand/ they do only one aspect
    3. ** ElasticSearch capabilities  (uncommon need-  Compose/ Bonasi / Qbox)
  5. Files hosting (images/videos, but not the app itself)
    1. simple upload
    2. good download performance using CDNs.
  6. Chat messaging
    1. real-time chat (only a second delay between messages)
    2. history of the chat
    3. ** Pusher/PubNub

BaaS providers, see here
The most known: "Parse" and "Firebase"  (but they do not prvoide it all...)
The less known: Buddy, QuickBlox , appriaries

Parse (now part of facebook)

REST Based (polling)
Mature with great modules for Users,Analytics, Push,DB similiar to MongoDB (not sure if it is actually same interface) with great online editor.
Files hosing is lacking, Polling based (no chat!) , can be expensive.

Firebase (now part of Google)

Persistent connection based, which is totally great, except the cost per connection (user)
As of (11-Mar-2015) free: 50 Max connections,5 GB transfer, 100MB storage.
For 49$ a month -> 200 Max connections, 20GB transfer , 3GB storage

Free: Max 20 connections, 100K messages.
For 49$ Max of 500 connections, 1M messages

HTML5 cross-platform (mobile/desktop) games

CocoonJS by Ludei

Replaces the PhoneGap(+PG Build) and supply Chromium browser.
It also provide libraries for some native/payment/push etc.
Your code should work on both desktop(without-it) and Mobile(with-it) . Need to look how the native/payment/push  works on desktop.
No special API for the game code itself, you can use any js libraries you want.

Compilation is done on the cloud. The result is a zip with debug&release apks, but sadly they are unsigned. you need to download them and using android java sdk:
%JAVA_HOME%\bin\jarsigner -verbose -keystore <keystore> -storepass <store-pass>  -keypass <key-pass> <apk_unsigned.apk> <key>
<android-sdk>\build-tools\21.1.2\zipalign.exe -v 4 <apk_unsigned.apk> <apk_signed.apk>
Then copy the signed_sdk to the android device (or send via drop-box) and install

Again replace the deployment with Chromium.
Have a special API for the game/app code, you don't work with the DOM regualrly and thus can't use JQuery for example, see here , not sure about the the level of abstraction of prepreated widgets (ready-to-use-list-view etc)
Great performance see this codepen on mobile

Game engines
nice list . another list review

pixi.js is a popular low-level engine
createjs - set of libs (easeljs,tweenks,soundjs,preloadjs)

turbulenz (no real mobile support)  - (like the game 'polycraft')
It uses WebGL for rendering, even on 2d games, with great performance on desktops, see this serious of articles on moving to HTML5.
But this means that on a lot of mobile devices the game will not work at all, not even chrome... There is still no native-app for it but the 2013 news says they are working on it.
Another note on the platform, the game engine can be used free of charge, and they encourage you to use their servers for hosting,multi-player, badges etc. The way they make money is 30% of the payment services, if you use theirs.  Note that you don`t need to.

TO TEST:  ,   , ?? -movie - - comes with few pre-made "engines" for platformer/menu/maps etc

TOO BASIC  - , - no big games as of yet

Highlight UTF text in Javascript

How can I highlight text in a page which contains non-english text?

Javascript regexp do not support such text at all.  It is planned to be supported at ECMAScript6, which currently, May-2015, is not supported by any-browser\node.js, but there are build tools to mimic it`s functionality.

How to highlight in Angular? using filters  

The solution is from this link, and a filter-tutorial is here.

1. Assuming you have a style like:
       .highlighted { background: yellow }

2. Create a filter
.filter('highlight', function($sce) {
    return function(text, phrase) {
      if (phrase) text = text.replace(new RegExp('('+phrase+')', 'gi'),
        '<span class="highlighted">$1</span>')
        return $sce.trustAsHtml(text)

3. In the html, Instead of using the regular binding:
{ model.test-text | highlight}
make sure to use the ng-bind-html
<ANY ng-bind-html="model.test-text | highlight"></ANY>


Installation on windows is possible, but takes time.  You can use to dev-test it (although the cordova part can't be used). Use:
npm install -g cordova ionic
ionic start myApp tabs (to create sample app)
ionic serve $PORT --nolivereload   (to shed few seconds on reload stuck)

Good tutorials are hard to find, and when I say good, I mean that the ratio of explanations rows to code rows is bigger than 1.
TODO: maybe this one:

States and Routing between pages

Let`s talk about the ionic starter tab apps (ionic start myApp tabs)
It uses state notion, which actually comes down to three built-in parameters behind the scenes:
$state = contacts.detail
$stateParams = {"":"1"}
state full url = /tab/chats/1

so if you have a list of people to chat with, and clicking on a person from the list will switch you to the chat-details, you will put a href="#/tab/chats/{{}}" on the item.
The state will know to search for the /tabs/chars/X , in our case this url is

see this link to see how they change in realtime.
from app.js:
.state('', {    --> $state . it`s just the name, using X.Y let you inherit from abstract X 
      url: '/chats/:chatId',
      views: {
        'tab-chats': {
          templateUrl: 'templates/chat-detail.html',
          controller: 'ChatDetailCtrl'
In the controllers.js , this how the controller asks for the id of the person:
.controller('ChatDetailCtrl', function($scope, $stateParams, Chats) {
  $ = Chats.get($stateParams.chatId);

Template for List


Images CSS

Let`s discuss few different image usage patterns, and their relevant CSS.
In all cases, we assume source image size is different from the div we want it to be in the page, and that aspect-ratio must remain.

  • Full site background image (all-page width and height)
  • Full width but small-height header/middle-page image, common in bootstrap templates.
  • User image by itself, sometimes with circle around it
  • User image inside a gallery of images, where we have known X images in each row, but almost infinite number of scrolling rows.

good start at:

Full site background image (all-page width and height)

.your-div-class {
    background-size: cover;

    background: url(splash-bunny.jpg) no-repeat  center center;

Few notes:
1. "background-size: cover"  expects your image to be always greater than the size of the div.  So if your max target size is Samsung note 2 (720x1280), make sure that your image is equal or bigger than that.  If not, the image will be at it`s maximum size on the center of the page, which you don`t want!
2.  Cover will only show a fraction of your image, think of it as a "sliding-window" above your image.  Ask your art dude to make sure the important stuff is in one postion, so that users with small screen can still understand it. It is common to put the important stuff in the center of the image, and then use "center center".
If the important stuff is on the top left, use "top left", see this lovely bunny:

 If you use "center center", on small screens you will only see his pink belly, and no one will understand what the cover image means.
If you use the "top center" , even on narrow screen, the bunny face will be shown (but the right/left side of it will not)
If you use the "top left" , on narrow screens, only the text will be shown.
If you use "center bottom" the feet will always be shown, but on small devices (and landscapes) the head will not.

3. This css does not make your div as big as the screen, this is your job.  I won't get into it , but only for pure testing, try min-height: -large-number-like-1800px;

Full width but small-height header/middle-page image, common in bootstrap templates.

see above, just one change: reduce the height of your div to be what ever you like.

User image by itself, sometimes with circle around it

If your div and your user-image aspect-ratio are the same (200x200 against 100x100 image source or 500x500 image-source.  this is trivial (use either contain or cover).
If your div is a square and the user-image is not, you can use 'cover' and then choose the right top/bottom/center right/left/center,  to catch the user face. in the bunny above top-center is best, but it will still show the text on the left and some of the belly.
Lot of sites allow the user to choose an crop a square.  When the user is doing it you can actually crop the image and generate a new one.
Making it circle is as easy as adding:  border-radius:50% 50% 50% 50%;
Note:  Make sure, again, that the image size is BIGGER than the div size. cover will not work otherwise.

Thoughts on mobile app prototyping

I`ll discuss two questions

  1. Why it is so hard to create good cross-platform mobile app.
  2. Why it is hard to take the leap from prototype to real app, without re-coding.

Why it is so hard to create good cross-platform mobile app.

All the solutions for cross-platform apps try to use pre-exiting technology for cross-platform development. The most popular is Html5.  The next one is Xamarin (C#).
Sadly this solution is sub-par, as app development uses 3 "modules", and sadly, most don`t cover them.

  • The Programming language itself(Object-C , iOS)
  • The controllers and layout library (buttons, input-text, time-picker, page-transitions, menu-drawers)
  • System libraries like browsing-contacts, changing-contact-ringtome, play-music, take-a-picture

It will be clearer, we we go over the current solution:
Html5 + Cordova/phone-gap

  • Programming language is Java-script. Code will run on an internal-browser( WebView) [good]
  • Controllers and layout use Html/Css based , as desktop controllers/layout don`t work well out-of-the-box there are libraries which re-implement them like Ionic, Jquerymobile, but both don`t cover all the components, for example: date-time-picker is missing (mobi-scroll can cover this for a price), map location is bad . page transitions are slow. [bad- clearly sub-par components ]
  • Cordova create native-code wrappers around some of the system-libraries, and you can write new ones, but it actually means that for non-trivial things, you will have to write quite the code yourself.
  • Programming language is C#. On Android is is run in an interperter, on iOS compiled to native code.
  • The controllers and layout are Xamarin`s . They are not exactly like native, and sometimes look bad (time-pciker android version as an example). transitions look good, as performance is better.
  • Again, native-code wrappers, with limited system-library usage. The wrappers here are better, and usually expose more details than the js version, but still not all are exposed.
Corona labs (used for mobile games
  • Programming lanaguge is C++ (natively supported) or internal scripts
  • There is no real need to controlelrs/layout, so they are not really supported. Games UI is different.
  • Limited Native code wrappers

Is there another way?
Currently no, but let`s discuss an "invented" one.  Assuming there is a close-correlation between the controllers and the layout of iOS and Android, can't we define it in one language ('xml') and compile it to the other languages?
Can't we write code in one langauge (simplified java/js like, with only if,loops and no special stuff) and generate java/object-code from it?
System libraries are pain, is there a way around them?