Monday, January 11, 2016

Priority based decision of event


This is a domain-specific problem, not a general technology post, so read is as a computer-science riddle...

The problem

Match-making site between 2 people (like a dating site) or 2+ people activity (like a community basketball game of 5 on 5).
  • Each person can have a list of properties and preferences for activities, for example: age-group, distance from home, activity-type.
  • A person can choose multiple overlapping activities with priorities between them.
  • The selection should be bi-directional and we should make sure that there will not be "sterlie" selections where the other person did not even had a chance to choose him back.
Few other requirements:
  • There will be a rating system for people, after enough activities were made.

Discussion

Approaches:
CLUSTERING - TBD
One approach will be to try and cluster the data before any user-query is given, so that later queries will be answered faster. Many search engines use this approach.
The issue here is to define the clustering criteria.  It is clear that we can cluster by location (NYC is in different cluster than San Francisco.
We may also cluster in NYC by age  (a year for cluster).
And by degrees (none , 1 or 2 and above).
So now in NYC there are 40 clusters (20-60 age group) x 3 degree-types.
Although it can reduce the query time, a query on NYC women, any-degrees, age 25-35 will still yield a lot of data. (8m people in NYC area. This can yield 8m(2x4)=1m. Considrable reduction but still a big cluster...

QUERY

Storage: Each (1M) user has document/description of up to 1K, of which only small amount can be used for filtering. For a total of 1GB of storage.
Calculation: We assume the DB sort-by field is not enough, and that we need a smarter score function (like this)

[V1] Let`s first start with the trivial but but slow solution.

  • On Init, we will load all the N persons into memory, and check match between them (N^2), and will save a mapping of  person1-person2 = 50%.  person1-person3=95%, etc. we will have N^2/2 such mapping.
  • On a change in person preferences, we will recheck all his matches (N)
  • On a new person, we will check all his matches (N).

Assuming there are 1M persons, the match matrix can grow to be quite big. (1Mx1M)
In addition we need to save the history choices of every user (let`s say 1K for each) and filter them out, so they will not appear again. The issue here is that when we sort the results, for an old user, we will always filter out the first few thousand results.


[V2] Let`s try to refine this. As we know that only people with close-location proximity and age proximity are relevant, we first filter out big mismatches (using database-query)

On-demand, but cache query results
On Init, we do nothing
On change in preferences, we do the N/100 query, sort, and cache the results which will serve our paging result.
On new-person we do another query, and add the recent results to any cached result, as a secondary list.

To illustrate:
PersonA have a cached query  [10,000 sorted results, of them we choose the first 5,000] and a list of people already ranked by the user [the last 320 people he saw]
New PersonZ arrives and creates a query. Person C just changed it`s location to Far-away
PersonA now will also have a new-candidate [personF score 54] to combine during the next pagination, and [-MINUS- personF score ]

The first query (each time a user changes it`s preferences) requires ranking N/100. but it is cached.
The bigger issue, is how to save the updates to the cache. If a new user arrives, we may need to update 1,000-10,000 options.  No DB can simply persist this load. It may be solved by having the second-table in-memory with no-persistence.

[V3] Only on-demand, no-caching
Starting at a certain hour, only the logged-in users are the subset to query from. This will further considerably reduce the user set. Also use ElasticSearch to do the calculation in-server-farm
see function_score there.
There will be no caching  (except maybe client side cache of the next page of results, 5-10 tops).
Problem:  as we do re-ranking each time, of everything, it must be super fast, and it is hard to imagine how we can achieve it when needing to rank N/100 results.

[V4] On the first call, do a query on N/100 and save result and time of query.
On the second call, query only new data which changed preferences and merge to create a new snapshot.
Problem: snapshot change can be add-new-person.  update-up-score






BackEnd as a service (BaaS)

Standing on the shoulders of giants


Building software is becoming simpler and simpler.
Server-logic abstractions in the form of web-services (like google-apis, cloud databases).
Choosing the right level of abstraction will mean developing a website/app very fast, with most of the scaling problem already solved and instead of having 'opertaions' team, just use a credit card.

Why wasn't it done till now? why BaaS is becoming so popular today?
Cause of Mobile. When you had web, the developer already had his web-server and the skill-set to configure it to his needs.  iOS/Android applications are hosted in iOS/Google-play store, so there is no web-server, and their developers skill set is typically do not include web. Hence the need rose and many try to solve it.

A lot of the mobile apps, have common requirements from the BaaS:
Single-player games require: user-login, high-score, cloud save-game (upload file)
Social apps require: user-login, chat, uploading text and files,
All need analytics


  1. User Login abstraction
    1. Allow using identity from other site, like facebook/google.
    2. Allow email+password pair, including activation and password recovery
    3. Create the relevant user-tables and track their frequency of visiting 
  2. Analytics
  3. Push-notifications
  4. Database
    1. Usually non-SQL
    2. ** There are numerous hosted databases. MongolLab/Composeand/ they do only one aspect
    3. ** ElasticSearch capabilities  (uncommon need-  Compose/ Bonasi / Qbox)
  5. Files hosting (images/videos, but not the app itself)
    1. simple upload
    2. good download performance using CDNs.
  6. Chat messaging
    1. real-time chat (only a second delay between messages)
    2. history of the chat
    3. ** Pusher/PubNub



BaaS providers, see here
The most known: "Parse" and "Firebase"  (but they do not prvoide it all...)
The less known: Buddy, QuickBlox , appriaries

Parse (now part of facebook)

REST Based (polling)
Mature with great modules for Users,Analytics, Push,DB similiar to MongoDB (not sure if it is actually same interface) with great online editor.
Files hosing is lacking, Polling based (no chat!) , can be expensive.


Firebase (now part of Google)

Persistent connection based, which is totally great, except the cost per connection (user)
As of (11-Mar-2015) free: 50 Max connections,5 GB transfer, 100MB storage.
For 49$ a month -> 200 Max connections, 20GB transfer , 3GB storage



Pusher
Free: Max 20 connections, 100K messages.
For 49$ Max of 500 connections, 1M messages

HTML5 cross-platform (mobile/desktop) games


CocoonJS by Ludei

Replaces the PhoneGap(+PG Build) and supply Chromium browser.
It also provide libraries for some native/payment/push etc.
Your code should work on both desktop(without-it) and Mobile(with-it) . Need to look how the native/payment/push  works on desktop.
No special API for the game code itself, you can use any js libraries you want.

Compilation is done on the cloud. The result is a zip with debug&release apks, but sadly they are unsigned. you need to download them and using android java sdk:
%JAVA_HOME%\bin\jarsigner -verbose -keystore <keystore> -storepass <store-pass>  -keypass <key-pass> <apk_unsigned.apk> <key>
<android-sdk>\build-tools\21.1.2\zipalign.exe -v 4 <apk_unsigned.apk> <apk_signed.apk>
Then copy the signed_sdk to the android device (or send via drop-box) and install


Famo.us

Again replace the deployment with Chromium.
Have a special API for the game/app code, you don't work with the DOM regualrly and thus can't use JQuery for example, see here , not sure about the the level of abstraction of prepreated widgets (ready-to-use-list-view etc)
Great performance see this codepen on mobile


Game engines
nice list . another list review

low-level
pixi.js is a popular low-level engine
createjs - set of libs (easeljs,tweenks,soundjs,preloadjs)

turbulenz (no real mobile support)
http://biz.turbulenz.com/developers  - (like the game 'polycraft')
It uses WebGL for rendering, even on 2d games, with great performance on desktops, see this serious of articles on moving to HTML5.
But this means that on a lot of mobile devices the game will not work at all, not even chrome... There is still no native-app for it but the 2013 news says they are working on it.
Another note on the platform, the game engine can be used free of charge, and they encourage you to use their servers for hosting,multi-player, badges etc. The way they make money is 30% of the payment services, if you use theirs.  Note that you don`t need to.


TO TEST:
http://phaser.io/  ,   , http://www.kiwijs.org/ ??
http://www.gameclosure.com/ -movie - - comes with few pre-made "engines" for platformer/menu/maps etc
http://impactjs.com/


TOO BASIC  - http://craftyjs.com/ , http://melonjs.org/
 http://www.pandajs.net/ - no big games as of yet

Highlight UTF text in Javascript

How can I highlight text in a page which contains non-english text?

Javascript regexp do not support such text at all.  It is planned to be supported at ECMAScript6, which currently, May-2015, is not supported by any-browser\node.js, but there are build tools to mimic it`s functionality.


How to highlight in Angular? using filters  

The solution is from this link, and a filter-tutorial is here.

1. Assuming you have a style like:
       .highlighted { background: yellow }

2. Create a filter
angular.module('moduleName,[])
.filter('highlight', function($sce) {
    return function(text, phrase) {
      if (phrase) text = text.replace(new RegExp('('+phrase+')', 'gi'),
        '<span class="highlighted">$1</span>')
        return $sce.trustAsHtml(text)
  }
    })

3. In the html, Instead of using the regular binding:
{ model.test-text | highlight}
make sure to use the ng-bind-html
<ANY ng-bind-html="model.test-text | highlight"></ANY>




Ionic


Installation on windows is possible, but takes time.  You can use c9.io to dev-test it (although the cordova part can't be used). Use:
npm install -g cordova ionic
ionic start myApp tabs (to create sample app)
ionic serve $PORT --nolivereload   (to shed few seconds on reload stuck)

Good tutorials are hard to find, and when I say good, I mean that the ratio of explanations rows to code rows is bigger than 1.
TODO: maybe this one: http://www.htmlxprs.com/post/6/creating-a-realtime-image-sharing-app-with-ionic-and-socketio-tutorial

States and Routing between pages

Let`s talk about the ionic starter tab apps (ionic start myApp tabs)
It uses state notion, which actually comes down to three built-in parameters behind the scenes:
$state = contacts.detail
$stateParams = {"chat.id":"1"}
state full url = /tab/chats/1

so if you have a list of people to chat with, and clicking on a person from the list will switch you to the chat-details, you will put a href="#/tab/chats/{{chat.id}}" on the item.
The state will know to search for the /tabs/chars/X , in our case this url is tab.chat-details


see this link to see how they change in realtime.
from app.js:
.state('tab.chat-detail', {    --> $state . it`s just the name, using X.Y let you inherit from abstract X 
      url: '/chats/:chatId',
      views: {
        'tab-chats': {
          templateUrl: 'templates/chat-detail.html',
          controller: 'ChatDetailCtrl'
        }
      }
    })
In the controllers.js , this how the controller asks for the id of the person:
.controller('ChatDetailCtrl', function($scope, $stateParams, Chats) {
  $scope.chat = Chats.get($stateParams.chatId);
})


Template for List

see: http://ionicframework.com/docs/api/directive/ionList




Images CSS

Let`s discuss few different image usage patterns, and their relevant CSS.
In all cases, we assume source image size is different from the div we want it to be in the page, and that aspect-ratio must remain.

  • Full site background image (all-page width and height)
  • Full width but small-height header/middle-page image, common in bootstrap templates.
  • User image by itself, sometimes with circle around it
  • User image inside a gallery of images, where we have known X images in each row, but almost infinite number of scrolling rows.

good start at: http://www.w3schools.com/css/css3_backgrounds.asp


Full site background image (all-page width and height)

.your-div-class {
    background-size: cover;

    background: url(splash-bunny.jpg) no-repeat  center center;
}

Few notes:
1. "background-size: cover"  expects your image to be always greater than the size of the div.  So if your max target size is Samsung note 2 (720x1280), make sure that your image is equal or bigger than that.  If not, the image will be at it`s maximum size on the center of the page, which you don`t want!
2.  Cover will only show a fraction of your image, think of it as a "sliding-window" above your image.  Ask your art dude to make sure the important stuff is in one postion, so that users with small screen can still understand it. It is common to put the important stuff in the center of the image, and then use "center center".
If the important stuff is on the top left, use "top left", see this lovely bunny:

 If you use "center center", on small screens you will only see his pink belly, and no one will understand what the cover image means.
If you use the "top center" , even on narrow screen, the bunny face will be shown (but the right/left side of it will not)
If you use the "top left" , on narrow screens, only the text will be shown.
If you use "center bottom" the feet will always be shown, but on small devices (and landscapes) the head will not.





3. This css does not make your div as big as the screen, this is your job.  I won't get into it , but only for pure testing, try min-height: -large-number-like-1800px;


Full width but small-height header/middle-page image, common in bootstrap templates.

see above, just one change: reduce the height of your div to be what ever you like.


User image by itself, sometimes with circle around it


If your div and your user-image aspect-ratio are the same (200x200 against 100x100 image source or 500x500 image-source.  this is trivial (use either contain or cover).
If your div is a square and the user-image is not, you can use 'cover' and then choose the right top/bottom/center right/left/center,  to catch the user face. in the bunny above top-center is best, but it will still show the text on the left and some of the belly.
Lot of sites allow the user to choose an crop a square.  When the user is doing it you can actually crop the image and generate a new one.
Making it circle is as easy as adding:  border-radius:50% 50% 50% 50%;
Note:  Make sure, again, that the image size is BIGGER than the div size. cover will not work otherwise.






Thoughts on mobile app prototyping

I`ll discuss two questions

  1. Why it is so hard to create good cross-platform mobile app.
  2. Why it is hard to take the leap from prototype to real app, without re-coding.

Why it is so hard to create good cross-platform mobile app.


All the solutions for cross-platform apps try to use pre-exiting technology for cross-platform development. The most popular is Html5.  The next one is Xamarin (C#).
Sadly this solution is sub-par, as app development uses 3 "modules", and sadly, most don`t cover them.

  • The Programming language itself(Object-C , iOS)
  • The controllers and layout library (buttons, input-text, time-picker, page-transitions, menu-drawers)
  • System libraries like browsing-contacts, changing-contact-ringtome, play-music, take-a-picture

It will be clearer, we we go over the current solution:
Html5 + Cordova/phone-gap


  • Programming language is Java-script. Code will run on an internal-browser( WebView) [good]
  • Controllers and layout use Html/Css based , as desktop controllers/layout don`t work well out-of-the-box there are libraries which re-implement them like Ionic, Jquerymobile, but both don`t cover all the components, for example: date-time-picker is missing (mobi-scroll can cover this for a price), map location is bad . page transitions are slow. [bad- clearly sub-par components ]
  • Cordova create native-code wrappers around some of the system-libraries, and you can write new ones, but it actually means that for non-trivial things, you will have to write quite the code yourself.
C#/Xamarin
  • Programming language is C#. On Android is is run in an interperter, on iOS compiled to native code.
  • The controllers and layout are Xamarin`s . They are not exactly like native, and sometimes look bad (time-pciker android version as an example). transitions look good, as performance is better.
  • Again, native-code wrappers, with limited system-library usage. The wrappers here are better, and usually expose more details than the js version, but still not all are exposed.
Corona labs (used for mobile games
  • Programming lanaguge is C++ (natively supported) or internal scripts
  • There is no real need to controlelrs/layout, so they are not really supported. Games UI is different.
  • Limited Native code wrappers


Is there another way?
Currently no, but let`s discuss an "invented" one.  Assuming there is a close-correlation between the controllers and the layout of iOS and Android, can't we define it in one language ('xml') and compile it to the other languages?
Can't we write code in one langauge (simplified java/js like, with only if,loops and no special stuff) and generate java/object-code from it?
System libraries are pain, is there a way around them?





App Prototyping using appery.io . Which shortcuts work and which do not


appery.io had a great idea. Let`s make the next Wix/Webydo, but for apps.
Their solution tries to cover (almost) all aspects of app development, but it is lacking in many of them.

  • The app itself is based of JQueryMobile.  It contain a really great UI Editor (drag&drop components with properties editor) and a simple way to add events to it.
    This is a great feature, and it works fine for small apps prototyping.  
    • Problem:  JQueryMobile has it`s limits and although you can use Cordova, there are many things which are pain/impossible to do , like using native Facebook login intent.
    • Problem: Once you get to a bit more complex things, which is not part of the well defined components, it becomes messier.
  • Great tool (similar to curl) to test REST services, and to visually map properties from the REST result to the UI properties. This works very well, for most of the REST calls, as most services result JSON is  quite simple.
  • Built-in database, exposing REST api.  This one similar (maybe equal) to MongoDB.
  • Server code:  Most apps will require extra code on the server. You can write a hosted REST service quite easily (using js code) and schedule it.
  • Simple testing (on the browser) and exporting to .apk/iOS.
See a cool sample here.

Let`s prototype an app together. It will be "kickstarter" for events.
UI:
  • login&user-profile
  • create new event
  • search events and get event list
  • event details & add-me-to-event
  • push-notifications on kick-started event


Create event:
  • Input text for event-image-url (assuming the user knows how to click copy-image-url in the browser).
  • open-text for event data.
  • time-picker for start time. Date and time
  • event-type radio-buttons (sport / lecture /food)
  • "save event" button.
not MVP: add option to upload file from the phone and save it to appery.io FileAPI.
allowed age range (20-30). allowed gender (5 men, 5 women ... or any-gender)

Implementation issues:
  • There is a JQueryMobile date-picker , but not time (hour/min) picker. The forum recommends mobiscroll.com (195$ for few controllers). For now, I used open text.
  • During development, I added new columns to the database, after it was already created. Sadly the database-service does not get refreshed automatically and you have to re-create it (maybe to add parameters manually, didn`t try)
  • Create-service need to map the date+hour input fields into one date db field. This required custom js.
    //look at the date-format, currently it is yy-mm-dd , we convert to yyyy-mm-dd
    return "20"+ value + " "+Apperyio("hour").val();




Login&Profile screen. In the upper part: "Login to facebook" button.
  • In the lower part, the user image (taken directly from facebook profile picture) , a text which can be edited by the user, and the user age.  gender/age. will be taken from facebook
Search event request: choose location in a text box (like , 5th street, SF, California), the box by default will show device location.
event-type radio-button
Search event response: a list of events (image, text, date), for each one,
  •  a status: Open/Full 
  • combo-box for GOING/Maybe/NO 
  • a list of profiles which actually going/maybe.
Push notifications when an event had been kick-started, with the phone number of the event creator


  • TBD...............My events:  a list of the events the user created, clicking on one allows to edit it (the image and text)

The limitations:
Facebook login will not use the native-facebook app which exists on the device and will instead use a rather ugly facebook web page. This is pure ugly stuff.
(this is not appery.io limitation, but any js based implementation. PhoneGap does not fully covers this either)

Abstractions problems (tedious things you still need to do)
Android generate certificate





















Implementing a chat. It is not easy as it sounds...


Chat.  One of the most basic modules in any social app/site, but it still costs you. alot.

Requirements
Let`s go over the requirements of my typical 1on1 chat:
  • The user can have multiple 1x1 chats in parallel. He can see a list of the open chats, and see which chats have unread messages.
  • If it`s an app, a push notification to tell me someone is trying to reach me can be nice.
  • Every chat has history, I want to see, at least, the last 100 messages.
  • Low latency for new chat
  • Presence is optional (mark user as online/offline)
Client-Server protocol
There are two technologies to discuss here.
First, should we use custom chat messages and custom server code for message-format, presence keep-alive, user unique-id, or should we use the well known XMPP protocol (used by lots of IM, like google talk).  Using this known protocol, we can find well tested servers (like ejabberd) which implement it.
Second. what push/poll technology to use?
  • Regular Polling - to achieve low latency we will need to poll each 10 seconds, even if there is no update. This is a bit of a waste.
    Long Polling (also called Comet/Hanging Get) - the client calls a Get and the server will not respond until an update is ready, even if it will "wait on it" for an hour. This trick allows to have a one directional simulated "push" from the server to the client.
    WebSockets - open a bi-directional, always open, socket between the server and the client.  Not all browsers support it yet. The major unsupported version is Android prior to 4.4, this is still quite big market. And (TBD:verify) I heard it may work not as great on mobile, when switching network.

Implementation 

There are many ways to solve this. Let me discuss my solution, which is good for small apps/prototypes. In the appendix, I will describe few more, which can scale better.As my current back-end is based on Parse.com, we will use for the chat-history DB.  As Parse does not have realtime support (no long-polling or web-sockets), we will use a different back-end provider, quickblox, for realtime chat.

The data model:
The trivial one, is storing each message in a log-like table (from, to , time, message,opened-flag) and on query, read the last X(=100) message, and on the client-code combine them into Y(=15) chat conversations and mark those still not-opened.
This is the standard "SQL" way, let`s think about the "Non-SQL" / "Document" way. Another problem here is that X might need to be very large.

One option:

Using quickblox:
Integrate with quickblox users (you can use external users, as long as you saved the user id outside).
On signup/login, login with the same user&password to all platforms. Save in each platform, the user-id in the other platform ,for example use Parse id in Quickblox user.externalUserID, and vice versa.  If you already have some users, change the login code to sing-up quickblox and then login.


Use the chat api 2.0 , when current user P1 wants to communicate to P2 (parse-users-id), retrieve the curent user quickblox id (B1) on the current-user, and query P2 for it`s external B2 id.

Online/Offline - if you want to see the online state of users, you can use the roster chat option. You register for all the users you are intrested in the client (for example current 10 friends) and will get updates in realtime.
Note that this is a very resource intensive action. If a user roster contains 20 friends, although no chat messages is done, you still get updates. A less real-time option will be to use your server to see the states, and let the user query this once in few minutes. 


Other options...

Implement it using Google App Engine Channels (Comet/hanging GET).  Pricing

WebSockets - there are node.js open source implementations, 
or use (Still in alpha) Google App Engine has two option,  the  WebSocket-server on ManagedVM 

You can use a BaaS which provide chat or realtime-data services. This is usually free at first, but costly in large-scale. Generally a good option for rapid-development, for example: Firebase, Pusher (no history), Layer (no js.. coming soon)