Implementing a chat. It is not easy as it sounds...

Chat.  One of the most basic modules in any social app/site, but it still costs you. alot.

Requirements
Let`s go over the requirements of my typical 1on1 chat:

• The user can have multiple 1x1 chats in parallel. He can see a list of the open chats, and see which chats have unread messages.
• If it`s an app, a push notification to tell me someone is trying to reach me can be nice.
• Every chat has history, I want to see, at least, the last 100 messages.
• Low latency for new chat
• Presence is optional (mark user as online/offline)

Client-Server protocol
There are two technologies to discuss here.
First, should we use custom chat messages and custom server code for message-format, presence keep-alive, user unique-id, or should we use the well known XMPP protocol (used by lots of IM, like google talk).  Using this known protocol, we can find well tested servers (like ejabberd) which implement it.
Second. what push/poll technology to use?

• Regular Polling - to achieve low latency we will need to poll each 10 seconds, even if there is no update. This is a bit of a waste.
  Long Polling (also called Comet/Hanging Get) - the client calls a Get and the server will not respond until an update is ready, even if it will "wait on it" for an hour. This trick allows to have a one directional simulated "push" from the server to the client.
  WebSockets - open a bi-directional, always open, socket between the server and the client.  Not all browsers support it yet. The major unsupported version is Android prior to 4.4, this is still quite big market. And (TBD:verify) I heard it may work not as great on mobile, when switching network.

Implementation 

There are many ways to solve this. Let me discuss my solution, which is good for small apps/prototypes. In the appendix, I will describe few more, which can scale better.As my current back-end is based on Parse.com, we will use for the chat-history DB.  As Parse does not have realtime support (no long-polling or web-sockets), we will use a different back-end provider, quickblox, for realtime chat.

The data model:
The trivial one, is storing each message in a log-like table (from, to , time, message,opened-flag) and on query, read the last X(=100) message, and on the client-code combine them into Y(=15) chat conversations and mark those still not-opened.
This is the standard "SQL" way, let`s think about the "Non-SQL" / "Document" way. Another problem here is that X might need to be very large.

One option:

Using quickblox:

Integrate with quickblox users (you can use external users, as long as you saved the user id outside).

On signup/login, login with the same user&password to all platforms. Save in each platform, the user-id in the other platform ,for example use Parse id in Quickblox user.externalUserID, and vice versa.  If you already have some users, change the login code to sing-up quickblox and then login.

Use the chat api 2.0 , when current user P1 wants to communicate to P2 (parse-users-id), retrieve the curent user quickblox id (B1) on the current-user, and query P2 for it`s external B2 id.

Online/Offline - if you want to see the online state of users, you can use the roster chat option. You register for all the users you are intrested in the client (for example current 10 friends) and will get updates in realtime.
Note that this is a very resource intensive action. If a user roster contains 20 friends, although no chat messages is done, you still get updates. A less real-time option will be to use your server to see the states, and let the user query this once in few minutes. 

Other options...

Implement it using Google App Engine Channels (Comet/hanging GET).  Pricing

WebSockets - there are node.js open source implementations, 

or use (Still in alpha) Google App Engine has two option,  the  WebSocket-server on ManagedVM 

You can use a BaaS which provide chat or realtime-data services. This is usually free at first, but costly in large-scale. Generally a good option for rapid-development, for example: Firebase, Pusher (no history), Layer (no js.. coming soon)

Project time estimates and Murphy's law

As we saw in the last few iterations, Murphy's law: "Anything that can go wrong, will go wrong" tend to be 

very strong resident in our midst.

Let's take a client-server short-project as an example. When you ask the client developer how much time will it take he answers 10 hours of uninterrupted work.

Usually a developer will interact with the world and his team mates (eat-lunch, do code-review, attend a meeting) , so let's count this as 2 days.

Now let's go to Murphy

client-code:

1. Complexity in library-code:  You thought the default 1 row of code will do the work for you but it turns out it does not support your use-case, or t fails unexpectadly only in your input/
2. Complexity in code: changing one element caused an unexpected bug in another unrelated component.
3. Infrastructure failures:  your machine/network/server- can be down for few hours.
4. computer/network/coruppted-int server can take 0-5 hours to overrcome
5. Product requirements may slightly change when they see your finished feature. They don't ask for a new feature, they just explaining you what they ment before better.

UI related code:

1. UI/Art integration may not be flawless, depending on your workflow
2. Sometimes, you can't use the Art "as is" and it is not apparent until you actually try.

client-server potential issues:

1. Miss-understanding of the API usage, can take some time to debug and understand.
2. Sometimes it is not a miss-understanding, but a plain bug in the server-code.
3. Sometimes it's not a bug,but a miss-configuration / not latest-version of code etc

So, there are at least 10 potential risks. What should be the estimate in this case?

One approach is to look at the optimistic scenario and say 2 days. It will sometimes work, but a lot of the time will fail poorly.

Second, opposite approach is to look at the worst case scenario and assume all the problems will happen simultaneously. This approach will say let's estimate 8 days.  One might say "better safe than sorry" and if it will be faster, the developer will say he is done.

A third approach is to aim to be correct most of the time (80-90%) but assuming that some problems might happen , but not all of them at the same week. 
Someone, on the upper-management has to fully understand this, and to manage the risk accordingly.

If you actually want numbers.  Let's say this is an example of (a-low) risk-matrix for the 2 days task. No real human think of this in this manner, but you do have "hunches" about it. In this case, I include a 0.1 chance fo r 2 extra-days due to bad-library behavior. 0.2 extra-day if sever-code does not work etc.

risk	0.1	0.2	0.05	0.1	0.2	0.1
extra-day	2.00	1	5	1	1	3

The probability of failure for this risks is:

46% - 0 late

71% - up to 1d late

83% - up to 2d late

90% - up to 3d late

94% - up to 4d late

97% - up to 5d late

98.7% up to 6d late

99.1% up to 7d late

99.8% up to 8d late (remember it's on a 2d task....)

Now let's talk about who takes the buffer.  

Let's say a developer gives the matrix to his manager, and ask the manager to tell him how to estimate the task.  What should the manager answer?

Due to Parkinsons's law "work expands so as to fill the time available for its completion", a reasonable manager should do three things.: 

• Ask the developer to be 80% right, and in this case to add 2d to the estimate : 4d. Tell him that is is ok and fully expected to finish a day early, and they if all hell break loose, the manager got his back with few extra days.
• Keep a mangment buffer for the next 17% (3 more days) which can be used by the developer, and will be totally ok to use, if the developer explained that the risk actually happened.
  One buffer pool can sometimes be shared amongst multiple team members.
• Inform his managment about the chances.  If it's a live-or-die scenario, they may have their own buffer on the extra 3% risk.

Just few random notes on Parse and Mobile FileUpload

FileUpload on mobile

 <input type="file" accept="image/*">

Android (Chrome and 4.4 native) provides a reasonable, but not-pretty, interface.

If you want to use camera, add the capture camera option. Note that it will not allow other images.

Parse.com allows to upload the file and query it by URL. It`s very simple, but note that it means an image access counts as an API call, and as images usually come in packs, this can cost your quota.

Detect change on textarea

Use $('#textAreaId#).on('change',function() { do something here });

change will be called when the focus is changed and any of the inside-text change.

Parse model update

On many views, you will want to first refresh the view from a saved DB element, and then on edit update(re-save) the element and refresh the view again.

For this pattern, always keep a cached parse object which was returned from the DB and is connected to parse via the row-id.
On refresh, update the cached-element, if found in the DB.
On edit either re-save (on the cached one) if it already exists, or save a new one and create cache.

You can save multiple objects via Parse.Object.saveAll (array)

Parse queries

When querying a pointer, sadly, you can't just pass an instance of a previously returned item (item itself != item pointer). You have to create a pointer object. 

var pointerToUser = {  __type: "Pointer",  className: "_User",       objectId: myUserItem.id };

You can combine multiple queries using Query.or(q1, q2)

Parse weird save/query results

• Parse will not fail if you query a db class which do not exist at all. Beware of Parse.Object.extend("SpellingMistakeWillNotBeNoticed");
• It will also allow you to save db columns which do not exist at all. Beware of (item.set('spellingMistake',666).

Register/login

Jeff Atwood wrote a great post on creating the simplest login page. Read it!

Let`s use JQueryMobile for the widgests and Parse for the backed.

Insert this html to the JQueryMobile page:

<h3>Login/Register</h3>

<label for="usrnm" class="ui-hidden-accessible">Email:</label>

<input type="text" name="user" id="loginEmail" placeholder="Email">

<label for="pswd" class="ui-hidden-accessible">Password:</label>

<input type="password" data-inline="true" name="passw" id="loginPassword" placeholder="Password">

<input type="submit" data-inline="true" onclick="myForgotPassword();" value="reset pass"> 

<label id="loginMessage" style="color:red"></label>

<input type="submit" data-inline="true" onclick="myLogin();"       value="Log in">

<input type="submit" data-inline="true" onclick="myRegister();"    value="Create new account">

Use JQuery to extract user values and fill login message ($("#input_id".val() , or $(#message_id").text("new text")

We will use Parse user.signUp , Parse.User.logIn and Parse.User.requestUserPassword methods for the backend.

function myRegister() {

    console.log("myRegister");

    $('#loginMessage').text(""); //clean

    var email = $('#loginEmail').val();

    var password= $('#loginPassword').val();

    

    var user= new Parse.User();

    user.set("username",email);

    user.set("email" , email);

    user.set("password", password);    

    user.signUp(null, {

       success: function(user)  {

           console.log("register success");

           $('#loginMessage').text("success");

       },

       error: function(user,error) {

           console.log("register failed "+ error.code +" " +error.message );

           $('#loginMessage').text("failed "+error.message);

       }

       

    });

}

function myLogin() {

    console.log("myLogin");

    $('#loginMessage').text(""); //clean

    var email = $('#loginEmail').val();

    var password= $('#loginPassword').val();

    Parse.User.logIn(email, password, {

       success: function(user)  {

           $('#loginMessage').text("login success");

       },

       error: function(user,error) {

           $('#loginMessage').text("login failed "+error.message);

       }

    });

    

}

function myForgotPassword() {

    console.log("myLogin");

    var email = $('#loginEmail').val();

    

    Parse.User.requestPasswordReset(email , {

       success: function() {

           $('#loginMessage').text("password reset sent");

       },

       error: function(error) {

           $('#loginMessage').text("failed to reset password "+ error.message);

       }

    });

}

Tinder-clone in javascript

Tinder-like moving images:

Have a look at the end result here.
And now let`s go step by step:

[Step 1] http://jsfiddle.net/3y7fogdv/1/embedded/result/

• Check if it actually doing anything: Use requeststAnimationFrame (+big function to make it better)
• The actual moving of the element is done in js using: e.style.transform = 'translate3d('+newX+'px,'+ newY+'px,5px)';
• To return it to first position, add style{transition:all 0.3s;}
• To make sure the image is not dragged by itself:   user-drag: none;  -moz-user-select: none;  -webkit-user-drag: none;

[Step 2] http://jsfiddle.net/3y7fogdv/7/embedded/result/

Lets use a card-style (see here) and move the entire card.

[Step 3] http://jsfiddle.net/3y7fogdv/18/embedded/result/
Add tinder buttons  (V,X) below, add z-index to the .card ( z-index:1;    position: relative; ) so that it will move above them.
Also note that when we move the card down, it causes scrolling (solve by adding overflow:hidden to the main content) , and that the end of the content ends with the end of the buttons instead to extend to the full view. This is solved by a small js, to change main content size according to full size- header height.
Last change is addition of rotation to the transform  rotate( startX-newX)*0.05 + deg

[Step 4] http://jsfiddle.net/3y7fogdv/19/embedded/result/
check the release location on hammer "panend" event, and a check of ev.isFinal and ev.deltaX ev.deltaY.  If it is above a threshold, move it far away.
Also , let`s use multiple cards, so we will use multiple Hammer instances, one for each card. Luckily hammer supports it and hammer events are separated per card.


[Final notes]
On mobile, panning may cause scrolling the window down. To avoid this, add:
        $(document).bind('touchmove', function(e) {
            e.preventDefault();
        });

Searching for close matches to my choices

Problem:

Every user is asked a serious of binary questions:
What do you prefer, cats or dogs
Do prefer winter or summer
Do you consider computer-science gradatees as scientinst. Yes/No

There will be lots of users (1M), but not more than 300 questions.
Find the 10 closest matches for a certain user.

Solution:

This problem is a similar to search nearest-neighbors of  Hamming distance. It is simplified as there are only binary(yes/no questions), hamming works with more options, but nearest-neighbors is extremely difficult problem....

1. For a query for unkown, new user, this stackoverflow question says that it`s a tricky problem.  For short distances (4 from 32 bit integers) BK-tree and VP-tree can speed up queries up to 10 times, or even 1000 times for very short distance of 1.   For medium ranges and above, no good data structure exists, and brute-force is needed.

If we can assume that our users always have 10 close matches with short distance, we can use one of the suggested data-structure, otherwise, just use blunt-force each time.

2. For a big calculation of all the matches, there might be a better solution. Didn`t find one.

Page pagination in JqueryMobile

Goal:

  Infinite number of different, similar, pages like Tinder app, where each page contains images,labels and text.

One ajax request to the server contains the data for the next 10 pages, so we won`t do the "simple" solution of reading a new ajax page each time.

Solution:

We will have 3 pages  (match1, match2, match3) each one as a "next" button pointing to the next one.

We will save the data of the next 10 pages at any given point, let`s call it the model.

On each page pagebeforeshow event, we will change the data on the non visible page.

Once the model data is empty (all the 10 pages were seen), we will use an ajax call to fill the model.

model state= 50,51,52,53,...,59

[page1] [page2] [page3]

on the first time (when all pages are empty) and later, when one page becomes obsolete, we will fill the obsolete pages

  50            51         52

[page1] [page2] [page3]

  active

  53            51         52

[page1] [page2] [page3]

               active

...

 59            57         58

[page1] [page2] [page3]

              active   

when we get to this state, once 58 becomes active, we will want to replace page2 with 60, so we will do an ajax call and update the model with it,
delicate points:  

• If ajax call is on it`s way, there is no reason to make a new call.
• We should also clean the previous page to null data (default image and text), in case the user will be faster than our ajax call. We may even consider creating a "loading" progress bar, if the ajax is very slow.