On Software Development

C# ain`t too bad

2011-09-29T01:39:00.000-07:00

C# ain`t too bad

Nothing is really free in Microsoft`s world , but you can start by using the express editions of Visual C# 2010 and Visual Web Developer 2010 .
It amazing how much better the (java) free eclipse IDE is compared to the free visual C# , but if you are willing to put few hundred $`s , The Ultimate version+Resharper should give eclipse a very good competition.

Two notes:
LINQ is absolutely brilliant. It`s simply the best advancement in programming languages in the last 10 years. I do hope java will get it soon.

NUnit is a great testing tool , especially with the new attributes and constraints
But it does not integrate to VisualStudio Express (and Microsoft does not allow plugin/extensions to the epress edition)
Don`t worry , just do:
1. Add reference to NUnit in the project. (Add reference -> .Net -> component name=NUnit frameworl
2. Open the nunit-x86.exe (on windows7) , create a project and use "Add assembly" to target the relevant project dll/exe.
3. Don`t run it manually, instead change the Program.cs to run it, instructions here . Note that you will need to add the nunit-gui-runner.dll from the NUnit ../bin/../lib directory

Principles

2011-05-11T12:26:00.000-07:00

On requirements

1. You ain`t gonna need it - Always implement things when you actually need them, never when you just foresee that you need them.

2. MoSCoW method (Must Should Could Won`t)All requirements are important, but they are prioritized to deliver the greatest and most immediate business benefits early. Developers will initially try to deliver all the M, S and C requirements but the S and C requirements will be the first to go if the delivery timescale looks threatened.

On project scheduling

1. Brooks`s law - adding manpower to a late software project makes it later. "Nine women can't make a baby in one month"

2. Parkinson`s Law - always time each task. Work expands so as to fill the time available for its completion

On Design

1. Analysis paralysis - over-analyzing (or over-thinking) a situation, so that a decision or action is never taken, in effect paralyzing the outcome. A decision can be treated as over-complicated, with too many detailed options, so that a choice is never made, rather than try something and change if a major problem arises.

2. KISS principle - Keep it simple stupid. The principle is best exemplified by the story of Johnson handing a team of design engineers a handful of tools, with the challenge that the jet aircraft they were designing must be repairable by an average mechanic in the field under combat conditions with only these tools. Hence, the 'stupid' refers to the relationship between the way things break and the sophistication available to fix them.

.... It seems that perfection is reached not when there is nothing left to add, but when there is nothing left to take away

3. Rule of three ( not two as many think , three!) - Rule of three is a code refactoring rule of thumb to decide when a replicated piece of code should be replaced by a new procedure. It states that you are allowed to copy and paste the code once, but that when the same code is replicated three times, it should be extracted into a new procedure.

scala

2011-04-29T08:20:00.001-07:00

AKKA http://days2010.scala-lang.org/node/138/162

Eclipse plugin for Eclise 3.4 is incredibility bad.
Eclipse plugin for Eclipse 3.5.2 is better.

tutorials
http://www.scala-lang.org/docu/files/ScalaTutorial.pdf
http://www.scala-lang.org/docu/files/ScalaByExample.pdf

Postgres 9.0 hot-standby

2010-09-23T03:04:00.000-07:00

Postgres 9.0 have a new feature "streaming-replication" + "warm-standby". In few words , it allow asynchronic replication between master and slave (there is a delay , but it is a small one) and the slave can do read-only queries.

synchronios replication is only planned for 9.1

It`s a good time to recap the different sharding/master-slave options and their roles.

At first you develop the code for one DB instance and you assume three things about it:

1. It will never fail - the machine,process and disks are eternal.

2. data-scale: It can grow is size as much as you want - terra-bytes of disks space.
3. query(users) scale : The machine is as fast as you want - It can supports millions of fast queries in parallel.

After few months , you deploy it and then find out that these 3 things are not so easy to achieve. Lets see what are the (Postgres) solutions for them.

query-scale

Scale-Up : Still one machine, but buy the best one money can get - 8 cores,32GB are relatively cheap. But twice as strong machine will cost a lot more the twice the money. If you need x2 or x4 , switch to scale-out.
Scale-Out :
2.1 Scale reads not writes: (good for mainly read DB)
Use one master server (read&write) and multiple read-only slave servers which get periodically updated from the master server. The master and slave share the same data.
Postgres 9.0 supports it out-of-the-box
2.2 Scale writes by sharding

Fault-tolerance

For transaction-DB where no record can be lost , this is a hard task. For data-warehouse which may loose the latest few minutes of updates , this is much simpler.

No data loss:

Shared-disks : the master writes to a shared storage. When it fails the slave mount it and loads the data. It is a "code-standby" as it can takes few minutes to recover.
pgpool II - slows the updates on the master!
synchronios log shipping (should be in postgres 9.1) - does not (considerably) slow the master. Failover take 15 seconds (10 to detect failure , 5 to start the warm-standby)

Some data loss allowed:

Postgres 9.0 warm-standy & streaming-replication
Slony-1 updates the slave using triggers

Data scale

Start with more disks on the machine , then scale-out with sharding.

Axis2 RPC client perfromance

2010-07-13T00:27:00.000-07:00

The client time is consisted of few parts:
* Open TCP connection to the server (send sync, recieve syn/ack , send ack = 1 round trip)
* Send/Recieve WS xml (send at least one packaget , recieve at least one back = at least 1 round trip)

Let`s eliminate the TCP connection time. My ping to the server is 222 ms. The server calculation time is small (less than 15ms). So one round-trip = ~235.
The goal is to reduce each WS call to one round-trip.

1. Use high HTTP1.1 keepAlive value:
Pro : reduce the tcp-connection round-trip
Cons: Load-balancing based on IP will not function as intended , as after first connection the client will keep using the same host over and over.

Make sure HTTP1.1 keepAlive is turned on on both client and server.
Tomcat HTTP1.1 keepAlive is turned on by default ( you can shut it off in server.xml , Connector section , by adding maxKeepAliveRequests="1". [default is no parameter at all and means 100 requests )
Turning it off will cause 2 round-trips - [first=626] then 457, 455 , 458 , ...
Keep the connection alive by using it more frequently than the timeout.
The default keepAliveTimeout in Tomcat 6.0.18 is equals to connectionTimeout which is typically 20000ms=20 seconds.
It means that the TCP connection will stay alive only if it is used more frequently than this.
You can configure the tomcat to higher value (120seconds for example) in the server.xml, Connector section.
Note: adding the parameter keepAliveTimeout="500000" alone is not enough for tomcat6.0.18 ,cause of this bug. you will need to add disableUploadTimeout="false" keepAliveTimeout="500000"

2. If your data is more than few bytes, compress it to save transfer time. You need to configure both the server and the client to support compression.

Client - GZIP options
options.setProperty(HTTPConstants.MC_ACCEPT_GZIP ,true);

[note A. if you configure HTTPConstants.MC_GZIP_REQUEST , the request itself will be compressed . will work fine for a server which support gzip. will fail on other-servers.
If you requests are not huge , not worth the risk! ]
[note B. this option can cause you problems if you have LB which is problematic with chunking :options.setProperty(HTTPConstants.CHUNKED,Boolean.TRUE ) ]
Sever: on apache6.0.18 configure on server.xml, Connector . Add: compression="on" , which means try to use compression depending on the client.
Firefox 3.6.6 by default will not get it compressed, nor a WS client without GZIP options, but our optimized client will use it.
Note: this configuration should work with/without compression on both the client and the server side and should be "backward compatible". Transfer time can be considerably reduced (in my case 500ms on 6KB to 260ms on 1KB)

3. TCP connection optimization

TCP implementation increase the buffer size , sending more and more bytes per second , as long as the connection is not congested , so after a second or two , you will reach the optimal size. But , on high-latency, high-bandwidth connections (100MB/s , 50ms latency) the OS max tcp-buff limit the max throughput and can reduce by a factor of 2-10.
A good max value should be: round-trip[ping]* bandwidth .
If this is your use case (like connection between data-centers) , tune the system.
example = 100Mb/s = 12.5MB/s . ping 100ms --> 12.5MB/s*0.1s= 1.25MB. the default on UNIX is 256KB , so you can get a boost of x5.
If you are not using a persistent TCP connection , and the round-trip is large , for small files it will take few round-trips to get to optimal speed (and by then the file already finished). If this is your case, set the Socket receive/send buffer to high value immediately.
For low-latency , small-transmittion , also use Socket.setTcpNoDelay which disable Nagle`s algoritm. Wikipedia quote:
Nagle's algorithm works by combining a number of small outgoing messages, and sending them all at once. Specifically, as long as there is a sent packet for which the sender has received no acknowledgment, the sender should keep buffering its output until it has a full packet's worth of output, so that output can be sent all at once.

P.S.
1. The default Axis2 RPC Client do not use a pool of HTTP connectors. This default implementation is good only for tests and will cause problems in productions , follow the instruction here and use MultiThreadedHttpConnectionManager.
2. To see TCP connection do (linux) "netstat -nap | grep

RAID and SSD

2010-03-26T06:58:00.001-07:00

I really need a new desktop machine. The current dilemma is between large,cheap mechanical HD or small, costly, SSD.
The second question is "to raid" or not "to raid".

Solid-state-device VS mechanical disks:
Typical windows usage is faster with SSD. Question remains, is there a good&cheap drive. The new generation from SanDisk (g3) claims the following numbers:
Average sequential (higher is better) and Random-access latency(smaller is better)
Read- 220MB/s ,write 120MB/s . 0.1-0.2 ms latency. cost: for 60,120GB for 229$,399$

Mechanical disks: Some numbers:
VelociRaptor VR150 ,10,000 rpm, 300GB 102MB/s , 7ms - ~200$
Segate Baddacuda ST3320613AS , 7200 rpm , 230GB - 99.2MB/s ,17.1ms - 50$

Conclusion: SSD is worth the money

Raid
There are two goals for Raid. One is performance , Second is fault-tolerance. For a desktop owner the first tends to be the only relevant goal. Let`s assume you have a budget of up to 2 HD`s.
Raid 0 - two striped disks. If a file contains 4 sections , disk one contains 1,3 . The other 2,4.
on mechanical/ssd disks: x2 sequential-read , x0.9 random-read

P.S. What does the SATA150/SATA300 Interface means?
This is the BUS speed. SATA150 supports up to 150MB/s . SATA300 (or SATA-II) supports up to 300MB/s.
Make sure that the SATA interface max throughput > HD max throughput .

Why small objects cloud hosting is so slow? Amazon S3 and SimpleDB performance

2009-12-28T10:37:00.000-08:00

Is it possible to use S3 as a scalable storage for small objects?

S3 is perfect for large objects (1GB file) but is quite bad for small objects like those a media-server needs (songs,images).
Behind the scenes S3 uses standard disks with varying levels of replication and striping. Standard disks have poor random-access, but a good sequential throughput.
Each object has a minimal access time, caused by the disk seek time.see disk throughput and seek-time .
The real solution for small files is either caching the most-accessed in-memory (done by most of the Content-delivery-networks like CloudFront).
If you want to retrieve thousands of small-objects per second and not use a cache, then the only way to do it fast is to spread them across as many physical disks as possible. The number of disks does grows linearly with large files, as few files take a whole disk.
But with small files, the number of disks grows very slowly (500GB contains thousands of images) which may all sit in one machine. That`s why S3 is not appropriate for small files.
see read-performance of S3

For a real fast , cost-effective, random-access of small files the only solution will be different hardware: Solid-state disks. They are becoming cheaper every year and their random access speed is close to sequential performance.
Why there is no CDN with this solution yet? Because caching of the most-access is enough for the typical website. The access pattern of most users is not really random.

Command&Conquer managment approach

2009-11-24T12:20:00.000-08:00

Great two articles . A must read to anyone who manage anything more then himself.

Flex visualization libraries

2009-07-24T06:37:00.000-07:00

multiple layouts : http://labs.kapit.fr/label/diagrammer
Spring based map: http://mark-shepherd.com/blog/springgraph-flex-component/
Facebook integration: http://www.adobe.com/devnet/facebook/

Java update 6.14 and compressed Oops

2009-06-12T06:17:00.000-07:00

From the release notes:

Compressed Object Pointers
The -XX:+UseCompressedOops option can improve performance of the 64-bit JRE when the Java object heap is less than 32 gigabytes in size. In this case, HotSpot compresses object references to 32 bits, reducing the amount of data that it must process.
(from: http://java.sun.com/javase/6/webnotes/6u14.html)

This gives a huge boost for memory-intensive applications.
Take for example an application which heavily uses tree-maps.
Tree element: pointer to next + pointer to previos + pointer to key + pointer to value (+ some housekeeping bytes).
4 x64 bit pointers which can become 32 bits now.

It won`t reduce the memory footprint to half , but it can give you an incredible boost. so ... (-XX:+) UseCompressedOoops.

Fighting bytes - 64bit Lean hashmaps

2009-04-30T12:34:00.000-07:00

Lets say you need to implement a 375,000,000 (more than 3/8 GB) key-value mapping in live memory.
The good: It is integer based and you have a 40GB RAM machine.
The bad: It is a multi-map : integer to multiple integer mapping.

Lets start with the trivial solution:
First of all, as we use 40GB , we need a 64bit java implementation.
Lets use hashmap where key=Integer , value= LinkedList of Integers.
HashMap cost (64bit) = aroud 60 bytes per entry.
Integer cost(64 bit) = 24 byte. remeber we need one in key and at least one in value.
LinkedList(64bit) = starts with ~80byes , ~40 more per entry.

For fast calculation, lets assume one Integer in key and one in value.
We get N*(60+24+24+120) = N*228 = 85.5GB. Way above what we have - the 40GB include the linux , some GC room and few other applications.

But The real max we can have per entry is 100bytes (which will get us to 37.5GB)
So , what to do?
We must reduce the multi-map cost (LinkedList is a big waste). We can also reduce the Map implementation.

Lets start with the hashmap for a minute.
It mainly consists of an Entry[] = N*pointer_size
and each entry is a hash value(integer) and three pointer(to key,to value, to next-entry) = 16(minimal-per-java-object)+4bytes+3*pointer_size = (64bit machines) 48.
For a total of: N*56.
As the size of Entry[] is typically more than than the current size(load-factor ~0.75) , it will even be bigger.

Use of open-addressing , can reduce the size (although there are certain other issues):
N keys
N values
no need for hash(performance by caching), no need for next-entry.
This can reduce the size by 4+8=12 , for a total of N*44. Drawback is that load-factor need to decrease.

Other optimizations:
if key/value are primitives , save a lof of space by using the value instead a pointer and a heap-allocated value.
Integer as a key costs a pointer + 24 byte (if integers are not object-pooled)
int as a key(in a specialy designed map) only cost 4 byte!!!

array based - http://members.optusnet.com.au/~askitisn/CRPITV91Askitis.pdf

This post (http://stackoverflow.com/questions/629804/what-is-the-most-efficient-java-collections-library) gives a list of few customized hash implementation with up to 1/3 the java.util size.

In particular Trove supplies TIntIntHash for int to int hashin TIntObjectHash and TIntArrayList which can be used for (half) efficent integer based multi-maps.
It does not supports out-of-the-box multimaps.
I`m pretty sure it will take me under the 100 bytes threshold. If not, the only other options is to implement a custom int to multi-int hash table, using open-addresing. I hope I won`t get to that (re-inventing wheels is not my cup of tea)

Nice .... Google App Engine support java

2009-04-24T05:03:00.000-07:00

Good new: http://googleappengine.blogspot.com/2009/04/seriously-this-time-new-language-on-app.html
Good news for java web-developers . Sad news for the java cloud-based startups...

Going to port the python app to java. Yes, it will take 10 classes where in python it was 100 rows , but ....

Automation

2009-03-26T13:46:00.000-07:00

1. Deployment automation
2. Test automation

Automatic deployment and automatic tests are the developer`s best friend.
An application development has three stages: Development , Debugging and Maintanance.
In the first stage you have a lot of time to spare. In the secod you have less and in the third you have none.

Automatic test and deployment save you time and bugs on the 2nd and 3rd stage , when you really need them ( need to fix the bug and redeploy by noon tommorow morning otherwise your very important customer will drop you , and just now your QA is on vacation and your IT guy is on sick day ).
But automation takes time to develop. It will elongate your development time (usually by mere few days) and automation may not be the coolest feature to develop (usually its pretty boring) ... you understand where it is going...

When you are at maintanance mode and suffers greatly from the wasted time on the manual testing/deployment and you understands the need for this , you simply do not have time to spare on building it.

Conclusion: Think ahead , in the development period , if the feature can benfit from automation down the line. If so , add the required days to the (middle) of the development period - you will thanks yourself later for doing so.

Google App Engine

2009-03-14T15:31:00.000-07:00

Prediction:
Google App engine (or its copycats) will be the only way to develop web applications in the following years.
They have everything exactly right except the programming language. Note to google: the average industry developer does not know Python. In fact if he knew Python, he would have probably already been recruited as a Google employee... So your runtime environemnt is not usuable to us , the common programmers.

When(and if) they implement PHP , Java (and maybe even ASP) runtimes , this project will become te industry leader and will kill all the data-stores(including Amazons) and half of the IT guys in the world.

If they will be smart enough, it can even create a standard web-development kit , as long as it is not Python based!!!

Outsourcing

2009-03-14T14:06:00.000-07:00

In the current state of the economy, outsourcing become more popular every day.
Recruiting new workers is tough . Recruiting new workers from outsourcing is even tougher.

I'm doing a grouse generalization here and I apologize for it in advance for this observation:
Outsourcing workers can be divided to two groups:
The first group is extremely talented and experienced men who understands they can do a lot more money as outsourced-consultants than as regular workers.
The second group is of the "refugees" - the less talented/experienced men which could not find a regular job as company employee and turned to outsourcing companies as a last resort

The outsourcing H.R. will always tell you their workers are excellent for the job. So how can you distinguish between the extremely talented and the less-than-averages ones?
One way is to have a good technical interview, but as you usually outsource the field in which your group do not have the technology expertise , it will be hard for you to really estimate the interview results.
A second way is quite simple. The expert will demand a salary which is usually more than twice the salary of a regular employee whereas the refugee will usually ask for a salary similar or even less than a regular employee. So if the man you are interviewing wants a lot of money , there is a good chance he is the real deal. If he is cheap , there is a good chance he is not worth a penny.

I'm not saying anyone who demands lots of money is immediately good and that anyone who demands less is immediately bad , but pay attention to that...

Another Gem from Joel

2009-03-14T13:58:00.000-07:00

Joel is truly a genius. As I`m "maturing" in work experience , I see more and more how true his observations are.

Joel is has written a great post on "How to be a Product Manager"

And a remark of few other great ones:
1. UI - Never show a UI-screen to an "outsider" before it is fully polished.
2. The law of leaky Abstrsctions

Running a web-server from home

2009-01-24T08:56:00.000-08:00

So , you want to run a web-server from home (why? , for testing purposes, of course! , don`t deploy anything on home server. Your ISP will eventually block you...)

Behind a router:
1. Change tomcat back to port-80.
2. Assign a LAN static-ip to the web-server on the LAN.
3. configure your router to do port-forwarding on port 80 to your static-ip.
4. Use a dynamic-dns to map a domain into your ever-changing ISP provided ip address.
And you are ready to go.

Few resources:
Good ref (for MAC)
http://www.canyouseeme.org/ - external site which tests your open ports.
http://www.no-ip.com/ - dynamic-DNS (with free version for non-commercial usage)

Web Servers

2009-01-15T12:43:00.000-08:00

web-servers
Lots of buzz words are flying around when talking about web-servers. I assume you know what it is , and just want to find which of the free/commercial servers will better suit your need:

web-servers For PHP/CGI/Pyton (etc) developers
1. Apache HTTP server [ = HTTPd]- a C-based implementation of a web-server for which you can use modules with the well-known "script" languages: PHP , Perl , Pyton , Ruby.
2. NginX - 2nd generation server (faster,less known)A C-based implementation of web-server which allow you to use again the "Script" langugs. This is a rather new server with better io utilization which should outperforme the Apahce HTTP server.
3. Lighttpd - another 2nd generation server, less table than NginX.

web-servers For Java developers

1. Apache Tomcat - a JAva-based implementation of web-server for which you can write modules in Java (plain java servlets or JSP). The internal modules which support the servlets are called "catelina" and the JSP is called "jasper" , but these names are only important for knowning which log file to look. It is not related to the popular Apache-c-based-Http-server. ( they share the "Apache" prefix cause they come from the same open-source organization , but there are dozens of Apache projects which are not technology related to one another)
2. Jetty - a small webserver , usually used for web-applications which run on a very small scale ( some application need only to work only on the localhost , and Jetty is a good solution for them)
3. JRun
4. Resin

web-server for ASP.NET
Here, like in many microsoft applications, you have one and only one choice: IIS

Simple so far? lets make it bit more complex - Interconnections!!!!
On some sites, one web-server gets the original request and forward it to another web-server. Few milliseconds later , it gets the response and send it to the user.
In this setups it is possible that the "gateway" web-server will be a c-based one , and the other is java-based one. They will connect to each other in a different protocol (its a waste to use HTTP between two servers) .
For apache-http-server(c-based) to Apache-Tomcat(java-based) , the connector is using the AJP protocol. see The Apache Tomcat Connector - AJP Protocol Reference
http://tomcat.apache.org/connectors-doc/generic_howto/workers.html

Why using interconnections?
usually when the Apache(c-based) is the load-balancer and the Tomcat`s really to the dynamic-http generation.
Note that it is possible to run a web-app (pure java) of a load balancer on tomcat , but it (by rumers) does not get close to the Apache(c-based) performance at load balancing.

Performance Benchmarks

Between different web-server "languages"
It will be intersting to see which web server flavor (c-based, java-based or IIS) performs better in similiar tasks of static pages and similiar dynamic pages. Didn`t find a good comparison , but iI do do not see anyone chaning the implementation languge of his company site just because the other langugle web-server performance is better.

Inside java web-server
http://www.webperformanceinc.com/library/reports/ServletReport/index.html - bottom line is that Tomcat and Resin are close. 1000 concurrent users per second (when some have short sessions , some medium and some large -yes , it does not say much about the numbers in your site)
http://tomcat.apache.org/articles/performance.pdf
http://tomcat.apache.org/articles/benchmark_summary.pdf - static files (Tomcat VS apache-C-based-server)

conclusion
It`s hard to find reliable benchmark between all the web-server matrix. (There should be a "Tom`s Hardware" type of benchmark for web-servers)
That`s said , the actual server choices which are seem to be recommended.

Choose dynamic web-server , according to the language you are familiar with. If you use Java , the free Apache Tomcat should be great.
If you are not going to have a lot of static pages/images or load-balancing , you can use Tomcat for all the rest too.
Otherwise , Choose "static " web-server as one of the 2nd generation C-based servers. NginX is a good choice. Its probably better than the java based servers. You can also use its load balancing capability.

Final notes:

Look at the feature matrix of the servers you use. If you need advanced features (load balancing , proxing etc ) , your choice will be easier because the number of servers will be more limited.
Remeber to look for the bottlenecks: dynamic web-server usually uses a Database which tend to be the bottleneck. Static web-server bottleneck tend to be the bandwidth itself.

Joel on software

2008-10-26T18:04:00.000-07:00

If you did not read his blog, you need to. (http://joelonsoftware.com)
He is one of the best writers on software development , and my favorite blogger.

Not all the posts are on software managment. The best software blogs , are from few years back. I gurantee it is worth your time to go to his archieve and look at it.

A great recent pearl on standard API: martian headsets
enjoy!

coding and testing pipeline

2008-10-26T17:57:00.000-07:00

This is one good Agile/Scrum concepts which is rarely used in most of the teams. I recommend using it even if your working "waterfall".

Instead of releasing the full feature set for testing on beta /hand-over-to-test day, you need to have small releases , each one both stable and has additional testable feature.
Benefits:

This will allow the test team to start testing even before the regular date (more time to test means more bugs are found)
reduce the instability of the version , which save time for both developers and QA.
Allow faster feedback on bugs for the developers.

There should be be multiple releases in each version. Don`t go to far and release one each week (remember that it has an overhead which can never reduced to zero, no matter what the "Agile" methodology says).
I believe a once-in-month period is sufficent , for a 3 months-version perioid.

Tester expertize

2008-10-26T17:47:00.000-07:00

For each new feature , make sure the QA (quality assurance) team gets all the info/training material as the developer does.

If the tester learns what to test from the original developer , the testing level will be very poor , and will only verify that what the developer already knows.

Anecdote regarding this:

On of my first jobs, I was responsible of an old feature which someone else (which is long gone) wrote. The tester did not know the feature and asked me , the responsible developer , what does this feature do.

After asking around and understanding that no one around knew this feature, I had to dwell into the code and describe what the code does.

The tester happily wrote it down and then went back to his office and started testing it.

Anyone see`s the problem here?

The tester tested my understanding of the code and not what the feature should do. He found a bug and showed it to me. I then realized I failed to read what the code does in a corner case. Both he and I did not know whether the actual code behavior was ok or not , and we ended up telling that I was wrong understaning the code , not that the code has a bug.

Develolper Expertize

2008-10-26T17:11:00.000-07:00

Trivial principle ,yet usually ignored because lack of time in the short run.

Developers must have expertize in both the subject domain they are working on, whether it is finance/networking or music.
They should also have a minimal level of knowledge in the programming language they are using.
Sounds trivial? it is not. Most teams has at least one "novice" programmer strait from the university which, usually, don`t have the required expertise I mentioned. You can:

Let him code new features
Let him only fix old features
Let him watch but never touch production-code. He can touch non-critical scripts.
We do not hire inexperienced employees.

What happens in your team?
I bet most of you answer (2) , some answer (1) , know that it is bad but have tight deadlines and only small percentage will answer (3) , and almost no one answered (4).

What is the right answer? I have to go with (3) and in some cases even (4).
I think the best analogy to this questions , is a case where a father and son need to paint the house together.
Asking your 3-years old boy to help you paint the house is not helpfull. Yes , he is cute and eager to help , but we both know you can do it alone in half the time you will do it together , because you need to superwise him all the time , make sure he does not "eat the paint" and paint the furntiure.
When your kid is 10 years old you may start using him, knowing that although you will not save time yet, this will help your kid learn the craft.
When your kid is 12 , you still supervise him, but already gain a good time boost with his help.
When your kid is 15 , he can do the job himself , and if you work together , you get the job done in half the time.

Setting the analogy aside, you can can let your new employee write non-production code. This is either internal-company-code or even unit-test-code. At the first week or two , you probably won`t gain any time benefit from him doing that , but like the 10 years old kid , the developer learns the craft.
For those of you who answered (1) with tight-deadlines , please understand that it will only take you more time , in the medium-long run. What is code good for if it is written fast but does not pass first QA tests?
For those who answered (2) , another analogy: lets say you missed a very small spot when painting the cieling , will you give your 3-years old boy the brush and a ladder and let him loose? you will probably , again , loose time. Fixing bugs is not a good way to make new employees learn the craft. It is only the "easy" (not caring) way to do it.

Last emphasize , expertise of a new subject is needed even for an engineer with dozen years of experience. For example , multi-threading expertise is a hard and quite rare one.
Don`t let any programmer write multi-threaded code before s/he had a course/book in the matter and few weeks(months?) of experience with writing non-production multi-threaded code.

Increasing Quality - review and unit-tests

2008-10-26T16:45:00.000-07:00

These principles aim is to increase code-quality and thus overall productivy. They aim to reduce the number of bugs which slip the development team. We know that bugs take (exponentially) longer time to solve depending to how fast they were traced.

The 3 principles are:

design review
unit-testing
code review

Design review

The most important review is not the code-review but the design review. This stage is mainly ignored in most of the companies. After the design is ready (Im implicitly assuming you are doing some sort of a design before starting to code) , let the tech-leads/seniors/smart-people review it and give remarks.
The benefits of design review:

(a) change done in this change and not after the code-review , will save days of work later.

(b) change done here will most probably be actually done. A lot of code-reviews ends up saying "it should have been done differently but we do not have time to fix it and re-test it now , so leave it be". In the design-review stage these does not happen.

(c) will be used to communicate initial API between different components , helping other developers understand what to expect for.

Note that design-review must be enforced by the managers on the tech-leads/seniors , as it is usually done in the period when the tech-leads has the least time.The manager must press them on this issue.

unit test

The best way to know the quality of a component is to ask the developer what is the level of functionallity he tested. Let there be the following answers: "It may work" , "It probably works even on hard cases" , "It defentially works on all cases".

The good result should be: "It defenitally works on avegarage and it probably works even in corner cases". less then that and they did not test enough , more than that , and they wasted time on unnecerayy tests.

Developers should do extensive unit-testing , manaul , or better yet, automated ( in java the term "JUnits" is most familiar) . This developer testing goal should be to test the high-risk sections of the code. 80-90% coverage will be better than 100% coverage , if you have more tests of error-prone code.We must understand that on 99% of the features, full code coverage is both impossible and a waste of resouces!

It is OK for a developer to leave some tedious test cases to the quality-assurance people as long as the developers do make an effort to test few of these cases and knows that the code "probabily works".

code review

This is done quite commonly and its a good practice. This stage can help trace dozens of bugs very fast. It also helps to spread knowledge between team members , and provide coding feedback for the coder.

One uncommon practice in code review I want to recommend is that the code-review will be done"offline" without the original developer sitting next to the reviewer.

This will ensure the developer writes code that another human being can understand (usually using remarks, good naming conventions etc) , as he knows someone will read his code in few weeks and will suffer otherwise.

It will also ensure that no bugs are swept under the table, as you don`t talk with the code-reviewer directly.

The coder and the reviewer will meet twice : once before the code review for high-level explanation and once after the reviewer read the whole code, to share his remarks.

Bug fixing assignment - the greatest dilema of them all

2008-10-26T16:10:00.000-07:00

closing to the end of the first version development, the team find few bugs in the version. It always happen. When should these bugs be fixed?
Don`t listen to the engineer which will tell you: "I`m your best engineer , developing the cutting edge features the fastest in the company. Let the new programmers to solve the bugs. This way they will learn the system and make me available to new development".
Does the enginner sounds reasnible to you? A lot of managers falls to this pit. Its not that the engineer is evil , s/he usually really believes that is the best way to work.
It is not.

I got two simple principles for you regaring bug-fixing:

The bug-fixer should be the original coder if possible (and it is the manager job to make it possible)
The bug-fixing time should be close to the finding of the bug time . It does not have to be in that day , but shortly after. (and not in the next bug-fix stage which is 3 months later)

There is a great debate on these two principles, lots of manager will say the complete opposite. I will explain why I believe they are true.

Principle 1 - The bug-fixer should be the original coder (if possible)

feedback - The coder understand his own coding mistakes and becomes a better coder, which produces less bugs.
It will take the orginal coder a slight percentage of the time it will take to another programmer to solve it.
When solving a bug , there is always a small chance to introduce a new different one. When the fixer is not the original coder , the chances become much bigger.
Not all the bugs , but the nasty ones , are only simptoms to other ("chronic") bugs. The original coder will find the real bug and treat it. Other programmer will probably cure the symtoms , not the problem. One simple example: a race-condition bug can be solved locally , but can be a symptom of multi-threading problem across-the-system.

Principle 2 - The bug-fixing time should be close to the finding of the bug time
If you wait to end of the current version to fix the bug you can have few problems:

When all the bug-fixes are concentrated in the end of the version , it is guranteed , that the bug assignment will never be as good.
unstable development version until that time. Finding workaround for multiple developers will usually waste more time that fixing it on the first place.
unstable test version - the QA team may not be able to test a feature well , or waste time in discovering an already known bug. This will further reduce the quality of the version.

Focal points

2008-10-26T15:43:00.000-07:00

This is a very simple principle which take`s zero effort to implement. Surprisingly, a lot of teams do not implement it.

The problem:
When outsiders (non group guys) want to talk with insiders about a bug/feature/just-an-explanation , they rarely know who to refer to in the group.
In a lot of teams , the outsider need to go through all the group members , one-by-one , to find out who can help him with the relevant subject. When this one-by-one walk is done by email, the outsider will probably give up after three or four fwd`ed mails.
Add to that that sometimes the first-line-manager or 2nd-line-manager are also outsiders , and you can see what a horrific waste of time and bad intra-group communication is done here.
On some cases (aka "Agile") this can also happen inside the group , as a feature/bug does not have a clear go-to guy and instead the "whole group" is responsible to everything.

Solution:
It does not matter if you used the "circle-of-knowledge" principle / Agile / Scrum or any other methodologies, you must assign one "focal point" person for each feature/area.

The focal point will know which feature/bugs are being developed and will be able to say who developed it ,which bugs may be related and pinpoint the exact person which will be able to give you the final answer.
They must know the field well , but do not need to be the one which actually write the code or knows the exact small details of everything.

In a flat-hierarchy team , there can be a lot of focal-points , each one is responsible for his own area-of-expertize.
In a traditional hierarchy team , the group/team leaders tend to be the focal points for all the areas their team are doing.

It takes a MAX of 3 hops to get to the person which knows the answer:
first hop - Danny points you to the focal point of this subject, Joe.
second hop - Joe either knows the answer right away and we are done here, or knows exactly who solved the bug a month ago (Jenny) and send her a email, cc-ing himself regarding the problem.
third hop - Jenny answers the problem and will never "pass-it-on".

a final note: DO NOT MISTAKE FOCAL-POINT TO TEAM LEADER.
Focal-point can be a non-senior programmer too, if he is the main expert in his field (or the field is so un-intresting that no one wants to get near it)

How to fight against "job-security" employee-principle using the "Circles-of-knowledge" manager-principle

2008-10-26T14:20:00.000-07:00

This is a great principle I learned from a great manager in one of the last jobs I had.
I`ll start describing the problem that this principle solves by a true example:

A startup company , lets call it , "the guild" has 20 good dedicated workers which build a great and complex piece of software. As "the guild"`s buisness is blooming they want to recruit 20 new employees each year.
Some of the existing employees, usually the less bright ones , fear for their job/status when the new employees arrive and think of a clever way to have "job security". They do not teach the other employees anything about the systems they are working on and remain the only one knowning how to operate these systems.
At the first year, the company still manage fine. The new employees work tend to "flow" around the old employee`s systems and the new and the old work in hamoney.
At the second year, the old employees understand that they are the only ones knowing how to operate the core systems and they demand better salary/promotion etc. The manager ask himself (and some of his workers) can anyone else knows how to do thier jobs ? as the answer is negative - they did not allow anyone to get near those core-systems , the manager promotes the non-bright people to a team-leader jobs.
At the third year , those team-leaders , continue to use the principle that helped them advance so well , on their previous jobs (the core-systems) and the new jobs: their whole new team don`t allow anyone near their code.
At the fourth year , the "job security" principle is so bad , that no team can work with another team and no new feature which needs inter-team cooperation can be done.
The employess demand a raise again , and as they are now totally irriplacable , get it.
At the fifth year , the employees are getting bored and decide to move to another company. They allow one month notice in which they will be willing to explain all the core-systems that they wrote to the new employee replacing them, but one month is never enough to pass a five years body of knowledge.
At the sixth year, things in the core-systems , which now are not maintanaced by anyone , start to break. No one knows how to fix them and the core-systems are slowly getting deprecated and finally shut down.

This is a true story which happend in various teams in the same company. I`ll bet you met at least one employee which is creating his "job-security" in every big company you worked on.

Some of you are probably running to the "comments" and going to tell me about the great software development methadologies you are working on ( like "Agile/XP") who succesfully solve this problem by making sure that no one person will ever be responsible to one component.
So let me say one thing, Agile/XP solve this problem , but creates another, even more serious one: Instead of only one person knows how to work with the core-system , Agile/XP makes sure that NO ONE will know how to work with the core-system.

Lets me describe the "circles-of-knowledge" solution (again, a young manager I worked with created and perfected it. Im only describing his idea). It can be summed by three simple principles:

Inner circle principle - Every big feature will have few(1-2) people which are responsible for it and have excellent knowledge on the code/bugs of the feature.
Outer circle principle - When coding/bug-fixing a big feature, few(1-2) other people will get tasks related to the feature which they are not in its inner-circle.
Make sure the circles overlaps and interwind as much as you can, creating good redundency among your employees

Usage example:
On our team there there are 5 people ,lets call them A,B,C,D and E. The team task is to develop a client-server system with 4 components, with the following man-hours percentages:
server side framework (20%) , server-side application(big feature 40%) , client-side framework(10%) , client-side application(30%).
This is an example , of work-division according to the circle-of-knowledge principle:
A: inner circle - server-side FW outer circle - server-side apps
B: inner circle - server-side Apps outer circle - server-side FW
C: inner circle - server-side Apps outer circle - client-side FW,client-side apps
D: inner cirlce - client-side FW outer-circle- server-side apps
E: inner circle - clinet-side apps outer circle - client-side FW

This is only one example , of a possible division. The important thing to note is that big features will have more than one people in the inner-circle , and few on the outer circle too and that even the smallest feature will have someone in the outer-circle which have knowledge of it.
And as there are more employees (imagine same team with 10 people) the cirlcels overlaps become clearer.

Using these priciples , it is easy to see the end result:
Redundency - you can "loose" 2 employees and still have excellent knowledge coverage of all the features. Even if your "loose" half of your team, you will still have excellent knowledge about the inner-circle features , and reasnilblee knowledge of the other featues , they covered as part of their outer circle.
Resposibility - at any given time , each employee is directly responsible for a feature of his own. . It is clear who can help in a problem. It is clear who to praise for a job well done and whoto blaim for a system defacts.
Efficency - As all the features are covered with excellent level of knowledge , it is easy to develop new code or solve bugs. There is never a "black-box" feature which all dread to touch.
But you never incrase the outer-circle to the whole team, preventing the problem of "all-need-to-know everthing but end up knowing nothing".

A final note to clear things up: The division to the circle of knowledge is being created over a period of time , meaning , don`t go and divide each two-weeks feature into 4 different people (thats the mistake agile/xp guys tend to do). Instead, make sure that over a long period of time , like 3-6 months , you slowly create this coverage. Example; The first team (inner-circle) will do the core feature on the first 2 months. In the 3rd month a small feature needs to be added - assign one guy from the inner circle and one from the outer-circle. In the 4th month another small feature is added, again , put one from the inner circle and one from the outer-circle , and so on and so on.

Using this method , is not easy at first for the manager, cause it requires a lot of premeditation of his part, but it is worth it. Who said the work of a manager is easy ...

Good Manager

2008-10-26T14:03:00.000-07:00

You need a lot of character to become a good manager. I can`t teach you that (other blogs claim they can ...)
I can give you simple work principles ,to-do and don`t-do list, I collected from the good/bad managers I had the privilege/horror to work with.

Recruiting: Small excellent groups are far more productive than big moderate groups.
Following this rule allegedly contradicts a known principle: "The more people the manager has, the more important the manager is". Yes , known problem.
I can only suggest setting your ego aside when recruiting new employees and thinking what is really best, long run for the business and thus for you. I hope you will choose wisely.
Flatten the hierarchy of your team.
Find how many people can you effectively manage by yourself. The number is usually capped at 5-7. people. If you have that many people, treat all as equal (in importance not in knowledge). If you have more than that , create sub-hierarchy's by creating 2-4 groups in the team and provide one group leader for each group. Tell them to treat each one of their employees as equal [ this is a recursive principle :) ]
Do a weekly-review of one hour with the whole team. Share managerial decisions (those you can) in the weekly meetings. Don`t let rumors rise , and Send a summary of the weeky-review after it , so people who could not join won`t waist a full day asking what did they miss.
When dividing tasks/pay you can never be equal with everyone. thats a fact.
So Don`t hesitate to tell your employees that they will be compensaited in the next tasks/pay division. It will usually change their mood from sad "I wanna quit" to joy "The boss owes me a great thing next time" . Remember that they will wait for you to keep your promise.
keep your promises to your employees.

Hello World

2008-10-26T13:59:00.000-07:00

In this blog , I will talk about software development.
The good
The bad
and the Ugly things in software development.

It is not intended for specific programming language , but on principals related to managing a software project as a whole.
Trying to summarize here the work experience I had in quite a few projects , in small companies (start ups) and very big companies.

so , lets start.