Mac and Code coverage

Well maybe not many of you do code coverage on Mac but I was asked to set up code coverage so that we could check on the coverage of our automated tests. I have always been a big fan of code coverage since I feel there is a report that shows which are the lines of code that have been touched. Though I wouldn’t use this report alone to certify that testing is complete on code, it can be used wisely to help the testing process.

As I mentioned I had to get the code coverage setup and much to my surprise it wasn’t as easy as EclEmma that I had used on Eclipse. I spent atleast 3 days getting frustrated that I was heading nowhere close to getting the code coverage working using lcov for Mac. But thanks for all the info on stackoverfow and CubicleMuses I have code coverage working for both simulator and device(iPhone/iPad)!  Here are the steps and configuration that worked for me:

Configuration : Xcode 4

XCode project settings

Build Settings

• Other Linker Flags: add “-lgcov”
• GCC_GENERATE_TEST_COVERAGE_FILES: Set to YES
• GCC_INSTRUMENT_PROGRAM_FLOW_ARCS: Set to YES
• C/C++ Compiler Version: GCC 4.2 (if you are on XCode 4) iOS deployment target: 4.2
• Precompile prefix header: NO

Info.plist

• Set UIApplicationExitsOnSuspend flag in your Info.plist to YES

Above steps are same for Simulator and Device however, we have some extra work to make it work on Device.

Main.m: Copy paste the below code to main.m

const char *prefix = “GCOV_PREFIX”;

const char *prefixValue = [[NSHomeDirectory() stringByAppendingPathComponent:@"Documents"] cStringUsingEncoding:NSASCIIStringEncoding]; // This gets the filepath to the app’s Documents directory

const char *prefixStrip = “GCOV_PREFIX_STRIP”;
const char *prefixStripValue = “1″;
setenv(prefix, prefixValue, 1); // This sets an environment variable which tells gcov where to put the .gcda files.
setenv(prefixStrip, prefixStripValue, 1); // This tells gcov to strip the default prefix, and use the filepath that we just declared.

Note: Make sure the above code is before:

NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
int retVal = UIApplicationMain(argc, argv, nil, nil);
[pool release];    
return retVal;

Why we set the above coverage variables?

http://gcc.gnu.org/onlinedocs/gcc/Cross_002dprofiling.html

How to get the .gcda files?

Use the Organizer in Xcode to download app’s package from the device to get the .gcda files out of the Documents directory.

Note: I could not get the code coverage using Xcode 3.2.5 with the same settings. But Xcode 4 was a cake-walk

Vanilla ice-cream mystery

There are a lot of times when during our testing we come across a bug that’s weird in terms of occurrence.  Issues that occur in some special condition, which does not seem logical at all. But hey the issue is valid. How do we justify it? We would have to probe and isolate the issue in such a way that we find a pattern or a value that causes this issue. On similar lines, I came across this unique (weird) issue in the automobile industry. The story is as follows (for ppl who want the link: http://www.bairnet.org/footnote/981001pontiac.htm)

A complaint received by the Pontiac Division of General Motors:

“This is the second time I have written you, and I don’t blame you for not answering me, because I kind of sounded crazy, but it is a fact that we have a tradition in our family of ice cream for dessert after dinner each night. But the kind of ice cream varies so, every night, after we’ve eaten, the whole family votes on which kind of ice cream we should have and I drive down to the store to get it.

“It’s also a fact that I recently purchased a new Pontiac and since then my trips to the store have created a problem. You see, every time I buy vanilla ice cream, when I start back from the store my car won’t start. If I get any other kind of ice cream, the car starts just fine.

“I want you to know I’m serious about this question, no matter how silly it sounds: ‘What is there about a Pontiac that makes it not start when I get vanilla ice cream, and easy to start whenever I get any other kind?’”

The Pontiac President was understandably skeptical about the letter, but sent an engineer to check it out anyway. The latter was surprised to be greeted by a successful, obviously well-educated man in a fine neighborhood. He had arranged to meet the man just after dinner time so the two hopped into the car and drove to the ice cream store. It was vanilla ice cream that night and, sure enough, after they came back to the car, it wouldn’t start.

The engineer returned for three more nights. The first night, the man got chocolate. The car started. The second night, he got strawberry. The car started. The third night he ordered vanilla.

The car failed to start.

Now the engineer, being a logical man, refused to believe that this man’s car was allergic to vanilla ice cream. He arranged, therefore, to continue his visits for as long as it took to solve the problem. And toward this end he began to take notes: he jotted down all sorts of data, time of day, type of gas used, time to drive back and forth, etc.

In a short time, he had a clue: the man took less time to buy vanilla than any other flavor. Why? The answer was in the layout of the store. Vanilla, being the most popular flavor, was in a separate case at the front of the store for quick pickup. All the other flavors were kept in the back of the store at a different counter where it took considerably longer to find the flavor and get checked out.

Now the question for the engineer was why the car wouldn’t start when it took less time. Once time became the problem — not the vanilla ice cream — the engineer quickly came up with the answer: vapor lock. It was happening every night, but the extra time taken to get the other flavors allowed the engine to cool down sufficiently to start. When the man got vanilla, the engine was still too hot for the vapor lock to dissipate.

Moral of the story: Even insane-looking problems are sometimes real.

What I liked most in this case is the way the engineer looked at all factors from details of the car to the details of where the flavor of ice-cream was located in the store. There would be people who could overlook the latter detail, but that was THE key to solving the Vanilla ice-cream mystery.

So what are the ways:

1. Reproduce the issue as many times and check for different conditions like environment, background applications, time of the issue (you never know what an AM and PM could do). Basically think out of the box!
2. Once you have a pattern, try to put a logical link to the issue and get the technical reason for the same. (sounds simple but not easy)
3. Google! It is always good to check the online resources where others might have encountered similar kind of bugs and read through their findings.
4. Try to record the sequence of events that cause the issue and play it back. A visual recap of events always helps in triggering an idea. Reproducing is different from this. Here you are only watching the sequence and that could get something running.
5. Talk to people around you even if they don’t have an idea about your work. Believe me sometimes they are of best help.

There was a time when I was known for finding bugs on my Mac to the extent that developers wanted the Mac to be removed from the test team .  The frustrating part was that only the Mac that I had could reproduce the issue and not any other machine that we had in the team. A lot of times we couldn’t find the exact reason but in others it depended on the system configuration and order of executing the steps. Today when I think of it, I regret not having probed further to find out the actual cause.

It definitely is not easy to find the reason behind weird issues such as above, but in addition to the frustration involved it adds knowledge to one’s testing and probing skills. So the next time you find a weird issue step back and see how best you can find the cause and solution for it.

Explore testing!

I am not a superwoman!

Ok, so for all those ppl who think I am a superwoman to have published 3 blog posts in a day, no I am not a superwoman. Its just that I have been writing in my company’s internal blog and so it was just matter of copy-pasting .

I am hoping I write often and keep up the frequency of my posts. So till then Happy Reading and Testing!

Cascading bugs

QE:  Do you want this bug fixed?

Manager :Of course, you should. There should be no question about it.

QE: But what if this bug is going to have a cascading effect?

Manager: hmm…. How bad is it?

Developer: This might bring out other bad bugs

Management team?

Testers are always happy when they find a bug. But very few ask themselves is it worth fixing this bug? Why should one ask such question? When you find a bug it has to be fixed. Isn’t that obvious?

What if fixing a bug is going to throw open a whole bunch of new issues that was happily hiding under the one bug? Is it worth it? Well, it depends.

Few questions that you can ask yourself:

• Are you in the end cycle of a release?

• Does the client want all the bugs to be fixed?

• Can we live with this bug?

• Is there a workaround?

In medicine, cascade effect refers to a chain of events initiated by an unnecessary test, an unexpected result, or patient or physician anxiety, which results in ill-advised tests or treatments that may cause harm to patients as the results are pursued. The cascade effect can lead to a “chain of events (which) tends to proceed with increasing momentum, so that the further it progresses the more difficult it is to stop.

But in software we cannot term a bug an unnecessary test, but we can prioritize the bug as P4. So should we fix a P4 bug that might bring P1/P2/P3 bugs?

Are you in the end cycle of a release? Then defer the bug to next release and make sure it is entered in the release notes. In the next release look at ways to have it fixed.

Does the client want all the bugs to be fixed? Talk to the client explaining the effects of fixing the bug. If he/she is convinced; great! If not, too bad you will have to deal with fixing the cascading bugs and push the release.

Can we live with this bug? If it takes more than 7-8 steps to hit this issue, then the chances of hitting this issue is very low. Moreover if it is a P4 you are even lucky.

Is there is a workaround? – Document the bug in release notes with the workaround. A lot of times users are fine to live with a bug if there is a workaround.

So next time you hit a bug, think about the effects of fixing it. If you have other thoughts/ideas to deal with cascading effects of bugs, do share them.

Units Tests : Food for thought

 

If the above image surprised you, then you are among the very few lucky testers who have worked with developers who religiously write unit tests. To me it always is surprising and irritating when I find a very obvious bug in a code. The first thing that comes to my head “Why didn’t that guy test his code before submitting it”.  Just the same way developers feel when a bad bug is caught late in the cycle when it has always been sitting in the code all the while. Do you know how frustrating that is? I do. Luckily for me I have had the chance to wear the hat of a developer for a few years before I stuck with the tester hat.

Most people have a wrong notion that developers will only write code. What are testers for? #@#$@%@. Even while I type this I can feel the frustration I feel every time I hear the above line. Over the years I have learnt not to react to it, at least try not to. Yes testers are people who test code, I don’t need Google to tell that. But I really wish developers and management understand that they are not there to find out SILLY mistakes that are made by the devs. I have always argued with the developer leads and managers asking why developers are not writing unit tests for their code? The most common reply that I get is “Where is the time?”. WOW and the testers have all the time in the world to test their buggy code. Think about this, how much time would be saved and utilized in more productive ways if unit tests catch bugs?

While I worked at Adobe I was lucky enough to work with a number of developers with varied experience. Amongst them there was this one guy who believed unit tests were important. He practically wrote unit tests for all possible cases, yes unbelievable as it might sound. Every time I worked with him as his tester, it was a nightmare for me. I had to struggle to find a bug! I used to spend hours in trying to log at least one high priority bug and no, he would have never left a gap. In fact there was this one time where I called for a QA meeting and demoed the feature to the team. At the end of the meeting I asked my peers “Give me test cases to test this feature, I haven’t logged a single bug for this feature”. This actually helped. A couple of guys gave me some good test cases and I actually found bugs. Ok, for those of you who think I was an idiot not to think of those test cases. First and foremost, yes I never thought of them and next of all they were edge cases.  Sometime in my other posts I will write about buddy testing and bug bash. Coming back to the point. Because my dev was writing unit tests I was trying to test almost everything which was covering the most rare and edge cases. In fact while I worked with him I used to study and look for different ways to test a particular feature/module. And all this reading helped me to test better. So coming to my point, unit tests are to catch the most obvious bugs and testers are there to fine tune the application/product. I know that unit tests cannot cover everything but hey it does cover a lot. Food for thought.

In the past few years there has been a splurge of unit test frameworks that have come out. I really wish all the great developers use them and become greater developers .

Objects in Mirror are closer than they appear

How many of us think the meaning of above is this:

1. The objects are shown closer in the mirror and hence actually they are farther away.

2. The objects are much closer to your vehicle than what they are shown in the mirror.

If you think no one would think of the phrase like the first option, then you are wrong. I have come across a couple of people who strongly felt the first option was what it actually meant. After trying to explain, I just directed them to Wikipedia  :-)

In our day to day lives we all come across a lot of such lines where people interpret the sentences in a totally opposite manner. More often you cannot even convince them that they are interpreting it the wrong way.

How many times have we faced the situation that we submit a bug and the developer says he cant reproduce it? But you are able to reproduce it repeatedly. The reason? (Here I am not talking about details that are missing in the bug report) The way in which one understand the sentences and order matters when it comes to reproducing a issue. In our profession of requirements gathering,  writing specifications, test plans and most importantly writing bugs it is very vital that we put across our thoughts/views in the right way. It is not really important to use new or bombastic words for this. All this needs is plain and simple English and yet a lot of people don’t do it the right way. This post is not to teach anybody how to write. I don’t believe I can; for that matter I don’t know if somebody can teach how to write. Its something one needs to develop and learn from experience, reading and practicing. But one thing that we can do is to read what we write and make sure it is concise. Believe me it goes a long way in not only saving some else’s time but also yours. Think about it……

Connect to BlazeDS

Are you looking for help to be able to connect to your BlazeDS/LCDS server using Flash Builder?  Look no further .  I can bet you will find everything to help you out in Sujit’s blog!!