Flash Gordon wrote:
> Ian Collins wrote, On 27/08/07 21:51:
>> Flash Gordon wrote:
>>> Ian Collins wrote, On 27/08/07 08:14:
>>>> Ark Khasin wrote:
>>>>> Ian Collins wrote:
>>> <snip>
>>>
>>>>>>> And finally there is a port issue (it's an embedded type talking
>>>>>>> :)). I
>>>>>>> am proposing something that requires only the compiler.
>>>>>>>
>>>>>> Shouldn't matter for unit testing, develop and test on a hosted
>>>>>> system.
>>>>>> If you require bits of the target environment, mock (simulate) them.
>>>>>>
>>>>> The farthest I can go away from the target is a software simulator of
>>>>> the instruction set. Same compiler, same version, perhaps, more
>>>>> "memory". I think I am not alone in this...
>>>>>
>>>> Why?
>>> There are many possible *valid* reasons for this. One is that if you are
>>> not using the same version of the same compiler with the same switches
>>> then the code you are testing is not the same as the code that will be
>>> run. Since compilers *do* have bugs it is possible that the bug will be
>>> triggered in the real environment but not in the test environment unless
>>> you ensure that they are the same.
>>>
>> But one has to differentiate between developer unit testing (the subject
>> of this post) and QA (customer) acceptance testing.
>
> You have missed out internal formal testing which in many environments
> is far more complete than acceptance testing. For example, I've worked
> on projects where a formal test literally takes a week to complete but
> the customer acceptance testing takes only a few hours.
>
If performed, internal formal testing is still a step away from
developer testing.

>> The former can be
>> performed on any environment the developer chooses,
>
> Informal testing can be run in any environment the developer has
> available. Formal testing, which is the only sort of testing that you
> can guarantee will be available and working for those maintaining later,
> is another matter.
>
How so? A unit test suite doesn't just vanish when the code is
released, it is an essential part of the code base.
>>
>> Again, this is different from developer unit testing, I don't think
>> anyone would be daft enough to release a product that hadn't been
>> through acceptance testing on the target platform.
>
> Acceptance testing has very little to do with proving whether the system
> works, it is just to give the customer some confidence.

That depends on your definition of Acceptance tests. In our case, they
are the automated suite of tests that have to pass before the product is
released to customers.

> The real worth
> while formal testing has to be completed *before* doing customer
> acceptance testing and done with the correct compiler. At least, this is
> the case in many environments, including all the projects where I have
> been involved in QA, and on the safety critical project I was involved in.
>
Again, that depends on your process.

> If your customer acceptance testing is sufficient to prove the SW is
> sufficiently correct then your customer has either very little trust in
> your company or a lot of time to waste. If your customer acceptance
> testing is the only testing done with the correct compiler and it is not
> sufficient to prove your SW is sufficiently correct then your SW is not
> tested properly. At least, not according to any standard of testing I
> have come across.

Why? Our acceptance test are very comprehensive, written by
professional testers working with a product manager (the customer).

It sounds like you don't have fully automated acceptance tests. Where
ever possible, all tests should be fully automated.

--
Ian Collins.