[perl] playing with Test::Exception

More about testing: handling exceptions

Today I was updating some tests for my modules and I needed to test the exception handling. I was going to use 'eval{..}or do{..}' but decided to go for Try::Tiny, but once I entered in the path of using CPAN modules I decided to give a try (not pun intended) to Test::Exception.

I have read about some people complaining about Test::Exception using prototypes and talking about this would lead to some unexpected behaviors in very uncommon situations. The author of Test::Exception also implemented the methods without prototypes (they are less attractive) but I don't know if not using the prototyped methods are free of the edge cases errors. Any way chromatic explained in this blog entry how the prototyped methods in Test::Exception works.

Here I am playing with the test to see how can I handle the output of the tests:

* First try

Created a basic test "test_Test_Exception.t" to see if my method die

use strict;
#use warnings;
#use Data::Dumper;
use Test::More;
use Test::Exception;

plan tests => 3;

my $ok;


$ok = dies_ok (sub {die1()}
, 'testing die1'
);

$ok = dies_ok (sub {die2()}
, 'testing die2'
);
$ok = dies_ok (sub {live()}
, 'testing live expecting die1'
);

sub die1 {
die "cause1";
}

sub die2 {
die "cause2"
}

sub live {
return 1;
}

== result ==

> perl test_Test_Exception.t
1..3
ok 1 - testing die1
ok 2 - testing die2
not ok 3 - testing live expecting die1
# Failed test 'testing live expecting die1'
# at /nfs/users/nfs_p/pg4/programming/perl/test_Test_Exception.t line 29.

This is nice, if you expect something to die but doesn't, it tells you that.

* Second try

What would happen when you have some exceptions in a method and you want to test them and you want to be sure that they are the correct ones.

You have throws_ok and you test for a regex of the output

#
## trying throws_ok
#
$ok = throws_ok (sub {die1()}
, qr/cause1/
, 'testing die1'
);

$ok = throws_ok (sub {die2()}
, qr/cause1/
, 'testing die2 to see if die of cause 1'
);
$ok = throws_ok (sub {live()}
, qr/cause1/
, 'testing live expecting die1'
);

== result ==

> perl test_Test_Exception.t
1..3
ok 1 - testing die1
not ok 2 - testing die2 to see if die of cause 1
# Failed test 'testing die2 to see if die of cause 1'
# at /nfs/users/nfs_p/pg4/programming/perl/playing_with_test_exception.pl line 41.
# expecting: Regexp ((?-xism:cause1))
# found: cause2 at /nfs/users/nfs_p/pg4/programming/perl/playing_with_test_exception.pl line 86.
not ok 3 - testing live expecting die1
# Failed test 'testing live expecting die1'
# at /nfs/users/nfs_p/pg4/programming/perl/playing_with_test_exception.pl line 45.
# expecting: Regexp ((?-xism:cause1))
# found: normal exit

This is even better: test2 now fails if the exception text is not the expected one.

* third try

A more complicated case:
I have a subrutine, and I test for one of the arguments but I tested for lowercase 'a' instead uppercase 'A'. But this subrutine has another exception and dies when second argument is 'B'

#
## more about throws_ok
#

# testing an exception but obtaining other
# error in the method: testing a instead A
# - it should die when passing 'A' but does not die because that
$ok1 = dies_ok (sub {die3('A', 'B')}
, 'testing faulty method die3 with dies_ok: it should die BUT does it because another reason'
);

$ok2 = throws_ok (sub {die3('A', 'B')}
, qr/because A/
, 'testing faulty method die3 with throws_ok it should die BUT does it because another reason'
);

# testing the other exception
$ok3 = throws_ok (sub {die3( undef,'B')}
, qr/because B/
, 'testing faulty method die3 with throws_ok for another exception that works ok'
);

# testing exception but an error in the test (wrong argument so it does not die)
$ok4 = dies_ok (sub {die3( undef)}
, 'testing faulty test for die3: should die but it doesn\'t'
);


$ok5 = throws_ok (sub {die3( undef)}
, qr/because A/
, 'testing faulty test for die3: should die but it doesn\'t'
);


sub die3 {

my $in = shift;
my $out = shift;

# [error] testing if $in eq A but testing eq 'a' instead
die "die because A" if $in eq 'a';

# ups sometimes something else goes wrong
if ( $out eq "B" ) {
die "die because BLAH";
}

return "live";
}

== results ==

# with dies_ok wrong result, it is happy with the death (but by the wrong motives :-():

ok 1 - testing faulty method die3 with dies_ok: it should die BUT does it because another reason

# but the expected result (failed) with throws_ok

not ok 2 - testing faulty method die3 with throws_ok it should die BUT does it because another reason
# Failed test 'testing faulty method die3 with throws_ok it should die BUT does it because another reason'
# at /nfs/users/nfs_p/pg4/programming/perl/test_Test_Exception.t line 61.
# expecting: Regexp ((?-xism:because A))
# found: die because BLAH at /nfs/users/nfs_p/pg4/programming/perl/test_Test_Exception.t line 100.

#

ok 3 - testing faulty method die3 with throws_ok for another exception that works ok

# it fails because not die

not ok 4 - testing faulty test with dies_ok for die3: should die but it doesn't
# Failed test 'testing faulty test for die3: should die but it doesn't'
# at /nfs/users/nfs_p/pg4/programming/perl/test_Test_Exception.t line 73.

# It fails because does not die at all and tell you that: "found: normal exit"

not ok 5 - testing faulty test with throws_ok for die3: should die but it doesn't
# Failed test 'testing faulty test for die3: should die but it doesn't'
# at /nfs/users/nfs_p/pg4/programming/perl/test_Test_Exception.t line 77.
# expecting: Regexp ((?-xism:because A))
# found: normal exit

The throws_ok is more informative and prevents to think that your test failed as expected when indeed was not "as expected" but because another reason.

This is fine, but if I want double check the output messages for the exceptions when debugging, I need to add the 'and diag($@)' or if I want to see if it failed because other exception or because it didn't die I need to do the following:

my $ok = throws_ok( sub {...}, qr/check/, 'test desc') 
          or do {$@? diag("FAILED:\n Wrong exception.") 
                   : diag("FAILED:\n it was suppose to die")
                };
 diag("Exception_ok:\n". lines($@,3)) if $ok; 

 sub lines{
   my $txt = shift;
   my $num = shift;
   # add the [ok] to help the output to be taken as ok
   my $msg = '[ok] ' . join ("\n[ok] ",(split "\n",$txt)[0..($num-1)]);
   return $msg;
 }

And then you would have output like this for failed tests:

not ok 5 - \#Should die: parse_tab_file_to_AoH without headers array ref:
# Failed test '\#Should die: parse_tab_file_to_AoH without headers array ref: '
# at PMGBase.t line 69.
# expecting: Regexp ((?-xism:but not \$headers))
# found: normal exit
# FAILED:
# it was suppose to die
not ok 6 - \#Should die: parse_tab_file_to_AoH without headers array ref:
# Failed test '\#Should die: parse_tab_file_to_AoH without headers array ref: '
# at PMGBase.t line 76.
# expecting: Regexp ((?-xism:but no \$headers))
# found:
# -------------------- EXCEPTION --------------------
# MSG: # [FATAL] sorry but 'parse_tab_file()' called with type 'AoH' but not $headers_aref provided
#
# STACK PMGBase::_throw ..//PMGBase.pm:498
# STACK PMGBase::parse_tab_file ..//PMGBase.pm:612
# STACK PMGBase::parse_tab_file_to_AoH ..//PMGBase.pm:561
# STACK Test::Exception::throws_ok PMGBase.t:73
# STACK toplevel PMGBase.t:76
# ---------------------------------------------------
# FAILED:
# Wrong exception.

And some debug info for the ones ok:

ok 7 - \#Should die: parse_tab_file_to_AoH without headers array ref:
# Exception_ok:
# [ok]
# [ok] -------------------- EXCEPTION --------------------
# [ok] MSG: # [FATAL] sorry but 'parse_tab_file()' called with type 'AoH' but not $headers_aref provided

ok 9 - \#Should die: parse_tab_file_to_AoH with wrong number of header columns:
# Exception_ok:
# [ok]
# [ok] -------------------- EXCEPTION --------------------
# [ok] MSG: line 2 has different number of columns that the header

Next step is to explore the Test::Trap that seems that have better layers for handling and reporting the exceptions. But at the moment if you want a simple ok/fail for exceptions Test::Exception is very handy.