windows audiodg.exe memory leak

Windows7 still have a memory leak in audiodg.exe. Unfortunately I have discovered 4 month and a 4GB memory expansion late. You can solve it restarting Windows Audio service from time to time. 

I am a linux person but my computer was too new for linux so I ended with Windows  7 professional (even for windows I needed to install a lot of samsung's drivers and still are things that does not work properly). I usually does not shut down my laptop for months (suspend to ram works OK). But with this computer I needed to restart it once a week. When it had only 4GB it was getting quickly out of memory and windows, in its big desire to be helpful in its own ways, was closing programs at will with a polite notice ('sorry your computer has low memory I am going to close some programs') but without actually allowing you to decide. I bought another 4GB memory card, and then it takes 10 days before reaching to 31GB of virtual memory plus the 8GB of real memory and start to complain. 

I thought that this was due to my abuse of firefox tabs (usually 150-200 opened tabs) and the cache problem, plus a lot of other programs in use like  thunderbird, a linux virtual machine with 2GB, word, Xmind, excel, R....

I have seen several times one process with a lot of memory but always left aside (audiodg.exe). Finally today with all 31GB  virtual memory full and some windows warnings, closed firefox and chrome and I did not see any relevant change in the memory consumed, still 29GB after 1 hour. Looking at the Processes I saw that audiodg.exe had 25GB virtual memory allocated and 2GB real memory in use!!!!

A quick google search  shows that since 2009 is well known that audiodg.exe has a memory leak and there is a hot-fix for it (unfortunately people say that it does not work with windows 7). Some folks solved the problem restarting the audio server. I went to adminitrative_tools/componet_services and searched for audiodg. It was not there but there was a windows audio service. I restarted it and then the real memory used went from 8 to 3GB and the virtual memory from 29 to 4GB!!!!!!!!!!!

My computers just after restarting Windows Audio

VCFtools 0.1.9 perl Vcf.pm does not work with perl 5.14 or newer. use the VCFtools svn or this patch.

Error: bare qw() in a for loop: for $x qw() {} # deprecated. It needs to be surrounded by parents. . See my previous blog entry about this feature.
 patch:

$  diff Vcf.pm Vcf.pm-ori 
1622c1622
<         for my $key (qw(fmtA fmtG infoA infoG)) { if ( !exists($$out{$key}) ) { $$out{$key}=[] } }
---
>         for my $key qw(fmtA fmtG infoA infoG) { if ( !exists($$out{$key}) ) { $$out{$key}=[] } }

Now all tests work OK. [upate] the svn version 779 of Vcf.pm is fixed

for my $key (qw(fmtA fmtG infoA infoG)) { if ( !exists($$out{$key}) ) { $$out{$key}=[] } }

perl 5.14 and up will break some of well known programs! Think twice (test thrice) before use it!

Seems that at some point someone thought that a qw() in a for loop should be surrounded by parents (qw()).
Lines like

$  perl -e 'foreach $x qw(1 2 3){print $x}'
Use of qw(...) as parentheses is deprecated at -e line 1.

are not permitted any longer. People has revealed against this and patches were proposed, but perl 5.16 still does not permit it. Some people say that this was a 'feature' of the parser not an intended behaviour (see comments in Reini's post).
In my case I switched to perl 5.14 and VCFtools was not working any more. At least I discovered it testing the test suit. 20 test failed all with the same error:

[..]
not ok 30 - Testing vcf-consensus .. perl -I../perl/ -MVcf ../perl/vcf-consensus  consensus.vcf.gz < consensus.fa
#   Failed test 'Testing vcf-consensus .. perl -I../perl/ -MVcf ../perl/vcf-consensus  consensus.vcf.gz < consensus.fa'
#   at test.t line 418.
#     Structures begin differing at:
#          $got->[0] = 'Use of qw(...) as parentheses is deprecated at ../perl//Vcf.pm line 1622.
#     '
#     $expected->[0] = '>1:1-500
[...]

I am lucky to be using perlbrew so I can change quickly from one version of perl to another (not so lucky that I need to reinstall a lot of CPAN modules.)

Test::Most not automatically installed with cpanm. Some inner dependencies failed

After installing the the CVFtools, I went to test the perl scripts with

perl vcftools/perl/test.t

But failed becuase I needed Test::More. I tried to install it but failed:

$  cpanm Test::Most.pm
--> Working on Test::Most
Fetching http://search.cpan.org/CPAN/authors/id/O/OV/OVID/Test-Most-0.31.tar.gz ... OK
Configuring Test-Most-0.31 ... OK
==> Found dependencies: Exception::Class
--> Working on Exception::Class
Fetching http://search.cpan.org/CPAN/authors/id/D/DR/DROLSKY/Exception-Class-1.33.tar.gz ... OK
Configuring Exception-Class-1.33 ... OK
Building and testing Exception-Class-1.33 ... FAIL
! Installing Exception::Class failed. See /home/pmg/.cpanm/build.log for details.
! Bailing out the installation for Test-Most-0.31. Retry with --prompt or --force.

Looking at .cpanm/build.log I saw that Class/Data/Inheritable.pm was missing. The dependency was not automatically picked and installed so I installed by hand. Then tried again Test::Most, and again Exception-Class was failing. This time needed Devel::StackTrace. I installed this one also by hand and now all went OK.

cpanm Devel::StackTrace
cpanm Test::More

These missing dependencies are not uncommon in CPAN but are a bit annoying, mainly if your cpanm install action becomes large and the failed module is at the beginning and your last line says, 11 packages installed (all dependencies of your wanted package, but the wanted one is not installed). In situations like this, is easy to skip that your modules was not installed.

LD calculation

LD is a very very blurry concept and the ways of measure it is very diverse. In the following paper you would be able to gain a deeper view of the problem:

Slatkin M. Linkage disequilibrium--understanding the evolutionary past and
mapping the medical future. Nat Rev Genet. 2008 Jun;9(6):477-85. Review. PubMed
PMID: 18427557.

Just in case you don't have access here you have the starting paragraph rant:

Linkage disequilibrium (LD) is one of those unfortunate
terms that does not reveal its meaning. As every
instructor of population genetics knows, the term is a
barrier not an aid to understanding. LD means simply
a nonrandom association of alleles at two or more loci,
and detecting LD does not ensure either linkage or a
lack of equilibrium. The term was first used in 1960 by
Lewontin and Kojima1 and it persists because LD was
initially the concern of population geneticists who were
not picky about terminology as long as the mathematical
definition was clear. At first, there were few data with
which to study LD, and its importance to evolutionary
biology and human genetics was unrecognized outside
of population genetics. However, interest in LD grew
rapidly in the 1980s once the usefulness of LD for gene
mapping became evident and large-scale surveys of
closely linked loci became feasible. By then, the term
was too well established to be replaced.

and some formula:

From wikipedia:

Calculate years between two dates with Perl

Problem:
Giving two dates calculate the number of years between them.

Preamble:
This seems a trivial problem BUT it is not: you could take the year of each date subtract them and if > 0, then if the month of the second date is smaller than the first, add -1, and if it is equal, check the day. What could possibly go wrong?

First and more important, there are zillions of date formats and some very difficult to parse with RE. Then comes the problem that you would like to see the difference in years forward... and backwards!! and probably you would like to show the years with decimal point, so you need to count days .... and so on, and at some point your script would need to report days and months, that is life!.

Solution:
This is a task for the industry grade CPAN's DateTime family modules in Perl.

DateTime::Format::xxx is a family of modules specialised in parsing and formatting any date format you can imagine. Then you can calculate things with Date::Calc or DateTime::Duration.

#!/usr/bin/env perl


=head1 [progam_name]

 description: Calculate years between dates (two dates or one date and current time)

=head2 First version

  a file with three columns (id, start date, end date)

  - parse the date

      | strptime($strptime_pattern, $string)
      | Given a pattern and a string this function will return a new DateTime object.
      | %F
      | Equivalent to %Y-%m-%d. (This is the ISO style date)


  - check that first date < last date

  - output years as a new column

=cut


use feature ':5.10';
use strict;
use warnings;
use Getopt::Long;
use DateTime::Format::Strptime;
use File::Slurp;


my $prog = $0;
my $usage = <<EOQ;
Usage for $0:

  $prog [-test -help -verbose] file_with_dates_in_column_2_and_3

EOQ

my $has_header =1;

my $file_status;

my $file = shift;


my $fmt = DateTime::Format::Strptime->new(
    pattern => '%F',
    locale  => 'en_US',
);

# take care of windows end of line in a linux machine (need both chomp and s/\r$//)
my $dates = [map{chomp;s/\r$//;[split /\t/]} read_file($file)];

my $header = shift @$dates if $has_header;

my $line;
foreach my $date_aoa (@$dates) {

    # get the DateTime objects
    my $start = $fmt->parse_datetime($date_aoa->[1]);
    my $end   = $fmt->parse_datetime($date_aoa->[2]);

    unless ($start || $end){
        die "Error parising id '$date_aoa->[0]' at  line $line\n"
    }
    # get a DateTime::Duration object (that is automatic when doing math with DateTime objects)
    my $dur = $end - $start;

    $date_aoa->[3] = $dur->years;
    $line ++;
}

print_result($dates);


sub print_result {

    my ($dates) = @_;

    say join("\n", map{join("\t",@$_ )}@$dates);

}

Some Links:

The question http://stackoverflow.com/questions/6549522/how-to-make-datetimeduration-output-only-in-days and its answer

http://stackoverflow.com/a/6550372/427129 are also interesting

http://stackoverflow.com/questions/821423/how-can-i-calculate-the-number-of-days-between-two-dates-in-perl

http://stackoverflow.com/questions/3055422/calculating-a-delta-of-years-from-a-date

http://stackoverflow.com/questions/8308655/how-to-find-the-difference-between-two-dates-in-perl

http://stackoverflow.com/questions/3910858/using-perl-how-do-i-compare-dates-in-the-form-of-yyyy-mm-dd


http://datetime.perl.org/wiki/datetime/page/FAQ%3A_Date_Time_Formats#How_Do_I_Convert_between_Date::Manip_and_DateTime_Objects_-6


The Many Dates and Times of Perl

figshare: sharing your scientific results

Figshare is intended to be the place for all researcher to share its supplementary data or prepublication data so it can be reviewed. Also would be a way of sharing data between groups when working in large interdisciplinary and international multigroup projects.

http://figshare.com/

The only caveats that I see are:

- Only 1  GB of free space. This is nothing when talking about images and data for a lab. It is OK to pay for your own backup , but if this initiative is successful and people start to use it for backing their papers, you would like to have all the data there for long time. Who is going to pay for keep the research data when you change lab or your are forced to leave remunerated science?

- I hope they have easy ways of export your data and metadata generated so all the effort and time used in creating this sceintific corpus in the web can be kept safe and replicated if the site disappear

This site has the support from Nature’s sister company, Digital Science.


blog entries about figshare:

http://blog.f1000.com/2012/01/17/figshare-a-carrot-for-sharing/
http://blogs.bmj.com/bmj-journals-development-blog/2012/02/17/figshare-striving-for-greater-efficiency-in-scientific-research/

New edition of Chromatic's "Modern Perl" book

new version of Chromatic's book "Modern Perl" covering perl 5.14
http://onyxneon.com/books/modern_perl/index.html

Free PDF version.

Perl programming 4th edition published

http://shop.oreilly.com/product/9780596004927.do

$20 the PDF in O'Reilly
25 GBP the paper back in Amazon (but not available until march)

Adopted as the undisputed Perl bible soon after the first edition appeared in 1991, Programming Perl is still the go-to guide for this highly practical language. Perl began life as a super-fueled text processing utility, but quickly evolved into a general purpose programming language that’s helped hundreds of thousands of programmers, system administrators, and enthusiasts, like you, get your job done.

In this much-anticipated update to "the Camel," three renowned Perl authors cover the language up to its current version, Perl 5.14, with a preview of features in the upcoming 5.16. In a world where Unicode is increasingly essential for text processing, Perl offers the best and least painful support of any major language, smoothly integrating Unicode everywhere—including in Perl’s most popular feature: regular expressions.

Important features covered by this update include:

New keywords and syntax
I/O layers and encodings
New backslash escapes
Unicode 6.0
Unicode grapheme clusters and properties
Named captures in regexes
Recursive and grammatical patterns
Expanded coverage of CPAN
Current best practices

Prof. dr. N.G. de Bruijn has passed away

In his obituary at Korteweg-de Vries Institute for Mathematics they mention that his de Bruijn sequences had practical applications in magic tricks but does not mention de novo assembly :-(

http://www.science.uva.nl/math/#item1329781416

Prof. dr. N.G. de Bruijn, 1918-2012

N.G. (Dick) de Bruijn passed away on 17 February 2012. He was an emeritus professor of the Technical University Eindhoven, where he was a professor of mathematics from 1960 until 1984. He came to Eindhoven from the University of Amsterdam, where he was a professor of mathematics during 1952-1960. He published in many different areas, in particular in analysis, number theory, combinatorics and logic. Famous is his book Asymptotic methods in analysis (1958). In later years he developed Automath, a computer program for automatic theorem proving. The De Bruijn sequences, named after him, are widely discussed in literature and have even applications in magic (see the recent book Magical mathematics by Persi Diaconis and Ron Graham).

For more information, see http://tinyurl.com/89yn946 (pdf file from Kleine TUE Encyclopedie) and http://tinyurl.com/6pkuuol (MacTutor History of Mathematics biography). See also on
http://www.win.tue.nl/debruijn90/video/debruijn.html the video of the lecture by de Bruijn during the symposium on the occasion of his ninetieth birthday.