Showing posts with label software. Show all posts
Showing posts with label software. Show all posts

Wednesday, 24 March 2010

MAPPING files into memory vs READING them

Brian d Foy 'efective perler' blog talks about mapping files into memory to avoid IO and memory footprint,

Memory-map files instead of slurping them

It uses the module File::Map


use File::Map qw(map_file);

{
my $start = time;
map_file my $map, '/Volumes/Hercules/Red/revealing_it_all_big.mov';
my $loadtime = time - $start;
print "Loaded file in $loadtime seconds\n";
my $count = () = $map =~ /abc/;
print "Found $count occurances\n";
}

[copy paste from the blog]
The $map acts just like a normal Perl string, and you don’t have to worry about any of the mmap details. When the variable goes out of scope, the map is broken and your program doesn’t suffer from a large chunk of unused memory.
In Tim Bray’s Wide Finder contest to find the fatest way to process log files with “wider” rather than “faster” processors, the winning solution was a Perl implementation using mmap (although using the older Sys-Mmap). Perl had nothing special in that regard because most of the top solutions used mmap to avoid the I/O penalty.
The mmap is especially handy when you have to do this with several files at the same time (or even sequentially if Perl needs to find a chunk of contiguous memory). Since you don’t have the data in real memory, you can mmap as many files as you like and work with them simultaneously.
Also, since the data actually live on the disk, different programs running at the same time can share the data, including seeing the changes each program makes (although you have to work out the normal concurrency issues yourself). That is, mmap is a way to share memory.
The File::Map module can do much more too. It allows you to lock filehandles, and you can also synchronize access from threads in the same process.
If you don’t actually need the data in your program, don’t ever load it: mmap it instead.
[/end copy paste]

put perl under git and create branches for diferent module sets

Instead of having a core perl distribution and then put directories in your home for different sets of cpan-modules installations and deal with it changing @INC, other solution could be to put your perl installation under git control.

Manage your Perl modules with git

Read this Brian d Foy blog entry to find out how to do it and some advantages of this approach.

perl get-options good practices

Rading a recent post at  http://perlbuzz.com/ :
The horrible bug your command line Perl program probably has 

 It talks about the best practice to test always system call return values. But also make a good point that many user forget about to test the result of get-option. If you don't know what get_option is, and you are writing perl scripts for comand line usage, then you are missing a very important tool.


For the record, I am putting here my standard get_option scaffold (I have it as a template in my .emacs)

use Getopt::Long;  

my $prog = $0;
my $usage = <<eoq;
Usage for $0:

  >$prog [-test -help -verbose]

EOQ

my $help;
my $test;
my $debug;
my $verbose =1;
my $log;
my $stdout;
my $ok = GetOptions(
                    'test'      => \$test,
                    'debug:i'   => \$debug,
                    'verbose:i' => \$verbose,
                    'help'      => \$help,
                    'log'       => \$log,
                    'stdout'    => \$stdout,
                   );

if ($help || !$ok ) {
    print $usage;
    exit;
}

I am capturing the return value of the getoptions and printing a 'usage' message if error or help

I encourage you to put this in your editor's perl templates.