path: root/lib/MCE.pod

=head1 NAME

MCE - Many-Core Engine for Perl providing parallel processing capabilities

=head1 VERSION

This document describes MCE version 1.888

Many-Core Engine (MCE) for Perl helps enable a new level of performance by
maximizing all available cores.

=begin html

<p><img src="https://raw.githubusercontent.com/marioroy/mce-assets/master/images_README/MCE.png" width="630" height="444" alt="MCE" /></p>

=end html

=head1 DESCRIPTION

MCE spawns a pool of workers and therefore does not fork a new process per
each element of data. Instead, MCE follows a bank queuing model. Imagine the
line being the data and bank-tellers the parallel workers. MCE enhances that
model by adding the ability to chunk the next n elements from the input
stream to the next available worker.

=begin html

<p><img src="https://raw.githubusercontent.com/marioroy/mce-assets/master/images_README/Bank_Queuing_Model.png" width="664" height="446" alt="Bank Queuing Model" /></p>

=end html

=head1 SYNOPSIS

This is a simplistic use case of MCE running with 5 workers.

 # Construction using the Core API

 use MCE;

 my $mce = MCE->new(
    max_workers => 5,
    user_func => sub {
       my ($mce) = @_;
       $mce->say("Hello from " . $mce->wid);
    }
 );

 $mce->run;

 # Construction using a MCE model

 use MCE::Flow max_workers => 5;

 mce_flow sub {
    my ($mce) = @_;
    MCE->say("Hello from " . MCE->wid);
 };

The following is a demonstration for parsing a huge log file in parallel.

 use MCE::Loop;

 MCE::Loop->init( max_workers => 8, use_slurpio => 1 );

 my $pattern  = 'something';
 my $hugefile = 'very_huge.file';

 my @result = mce_loop_f {
    my ($mce, $slurp_ref, $chunk_id) = @_;

    # Quickly determine if a match is found.
    # Process the slurped chunk only if true.

    if ($$slurp_ref =~ /$pattern/m) {
       my @matches;

       # The following is fast on Unix, but performance degrades
       # drastically on Windows beyond 4 workers.

       open my $MEM_FH, '<', $slurp_ref;
       binmode $MEM_FH, ':raw';
       while (<$MEM_FH>) { push @matches, $_ if (/$pattern/); }
       close   $MEM_FH;

       # Therefore, use the following construction on Windows.

       while ( $$slurp_ref =~ /([^\n]+\n)/mg ) {
          my $line = $1; # save $1 to not lose the value
          push @matches, $line if ($line =~ /$pattern/);
       }

       # Gather matched lines.

       MCE->gather(@matches);
    }

 } $hugefile;

 print join('', @result);

The next demonstration loops through a sequence of numbers with MCE::Flow.

 use MCE::Flow;

 my $N = shift || 4_000_000;

 sub compute_pi {
    my ( $beg_seq, $end_seq ) = @_;
    my ( $pi, $t ) = ( 0.0 );

    foreach my $i ( $beg_seq .. $end_seq ) {
       $t = ( $i + 0.5 ) / $N;
       $pi += 4.0 / ( 1.0 + $t * $t );
    }

    MCE->gather( $pi );
 }

 # Compute bounds only, workers receive [ begin, end ] values

 MCE::Flow->init(
    chunk_size  => 200_000,
    max_workers => 8,
    bounds_only => 1
 );

 my @ret = mce_flow_s sub {
    compute_pi( $_->[0], $_->[1] );
 }, 0, $N - 1;

 my $pi = 0.0;  $pi += $_ for @ret;

 printf "pi = %0.13f\n", $pi / $N;  # 3.1415926535898

=head1 CORE MODULES

Four modules make up the core engine for MCE.

=over 3

=item L<MCE::Core>

This is the POD documentation describing the core Many-Core Engine (MCE) API.
Go here for help with the various MCE options. See also, L<MCE::Examples>
for additional demonstrations.

=item L<MCE::Mutex>

Provides a simple semaphore implementation supporting threads and processes.
Two implementations are provided; one via pipes or socket depending on the
platform and the other using Fcntl.

=item L<MCE::Signal>

Provides signal handling, temporary directory creation, and cleanup for MCE.

=item L<MCE::Util>

Provides utility functions for MCE.

=back

=head1 MCE EXTRAS

There are 5 add-on modules for use with MCE.

=over 3

=item L<MCE::Candy>

Provides a collection of sugar methods and output iterators for preserving
output order.

=item L<MCE::Channel>

Introduced in MCE 1.839, provides queue-like and two-way communication
capability. Three implementations C<Simple>, C<Mutex>, and C<Threads> are
provided. C<Simple> does not involve locking whereas C<Mutex> and C<Threads>
do locking transparently using C<MCE::Mutex> and C<threads> respectively.

=item L<MCE::Child>

Also introduced in MCE 1.839, provides a threads-like parallelization module
that is compatible with Perl 5.8. It is a fork of L<MCE::Hobo>. The difference
is using a common C<MCE::Channel> object when yielding and joining.

=item L<MCE::Queue>

Provides a hybrid queuing implementation for MCE supporting normal queues and
priority queues from a single module. MCE::Queue exchanges data via the core
engine to enable queuing to work for both children (spawned from fork) and
threads.

=item L<MCE::Relay>

Provides workers the ability to receive and pass information orderly with zero
involvement by the manager process. This module is loaded automatically by
MCE when specifying the C<init_relay> MCE option.

=back

=head1 MCE MODELS

The MCE models are sugar syntax on top of the L<MCE::Core> API. Two MCE options
(chunk_size and max_workers) are configured automatically. Moreover, spawning
workers and later shutdown occur transparently behind the scene.

Choosing a MCE Model largely depends on the application. It all boils down
to how much automation you need MCE to handle transparently. Or if you prefer,
constructing the MCE object and running using the core MCE API is fine too.

=over 3

=item L<MCE::Grep>

Provides a parallel grep implementation similar to the native grep function.

=item L<MCE::Map>

Provides a parallel map implementation similar to the native map function.

=item L<MCE::Loop>

Provides a parallel for loop implementation.

=item L<MCE::Flow>

Like C<MCE::Loop>, but with support for multiple pools of workers. The pool
of workers are configured transparently via the MCE C<user_tasks> option.

=item L<MCE::Step>

Like C<MCE::Flow>, but adds a C<MCE::Queue> object between each pool of
workers. This model, introduced in 1.506, allows one to pass data forward
(left to right) from one sub-task into another with little effort.

=item L<MCE::Stream>

This provides an efficient parallel implementation for chaining multiple maps
and greps transparently. Like C<MCE::Flow> and C<MCE::Step>, it too supports
multiple pools of workers. The distinction is that C<MCE::Stream> passes
data from right to left and done for you transparently.

=back

=head1 MISCELLANEOUS

Miscellaneous additions included with the distribution.

=over 3

=item L<MCE::Examples>

Describes various demonstrations for MCE including a Monte Carlo simulation.

=item L<MCE::Subs>

Exports functions mapped directly to MCE methods; e.g. mce_wid. The module
allows 3 options; :manager, :worker, and :getter.

=back

=head1 REQUIREMENTS

Perl 5.8.0 or later.

=head1 SOURCE AND FURTHER READING

The source and examples are hosted at GitHub.

=over 3

=item * L<https://github.com/marioroy/mce-perl>

=item * L<https://github.com/marioroy/mce-examples>

=back

=head1 SEE ALSO

Refer to the L<MCE::Core> documentation where the API is described.

C<MCE::Shared> provides data sharing capabilities for C<MCE>. It includes
C<MCE::Hobo> for running code asynchronously with the IPC handled by the
shared-manager process.

=over 3

=item * L<MCE::Shared>

=item * L<MCE::Hobo>

=back

=head1 AUTHOR

Mario E. Roy, S<E<lt>marioeroy AT gmail DOT comE<gt>>

=head1 COPYRIGHT AND LICENSE

Copyright (C) 2012-2023 by Mario E. Roy

MCE is released under the same license as Perl.

See L<https://dev.perl.org/licenses/> for more information.

=cut