To RedHat Engineers: Paul Howarth, Denis Fateyev, Near the very end of the 1.8 cycle were several revisions. Each time, an email was sent to you stating that MCE was completed. But each morning, I woke up with a sense that something else was missed. (a) Running MCE and MCE::Shared inside an eval block. (b) Check for EINTR condition during sysread and syswrite. (c) Check the OS while running with ~ 200 workers. (d) Test MCE with the Wx toolkit - wxWidgets. (e) Test MCE and MCE::Shared by running taint mode via perl -T. (f) Validate DESTROY for shared Condvar and Queue objects. (g) Improve MCE::Shared::Handle for non-shared use, was not 100%. Going from 99.7% to 100.0% required so much effort. I am deeply sorry for the many emails sent. Although far away in miles, you were there in keeping me from going insane. For this I thank you. Best, Mario -- ############################################################################### # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # ############################################################################### What follows are many people reporting problems, fixes or solutions. More specifically in alphabetical order by last name. Also, not to forget all the people mentioned in the change log. Baumgart, Philippe For trying the MCE interval option and calling MCE->yield. I felt bad for Philippe spending many hours that it caused me to simulate the interval option over and over again. It turns out that there can be unnecessary delays from a worker taking extra time. MCE 1.815 resolves the issue by allowing the next available worker to run versus workers running orderly. Thank you, Philippe for your patience. Berger, Joel The inspiration for the sixth model (MCE::Step) came from reading a blog by Joel, "Writing Non-Blocking Applications with Mojolicious: Part 3": https://blogs.perl.org/users/joel_berger/2014/01/ writing-non-blocking-applications-with-mojolicious-part-3.html Bouras, George For reporting sockets failing under the Windows environment due to a period of inactivity. An example is when workers execute an external command taking beyond 4 minutes to complete. On returning, the sockets have gone stale with a CPU core going 100% at that point. The MCE 1.601 release is attributed to the many use cases I received from George. Cantrell, David (DrHyde on perlmonks.org) For reporting on MCE hanging with cpan -t MCE under his environment. Also, thank you for providing me a guest account via ssh. I was able to reproduce the issue. Thus, added writeable check and croak if tmp dir is not writeable. One cannot assume that /dev/shm is writeable by everyone. :) Collet, Eric For the inspiration to the tre-agrep binary. I first learned of tre-agrep from Eric. He emailed me running times for a couple scenarios utilizing a single core. Thus increasing the need to have a MCE-driven wrapper to run tre-agrep in parallel. The bin/mce_grep script was created to show folks how one may go about chunking data between Perl and an external C binary efficiently without invoking the shell. The script supports grep, egrep, fgrep, agrep and tre-agrep. Eldai, Hisham While working on the biofasta examples, Hisham introduced me to hg19.fa. A couple records inside hg19 (driven by $/ = "\n>") are 250 MiB in size. Memory optimizations in v1.600 came about by running against hg19.fa hundreds of times. ############################################################################### # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # ############################################################################### Farrell, David David wrote a nice article about MCE (love the picture). What stood out was his example. I never thought about using mce_grep { ... } <$fileHandle> without the _f (mce_grep_f). At the time, updated MCE::Grep and MCE::Map to be fully aware of wantarray. The blog "Make your code run faster with Perl's secret turbo module" is located at: https://www.perl.com/article/61/2014/1/21/ Make-your-code-run-faster-with-Perl-s-secret-turbo-module/ Gasper, Felipe For test scripts found in IO::SigGuard. I was able to get MCE::Mutex to fail with similar testing. The result from testing is that MCE too, must check for EINTR during sysread and syswrite operations. The same is true for MCE::Shared. Golden, David For the Hash::Ordered module. I learned a lot from this *GEM* on CPAN. Particularly, the use of overload. I didn't know that was possible. For the time during collaborative efforts in making Hash::Ordered faster. I learned some new tricks along the way. Thank you for alarming me of the memory leaks with the on-demand hash dereferencing inside MCE::Shared::Ordhash. MCE::Shared::Object was also impacted. I'd never forget that dreadful day. MCE::Shared::{ Array, Hash, and Ordhash } are inspired by Hash::Ordered. In particular, the clone and iterator methods. Also, keys and values on taking arguments. I confess for never getting used to the "as_list" method name, initially. :) Gorwits, Oliver For reporting on a couple issues with MCE along with a solution for each. Pointed me in the right direction for improving the logic for the die handler. Basically, eval { die 'this should not cause MCE to die' }; This has reached 100% (all cases, threads and processes) in MCE 1.600. Halpenny, Shawn For reporting an issue (bug RT#94706) with signal handling in MCE::Signal. Also, thank you for tip on getpgrp. Hedden, Jerry From reading threads::shared. In particular the make_shared function. That was helpful in getting MCE::Shared working. Hluchan, Yary For providing a benchmark script which I used to compare IPC performance between 1.6 and 1.7. For many kind words near the end of the development cycle for MCE 1.7 and MCE::Shared. ############################################################################### # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # ############################################################################### Jacobsen, Dana For reporting an issue with get_ncpu on AIX. In addition, suggesting various solutions. Also, thank you for your time with Sandboxing with Perl + MCE + Inline::C at https://github.com/marioroy/mce-sandbox/. Karasik, Dmitry For bug (RT#102802) and elegant solution for the die handler inside MCE::Core::Worker and MCE::Signal. Kharchenko, Oleksandr Someone once asked Oleksandr why not use MCE. Oleksandr created 2 modules on CPAN, Parallel::DataPipe and Parallel::parallel_map. I used his example heavily in comparing IPC between pipe-driven and socket-driven. Not pleased with MCE's performance with chunk_size => 1, although 2 was as fast if not faster, I tried various things to include IO::Select. Finally, realized increasing the number of data channels was all that was needed (v1.500). Kulow, Stephan For making the OpenSUSE package for MCE. https://build.opensuse.org/package/show/devel:languages:perl:CPAN-M/perl-MCE Mabon, Philip For reporting on a couple issues with MCE in particular bug RT#92627. McKeown, Benjamin For reporting several edge cases with MCE in particular bugs RT#105557, RT#105559, and RT#111780. Mertens, David I am thankful for the tip on making PDL + MCE a possibility. Workers were crashing during exiting. Adding "sub PDL::CLONE_SKIP { 1 }" inside MCE.pm solved this issue entirely. Also, for tips on effectively using PDL in his example at https://gist.github.com/run4flat/4942132. Barrier synchronization in MCE came from reading parallel_sync in David's Perl module; PDL::Parallel::threads::SIMD. Mughal, Zakariyya First, Demian Riccardi introduced me to Devel::IPerl by Zakariyya. The die handlers in MCE broke the user experience. It seems that looking at $^S (for Perl state) is not enough. MCE v1.600 fixes this once and for all. The answer is in perl-5.20.1/cpan/CGI/lib/CGI/Carp.pm (Carp::longmess). ############################################################################### # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # ############################################################################### Ogulin, Joseph For providing the initial perl-MCE.spec file. For the cool remarks at work about MCE being really big :) Riccardi, Demian I received a message from Demian asking for mce_map. His message requested ( my @a = mce_map { ... } 1..100 ). Five models were introduced in v1.500; MCE::Flow, MCE::Grep, MCE::Loop, MCE::Map, and MCE::Stream. Find capability were limited in MCE::Shared 1.001. For example, not able to mix :AND(s) and :OR(s) together. One day, I clicked on recent distributions on metacpan.org and came across HackaMol::Roles::SelectionRole. The trick is to eval the query string together with grep or map. Thank you Demian. Rowe, Jillian For reporting on IO::File handles not working with MCE. Fixed in 1.515. For believing in possibilities beyond text-book thinking. Roy, Sylvia (spouse) For supporting me from v1.0, especially during later 1.6 revisions and 1.7. But not to forget the journey during 1.8. So little was known on the hurdles to cross. It's all over now with MCE 1.827 and MCE::Shared 1.823. Šabata, Petr For initial package submission at Red Hat. https://bugzilla.redhat.com/show_bug.cgi?id=1162531 Sasser, Tom For reporting on bin/mce_grep failing with Perl 5.8.x. Schlichting, Florian For making a Debian package for MCE: libmce-perl Shen, Wei For the BioUtil CPAN module. BioUtil::Seq::FastaReader inspired me to create parallel examples (biofasta folder) in MCE 1.600. We tried to make FastaReader in BioUtil::Seq even faster. ############################################################################### # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # ############################################################################### Sjøgren, Adam For reporting on running /usr/bin/time mce_script.pl and pressing Ctrl-C failing. The default behavior in MCE::Signal is to call setpgrp. MCE 1.403 adds a new option to MCE::Signal to not call setpgrp. This also gives the developer finer control as to where setpgrp should be called, if ever needed, such as in the child process and not the manager process. use MCE::Signal qw(-no_setpgrp); use MCE; Smith, Marcus For reporting a problem with MCE including a sample script demonstrating MCE not 100% UTF-8 safe. All this time, totally overlooked the fact that the length function deals in logical characters, not physical bytes. :) Timmermans, Leon The inspiration for not having to backslash variables for MCE::Shared came from reading Const::Fast: e.g. mce_share my @array => ( list ); Thalhammer, Jeffrey Ryan For submitting a feature request for lazy arrays support. Although a lazy array cannot be used directly for input_data, support for iterator objects was added to MCE 1.505; e.g. input_data => $iterator_ref; This enabled support for lazy arrays using closures. I am thankful for the small code snippet highlighting a use-case of MCE. I would have failed initially without it. Your example was placed under the "SYNTAX for INPUT_DATA" section in MCE::Core for others to see on how to process output from Path::Iterator::Rule in parallel. Warm regards, Mario