libtbx.easy_mp
index
/net/chevy/raid1/nat/src/cctbx_project/libtbx/easy_mp.py

 
Modules
       
os
sys
traceback

 
Classes
       
__builtin__.object
fixed_func_proxy
func_wrapper_simple
func_wrapper_simple_impl
func_wrapper_sub_directories
func_wrapper_sub_directories_impl
python_exception_safe_run
multiprocessing.pool.Pool(__builtin__.object)
Pool

 
class Pool(multiprocessing.pool.Pool)
    Subclass of multiprocessing.Pool, used internally by pool_map.
 
 
Method resolution order:
Pool
multiprocessing.pool.Pool
__builtin__.object

Methods defined here:
__init__(self, processes=None, initializer=None, initargs=(), maxtasksperchild=None, fixed_func=None)
map_fixed_func(self, iterable, chunksize=None)
Uses fixed_func as passed to the constructor.
Avoids repeated pickling/unpickling of func, which can be rate-limiting
if func is large and the amount of work per call is relatively small.

Methods inherited from multiprocessing.pool.Pool:
__reduce__(self)
apply(self, func, args=(), kwds={})
Equivalent of `apply()` builtin
apply_async(self, func, args=(), kwds={}, callback=None)
Asynchronous equivalent of `apply()` builtin
close(self)
imap(self, func, iterable, chunksize=1)
Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()`
imap_unordered(self, func, iterable, chunksize=1)
Like `imap()` method but ordering of results is arbitrary
join(self)
map(self, func, iterable, chunksize=None)
Equivalent of `map()` builtin
map_async(self, func, iterable, chunksize=None, callback=None)
Asynchronous equivalent of `map()` builtin
terminate(self)

Data descriptors inherited from multiprocessing.pool.Pool:
__dict__
dictionary for instance variables (if defined)
__weakref__
list of weak references to the object (if defined)

Data and other attributes inherited from multiprocessing.pool.Pool:
Process = <class 'multiprocessing.process.Process'>
Process objects represent activity that is run in a separate process
 
The class is analagous to `threading.Thread`

 
class fixed_func_proxy(__builtin__.object)
    Implementation detail
 
  Methods defined here:
__call__(self, arg)
__init__(self, key, func)

Data descriptors defined here:
__dict__
dictionary for instance variables (if defined)
__weakref__
list of weak references to the object (if defined)

 
class func_wrapper_simple(__builtin__.object)
    Implementation detail
 
  Methods defined here:
__init__(O, buffer_stdout_stderr=False)
wrap(O, func)

Data descriptors defined here:
__dict__
dictionary for instance variables (if defined)
__weakref__
list of weak references to the object (if defined)

 
class func_wrapper_simple_impl(__builtin__.object)
    Implementation detail
 
  Methods defined here:
__call__(O, index_and_arg)
__init__(O, options, func)

Data descriptors defined here:
__dict__
dictionary for instance variables (if defined)
__weakref__
list of weak references to the object (if defined)

 
class func_wrapper_sub_directories(__builtin__.object)
    Implementation detail
 
  Methods defined here:
__init__(O, sub_name_format='mp%03d', makedirs_mode=511)
wrap(O, func)

Data descriptors defined here:
__dict__
dictionary for instance variables (if defined)
__weakref__
list of weak references to the object (if defined)

 
class func_wrapper_sub_directories_impl(__builtin__.object)
    Implementation detail
 
  Methods defined here:
__call__(O, index_and_arg)
__init__(O, options, func)

Data descriptors defined here:
__dict__
dictionary for instance variables (if defined)
__weakref__
list of weak references to the object (if defined)

 
class python_exception_safe_run(__builtin__.object)
    Thin wrapper for catching Python exceptions
 
  Methods defined here:
__call__(self, *args, **kwargs)
__init__(self, func)

Data descriptors defined here:
__dict__
dictionary for instance variables (if defined)
__weakref__
list of weak references to the object (if defined)

 
Functions
       
StringIO(...)
StringIO([s]) -- Return a StringIO-like stream for reading or writing
detect_problem()
Identify situations where multiprocessing will not work as required.
enable_multiprocessing_if_possible(nproc=<libtbx.AutoType object>, log=None)
Switch for using multiple CPUs, usually called at the beginning of an app.
If nproc is Auto or None and we are running Windows, it will be reset to 1.
get_processes(processes)
Determine number of processes dynamically: number of CPUs minus the current
load average (with a minimum of 1).
parallel_map(func, iterable, params=None, processes=1, method='multiprocessing', qsub_command=None, asynchronous=True, callback=None, preserve_order=True, preserve_exception_message=False, use_manager=True)
Generic parallel map() implementation for a variety of platforms, including
the multiprocessing module and supported queuing systems, via the module
libtbx.queuing_system_utils.scheduling.  This is less flexible than pool_map
above, since it does not provide a way to use a non-pickleable target
function, but it provides a consistent API for programs where multiple
execution methods are desired.  It will also work on Windows (if the method
is multiprocessing or threading).
 
Note that for most applications, the threading method will be constrained
by the Global Interpreter Lock, therefore multiprocessing is prefered for
parallelizing across a single multi-core system.
 
See Computational Crystallography Newsletter 3:37-42 (2012) for details of
the underlying method.
 
:param func: target function (must be pickleable)
:param iterable: list of arguments for func
:param processes: number of processes/threads to start
:param method: parallelization method (multiprocessing|threading|sge|lsf|pbs)
:param qsub_command: command to submit queue jobs (optional)
:param asynchronous: run queue jobs asynchronously
:param preserve_exception_message: keeps original exception message
:returns: a list of result objects
pool_map(processes=None, initializer=None, initargs=(), maxtasksperchild=<libtbx.AutoType object>, func=None, fixed_func=None, iterable=None, args=None, chunksize=<libtbx.AutoType object>, func_wrapper='simple', index_args=True, log=None, call_back_for_serial_run=None)
Parallelized map() using subclassed multiprocessing.Pool.  If func is not
None, this function essentially calls the Pool's own map method; this means
that both func and iterable/args must be pickle-able.  If fixed_func is not
None, it will not be pickled but instead saved as an attribute of the Pool,
which will be preserved after the fork() call.  Additional features include
optional redirection of output and automatic process number determination.
 
Note that because of the reliance on fork(), this function will run in serial
on Windows, regardless of how many processors are available.
 
:param processes: number of processes to spawn; if None or Auto, the
  get_processes() function will be used.
:param func: target function (will be pickled)
:param fixed_func: "fixed" target function, which will be be propagated to
  the child process when forked (instead of pickling)
:param iterable: argument list
:param args: same as iterable (alternate keyword)
:param chunksize: number of arguments to process at once
show_caught_exception(index, arg)

 
Data
        Auto = <libtbx.AutoType object>
division = _Feature((2, 2, 0, 'alpha', 2), (3, 0, 0, 'alpha', 0), 8192)
fixed_func_registry = <WeakValueDictionary at 31877960>
fixed_func_registry_key_generator = count(0)
parallel_captions = ['Multiprocessing', 'Sun_Grid_Engine', 'LSF', 'PBS', 'Condor', 'SLURM']
parallel_methods = ['*multiprocessing', 'sge', 'lsf', 'pbs', 'condor', 'slurm']
parallel_phil_str = '\nnproc = 1\n .type = int\n .short_caption = Numb...ne\n .type = str\n .short_caption = qsub command\n'
parallel_phil_str_base = '\nnproc = 1\n .type = int\n .short_caption = Numb...ne\n .type = str\n .short_caption = qsub command\n'
parallel_phil_str_no_threading = '\nnproc = 1\n .type = int\n .short_caption = Numb...ne\n .type = str\n .short_caption = qsub command\n'