- StringIO([s]) -- Return a StringIO-like stream for reading or writing
- Identify situations where multiprocessing will not work as required.
- enable_multiprocessing_if_possible(nproc=<libtbx.AutoType object>, log=None)
- Switch for using multiple CPUs, usually called at the beginning of an app.
If nproc is Auto or None and we are running Windows, it will be reset to 1.
- Determine number of processes dynamically: number of CPUs minus the current
load average (with a minimum of 1).
- parallel_map(func, iterable, params=None, processes=1, method='multiprocessing', qsub_command=None, asynchronous=True, callback=None, preserve_order=True, preserve_exception_message=False, use_manager=True)
- Generic parallel map() implementation for a variety of platforms, including
the multiprocessing module and supported queuing systems, via the module
libtbx.queuing_system_utils.scheduling. This is less flexible than pool_map
above, since it does not provide a way to use a non-pickleable target
function, but it provides a consistent API for programs where multiple
execution methods are desired. It will also work on Windows (if the method
is multiprocessing or threading).
Note that for most applications, the threading method will be constrained
by the Global Interpreter Lock, therefore multiprocessing is prefered for
parallelizing across a single multi-core system.
See Computational Crystallography Newsletter 3:37-42 (2012) for details of
the underlying method.
:param func: target function (must be pickleable)
:param iterable: list of arguments for func
:param processes: number of processes/threads to start
:param method: parallelization method (multiprocessing|threading|sge|lsf|pbs)
:param qsub_command: command to submit queue jobs (optional)
:param asynchronous: run queue jobs asynchronously
:param preserve_exception_message: keeps original exception message
:returns: a list of result objects
- pool_map(processes=None, initializer=None, initargs=(), maxtasksperchild=<libtbx.AutoType object>, func=None, fixed_func=None, iterable=None, args=None, chunksize=<libtbx.AutoType object>, func_wrapper='simple', index_args=True, log=None, call_back_for_serial_run=None)
- Parallelized map() using subclassed multiprocessing.Pool. If func is not
None, this function essentially calls the Pool's own map method; this means
that both func and iterable/args must be pickle-able. If fixed_func is not
None, it will not be pickled but instead saved as an attribute of the Pool,
which will be preserved after the fork() call. Additional features include
optional redirection of output and automatic process number determination.
Note that because of the reliance on fork(), this function will run in serial
on Windows, regardless of how many processors are available.
:param processes: number of processes to spawn; if None or Auto, the
get_processes() function will be used.
:param func: target function (will be pickled)
:param fixed_func: "fixed" target function, which will be be propagated to
the child process when forked (instead of pickling)
:param iterable: argument list
:param args: same as iterable (alternate keyword)
:param chunksize: number of arguments to process at once
- show_caught_exception(index, arg)