Module: vemomoto_core.concurrent.sharedmem_ext

Created on 07.12.2016

@author: Samuel

Classes:

MapAsyncResult(fetcher, result, exceptions)

MapReduce([backend, np])

classdocs

class MapAsyncResult(fetcher, result, exceptions)[source]

Bases: object

Methods:

fetch()

fetch()[source]
class MapReduce(backend=<class 'sharedmem.sharedmem.ProcessBackend'>, np=None)[source]

Bases: sharedmem.sharedmem.MapReduce

classdocs

Methods:

map_async(func, sequence[, result, reduce, …])

Map-reduce with multile processes.

map_async(func, sequence, result=[], reduce=None, star=False, minlength=0)[source]

Map-reduce with multile processes.

Apply func to each item on the sequence, in parallel. As the results are collected, reduce is called on the result. The reduced result is returned as a list.

Parameters
  • func (callable) –

    The function to call. It must accept the same number of arguments as the length of an item in the sequence.

    Warning

    func is not supposed to use exceptions for flow control. In non-debug mode all exceptions will be wrapped into a SlaveException.

  • sequence (list or array_like) – The sequence of arguments to be applied to func.

  • reduce (callable, optional) – Apply an reduction operation on the return values of func. If func returns a tuple, they are treated as positional arguments of reduce.

  • star (boolean) – if True, the items in sequence are treated as positional arguments of reduce.

  • minlength (integer) – Minimal length of sequence to start parallel processing. if len(sequence) < minlength, fall back to sequential processing. This can be used to avoid the overhead of starting the worker processes when there is little work.

Returns

results – The list of reduced results from the map operation, in the order of the arguments of sequence.

Return type

list

Raises

SlaveException – If any of the slave process encounters an exception. Inspect SlaveException.reason for the underlying exception.