Module: vemomoto_core.concurrent.concurrent_futures_ext#

Created on 03.01.2017

@author: Samuel

Classes:

ProcessPoolExecutor([max_workers, ...])

classdocs

Functions:

get_cpu_chunk_counts(task_length[, ...])

class ProcessPoolExecutor(max_workers=None, const_args=[], shared_np_arrs=[])[source]#

Bases: concurrent.futures.process.ProcessPoolExecutor

classdocs

Methods:

get_shared_arrays()

map(fn, *iterables[, timeout, chunksize, ...])

Returns an iterator equivalent to map(fn, iter).
get_shared_arrays()[source]#
map(fn, *iterables, timeout=None, chunksize=None, tasklength=None, chunknumber=5, min_chunksize=1)[source]#

Returns an iterator equivalent to map(fn, iter).

Parameters

  • fn – A callable that will take as many arguments as there are passed iterables.

  • timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.

  • chunksize – If greater than one, the iterables will be chopped into chunks of size chunksize and submitted to the process pool. If set to one, the items in the list will be sent one at a time.

  • tasklength – length of the iterable. If provided, the cpu count and the chunksize will be adjusted approprietly, if they are not explicietely given.

Returns

map(func, *iterables) but the calls may be evaluated out-of-order.

Return type

An iterator equivalent to

Raises

  • TimeoutError – If the entire result iterator could not be generated before the given timeout.

  • Exception – If fn(*args) raises for any values.

get_cpu_chunk_counts(task_length, chunk_number=5, min_chunk_size=1)[source]#