我應該如何傳遞一個運行外部檔案的函式才能使用多處理？

我最近看到了一個模塊，它允許我同時運行我的代碼，這恰好是我需要的。然而，當我在測驗功能時，我遇到了一些錯誤并需要幫助。

所以基本上我需要從名為genODE. 基本上genODE是我的模擬檔案，所以它找到檔案規范，處理它，并生成一個輸出檔案。

目前這是我的代碼（我在 spyder(anaconda3) 順便說一句，在 Windows 中運行它，如果有幫助的話），

import multiprocessing
from astools import generateODEScript as genODE


start = time.perf_counter()

def simulation(niteration):
        print(f'Running simulation number: {niteration}')
        genODE(modelfile=f'./models/organism/Organism_{niteration}.modelspec', mtype='ASM', solver='RK4', timestep='1', endtime='21600', lowerbound='0;0', upperbound='1e-3;1e-3', odefile=f'organism_{niteration}.py')


p1 = multiprocessing.Process(target=simulation, args=[1])
p2 = multiprocessing.Process(target=simulation, args=[2])

if __name__ == '__main__':
    p1.start()
    p2.start()
    p1.join()
    p2.join()
print('process completed')
    
end = time.perf_counter()

print(end-start)

但是，我遇到了錯誤，無法讓它作業。目前我收到一個錯誤Can't attribute 'simulation' on module '__main__'。

更新：設法解決了歸因錯誤。但是，我收到了一條新的錯誤訊息Type error: generateODEscript() missing required positional argument

import time
from astools import generateODEScript as genODE
import multiprocessing

if __name__ == '__main__':
    start = time.perf_counter()
    p1 = multiprocessing.Process(target=genODE, args=())
    p1.start()
    p1.join()
    print('process completed')
    end = time.perf_counter()
    print(end-start)

如何在(modelfile=f'./models/organism/Organism_{niteration}.modelspec', mtype='ASM', solver='RK4', timestep='1', endtime='21600', lowerbound='0;0', upperbound='1e-3;1e-3', odefile=f'organism_{niteration}.py')這個多處理代碼中輸入所有 8 個引數？

uj5u.com熱心網友回復：

這是一個為這個問題引入一些并發性的框架建議：

from concurrent.futures import ProcessPoolExecutor

def genODE():
    pass
def runODE():
    pass
def sim(n):
    print(n)
    genODE()
    runODE()

def main():
    with ProcessPoolExecutor() as executor:
        executor.map(sim, range(1, 31))

if __name__ == '__main__':
    main()

讓OP來“填補空白”。

根據作業系統和/或潛在的記憶體限制，應考慮在 ProcessPoolExecutor 建構式中指定 max_workers=N

uj5u.com熱心網友回復：

這是使用佇列和多執行緒的生產者消費者模式解決方案的完整作業示例。

from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed
from queue import Queue
import numpy as np
import pandas as pd


def genODE(itmun):
    dates = pd.date_range('20130101', periods=6)
    pdf = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    pdf['itmun'] = itmun
    return pdf

def runODE(pdf):
    itnum = pdf['itmun'][0]
    pdf['itmun'] = pdf['itmun']   1
    return pdf

def producer(queue, itmun):
    # apply your function to create the data
    data = genODE(itmun)
    # put the data in the queue
    queue.put(data) 

def consumer(queue):
    while not queue.empty(): 
        data = queue.get() 
        # do somework on the data by running your function
        result_data = runODE(data)
        itnum = result_data['itmun'][0]
        result_data.to_csv(f'{itnum}test.csv')
        queue.task_done()
        return result_data 


def main():
    q = Queue() 
    futures = []
    with ThreadPoolExecutor(max_workers=15) as executor:
        for p in range(30):
            producer_future = executor.submit(producer, q, p)
            futures.append(producer_future) 
        for c in range(30):
            consumer_future = executor.submit(consumer, q)
            futures.append(consumer_future)

        for future in as_completed(futures):
            # do something with the result as soon as it is available such as saving or printing or nothing if you already done what you need in runODE()
            print(future.result()) 

if __name__ == "__main__":
    main()

uj5u.com熱心網友回復：

由于genODE需要關鍵字引數，因此您需要傳遞給multiprocessing.Processkwargs引數，指定關鍵字/值對的字典。niteration因此，要使用值 1 和 2開始您想要的兩個程序：

if __name__ == '__main__':
    import time
    import multiprocessing
    from astools import generateODEScript as genODE

    start = time.perf_counter()

    processes = []
    for niteration in (1, 2):
        kwargs = {
            'modelfile': f'./models/organism/Organism_{niteration}.modelspec',
            'mtype': 'ASM',
            'solver': 'RK4',
            'timestep': '1',
            'endtime': '21600',
            'lowerbound': '0;0',
            'upperbound': '1e-3;1e-3',
            'odefile': f'organism_{niteration}.py'
            
        }
        processes.append(multiprocessing.Process(target=genODE, kwargs=kwargs))

    for process in processes:
        process.start()
    for process in processes:
        process.join()
    print('process completed')
    end = time.perf_counter()
    print(end-start)

標籤：Python 视窗 功能 多处理

上一篇：如何從windowsCMD提示符中關閉標準輸入？

下一篇：在Spyder中添加庫