def my_function(b): x = np.random.normal(0, b, 1024) A = np.random.normal(0, b, (1024, 1024)) return np.dot(A, x) pwex = pywren.default_executor() res = pwex.map(my_function, np.linspace(0.1, 100, 1000))
First, make sure you have an account with Amazon Web Services. Then download and install pywren via PIP as outlined in the getting started materials Then enjoy running your code on thousands of cores simultaneously!
Key technologies leveraged include:
- AWS Lambda for fast, containerized, stateless compute
- AWS S3 for event coordination
- Continuum's Anaconda python distribution for up-to-date python packages
- cloudpickle for shipping functions back and forth
The overall goal is to mimic the Python 3.x futures interface as much as make sense.
- low limit of simultaneous workers (maybe 3k if you reserve ahead)
- finite amount of time per worker (300 seconds), but [see support for stand-alone workers!]
- non-trivial function invocation overhead, sometimes 15 sec!
"Occupy the Cloud: Distributed computing for the 99%" arXiv 1702.0402 Eric Jonas, Shivaram Venkataraman, Ion Stoica, Benjamin Recht