Today we're excited to annouce the latest version of PyWren, version 0.1. This
is our first release with a stand-alone mode, a
reduce verb, and Python 3 support.
A lot has happened over the past few months with PyWren. There was a a
talk AnacondaCon and we
written up at the the New Stack.
We posted our latest paper on the
Occupy the Cloud: Distributed Computing for the 99%. We
moved the project to a github organization and
and organized the examples.
We sometimes run into situations where we can't get around Lambda's
limits of runtime and memory, or need a GPU, or want to run our code
on a machine with 2 TB of RAM. We created a new type of executor,
standalone, which runs PyWren jobs inside our Anaconda runtime on
arbitrary EC2 instances, using Amazon's SQS as a queue dispatch
system. Jobs can run for up to 12 hours, and the available memory is
constrained only by the instance type. When the queue is empty, the
instance will shut itself down automatically.
Note that this feature is still somewhat experimental, and we're actively soliciting feedback. We'll have a complete example of use soon.
Up till now, PyWren has largely supported
functionality. Many machine-learning workloads consist of a
featurization step that is embarrassingly parallel, followd by a large
distributed machine-learning operation that aggregates the result of
reduce. Now you can write
lexec = pywren.lambda_executor() sexec = pywren.standalone_executor() features = lexec.map(create_feature, data) model = sexec.reduce(train_model, features)
The benefit here is the standalone executor can be one of the massive
AWS instances, like the
x1.32xlarge, with 2 TB of ram and 64 cores. Even
if training your model takes an hour, it's only $14.
This functionality is also experimental, and we'll have a good example soon!
Python 3 support
Based on great work by @washcycle, we now have full Python 3 support and Python 3.5 and 3.6 are fully-supported runtimes.
- New website, pywren.io
- Additional contributors, especially Qifan Pu and Shivaram Venkataraman, and support from both the Berkeley Center for Computational Imaging (via Ben Recht) and the UC Berkeley RISE Lab (thanks to Ion Stoica )
- Installable via PyPi
- Improved error handling and catching of exceptions