我的问题是:如何使用aws-sdk(或knox)使用这些库的最新功能处理流来正确使用强大的功能?
> How to receive an uploaded file using node.js formidable library and save it to Amazon S3 using knox?
> node application stream file upload directly to amazon s3
> Accessing the raw file stream from a node-formidable file upload(以及关于覆盖form.onPart()的非常有用的接受答案)
但是,我认为答案有点过时和/或偏离主题(即CORS支持,我现在不希望出于各种原因使用)和/或最重要的是,没有提及来自的最新功能aws-sdk(参见:https://github.com/aws/aws-sdk-js/issues/13#issuecomment-16085442)或knox(特别是putStream()或其readableStream.pipe(req)变体,both explained in the doc).
经过几个小时的挣扎,我得出结论,我需要一些帮助(免责声明:我是一个流媒体的新手).
HTML表单:
<form action="/uploadPicture" method="post" enctype="multipart/form-data"> <input name="picture" type="file" accept="image/*"> <input type="submit"> </form>
Express bodyParser中间件以这种方式配置:
app.use(express.bodyParser({defer: true}))
POST请求处理程序:
uploadPicture = (req,res,next) -> form = new formidable.IncomingForm() form.parse(req) form.onPart = (part) -> if not part.filename # Let formidable handle all non-file parts (fields) form.handlePart(part) else handlePart(part,form.bytesExpected) handlePart = (part,fileSize) -> # aws-sdk version params = Bucket: "mybucket" Key: part.filename ContentLength: fileSize Body: part # passing stream object as body parameter awsS3client.putObject(params,(err,data) -> if err console.log err else console.log data )
但是,我收到以下错误:
{ [RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.]
message: ‘Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.’,
code: ‘RequestTimeout’,
name: ‘RequestTimeout’,
statusCode: 400,
retryable: false }
以这种方式定制的knox版本的handlePart()函数也惨遭失败:
handlePart = (part,fileSize) -> headers = "Content-Length": fileSize "Content-Type": part.mime knoxS3client.putStream(part,part.filename,headers,res) -> if err console.log err else console.log res )
我还得到一个带有400 statusCode的大型res对象.
在两种情况下,区域都配置为eu-west-1.
补充笔记:
node 0.10.12
latest formidable from npm (1.0.14)
latest aws-sdk from npm (1.3.1)
latest knox from npm (0.8.3)
解决方法
The S3 API requires you to provide the size of new files when creating them. This information is not available for multipart/form-data files until they have been fully received. This means streaming is impossible.
实际上,form.bytesExpected指的是整个表单的大小,而不是单个文件的大小.
因此,在上传到S3之前,数据必须首先点击服务器上的内存或磁盘.