How to limit file size on upload to S3 using Java
If you have an S3 bucket with public write access, you probably would like to limit the uploaded file size. If you are using AWS Java SDK, you probably figured there isn’t a straight forward solution for that. In the following article, I will explain exactly how it can be done.
If your bucket has public write access, limitation on the uploaded file size is crucial to prevent abuse of your system. Without those limitations, clients are able to upload huge files to your system, which you will be charged for.
A possible solution one can think of is creating an AWS Lambda that will be triggered when a file is uploaded to S3, check its size and delete it if the file is too big. Although that solution will work, it is not ideal as you are being charged for (among other things) the file size.
POST Policy
POST policy is the policy required for submitting authenticated HTTP POST requests to AWS. It is a base64 encoded JSON that specifies conditions the request should meet. For example, one condition might be the content-length-range which specifies that the uploaded file size must be in that range.
As for the moment of writing those lines, generating this presigned post request to S3 is not supported by AWS Java SDK (it is on Python and JS), hence the best solution is generating the request in the structure S3 expects.
Creating the presigned post request
The process consist of 3 parts:
- Creating the policy with the desired conditions.
- Encoding the policy to base64.
- Creating the request signature from the policy.
To achieve that, let’s implement the following method:
public PreSignedPostRequest generatePresignedPost(String bucketName, String objectKey, Instant expiration, long fileMinSize, long fileMaxSize) throws PresignedUrlException {SimpleDateFormat dateTimeFormat =
new SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'");
dateTimeFormat.setTimeZone(new SimpleTimeZone(0, "UTC"));
SimpleDateFormat dateStampFormat = new SimpleDateFormat("yyyyMMdd");
dateStampFormat.setTimeZone(new SimpleTimeZone(0, "UTC"));
Date now = new Date();
String dateTimeStamp = dateTimeFormat.format(now);
String dateStamp = dateStampFormat.format(now);Map<String, Object> policy = generatePolicy(expiration,fileMaxSize, fileMinSize, bucketName, objectKey, dateTimeStamp, dateStamp);
String encodedPolicy = encodePolicy(policy);
String signature = sign(encodedPolicy, dateStamp, "s3");
return buildSignedPostRequest(encodedPolicy, signature, dateTimeStamp,dateStamp, bucketName, objectKey);
}
The method generatePresignedPost receives the bucket name, the object name, the expiration time of the signature, and the range of file size allowed. It creates timestamps in the needed formats, generates the policy, and signs it. We will now implement the methods generatePolicy, encodePolicy, and sign.
Create the policy:
the POST policy always contains “expiration” which is a string and “conditions” which is a list of objects. In addition, we will also add the following AWS mandatory fields:
x-amz-credential — provides the scope the signature is valid in.
x-amz-algorithm — identifies the version of AWS signature and the algorithm you used.
x-amz-date — the timestamp of the moment you signed the request in ISO 8601 standard.
bucket — the target bucket name.
key — the name of the object in the bucket.
x-amz-security-token — optional field needed if you are using session credentials such as IAM role.
Therefore our policy should look like that:
{"expiration":"2021-01-28T22:07:303Z",
"conditions":[["content-length-range",1,1000000],{"bucket":"medium-example"},{"key":"test/test.json"},{"x-amz-algorithm":"AWS4-HMAC-SHA256"},{"x-amz-credential":"AKIAYY********/20210128/us-east-2/s3/aws4_request"}]}Generated by the following code:
private Map<String, Object> generatePolicy(Instant expiration, long fileMaxSize, long fileMinSize, String bucketName, String objectKey, String dateTimeStamp, String dateStamp) {
List<?> conditions = generateConditions(bucketName, objectKey, fileMaxSize, fileMinSize, dateTimeStamp, dateStamp);
return Map.of(
"expiration", new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:SS'Z'").format(Date.from(expiration)),
"conditions", conditions
);
}private Map<String, String> generateAwsDefaultParams(String bucketName, String objectKey, String dateTimeStamp, String dateStamp) {
String scope = String.format("%s/%s/%s/%s/%s", awsCredentials.getAWSAccessKeyId(), dateStamp, REGION, "s3", "aws4_request");
Map<String, String> params = new HashMap<>();
params.put("bucket", bucketName);
params.put("x-amz-algorithm", "AWS4-HMAC-SHA256");
params.put("x-amz-credential", scope);
params.put("x-amz-date", dateTimeStamp);
params.put("key", objectKey);
if (this.awsCredentials instanceof AWSSessionCredentials) {
params.put("x-amz-security-token", ((AWSSessionCredentials) this.awsCredentials).getSessionToken());
}
return params;
}
private List<?> generateConditions(String bucketName, String objectKey, long fileMaxSize, long fileMinSize, String dateTimeStamp, String dateStamp) {
List<?> contentLengthRange = List.of("content-length-range", fileMinSize, fileMaxSize);
List<Object> conditions = new LinkedList<>();
conditions.add(contentLengthRange);
Map<String, String> awsDefaults = generateAwsDefaultParams(bucketName, objectKey, dateTimeStamp, dateStamp);
conditions.addAll(awsDefaults.entrySet());
return conditions;
}
Notice I have used the following data member:
private AWSCredentials awsCredentials = DefaultAWSCredentialsProviderChain.getInstance().getCredentials();Encoding the policy
The next step is to implement encodePolicy method:
private String encodePolicy(Map<String, Object> policy) throws JsonProcessingException {
String policyAsStr = mapper.writeValueAsString(policy);
byte[] policyAsBytes = policyAsStr.getBytes(StandardCharsets.UTF_8);
return BinaryUtils.toBase64(policyAsBytes);
}where mapper is Jackson ObjectMapper.
Signing the policy
What we have left to do here after preparing the policy is:
- Derive the signing key from AWS secret access key.
- use the policy and the signing key to create a signature.
When an AWS service receives the request, it performs the same steps to calculate the signature you sent in your request. AWS then compares its calculated signature to the one you sent with the request. If the signatures match, the request is processed. If the signatures don’t match, the request is denied.
public String sign(String toSign, String dateStamp, String service)
throws NoSuchAlgorithmException, InvalidKeyException, UnsupportedEncodingException {
String awsSecretKey = awsCredentials.getAWSSecretKey();
byte[] kSecret = ("AWS4" + awsSecretKey).getBytes(StandardCharsets.UTF_8);
byte[] kDate = hmacSHA256(dateStamp, kSecret);
byte[] kRegion = hmacSHA256(REGION, kDate);
byte[] kService = hmacSHA256(service, kRegion);
byte[] kSigning = hmacSHA256("aws4_request", kService);
byte[] signature = hmacSHA256(toSign, kSigning);
return BinaryUtils.toHex(signature);
}
private byte[] hmacSHA256(String data, byte[] key)
throws NoSuchAlgorithmException, InvalidKeyException {
String algorithm = "HmacSHA256";
Mac mac = Mac.getInstance(algorithm);
mac.init(new SecretKeySpec(key, algorithm));
return mac.doFinal(data.getBytes(StandardCharsets.UTF_8));
}That's it! we are done. All we have left is to build the response object. S3 expects to receive with your POST request all the mandatory fields we added to the policy, with the encoded policy and the signature.
private PreSignedPostRequest buildSignedPostRequest(String encodedPolicy, String signature,String dateTimeStamp, String dateStamp, String bucketName, String objectKey) {
Map<String, String> body =
generateAwsDefaultParams(bucketName, objectKey, dateTimeStamp, dateStamp);
body.put("x-amz-signature", signature);
body.put("policy", encodedPolicy);
String url =
String.format("https://%s.s3.amazonaws.com/", bucketName);
return PreSignedPostRequest.builder()
.body(body).url(url).build();
}where PreSignedPostRequest is just a POJO contains the URL for submitting the request, and body field represents the request body (without the file).
To upload the file to S3 just attach the file to the returned body, and fire the request to the returned URL.
Notice that if you forget to add one of AWS mandatory fields you are going to get some unclear error message when submitting the request to S3 such as:
֫Invalid according to Policy: Extra input fields: x-amz-dateConclusion
AWS supports creating a presigned URL for upload to S3 using a PUT request, that at the moment can’t have any restrictions on the uploaded object. If we take a look at the use case for presigned URL to S3 — enable upload to a bucket from outside, it is clear that some restrictions must exist to prevent abuse of your system. Hence until that feature will be added to Java SDK, the above solution seems to be ideal.
