Some customers have workflows or SLAs that require setting a maximum size of a bucket. While BlackPearl Nearline Gateway offers no built-in functionality to limit the bucket size, this operation can be accomplished via ACLs and scripting. If user or group access is defined at the bucket-level for non-bucket owners, privileges can be reduced to LIST, READ, and JOB. This allows the read access to the system, while preventing new data from being written to the bucket. This effectively makes the bucket read-only. A script can be used to monitor the size of the bucket and revoke WRITE and DELETE privileges once the bucket grows past a certain size. This functionality is included in version 1.4.0 of the BlackPearl Python Tool.
The script stores the desired bucket limits in its configuration file, nacre.yml. The bucket_size_limits document allows an administrator to specify the maximum size of the bucket. The yaml file requires only two fields per bucket, the name and the maximum size, which use the keys bucket and limit respectively. A hyphen before the bucket is required as it allows for the entry of multiple bucket values. The limit field can be specified in either the binary system or base-10 nomenclature, e.g. GiB or GB. The units are case sensitive. Once the script applies the read-only ACLs to the bucket, a triggered field is added to the bucket object. This prevents the script from checking the bucket and re-applying fields on subsequent executions.
Executing the command, works the same as any other command for the python tool. The required flags for the script are endpoint (-e), credentials, and command (-c). As it operates entirely on the data path, either an Amazon S3 access key and secret key must be specified or an administrator username and password can be specified. If the username and password combination are used, the script will query the management path for the Amazon S3 credentials before processing the script. Depending on what credentials are used, the management address or data path address should be specified in the endpoint. The command flag is used to specify the “check-bucket-limits” command. Assuming all required parameters are entered, the script will execute and provide a summary of how many bucket policies were modified.
In order to automate this process, CRON or Windows Task Schedule can be used to execute the script at the desired interval.
Sample configuration file: nacre.yml
logging:
log_count: 3
log_level: INFO
log_location: ../log/
log_name: nacre-main.log
log_size: 100 KB
---
bucket_size_limits:
- bucket: seans
limit: 10 GB
triggered: true
- bucket: sean-test
limit: 5 GiBScript execution on a Linux machine to a BlackPearl with a domain name of test-bp with the username script-user and password scriptpass. The script's parameters can be entered in any order.
./nacre -e test-bp.local -u script-user -p scriptpass -c check-bucket-limitsSample for setting up a CRON job and for setting up a scheduled task.