Function Definition Language (SCAR) ==================================== Example:: functions: aws: - lambda: boto_profile: default region: us-east-1 name: function1 memory: 1024 timeout: 300 execution_mode: lambda log_level: INFO layers: - arn:.... environment: Variables: KEY1: val1 KEY2: val2 init_script: ffmpeg-script.sh container: image: jrottenberg/ffmpeg:4.1-ubuntu timeout_threshold": 10 environment: Variables: KEY1: val1 KEY2: val2 input: - storage_provider: minio.my_minio path: my-bucket/test output: - storage_provider: s3.my_s3 path: my-bucket/test-output suffix: - wav - srt prefix: - result- supervisor: version: latest iam: boto_profile: default role: "" api_gateway: boto_profile: default region: us-east-1 cloudwatch: boto_profile: default region: us-east-1 log_retention_policy_in_days: 30 batch: boto_profile: default region: us-east-1 vcpus: 1 memory: 1024 enable_gpu: False service_role: "arn:..." environment: Variables: KEY1: val1 KEY2: val2 compute_resources: security_group_ids: - sg-12345678 desired_v_cpus: 0 min_v_cpus: 0 max_v_cpus: 2 subnets: - subnet-12345 subnet-67891 instance_types: - "m3.medium" instance_role: "arn:..." oscar: - my_oscar: name: service1 memory: 1Gi cpu: '1.0' log_level: INFO image: grycap/darknet script: my-script.sh environment: Variables: KEY1: val1 KEY2: val2 input: - storage_provider: minio.my_minio path: my-bucket/test output: - storage_provider: s3.my_s3 path: my-bucket/test-output suffix: - wav - srt prefix: - result- storage_providers: s3: my_s3: access_key: awsuser secret_key: awskey region: us-east-1 minio: my_minio: endpoint: minio-endpoint verify: True region: us-east-1 access_key: muser secret_key: mpass onedata: my_onedata: oneprovider_host: op-host token: mytoken space: onedata_space .. _OSCAR-FDL-StorageProvider: https://docs.oscar.grycap.net/fdl/#storageproviders .. _OSCAR-FDL-Enviroment: https://docs.oscar.grycap.net/fdl/#envvarsmap .. _OSCAR-FDL-StorageIOConfig: https://docs.oscar.grycap.net/fdl/#storageioconfig .. _OSCAR-FDL: https://docs.oscar.grycap.net/fdl/ Top level parameters -------------------- +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +================================================================+==============================================================================================================================+ | functions | Map to define the credentials for a MinIO storage provider, being the key the user-defined identifier for the provider | + | | | `Functions`_. | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | storage_providers | Parameter to define the credentials for the storage providers to be used in the services, in lambda function only s3 buckets | + | are allowed. | | `OSCAR-FDL-StorageProvider`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ Functions --------- +-----------------------------------------------------+-----------------------------------------------+ | Field | Description | +=====================================================+===============================================+ | AWS Elements | Parameters to define manage AWS resources | + | | | `Lambda`_ | | +-----------------------------------------------------+-----------------------------------------------+ | oscar | Parameters to define OSCAR services | + | | | `OSCAR-FDL`_ | | +-----------------------------------------------------+-----------------------------------------------+ AWS Elements ------------ +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +================================================================+==============================================================================================================================+ | Lambda | Set Lambda properties. | + | | | `Lambda`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | iam | Set IAM properties. | + | | | `IAM`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | api_gateway | Set API Gateway properties. | + | | | `API Gateway`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | cloudwatch | Set CloudWatch properties. | + | | | `Cloudwatch`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | batch | Set AWS Batch properties. | + | | | `AWS Batch`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ Lambda ------ +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +================================================================+==============================================================================================================================+ | boto_profile | Boto profile used for the lambda client. Default 'default'.Must match the profiles in the file ~/.aws/credentials | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | region | Region of the function, can be any region supported by AWS. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | name | Function's name. REQUIRED | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | memory | Memory of the function, in MB, min 128, max 3008. Default '512' | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | timeout | Maximum execution time in seconds, max 900. Default '300' | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | execution_mode | Set job delegation or not. Possible values 'lambda', 'lambda-batch', 'batch'. Default 'lambda'. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | log_level | Supervisor log level. Can be INFO, DEBUG, ERROR, WARNING. Default 'INFO' | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | layers | Lambda function's layers arn (max 4). SCAR adds the supervisor layer automatically. | + | + | *string array* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | environment | Environment variables of the function. This variables are used in the lambda's environment, not the container's environment. | + | + | `OSCAR-FDL-Enviroment`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | init_script | Script executed inside of the function's container. | + | + | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | container | Define udocker container properties | + | + | `Container`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | input | Define input storage providers linked with the function. | + | + | `OSCAR-FDL-StorageIOConfig`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | output | Define input storage providers linked with the function. | + | + | `OSCAR-FDL-StorageIOConfig`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | supervisor | Properties for the faas-supervisor used in the inside the lambda function | + | + | `Supervisor`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ Container --------- +--------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +==========================+==============================================================================================================================+ | image | Container image to use. REQUIRED | + | | | *string* | | +--------------------------+------------------------------------------------------------------------------------------------------------------------------+ | timeout_threshold | Time used to post-process data generated by the container. This time is substracted from the total time set for the function.| + | If there are a lot of files to upload as output, maybe this value has to be increased. Default '10' seconds. | | *string* | | +--------------------------+------------------------------------------------------------------------------------------------------------------------------+ | environment | Environment variables of the container. These variables are passed to the container environment, | + | that is, can be accessed from the user's script. | | `OSCAR-FDL-Enviroment`_ | | +--------------------------+------------------------------------------------------------------------------------------------------------------------------+ Supervisor ---------- +--------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +==========================+==============================================================================================================================+ | version | Must be a Github tag or "latest". Default 'latest'. | + + + | *string* | | +--------------------------+------------------------------------------------------------------------------------------------------------------------------+ IAM --- +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +================================================================+==============================================================================================================================+ | boto_profile | Boto profile used for the iam client. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | role | The Amazon Resource Name (ARN) of the function's execution role. | + | This value is usually set for all the functions in the SCAR's default configuration file. REQUIRED | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ API Gateway ----------- +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +================================================================+==============================================================================================================================+ | boto_profile | Boto profile used for the iam client. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | region | Region of the function, can be any region supported by AWS. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ Cloudwatch ---------- +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +================================================================+==============================================================================================================================+ | boto_profile | Boto profile used for the iam client. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | region | Region of the function, can be any region supported by AWS. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | log_retention_policy_in_days | Number of days that the functions logs are stored. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ AWS Batch --------- +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +================================================================+==============================================================================================================================+ | boto_profile | Boto profile used for the iam client. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | region | Region of the function, can be any region supported by AWS. | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | vcpus | The number of vCPUs reserved for the container. Used in the job definition. Default 1 | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | memory | The hard limit (in MiB) of memory to present to the container. Used in the job definition. Default 1024 | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | enable_gpu | Request GPU resources for the launched container. Default 'False'. Values 'False', 'True' | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | service_role | Environment variables passed to the batch container | + | | | *string* | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | environment | Number of days that the functions logs are stored. | + | | | `OSCAR-FDL-Enviroment`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | compute_resources | Parameters that specifies all the resources is going to use. | + | | | `Compute Resources`_ | | +----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------+ Compute Resources ----------------- +----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +================================================================+=========================================================================================================================================+ | security_group_ids | List of the Amazon EC2 security groups associated with instances launched in the compute environment. | + | REQUIRED when using batch. | | *string array* | | +----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | desired_v_cpus | The desired number of Amazon EC2 vCPUS in the compute environment. Default 0 | + | | | *string* | | +----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | min_v_cpus | The minimum number of Amazon EC2 vCPUs that an environment should maintain. Default 0 | + | | | *string* | | +----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | max_v_cpus | The maximum number of Amazon EC2 vCPUs that an environment should maintain. Default 2 | + | | | *string* | | +----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | subnets | List of the VPC subnets into which the compute resources are launched. REQUIRED when using batch | + | | | *string array* | | +----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | instance_types |The instances types that may be launched. You can specify instance families to launch any instance type within those families. | + |(for example, c5 or p3 ), or you can specify specific sizes within a family (such as c5.8xlarge). You can also choose optimal | | *string array* |to pick instance types (from the C, M, and R instance families) on the fly that match the demand of your job queues. Default 'm3.medium' | +----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | instance_role | The Amazon ECS instance profile applied to Amazon EC2 instances in a compute environment. | + | | | *string* | | +----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+