Function Definition Language (SCAR)

Example:

functions:
  aws:
  - lambda:
      boto_profile: default
      region: us-east-1
      name: function1
      memory: 1024
      timeout: 300
      execution_mode: lambda
      log_level: INFO
      layers:
      - arn:....
      environment:
        Variables:
          KEY1: val1
          KEY2: val2
      init_script: ffmpeg-script.sh
      container:
        image: jrottenberg/ffmpeg:4.1-ubuntu
        timeout_threshold": 10
        environment:
          Variables:
            KEY1: val1
            KEY2: val2
      input:
      - storage_provider: minio.my_minio
        path: my-bucket/test
      output:
      - storage_provider: s3.my_s3
        path: my-bucket/test-output
        suffix:
          - wav
          - srt
        prefix:
          - result-
      supervisor:
        version: latest


    iam:
      boto_profile: default
      role: ""
    api_gateway:
      boto_profile: default
      region: us-east-1
    cloudwatch:
      boto_profile: default
      region: us-east-1
      log_retention_policy_in_days: 30
    batch:
      boto_profile: default
      region: us-east-1
      vcpus: 1
      memory: 1024
      enable_gpu: False
      service_role: "arn:..."
      environment:
        Variables:
          KEY1: val1
          KEY2: val2
      compute_resources:
        security_group_ids:
        - sg-12345678
        desired_v_cpus: 0
        min_v_cpus: 0
        max_v_cpus: 2
        subnets:
        - subnet-12345
          subnet-67891
        instance_types:
        - "m3.medium"
        instance_role: "arn:..."
  oscar:
  - my_oscar:
      name: service1
      memory: 1Gi
      cpu: '1.0'
      log_level: INFO
      image: grycap/darknet
      script: my-script.sh
      environment:
        Variables:
          KEY1: val1
          KEY2: val2
      input:
      - storage_provider: minio.my_minio
        path: my-bucket/test
      output:
      - storage_provider: s3.my_s3
        path: my-bucket/test-output
        suffix:
          - wav
          - srt
        prefix:
          - result-
storage_providers:
  s3:
    my_s3:
      access_key: awsuser
      secret_key: awskey
      region: us-east-1
  minio:
    my_minio:
      endpoint: minio-endpoint
      verify: True
      region: us-east-1
      access_key: muser
      secret_key: mpass
  onedata:
    my_onedata:
      oneprovider_host: op-host
      token: mytoken
      space: onedata_space

Top level parameters

Field	Description
functions Functions.	Map to define the credentials for a MinIO storage provider, being the key the user-defined identifier for the provider
storage_providers OSCAR-FDL-StorageProvider	Parameter to define the credentials for the storage providers to be used in the services, in lambda function only s3 buckets are allowed.

Functions

Field	Description
AWS Elements Lambda	Parameters to define manage AWS resources
oscar OSCAR-FDL	Parameters to define OSCAR services

AWS Elements

Field	Description
Lambda Lambda	Set Lambda properties.
iam IAM	Set IAM properties.
api_gateway API Gateway	Set API Gateway properties.
cloudwatch Cloudwatch	Set CloudWatch properties.
batch AWS Batch	Set AWS Batch properties.

Lambda

Field	Description
boto_profile string	Boto profile used for the lambda client. Default ‘default’.Must match the profiles in the file ~/.aws/credentials
region string	Region of the function, can be any region supported by AWS.
name string	Function’s name. REQUIRED
memory string	Memory of the function, in MB, min 128, max 3008. Default ‘512’
timeout string	Maximum execution time in seconds, max 900. Default ‘300’
execution_mode string	Set job delegation or not. Possible values ‘lambda’, ‘lambda-batch’, ‘batch’. Default ‘lambda’.
log_level string	Supervisor log level. Can be INFO, DEBUG, ERROR, WARNING. Default ‘INFO’
layers string array	Lambda function’s layers arn (max 4). SCAR adds the supervisor layer automatically.
environment OSCAR-FDL-Enviroment	Environment variables of the function. This variables are used in the lambda’s environment, not the container’s environment.
init_script string	Script executed inside of the function’s container.
container Container	Define udocker container properties
input OSCAR-FDL-StorageIOConfig	Define input storage providers linked with the function.
output OSCAR-FDL-StorageIOConfig	Define input storage providers linked with the function.
supervisor Supervisor	Properties for the faas-supervisor used in the inside the lambda function

Container

Field	Description
image string	Container image to use. REQUIRED
timeout_threshold string	Time used to post-process data generated by the container. This time is substracted from the total time set for the function. If there are a lot of files to upload as output, maybe this value has to be increased. Default ‘10’ seconds.
environment OSCAR-FDL-Enviroment	Environment variables of the container. These variables are passed to the container environment, that is, can be accessed from the user’s script.

Supervisor

Field	Description
version string	Must be a Github tag or “latest”. Default ‘latest’.

IAM

Field	Description
boto_profile string	Boto profile used for the iam client.
role string	The Amazon Resource Name (ARN) of the function’s execution role. This value is usually set for all the functions in the SCAR’s default configuration file. REQUIRED

API Gateway

Field	Description
boto_profile string	Boto profile used for the iam client.
region string	Region of the function, can be any region supported by AWS.

Cloudwatch

Field	Description
boto_profile string	Boto profile used for the iam client.
region string	Region of the function, can be any region supported by AWS.
log_retention_policy_in_days string	Number of days that the functions logs are stored.

AWS Batch

Field	Description
boto_profile string	Boto profile used for the iam client.
region string	Region of the function, can be any region supported by AWS.
vcpus string	The number of vCPUs reserved for the container. Used in the job definition. Default 1
memory string	The hard limit (in MiB) of memory to present to the container. Used in the job definition. Default 1024
enable_gpu string	Request GPU resources for the launched container. Default ‘False’. Values ‘False’, ‘True’
service_role string	Environment variables passed to the batch container
environment OSCAR-FDL-Enviroment	Number of days that the functions logs are stored.
compute_resources Compute Resources	Parameters that specifies all the resources is going to use.

Compute Resources

Field	Description
security_group_ids string array	List of the Amazon EC2 security groups associated with instances launched in the compute environment. REQUIRED when using batch.
desired_v_cpus string	The desired number of Amazon EC2 vCPUS in the compute environment. Default 0
min_v_cpus string	The minimum number of Amazon EC2 vCPUs that an environment should maintain. Default 0
max_v_cpus string	The maximum number of Amazon EC2 vCPUs that an environment should maintain. Default 2
subnets string array	List of the VPC subnets into which the compute resources are launched. REQUIRED when using batch
instance_types string array	The instances types that may be launched. You can specify instance families to launch any instance type within those families. (for example, c5 or p3 ), or you can specify specific sizes within a family (such as c5.8xlarge). You can also choose optimal to pick instance types (from the C, M, and R instance families) on the fly that match the demand of your job queues. Default ‘m3.medium’
instance_role string	The Amazon ECS instance profile applied to Amazon EC2 instances in a compute environment.