Configure AWS access key and secret access key for enabling pyspark to read-write from an S3 bucket.

• Step 1: Create an S3 bucket
• Step 2: Create an IAM policy with specific S3 permissions (example below) and attach it to an IAM role. Sample IAM policy which grants programmatic read-write access to the "test" S3 bucket:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::test"] }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": ["arn:aws:s3:::test/*"] } ] } For more information on IAM roles and policies, please refer to the AWS article: https://aws.amazon.com/blogs/security/writing-iam-policies-how-to-grant-access-to-an-amazon-s3-bucket/
• Step 3: Once the IAM user is created, note down the Access key ID and the Secret Access key.
• Step 4: To enable pyspark to read-write from an S3 bucket
Login as root user. Navigate to /etc/profile.d/ Create any bash file e.g spark-env.sh and add the following lines: #!/bin/bash export AWS_ACCESS_KEY_ID=<the Access key ID when IAM user was created> export AWS_SECRET_ACCESS_KEY=<the Secret Access key when the IAM user was created> Save the file and exit. Run "source /etc/profile"
• This will permanently save the environment variables for all users and user sessions. The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are necessary for pyspark code to be able to read/write to an S3 bucket.