One of our clients has hosted his application on AWS and it uses EC2 instances behind the load balancer. The application provides an interface where users are suppose to upload files. For saving the same on centralize location we use S3 bucket and mount the same on the EC2 instances. Following are the step which we follow to get the things working:
This guide shows how to mount the S3 bucket using s3fs file system to an Amazon EC2 server and access it using WinSCP. Creating Access Server If you do not have a Linux server available for the mounting, launch a new Amazon EC2 server. MinFS lets you mount a remote bucket (from a S3 compatible object store), as if it were a local directory. This allows you to read and write from the remote bucket just by operating on the local mount directory. MinFS helps legacy applications use modern object stores with minimal config changes.
Prerequisite : First we execute the following command, which will install the required packages necessary to install S3fs command.
Debian: Reformat mac os.
Amazon AMI:
Now we download the S3fs code, untar it and install it by using following command:
Now we will store our secret key and access key like following:
Tag -
Mounting S3 bucketMounting S3 bucket into an EC2Mounting S3 bucket into an EC2 instanceS3 bucket into an EC2 instanceLearn to mount an S3 bucket in RHEL, Ubuntu, CentOS Linux server. Understand how to debug issues while mounting an S3 bucket.
In this article we will walk through the process by which you can mount an S3 bucket on the Linux server. S3 bucket is a storage container of S3 (simple storage service) AWS service. As all traditional data centers are moving to cloud computing, it is necessary to know how to interconnect cloud and traditional services. Let's dive into the process to mount an S3 bucket in RHEL, Ubuntu, CentOS Linux. The complete process can be done in below 3 steps:
- Install fuse and s3fs packages
- Configure access keys of your AWS account
- Mount S3 bucket
For this tutorial, we are assuming you have an S3 bucket ready in your AWS account with a proper permission setup. https://obbbu.over-blog.com/2021/01/airfoil-and-airfoil-speakers-4-9-2-download-free.html. If not, follow this tutorial to create an S3 bucket in AWS.
Install fuse and s3fs packages
These packages can be found here: fuse & s3fs. You have to download them on the Linux server with tools like wget and compile them. Make sure you have their dependencies installed before you try to compile them.
Dependencies are :
for RedHat based : automake
gcc
gcc-c++
git
libcurl-devel
libxml2-devel
make
openssl-devel
mailcap
curl-devel
libstdc++-devel
for Debian based : automake
autotools-dev
g++
git
libcurl4-gnutls-dev
libfuse-dev
libssl-dev
libxml2-dev
make
pkg-config
Install all these packages and follow the below steps to configure fuse
and s3fs
. Make sure there is no package named fuse of s3fs
exist in the server before you proceed. This is to avoid conflicts in installation.
Fuse installation –
Use the below commands. Use the latest Fuse download link used in wget command. You can obtain it from Github here.
The fuse installation output is below for your reference. Click and expand if you want to view it.
s3fs installation-
Use the below commands. We are cloning git here so no need to check the latest release from GitHub website.
s3fs installation output is below for your reference. Click and expand if you want to refer it.
Configure access keys of your AWS account
Now you need to configure your AWS account security keys in s3fs utility. For that you need to visit your AWS account's IAM (Identity and Access Management) page and get those keys under ‘Access Keys (Access Key ID and Secret Access Key)‘. If you haven't created any till the time, then you can create a new key pair and use it.
Access S3 Bucket Windows
There are two keys: Access Key ID and Secret Access Key. Add those keys in file/etc/s3fs-keys
separated by a colon. You can keep them in any file name and path of your choice. I prefer to keep it in /etc
where other OS imp files reside.
Remove other permission to read this key file. If you don't set permission 0 to others then s3fs command utility will warn you about it like below. For extra security you can make this file hidden by adding . in the filename. Free podcast recording.
Now you are ready to mount your bucket.
Mount S3 bucket
Now run s3fs utility with bucket name you want to mount followed by directory on which you want to mount it. -o switch to be used to specify a key file path. If you used another filename and path than /etc/s3fs-keys then use it accordingly. There are many other options that can be supplied with this command to control cache and permission on the Linux server (OS side) which I haven't mentioned here.
Here –
- kerneltalks.bucket is my bucket name in S3
/my_s3_bucket
is a directory on the server on which I mounted S3 bucket- passwd_file is the path where I kept my AWS account keys
Your bucket is mounted on your server! You can check it in df output. You will see the filesystem as s3fs.
You can perform all file and directory operations as you do on normal mount point. Observe size it showing – 256T! That's huge, that's S3 – almost unlimited storage!
You can add an entry in /etc/fstab to mount your bucket on boot too. Use below entry :
Bonus tip :
If your bucket name includes .
then s3fs may fail to mount your bucket. Curl will exist the mount operation as SSL on Amazon won't be matched as certificate wild card matching fails due to extra . Can you get office on a macbook. in your bucket name. To understand, if I have bucket name kerneltalks.bucket then SSL wildcard mismatch will happen as indicated by below error –
SSL: certificate subject name (*.s3.amazonaws.com) does not match target host name ‘kerneltalks.bucket.s3.amazonaws.com'
To see errors, warning during mount operation you can run s3fs command with debugging on. Append below switches to the end of your s3fs mount command:
This will help you troubleshoot any issues you faced during the S3 bucket mount.
Windows Mount S3 Bucket
This is how we mount S3 bucket in Linux servers like Redhat, Ubuntu, Debian, etc. Let us know if you have any feedback/comments/suggestions in below comment box. https://downafile544.weebly.com/affinity-designer-1-5-4-vector-graphic-design-software.html.