Large File Upload from On-premises into Oracle
Cloud Object Storage using REST API
In this tutorial(for users of mac os ,linux) , I will walk through about uploading .CSV file from on-premises to Oracle Cloud Object Storage using REST APIs.
When to use this approach ?
Suppose you have large CSV file or else MySQL backup file and you wanted to export into MySQL database service(MDS) , then it’s easy to lift-shift the backup file into Object Storage and from object storage to MySQL database service.
Next blog, we will explore how to import CSV file from Object Storage into MySQL Database Service(MDS).
In case you wanted to know what is object storage then below web link will have detailed info:-
What is Object Storage ?
more info:-
https://docs.oracle.com/en-us/iaas/Content/Object/Concepts/objectstorageoverview.htm
Let’s get started...
High level work flow as below:-
-
Make sure cURL is install in your On-P machine.
-
Make sure bucket is created in the object storage of Oracle Cloud
-
Generate Authentication Token
-
Generate Pre-Authenticated Requests in side the bucket of Oracle
Cloud
-
Run the cURL- PUT command from the On-Premises.
The Oracle Cloud Infrastructure Object Storage service is an internet-
scale, high-performance storage platform that offers reliable and cost-
efficient data durability. The Object Storage service can store an unlimited
amount of unstructured data of any content type, including analytic data
and rich content, like images and videos.
Verify cURL is install or not run curl –help , in the command prompt or else install it.
Create bucket in the Oracle Cloud
Bucket is the location where CSV file will be exported from On-Premises.
It is a container for storing objects in a compartment within an Object
Storage namespace.
You may use for storing data, images, logs, and video etc.
Each object inside bucket is encrypted with its own data encryption key.
Data encryption keys are always encrypted with a master encryption key
that is assigned to the bucket.
Encryption is enabled by default and cannot be turned off. By default,
Oracle manages the master encryption key.
Generate Authentication Token
By going to :- user settings→Auth tokens→Generate Token
|
Copy the Auth Token and save into text file and looks like below:- |
Czhfc(g_8i#h{8CAW2Gr
|
Create a pre-Authenticate Request on Object Storage
Pre-authenticated requests provide a way to let you to access a bucket or
an object without having their own credentials.
you continue to have access to the bucket or object as long as the creator
of the request has permissions to access those resources.
Go to > storage →Buckets→Create Bucket→Pre-Authenticated Requests
Copy the URL as below it will look like
https://objectstorage.ap-mumbai- 1.oraclecloud.com/p/EcXPtgNE6vRxSwJB8buWVXpW-- KNUTviigbc0QlLcQ2IvJXqE5hrZQPC2hsoxkxt/n/idazzjlcjqzj/b/bucket-csv- file-upload/o/
Run the CURL command from On-Premises
cURL is one of the tools used to access the REST interface. cURL is an open source tool used for transferring data which supports various protocols including HTTP and HTTPS.
curl -v -X PUT -u '<user>:<token string from step 2>' --upload-file <local file location> <Pre-authenticated Request URL from step 3>
For example:#1. >> create Pre Authentication Request for Objects
curl -v -X PUT -u'chandan.d.kumar@oracle.com:Czhfc(g_8i#h{8CAW2Gr' --upload-file /Users/cjkumar/Documents/Mac-key/bootcamp.pub https://objectstorage.ap-mumbai- 1.oraclecloud.com/p/vDU3iX0C2yOKjNJCkc9yCp4CvYVCrY_FTDy1Yh6yVwqpIn JO4Ve_R-nAefZHhWlx/n/idazzjlcjqzj/b/bucket-fileUpload/o/On-P- MySQLbackupfile
For example:#2. >> create Pre Authentication Request for Bucket
curl -v -X PUT -u'chandan.d.kumar@oracle.com:Czhfc(g_8i#h{8CAW2Gr' --upload-file /Users/cjkumar/Downloads/employee.csv https://objectstorage.ap-mumbai- 1.oraclecloud.com/p/EcXPtgNE6vRxSwJB8buWVXpW-- KNUTviigbc0QlLcQ2IvJXqE5hrZQPC2hsoxkxt/n/idazzjlcjqzj/b/bucket-csv-file-upload/o/
Once done, remember to delete the pre-authenticated Request as it is no longer
needed.
Above screenshot show that , we have successfully uploaded CSV file from On-Premises to Oracle Cloud.
Next Blog , I will walk through how to import CSV file from Object storage into MySQL Database Service
Conclusion
Object storage is very easy way to store the data – could be
structured/unstructured from on-premises to oracle cloud and use with
cloud native applications.
REST API uses cURL which simplifying the task of lift-shift from different
sources into Oracle Cloud object storage.
REST API is used to manage containers and objects in the Oracle Storage
Cloud Service instance.
Anyone can access the REST API from any application or programming
platform that understands the Hypertext Transfer Protocol (HTTP) and has
Internet connectivity.
