Tuesday, January 18, 2022

How to Migrate From Azure for MySQL to MDS

Let's walk through on how to migrate sakila database from Azure for MySQL into Oracle Cloud MySQL Database Service(MDS).

For the migration ,let's use MySQL Shell Utility features

1.       Dump Schema Utility

a.       This will help us to export the schema of Azure for MySQL Instance into Oracle Cloud Object Storage.

2.       Load Dump Utility

a.       This is help us to Import the schema from Object Storage to MySQL Database Service(MDS) at Oracle Cloud .

 

How does Migration Work ?

Suppose you wanted to do lift-shift of database called “Sakila” , so “Schema Dump” utility will export sakila database from sources(in this example ,source is AZURE Cloud) to OCI(oracle cloud infrastructure) object storage.

Then “Load Dump” utility will import directly to MySQL Database Service running in Oracle Cloud as which is managed service of MySQL.

Below diagram I have made for clear understanding…






What do we needed handy ?

1.       MySQL Shell 8.0.x Version .

2.       Azure for MySQL Up and Running.

3.       MySQL database service(MDS) Up and Running.

4.       Install OCI CLI on Azure VM Machine.

5.       Install OCI CLI on OCI(Oracle Cloud Compute Instance)

6.       local_infile variables must be ON  for destination machine.

How to Launch MySQL database service (MDS) ?

Below is my another blog will help us to create MDS instance:-

https://mysqlsolutionsarchitect.blogspot.com/

 Additional Details

Azure VM details

Azure for MySQL Details

Oracle Cloud Compute Instance Details

MDS

Details

Public IP address:-

20.198.96.xx

SSH PPK file

User: azureuser

Database Name:- Sakila

End point IP:- azuremysql-dbs.mysql.database.azure.com

User:- admin123

Port: 3306

No of Tables: 23

Public IP Address: 

SSH ppk file

User:- opc

MDS Private IP

Address:- 10.0.0.xx

 

 First Install OCI CLI in Azure compute instance

bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)"

 

exec -l $SHELL

 cd /root/bin/

oci setup config

>>follow the steps

cat /root/.oci/oci_api_key_public.pem

 Paste the o/p of public key into OCI console->user settingsà paste

 


 

 

Command to Export the backup from Azure for MySQL to OCI Object Storage

Open MySQL Shell and connect to Azure compute instance(VM) then execute below commands

Connect azure for mysql Instance

 mysqlsh -hazuremysql-dbs.mysql.database.azure.com -uadmin123@azuremysql-dbs -pWelcome1!

[root@chandan-vm oltp_legacy]# mysqlsh -hazuremysql-dbs.mysql.database.azure.com -uadmin123@azuremysql-dbs -pWelcome1!

MySQL Shell 8.0.23

 Copyright (c) 2016, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates.

Other names may be trademarks of their respective owners.

 Type '\help' or '\?' for help; '\quit' to exit.

WARNING: Using a password on the command line interface can be insecure.

Creating a session to 'admin123%40azuremysql-dbs@azuremysql-dbs.mysql.database.azure.com'

Fetching schema names for autocompletion... Press ^C to stop.

Your MySQL connection id is 64630

Server version: 8.0.15 Source distribution

No default schema selected; type \use <schema> to set one.

 MySQL  azuremysql-dbs.mysql.database.azure.com:3306 ssl  JS >


Check for Compatibility issues and gather the problems in advance before migration by using DRYRUN

 

MySQL  azuremysql-dbs.mysql.database.azure.com:3306 ssl  JS > util.dumpSchemas(["sakila"], "sakilaDBdump", {dryRun: true, ocimds: true})

MySQL  azuremysql-dbs.mysql.database.azure.com:3306 ssl  JS > util.dumpSchemas(["sakila"], "sakilaDBdump", {dryRun: true, ocimds: true})

Acquiring global read lock

Global read lock acquired

Gathering information - done

All transactions have been started

Locking instance for backup

NOTE: Backup lock is not supported in MySQL 5.7 and DDL changes will not be blocked. The dump may fail with an error or not be completely consistent if schema changes are made while dumping.

Global read lock has been released

Checking for compatibility with MySQL Database Service 8.0.23

NOTE: MySQL Server 5.7 detected, please consider upgrading to 8.0 first. You can check for potential upgrade issues using util.checkForServerUpgrade().

ERROR: Procedure sakila.film_not_in_stock - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

ERROR: Procedure sakila.rewards_report - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

ERROR: Procedure sakila.film_in_stock - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

ERROR: View sakila.nicer_but_slower_film_list - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

ERROR: View sakila.sales_by_film_category - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

ERROR: View sakila.actor_info - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

ERROR: View sakila.film_list - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

ERROR: View sakila.sales_by_store - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

ERROR: View sakila.staff_list - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

ERROR: View sakila.customer_list - definition uses DEFINER clause set to user `admin123`@`%` which can only be executed by this user or a user with SET_USER_ID or SUPER privileges (fix this with 'strip_definers' compatibility option)

Compatibility issues with MySQL Database Service 8.0.23 were found. Please use the 'compatibility' option to apply compatibility adaptations to the dumped DDL.

Util.dumpSchemas: Compatibility issues were found (RuntimeError)

util.dumpSchemas(["sakila"], "sakilaDBdump", {dryRun: true, ocimds: true, "compatibility": ["strip_definers", "strip_restricted_grants"]})

 

FIX:-

MySQL  azuremysql-dbs.mysql.database.azure.com:3306 ssl  JS > util.dumpSchemas(["sakila"], "sakilaDBdump", {dryRun: true, ocimds: true, "compatibility": ["strip_definers", "strip_restricted_grants"]})

Acquiring global read lock

Global read lock acquired

Gathering information - done

All transactions have been started

Locking instance for backup

NOTE: Backup lock is not supported in MySQL 5.7 and DDL changes will not be blocked. The dump may fail with an error or not be completely consistent if schema changes are made while dumping.

Global read lock has been released

Checking for compatibility with MySQL Database Service 8.0.23

NOTE: MySQL Server 5.7 detected, please consider upgrading to 8.0 first. You can check for potential upgrade issues using util.checkForServerUpgrade().

Writing DDL for table `sakila`.`inventory`

Writing DDL for table `sakila`.`rental`

Writing DDL for table `sakila`.`category`

Writing DDL for table `sakila`.`address`

Writing DDL for table `sakila`.`city`

 MySQL  azuremysql-dbs.mysql.database.azure.com:3306 ssl  JS >

MySQL  azuremysql-dbs.mysql.database.azure.com:3306 ssl  JS >util.dumpSchemas(["sakila"],"sakilaDBdump",{threads:30,"osBucketName": "chandanBucket", "osNamespace": "idazzjlcjqzj",  "ocimds": "true","ociConfigFile":"/root/.oci/config", "compatibility": ["strip_definers", "strip_restricted_grants"]})

Output

 

MySQL  azuremysql-dbs.mysql.database.azure.com:3306 ssl  JS > util.dumpSchemas(["sakila"],"sakilaDBdump",{threads:30,"osBucketName": "chandanBucket", "osNamespace": "idazzjlcjqzj",  "ocimds": "true","ociConfigFile":"/root/.oci/config", "compatibility": ["strip_definers", "strip_restricted_grants"]})

Acquiring global read lock

Global read lock acquired

Gathering information - done

All transactions have been started

Locking instance for backup

Data dump for table `sakila`.`payment` will be written to 1 file

Data dump for table `sakila`.`film_text` will be written to 1 file

Data dump for table `sakila`.`film_actor` will be written to 1 file

Data dump for table `sakila`.`rental` will be written to 1 file

Data dump for table `sakila`.`inventory` will be written to 1 file

Data dump for table `sakila`.`category` will be written to 1 file

Data dump for table `sakila`.`address` will be written to 1 file

Data dump for table `sakila`.`city` will be written to 1 file

1 thds dumping - 100% (46.27K rows / ~46.27K rows), 1.61K rows/s, 96.26 KB/s uncompressed, 0.00 B/s compressed

Duration: 00:00:01s

Schemas dumped: 1

Tables dumped: 16

Uncompressed data size: 2.92 MB

Compressed data size: 557.10 KB

Compression ratio: 5.2

Rows written: 46273

Bytes written: 557.10 KB

Average uncompressed throughput: 1.87 MB/s

Average compressed throughput: 356.39 KB/s

 MySQL  azuremysql-dbs.mysql.database.azure.com:3306 ssl  JS >

 

You may cross check whether DUMP has been exported to Oracle Cloud Object Storage or not

 





Import Dump file into MySQL Database Service (MDS), from OCI Object Storage

 

First is to install OCI CLI by following below web links

https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm

 For import into a MySQL DB Service (MDS), the MySQL Shell instance where you run the dump loading utility must be installed on an Oracle Cloud Infrastructure Compute instance that has access to the MySQL DB System.

Let’s connect Oracle Cloud Compute Instance ,a nd check whether MDS Instance you are able to connect or not ?

How to connect Compute Instance- open Putty and Public IP into it , see below screen shot.
















How to Deploy MDS Instance ? Below is my another blog which will help us to get through.

https://mysqlsolutionsarchitect.blogspot.com/2020/09/how-to-launch-mysql-database-service.html

Let’s Connect MDS instance ,

sudo su root

mysql -h10.0.1.3 -uadmin –pXXXXX

 

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mysql              |

| performance_schema |

| sys                |

+--------------------+

4 rows in set (0.00 sec)

 

Connect MySQL Shell to Import

# mysqlsh -h10.0.1.3 -uadmin -pWelcome1!






DRY RUN scripts:-

MySQL  10.0.1.3:33060+ ssl  JS >util.loadDump("sakilaDBdump",{dryRun: true, osBucketName: "chandanBucket", osNamespace: "idazzjlcjqzj","ociConfigFile":"/root/.oci/config"})

OUTPUT

 MySQL  10.0.1.3:33060+ ssl  JS > util.loadDump("sakilaDBdump",{dryRun: true, osBucketName: "chandanBucket", osNamespace: "idazzjlcjqzj","ociConfigFile":"/root/.oci/config"})

Loading DDL and Data from OCI ObjectStorage bucket=chandanBucket, prefix='sakilaDBdump' using 4 threads.

Opening dump...

dryRun enabled, no changes will be made.

Target is MySQL 8.0.23-u2-cloud. Dump was produced from MySQL 8.0.15

Fetching dump data from remote location...

Fetching 16 table metadata files for schema `sakila`...

Checking for pre-existing objects...

Executing common preamble SQL

Executing DDL script for schema `sakila`

[Worker001] Executing DDL script for `sakila`.`city`

[Worker003] Executing DDL script for `sakila`.`category`

[Worker002] Executing DDL script for `sakila`.`sales_by_store` (placeholder for view)

[Worker003] Executing DDL script for `sakila`.`staff_list` (placeholder for view)

[Worker000] Executing DDL script for `sakila`.`actor_info` (placeholder for view)

Executing common postamble SQL

 No data loaded.

0 warnings were reported during the load.

 MySQL  10.0.1.3:33060+ ssl  JS >

 

 Script to Import DUMP file from Object Storage to MySQL Database System


util.loadDump("sakilaDBdump",{osBucketName: "chandanBucket", osNamespace: "idazzjlcjqzj","ociConfigFile":"/root/.oci/config"})

 

OUTPUT

 

MySQL  10.0.1.3:33060+ ssl  JS > util.loadDump("sakilaDBdump",{osBucketName: "chandanBucket", osNamespace: "idazzjlcjqzj","ociConfigFile":"/root/.oci/config"})

Loading DDL and Data from OCI ObjectStorage bucket=chandanBucket, prefix='sakilaDBdump' using 4 threads.

Opening dump...

Target is MySQL 8.0.23-u2-cloud. Dump was produced from MySQL 8.0.15

Fetching dump data from remote location...

Fetching 16 table metadata files for schema `sakila`...

[Worker000] sakila@payment@@0.tsv.zst: Records: 16049  Deleted: 0  Skipped: 0  Warnings: 0

Executing common postamble SQL

 15 chunks (46.27K rows, 2.92 MB) for 15 tables in 1 schemas were loaded in 2 sec (avg throughput 1.46 MB/s)

0 warnings were reported during the load.

 MySQL  10.0.1.3:33060+ ssl  JS >

Finally Import Has been done successfully.

Let’s verify The results

MySQL  10.0.1.3:33060+ ssl  JS > \sql

Switching to SQL mode... Commands end with ;

 MySQL  10.0.1.3:33060+ ssl  SQL > show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mysql              |

| performance_schema |

| sakila             |

| sys                |

+--------------------+

5 rows in set (0.0020 sec)

 MySQL  10.0.1.3:33060+ ssl  SQL > use sakila;

Default schema set to `sakila`.

Fetching table and column names from `sakila` for auto-completion... Press ^C to stop.

 MySQL  10.0.1.3:33060+ ssl  sakila  SQL > select count(*) from city;

+----------+

| count(*) |

+----------+

|      600 |

+----------+

1 row in set (0.0025 sec)

 MySQL  10.0.1.3:33060+ ssl  sakila  SQL > show tables;

+----------------------------+

| Tables_in_sakila           |

+----------------------------+

| actor                      |

| actor_info                 |

| address                    |

| category                   |

| city                       |

| country                    |

| customer                   |

| customer_list              |

| film                       |

| film_actor                 |

| film_category              |

| film_list                  |

| film_text                  |

| inventory                  |

| language                   |

| nicer_but_slower_film_list |

| payment                    |

| rental                     |

| sales_by_film_category     |

| sales_by_store             |

| staff                      |

| staff_list                 |

| store                      |

+----------------------------+

23 rows in set (0.0019 sec)

 MySQL  10.0.1.3:33060+ ssl  sakila  SQL >

 Conclusion:-

Migration happened successfully!!!

MySQL Shell utility makes MySQL easier to use, by providing an interactive MySQL client supporting SQL, Document Store, JavaScript & Python interface with support for writing custom extensions.

MySQL Database Service is a fully managed database service that enables organizations to deploy cloud-native applications using the world's most popular open source database. It is 100% developed, managed and supported by the MySQL Team.


Friday, May 21, 2021

How to run MySQL Enterprise Edition on Oracle Cloud Infrastructure from Marketplace ?

 Deploy MySQL Enterprise Edition with OCI Market Place Application


In this blog we will look into the process to deploy MySQL Enterprise Edition in to Oracle Cloud Infrastructure.

In this blog we will discuss the below agenda
1. Introduction about MySQL EE and OCI Market place

2. Deploy MySQL EE into Oracle Cloud Infrastructure

3. Accessing MySQL database from On-premises.

4. Licensing/Purchase model of using MySQL EE from OCI marketplace

5. Conclusion

Introduction

If you wanted to run MySQL Enterprise Edition (EE) in Oracle Cloud Compute Instance (VM) then generally what we do is first download the MySQL EE binaries from the oracle website and manually push to the Oracle Cloud Instance where, MySQL EE needed to be installed and follow the steps to get installed.
Another approach is going with MySQL EE image from Oracle Cloud Marketplace.
It automates this manual process to download the binaries and push to oracle cloud and then follow the step to install the MySQL binaries, it is pre-loaded package which by default install below software
 MySQL Server EE
 MySQL Enterprise Backup
 MySQL Shell
 MySQL Router
 MySQL Enterprise Thread pool

The MySQL Enterprise Edition Marketplace Application is an OCI compute instance, running Oracle Linux 7.7, with MySQL EE 8.0.x.
The MySQL EE installation on the deployed image is similar to the RPM installation.

This solutions is user-managed, meaning you are responsible for upgrades and maintenance.

MySQL Enterprise Edition version 8.0 is available in OCI Marketplace. MySQL 5.7 nor 5.6 are not available in Oracle Cloud Infrastructure Marketplace. If the user needs previous versions, you need to create an OCI Compute Linux, download and install MySQL EE by themselves, similar to on-prem.

What is Oracle Cloud Infrastructure Marketplace?
Oracle Cloud Infrastructure Marketplace is an online store that offers solution specifically for customers of Oracle Cloud Infrastructure.
you can find listings for two types of solutions from Oracle and trusted partners:

1. Image
Images are templates of virtual hard drives that determine the operating system and software to run on an instance. You can deploy image listings on an Oracle Cloud Infrastructure Compute instance.

2. Stack

Stacks represent definitions of groups of Oracle Cloud Infrastructure resources that you can act on as a group.
More info:- https://docs.oracle.com/en-us/iaas/Content/Marketplace/Concepts/marketoverview.htm

What is MySQL Enterprise Edition?

MySQL Enterprise Edition is a commercial product. Like MySQL Community Edition, MySQL Enterprise Edition includes MySQL Server, a fully integrated transaction-safe, ACID-compliant database with full commit, rollback, crash-recovery, and row-level locking capabilities. In addition, MySQL Enterprise Edition includes the following components designed to provide monitoring and online backup, as well as improved security and scalability, high availability.

More info:- https://dev.mysql.com/doc/refman/8.0/en/mysql-enterprise.html

Deploy MySQL EE into Oracle Cloud Infrastructure

Prerequisite:-

 Make sure You have have create an account with Oracle Cloud.
 Make sure virtual cloud network(VCN) is created
https://docs.oracle.com/en-us/iaas/Content/GSG/Tasks/creatingnetwork.htm

Step 1:- Create compute instance 





































Step 2 : Change the shape based on the requirements(in terms of CPU , RAM )



Step 03:-Configure the Network


Step 04:- Add SSH key files


Step 05 :- Create the Instance



Step 06:- View the created Instance


Step 07:- Note down the Instance details

Instance Public IP : 140.238.166.106
Instance User Name:   opc
Instance Private IP: 10.0.0.92
MySQL Host Name: localhost
MySQL UserName: root
MySQL Port: 3306


Step 08:- Let’s connect the Instance from On-premises network using Putty


Make sure putty is installed in your local machine, if not then please install by downloading from the web links:- https://www.putty.org/
Enter the correct public IP address as well as .ppk file to connect , refer below screenshot .










































When you click on OPEN, you will see that you are connected to Oracle Cloud Infrastructure.

















Step 09:- Get the temporary password

$sudo grep 'temporary password' /var/log/mysqld.log









Step 10:- Login to MySQL DB























Step 11:- Change the Password for the root user from temporary password to NEW password

alter user root@'localhost' identified by 'MySQL8.0';

Step 12:- Next is you can ready to execute the commands with MySQL Enterprise Edition







That’s it.
You can also connect MySQL Server using MySQL shell utility




So you may noticed that I have not install manually any mysql clients , libraries its dependencies , MySQL shell because it is bundled and available as pre-installed image at Oracle Cloud Marketplace.
It’s easy to plug in and play with MySQL Enterprise Edition.
More info available at MySQL Website:-

https://dev.mysql.com/doc/refman/8.0/en/mysql-oci-marketplace.html


Licensing Model

This is a BYOL (bring your own license) product with there are additional fees for the infrastructure usage."
Hence,

Price of MySQL EE OCI Marketplace = Price of MySQL EE + Price of Oracle Cloud Infrastructure VM usage

Price for MySQL EE goes via MySQL Sales , you can talk with sales to get the details pricing matrix(https://www.mysql.com/about/contact/).

More info :- https://www.mysql.com/products/

Price of Oracle Cloud Infrastructure VM usage, it depends on kind of shape (comprises of OCPU, RAM, Storage) you are going to configured to run MySQL Enterprise Edition (EE).

You can calculate using OCI cost estimator: - https://www.oracle.com/in/cloud/cost-estimator.html

If you are bringing your “on-premise license”, so this offering is available at no additional charge. Fees for usage of Oracle infrastructure resources, such as compute instances, are charged separately and based on the rate of usage. Learn about pricing" linked to

https://www.oracle.com/cloud/compute/pricing.html


Conclusion:-

MySQL Enterprise Edition include most comprehensive set of advanced features (solutions for scalability, performance, high availability, securities), management tools (unified monitoring and hot/online backup) and technical supports(24*7 support backed by MySQL engineering team)
It’s easy to deploy on private cloud , public cloud and available as “Managed Service” at Oracle Cloud named as MySQL Database Service (MDS).
MySQL Enterprise Edition is BYOL product and it is annual subscription product which has to be renewed annually.