Few weeks back, we decided to migrate our complete infrastructure to a single AWS region (which is closer to our client) for achieving better latencies and less data transfer cost. There were multiple components migrated to a single AWS region and ElasticSearch was one of them. Although migration steps are documented on ElasticSearch website, there are multiple AWS concepts one needs to understand  to execute it properly. Here, we have tried to dive deep into migration steps and login behind those steps. ElasticSearch data migration is trivial and only requires knowledge of ElasticSearch in case of a self managed publicly accessible cluster but things get tricky when it comes to AWS managed clusters as AWS doesn’t allow you to access config yml file and there are a lot of other AWS IAM concepts one needs to understand for executing it  properly. In this blog we are going to discuss the step-by-step AWS managed ElasticSearch data migration process using  snapshot and restore methods  and how it is different from self hosted ElasticSearch data migration in detail. ## Overview **Assumtion**: ElasticSearch domain need to be created in destination region with same cluster configuration. - Take manual snapshot of  ElasticSearch domain data into source S3 bucket - Copy source S3 bucket data into destination S3 bucket - Restore data from S3 bucket into new ElasticSearch domain ### Take snapshot _**Snapshot**: A snapshot is a backup taken from a running ElasticSearch cluster. You can take snapshots of an entire cluster, including all its data streams and indices. You can also take snapshots of only specific data streams or indices in the cluster._ _**Repository**: The repository is a place where you can store all your snapshots data._ **Note**: You must register a snapshot repository before you can perform snapshot and restore operations.You have multiple options available for defining a repository given that both old and new ElasticSearch should be able to access that location like a shared file system, AWS S3, HDFS etc. We will be discussing only the AWS S3 option in this blog post. ### Register snapshot repository using AWS S3 bucket > ElasticSearch provides APIs for both registering repositories and taking snapshots. ### How roles and user works in AWS - **Role**: An IAM role is an AWS Identity and Access Management (IAM) entity with permissions to make AWS service requests. IAM roles cannot make direct requests to AWS services; they are meant to be assumed by authorized entities, such as IAM users, applications, or AWS services such as EC2, ES etc - **AssumeRole**: Trusted entity can assume role and performs authorized action using temporary credentials. - **PassRole** To allow a user to delegate role to an AWS service, you must grant the PassRole permission to the user. Let’s create a role to delegate permission to Amazon Elasticsearch Service to take a snapshot into S3. ### Create S3 Bucket and Configure IAM Role 1. Create S3 bucket in the same region where the ElasticSearch domain exists and another bucket in destination region. 2. Create a new role theSnapshotRole without any permissions. 3. Create below Policy and add it to the newly created role. It will enable ElasticSearch to access both S3 bucket. 4. Update trust relationship for theSnapshotRole role. **Note**: We need to update trust relationships for newly created roles which will enable ElasticSearch to assume roles. This assume role permission will allow ElasticSearch service to use theSnapshotRole as its own. ### Configure IAM user > Once we are done with role creation, we need to create new user or we need to give PassRole permission to existing user which will allow user to pass newly created theSnapshotRole to ElasticSearch service and as theSnapshotRole already configured with assumeRole permission, it can access S3 bucket. - Create below policy and attach it with user ### Register snapshot repository Once we are done with the new role and user creation, let's start with repository registration. There are two ways of registering repositories on ElasticSearch. 1. Using postman if your ElasticSearch cluster can be accessed directly. ### Postman 2. If your ElasticSearch cluster is in a VPC that you cannot access directly, you will need access to EC2 which is in the same VPC and that can execute below scripts. ### Python script ### Response ### Take snapshot of current ElasticSearch data After registering repository successfully, we need to run below script for taking snapshot of current ElasticSearch data. ### Response ### Verify snapshot ### Copy S3 data to new region s3 You can copy s3 bucket data using AWS console > **aws s3 cp** s3://awsexamplesourcebucket s3://awsexampledestinationbucket ### Restore Snapshot 1. Register current snapshot repository with new ElasticSearch domain 2. Restore snapshot of old ElasticSearch data into new ElasticSearch 3. Verify restored indexes and alias. 4. Verify index data ### Take-away! - AWS managed ElasticSearch provides us with a lot of benefits in terms of managing and scaling our ElasticSearch cluster but also restricts us to access configuration files which leaves us with snapshot-restore method when we want to migrate from one domain to another. Snapshot-restore method has its own downside in terms of write downtime during the migration process which is ranging from a few minutes to hours depending on data size. - Make sure you are running your services using ElasticSearch  in read-only mode during the migration process to avoid data loss. ## References: [Migrate data from an Amazon ES domain\\ \\ Amazon Web Services, Inc.\\ \\ [Migrating from Amazon Elasticsearch Service \| Elasticsearch Service Documentation \| Elastic\\ \\ Elastic\\ \\ Disqus Comments We were unable to load Disqus. If you are a moderator please see our [troubleshooting guide](https://docs.disqus.com/help/83/). G Join the discussion… Comment ###### Log in with ###### or sign up with Disqus or pick a name ### Disqus is a discussion network - Don't be a jerk or do anything illegal. Everything is easier that way. [Read full terms and conditions](https://docs.disqus.com/kb/terms-and-policies/) This comment platform is hosted by Disqus, Inc. I authorize Disqus and its affiliates to: - Use, sell, and share my information to enable me to use its comment services and for marketing purposes, including cross-context behavioral advertising, as described in our [Terms of Service](https://help.disqus.com/customer/portal/articles/466260-terms-of-service) and [Privacy Policy](https://disqus.com/privacy-policy), including supplementing that information with other data about me, such as my browsing and location data. - Contact me or enable others to contact me by email with offers for goods or services - Process any sensitive personal information that I submit in a comment. See our [Privacy Policy](https://disqus.com/privacy-policy) for more information Acknowledge I am 18 or older - [5](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default# "Favorite this discussion") - ## Discussion Favorited! Favoriting means this is a discussion worth sharing. It gets shared to your followers' Disqus feeds, and gives the creator kudos! [Find More Discussions](https://disqus.com/home/?utm_source=disqus_embed&utm_content=recommend_btn) [Share](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default#) - Tweet this discussion - Share this discussion on Facebook - Share this discussion via email - Copy link to discussion - [Best](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default#) - [Newest](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default#) - [Oldest](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default#) - - [−](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default# "Collapse") - [+](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default# "Expand") - [Flag as inappropriate](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default# "Flag as inappropriate") isn't iam user require access/grant **_es:ESHttpPut_** action, attached as a policy to the IAM user? And to the last part to restore snapshot of old ElasticSearch data into new ElasticSearch (**2**.), which _elasticsearch-domain-endpoint_ are we suppose to add? is old one or new one? tried the new or destination domain endpoint but not works for me. Thank you btw for these clear and wonderful instruction. see more [Show more replies](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default#) [Load more comments](https://disqus.com/embed/comments/?base=default&f=eng-blackbuck-com&t_i=ghost-5fc4ad1b616671064aa61892&t_u=https%3A%2F%2Feng.blackbuck.com%2Faws-managed-elasticsearch-data-migration-to-another-domain-2%2F&t_d=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&t_t=AWS%20Managed%20ElasticSearch%20Data%20Migration%20to%20another%20domain&s_o=default#) live.rezync.com # live.rezync.com is blocked This page has been blocked by an extension - Try disabling your extensions. ERR\_BLOCKED\_BY\_CLIENT Reload This page has been blocked by an extension pippio.com # pippio.com is blocked This page has been blocked by an extension - Try disabling your extensions. ERR\_BLOCKED\_BY\_CLIENT Reload This page has been blocked by an extension