{"id":5792,"date":"2023-04-28T09:30:00","date_gmt":"2023-04-28T07:30:00","guid":{"rendered":"https:\/\/blog.besharp.it\/?p=5792"},"modified":"2024-01-18T09:13:22","modified_gmt":"2024-01-18T08:13:22","slug":"when-amazon-s3-is-not-enough-a-look-into-storage-services-on-aws","status":"publish","type":"post","link":"https:\/\/blog.besharp.it\/when-amazon-s3-is-not-enough-a-look-into-storage-services-on-aws\/","title":{"rendered":"When Amazon S3 is not enough: a look into storage services on AWS"},"content":{"rendered":"\n
42 is “the answer to life, the universe, and everything else”, as Douglas Adams said in The Hitchhiker’s Guide to the Universe. It goes the same way for storage needs on AWS: Amazon S3 is the answer for everything.<\/p>\n\n\n\n
As we’ll see, this isn’t an absolute truth: sometimes we need to use something different: variety is the spice of life.<\/p>\n\n\n\n
When dealing with lift and shift projects, hybrid environments, and Windows workloads, you’ll always find file shares accessed by users and services. Many off-the-shelf software solutions rely on that technology; sometimes, refactoring is not an option.<\/p>\n\n\n\n
As the name suggests, a file share is not equivalent to object storage: it has different properties, behaviors, and usage scenarios. In this article, we will look at the other options to migrate and adapt workloads on the Cloud. <\/p>\n\n\n\n
First, let’s clarify the key differences between file and object storage.<\/p>\n\n\n\n
File storage is our “traditional” idea: depending on the operating system (Windows, Linux, macOS), you’ll have a system that stores data in a hierarchical structure, identified by name and path. Metadata (like permissions and file properties) is stored separately, and its design depends on the filesystem in use. <\/p>\n\n\n\n
Object storage is different: all data (including metadata and properties) is stored in a flat namespace, accessed using APIs by referencing an identifier. Once data is stored, you need to write a new version to modify the object, as there’s no way to append or delete portions of the data as you do in a traditional filesystem.<\/p>\n\n\n\n
This different approach makes object storage more scalable and cost-effective, but you can’t switch between the two technologies seamlessly.<\/p>\n\n\n\n
What are the options once you have a Windows workload that needs shared file storage? As we always say, we’ll see that there’s no default answer. <\/p>\n\n\n\n
AWS gives us three different services: <\/p>\n\n\n\n
Let’s see what they offer and their use case.<\/p>\n\n\n\n
Amazon S3 File Gateway combines the world of object and file storage, so you could think it unlocks unlimited scalable storage with the famous “11 9s” of durability<\/a>. Unfortunately, as we’ll see, all that glitters is no gold. <\/p>\n\n\n\n With Amazon S3 File Gateway, you can use the SMB protocol to store files in S3, taking advantage of its scalability and automatically mapping API calls.<\/p>\n\n\n\n You can even use lifecycle policies<\/strong> to lower storage costs and archive or delete all files. There are no license costs, and you can deploy the solution into your on-premise environment using its internal cache to speed up access to files.<\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n This solution has some downsides: modifying large data files will create new versions of the S3 object every time, impacting performance and costs. <\/p>\n\n\n\n There’s a hard limit of 50 shares per Amazon S3 File Gateway appliance, and additionally, since its backend storage is S3, Amazon S3 File Gateway uses object metadata to map filesystems attributes. Hence, everything, including File Access Control Entries (ACE), has to be stored in metadata that is limited to 2KB in size. when using robocopy and transferring files with more than 10 Access Control Entries<\/p>\n\n\n\n Let’s see how to overcome these limitations.<\/p>\n\n\n\n Amazon FSx for Windows gives you a fully managed high-performance Windows fileserver compatible with standard Windows Management Instrumentation. You can deploy it in Multi-AZ mode to be highly available, even during the patch operations (managed by AWS).<\/p>\n\n\n\n It supports all the features a native solution can offer, like Shadow Copies, data deduplication, and DFS namespaces, while offloading management operations and automating encryption using AWS KMS. <\/p>\n\n\n\n
The consequence is the error: <\/p>\n\n\n\n\"ERROR 1344 (0x00000540)\" <\/code><\/pre>\n\n\n\n
Amazon FSx for Windows<\/h2>\n\n\n\n