Automate File Processing with S3 Event Notifications

Consider a scenario where you need to perform some kind of automated processing on files that arrive at a particular location as soon as they come in. It could be as small as two business processes talking to each other by dropping files at a predetermined location or as large as two businesses exchanging EDI files at an FTP server.

In the pre-cloud days, this would be accomplished by building a watcher service that polls the location for new files & takes actions when it finds something. With Amazon S3 though, all it takes is a few clicks to configure an “event notification.”

As shown below, event notifications can be configured for various S3 events, the most common use case being object create events. The notifications can be sent to either an SNS topic, an SQS queue or you can have a Lambda function execute your custom business logic on the file.

You can provide a prefix if you only want notifications for object events in a particular directory. You can also provide suffix to get notified of only certain file types (.csv, .png, etc.). All event notifications, regardless of their configured destination, always contain all details needed for a receiver to identify, fetch & process the file that triggered the notification. For more details, see the official documentation.

Harish KM is a Cloud Evangelist & a Full Stack Engineer at QloudX. He is very passionate about cloud-native solutions & using the best tools for his projects. With 10+ cloud & IT certifications, he is an expert in a multitude of application languages & is up-to-date with all new offerings & services from cloud providers, especially AWS.

Leave a Reply