我想為我的 s3 存盤桶中的每個物件或鍵生成一條訊息。我確實有十把鑰匙。這就是為什么我想用 list_objects_v2 列出所有這些，然后將它們傳遞給 SQS 佇列。下面是我嘗試使用的代碼示例：
import json
import boto3
region = "us-east-2"
bucket = "s3-small-files-fiap"
prefix = 'folder/'
s3_client = boto3.client('s3', region_name=region)
response = s3_client.list_objects_v2(Bucket=bucket,
Prefix=prefix)
settings = {
"bucket_name": "s3-small-files-fiap",
"queue_name": "sqs-csv-to-json",
"region": region,
"account_number": <my_account_number>
}
bucket_notifications_configuration = {
'QueueConfigurations': [{
'Events': ['s3:ObjectCreated:*'],
'Id': 'Notifications',
'QueueArn':
'arn:aws:sqs:{region}:{account_number}:{queue_name}'.format(**settings)
}]
}
qpolicy = {
"Version": "2012-10-17",
"Id":
"arn:aws:sqs:{region}:{account_number}:{queue_name}/SQSDefaultPolicy".format(
**settings),
"Statement": [{
"Sid": "allow tmp bucket to notify",
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": "SQS:SendMessage",
"Resource": "arn:aws:sqs:{region}:{account_number}:{queue_name}".format(
**settings),
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:{bucket_name}".format(
**settings)
}
}
}]
}
print("Bucket notify", bucket_notifications_configuration)
print("Queue Policy", qpolicy)
queue_attrs = {"Policy": json.dumps(qpolicy), }
sqs_client = boto3.resource("sqs",
region_name=region).get_queue_by_name(
QueueName=settings["queue_name"])
sqs_client.set_attributes(Attributes=queue_attrs)
sqs_client.attributes
s3_client.put_bucket_notification_configuration(
Bucket=bucket,
NotificationConfiguration=bucket_notifications_configuration)
出于某種原因，它的輸出只生成一條通知訊息，如下所示。如何使用上面的代碼發送通知十次而不是一次？
這是輸出的示例：
Bucket notify {'QueueConfigurations': [{'Events': ['s3:ObjectCreated:*'], 'Id': 'Notifications', 'QueueArn': 'arn:aws:sqs:us-east-2:<my_account_number>:sqs-csv-to-json'}]}
Queue Policy {'Version': '2012-10-17', 'Id': 'arn:aws:sqs:us-east-2:<my_account_number>:sqs-csv-to-json/SQSDefaultPolicy', 'Statement': [{'Sid': 'allow tmp bucket to notify', 'Effect': 'Allow', 'Principal': {'AWS': '*'}, 'Action': 'SQS:SendMessage', 'Resource': 'arn:aws:sqs:us-east-2:<my_account_number>:sqs-csv-to-json', 'Condition': {'ArnLike': {'aws:SourceArn': 'arn:aws:s3:*:*:s3-small-files-fiap'}}}]}
uj5u.com熱心網友回復：
創建一個方法并移動您的 SNS 發送代碼
QUEUE_NAME = os.getenv("QUEUE_NAME")
SQS = boto3.client("sqs")
## Inside handler method
s3_resource = boto3.resource('s3')
response = s3_client.list_objects_v2(Bucket=bucket,Prefix=prefix)
for file in response:
# call SQS send method here
try:
#logger.debug("Recording %s", file)
u = getQueueURL()
logging.debug("Got queue URL %s", u)
resp = SQS.send_message(QueueUrl=u, MessageBody=file)
#logger.debug("Send result: %s", resp)
except Exception as e:
raise Exception("Raised Exception! %s" % e)
def getQueueURL():
"""Retrieve the URL for the configured queue name"""
q = SQS.get_queue_url(QueueName=QUEUE_NAME).get('QueueUrl')
#logger.debug("Queue URL is %s", QUEUE_URL)
return q
uj5u.com熱心網友回復：
我按照建議的示例進行操作，代碼如下： import boto3 import logging import os
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO,
format='%(asctime)s: %(levelname)s: %(message)s')
queue_name = 'https://sqs.us-east-2.amazonaws.com/<my account>/sqs-csv-to-json'
sqs_client = boto3.client('sqs', region_name='us-east-2')
s3_client = boto3.client('s3', region_name='us-east-2')
bucket = "s3-small-files-fiap"
prefix = 'folder/'
## Inside handler method
#
def listingObjects(bucket, prefix):
response = s3_client.list_objects_v2(Bucket=bucket,Prefix=prefix)
for file in response:
# call SQS send method here
try:
#logger.debug("Recording %s", file)
u = getQueueURL()
logging.debug("Got queue URL %s", u)
resp = sqs_client.send_message(QueueUrl=u, MessageBody=file)
#logger.debug("Send result: %s", resp)
except Exception as e:
raise Exception("Raised Exception! %s" % e)
else:
return resp
def getQueueURL():
"""Retrieve the URL for the configured queue name"""
q = sqs_client.get_queue_url(QueueName=queue_name).get('QueueUrl')
#logger.debug("Queue URL is %s", QUEUE_URL)
return q
但我面臨著這樣的輸出例外：
2022-09-11 13:33:06,565: INFO: Found credentials in shared credentials file: ~/.aws/credentials
Traceback (most recent call last):
File "/home/felipediego/Studies/FIAP/teste6.py", line 25, in listingObjects
u = getQueueURL()
File "/home/felipediego/Studies/FIAP/teste6.py", line 36, in getQueueURL
q = sqs_client.get_queue_url(QueueName=queue_name).get('QueueUrl')
File "/home/felipediego/.local/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/felipediego/.local/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.QueueDoesNotExist: An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the GetQueueUrl operation: The specified queue does not exist for this wsdl version.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/felipediego/Studies/FIAP/teste6.py", line 41, in <module>
list_objects = listingObjects(bucket, prefix)
File "/home/felipediego/Studies/FIAP/teste6.py", line 30, in listingObjects
raise Exception("Raised Exception! %s" % e)
Exception: Raised Exception! An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the GetQueueUrl operation: The specified queue does not exist for this wsdl version.
uj5u.com熱心網友回復：
更改佇列名稱后，我面臨一種新的錯誤，即
2022-09-11 21:42:52,097: INFO: Found credentials in shared credentials file: ~/.aws/credentials
Traceback (most recent call last):
File "/home/felipediego/Studies/FIAP/teste6.py", line 33, in listingObjects
resp = sqs_client.send_message(QueueUrl=u, MessageBody=file)
File "/home/felipediego/.local/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/felipediego/.local/lib/python3.8/site-packages/botocore/client.py", line 878, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/home/felipediego/.local/lib/python3.8/site-packages/botocore/client.py", line 939, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/home/felipediego/.local/lib/python3.8/site-packages/botocore/validate.py", line 381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter QueueUrl, value: {'QueueUrl': 'https://us-east-2.queue.amazonaws.com/<my_account>/sqs-csv-to-json', 'ResponseMetadata': {'RequestId': '9d160995-f088-5346-9c34-520d724565a1', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '9d160995-f088-5346-9c34-520d724565a1', 'date': 'Mon, 12 Sep 2022 00:42:53 GMT', 'content-type': 'text/xml', 'content-length': '337'}, 'RetryAttempts': 0}}, type: <class 'dict'>, valid types: <class 'str'>
轉載請註明出處，本文鏈接：https://www.uj5u.com/gongcheng/506174.html
標籤：Python 亚马逊网络服务 亚马逊-s3 博托3 亚马逊广场