精华内容
下载资源
问答
  • golang sdk调用aws s3对象存储,createmutipartupload 获取uploadid报错.golang sdk调用aws s3对象存储,createmutipartupload 获取uploadid报错.golang sdk调用aws s3对象存储,createmutipartupload 获取uploadid...
  • Golang 调用 aws-sdk 操作 S3对象存储

    千次阅读 2019-09-21 11:52:00
    Golang 调用 aws-sdk 操作 S3对象存储 Golang 调用 aws-sdk 操作 S3对象存储 前言 因为业务问题,要写一个S3对象存储管理代码,由于一直写Go,所以这次采用了Go,Go嘛,快,自带多线程,这种好处就不用多...

    Golang 调用 aws-sdk 操作 S3对象存储

    前言

    因为业务问题,要写一个S3对象存储管理代码,由于一直写Go,所以这次采用了Go,Go嘛,快,自带多线程,这种好处就不用多说了吧。

    基础的功能

    1. 查看S3中包含的bucket
    2. bucket中的文件/文件夹
    3. bucket的删除
    4. bucket的创建
    5. bucket的文件上传
    6. bucket的文件下载
    7. bucket的文件删除

    aws-sdk 的安装

    玩Golang你还能不会那啥?对吧,那啥?那飞机!那飞机场,安上~

    go get github.com/aws/aws-sdk-go

    aws-sdk-go 的基础使用

    构建基础的S3连接

    访问S3的时候,咱们需要access_key,secret_key,对象存储访问IP这三个参数,我们首先要创建一个aws的config,说白了,我们需要定义aws的配置,这样它才知道要怎么访问,去哪里访问等问题。
    构建一个S3连接代码如下

    package main
    import (
        "fmt"
        "os"
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/credentials"
        _ "github.com/aws/aws-sdk-go/service/s3/s3manager"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/service/s3"
    )
    func main() {
    
        access_key := "xxxxxxxxxxxxx"
        secret_key := "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        end_point := "http://xx.xx.xx.xx:7480" //endpoint设置,不要动
        
        sess, err := session.NewSession(&aws.Config{
            Credentials:      credentials.NewStaticCredentials(access_key, secret_key, ""),
            Endpoint:         aws.String(end_point),
            Region:           aws.String("us-east-1"),
            DisableSSL:       aws.Bool(true),
            S3ForcePathStyle: aws.Bool(false), //virtual-host style方式,不要修改
        })
    }

    这时候需要你自己去定义一下access_key,secret_key,end_point这三个参数
    接下来所有的代码,都是以这个连接模板,为核心,后面我就用同上代替配置,请注意!
    所有的代码都传到GIT上了,到时候会给出地址,不懂得copy下来吧!

    查看S3中包含的bucket

    查看所有的bucket

    package main
    import (
        导入包同上
    )
    
    func exitErrorf(msg string, args ...interface{}) {
        fmt.Fprintf(os.Stderr, msg+"\n", args...)
        os.Exit(1)
    }
    
    func main() {
    
        配置同上
    
        svc := s3.New(sess)
        result, err := svc.ListBuckets(nil)
        if err != nil {
            exitErrorf("Unable to list buckets, %v", err)
        }
        
        fmt.Println("Buckets:")
        
        for _, b := range result.Buckets {
            fmt.Printf("* %s created on %s\n",
                aws.StringValue(b.Name), aws.TimeValue(b.CreationDate))
        }
    
        for _, b := range result.Buckets {
            fmt.Printf("%s\n", aws.StringValue(b.Name))
        }
        
    }

    列出bucket中的文件/文件夹

    查看某个bucket中包含的文件/文件夹

    package main
    import (
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/aws/credentials"
        "github.com/aws/aws-sdk-go/service/s3"
        "fmt"
        "os"
    )
    
    func exitErrorf(msg string, args ...interface{}) {
        fmt.Fprintf(os.Stderr, msg+"\n", args...)
        os.Exit(1)
    }
    
    func main() {
    
        配置同上
    
        // bucket后跟,go run ....go bucketname
        bucket := os.Args[1]
        fmt.Printf(bucket)
        fmt.Printf("\n")
    
        svc := s3.New(sess)
    
        params := &s3.ListObjectsInput{
            Bucket:             aws.String(bucket), 
        }
        resp, err := svc.ListObjects(params)
    
        if err != nil {
            exitErrorf("Unable to list items in bucket %q, %v", bucket, err)
        }
        
        for _, item := range resp.Contents {
            fmt.Println("Name:         ", *item.Key)
            fmt.Println("Last modified:", *item.LastModified)
            fmt.Println("Size:         ", *item.Size)
            fmt.Println("Storage class:", *item.StorageClass)
            fmt.Println("")
        }
        
    }

    bucket的创建

    创建bucket

    package main
    
    import (
        导包同上
    )
    
    
    func exitErrorf(msg string, args ...interface{}) {
        fmt.Fprintf(os.Stderr, msg+"\n", args...)
        os.Exit(1)
    }
    
    func main() {
    
        配置同上
    
        bucket := os.Args[1]
    
        if len(os.Args) != 2 {
            exitErrorf("Bucket name required\nUsage: %s bucket_name",
                os.Args[0])
        }
        
    
        // Create S3 service client
        svc := s3.New(sess)
    
        params := &s3.CreateBucketInput{
            Bucket: aws.String(bucket),
     }
    
        _, err = svc.CreateBucket(params)
    
        if err != nil {
            exitErrorf("Unable to create bucket %q, %v", bucket, err)
        }
        
        // Wait until bucket is created before finishing
        fmt.Printf("Waiting for bucket %q to be created...\n", bucket)
        
        err = svc.WaitUntilBucketExists(&s3.HeadBucketInput{
            Bucket: aws.String(bucket),
        })
    
        if err != nil {
            exitErrorf("Error occurred while waiting for bucket to be created, %v", bucket)
        }
        
        fmt.Printf("Bucket %q successfully created\n", bucket)
    }

    bucket的文件上传

    往某个固定的bucket里传文件

    package main
    
    import (
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/aws/credentials"
        "github.com/aws/aws-sdk-go/service/s3/s3manager"
        "fmt"
        "os"
    )
    
    
    func exitErrorf(msg string, args ...interface{}) {
        fmt.Fprintf(os.Stderr, msg+"\n", args...)
        os.Exit(1)
    }
    
    func main() {
    
        配置同上
    
        if len(os.Args) != 3 {
            exitErrorf("bucket and file name required\nUsage: %s bucket_name filename",
                os.Args[0])
        }
        
        bucket := os.Args[1]
        filename := os.Args[2]
        
        file, err := os.Open(filename)
        if err != nil {
            exitErrorf("Unable to open file %q, %v", err)
        }
        
        defer file.Close()
    
        uploader := s3manager.NewUploader(sess)
    
        _, err = uploader.Upload(&s3manager.UploadInput{
            Bucket: aws.String(bucket),
            Key: aws.String(filename),
            Body: file,
        })
        if err != nil {
            // Print the error and exit.
            exitErrorf("Unable to upload %q to %q, %v", filename, bucket, err)
        }
        
        fmt.Printf("Successfully uploaded %q to %q\n", filename, bucket)
    }

    bucket的文件下载

    下载某个bucket中的某个文件

    package main
    
    import (
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/aws/credentials"
        "github.com/aws/aws-sdk-go/service/s3"
        "github.com/aws/aws-sdk-go/service/s3/s3manager"
    
        "fmt"
        "os"
    )
    
    
    func exitErrorf(msg string, args ...interface{}) {
        fmt.Fprintf(os.Stderr, msg+"\n", args...)
        os.Exit(1)
    }
    
    func main() {
    
        配置同上
    
        if len(os.Args) != 3 {
            exitErrorf("Bucket and item names required\nUsage: %s bucket_name item_name",
                os.Args[0])
        }
        
        bucket := os.Args[1]
        item := os.Args[2]
    
        file, err := os.Create(item)
        if err != nil {
            exitErrorf("Unable to open file %q, %v", err)
        }
    
        defer file.Close()
        
        downloader := s3manager.NewDownloader(sess)
    
        numBytes, err := downloader.Download(file,
        &s3.GetObjectInput{
            Bucket: aws.String(bucket),
            Key:    aws.String(item),
        })
    if err != nil {
        exitErrorf("Unable to download item %q, %v", item, err)
    }
    
    fmt.Println("Downloaded", file.Name(), numBytes, "bytes")
    }   

    bucket的文件删除

    删除某个bucket里面的某个文件

    package main
    
    import (
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/aws/credentials"
        "github.com/aws/aws-sdk-go/service/s3"
        "fmt"
        "os"
    )
    
    
    func exitErrorf(msg string, args ...interface{}) {
        fmt.Fprintf(os.Stderr, msg+"\n", args...)
        os.Exit(1)
    }
    
    func main() {
    
        配置同上
    
        if len(os.Args) != 3 {
            exitErrorf("Bucket and object name required\nUsage: %s bucket_name object_name",
                os.Args[0])
        }
        
        bucket := os.Args[1]
        obj := os.Args[2]
        
        svc := s3.New(sess)
    
        _, err = svc.DeleteObject(&s3.DeleteObjectInput{Bucket: aws.String(bucket), Key: aws.String(obj)})
        if err != nil {
            exitErrorf("Unable to delete object %q from bucket %q, %v", obj, bucket, err)
        }
    
        err = svc.WaitUntilObjectNotExists(&s3.HeadObjectInput{
            Bucket: aws.String(bucket),
            Key:    aws.String(obj),
        })
    
        fmt.Printf("Object %q successfully deleted\n", obj)
    }

    代码所在地

    https://github.com/Alexanderklau/Go_poject/tree/master/Go-Storage

    posted on 2019-09-09 12:08  Kilichko 阅读( ...) 评论( ...) 编辑 收藏

    转载于:https://www.cnblogs.com/Yemilice/p/11490918.html

    展开全文
  • AWS S3对象存储手册

    2019-02-15 09:27:21
    S3用户手册,主要介绍对对象存储的基础概念,及相关的使用说明。 离线版本的S3用户手册
  • AWS S3存储服务SDK

    千次阅读 2017-08-04 14:12:40
    Amazon S3 提供了一个简单 Web 服务接口,可用于随时在 Web 上的任何位置存储和检索任何数量的数据。此服务让所有开发人员都能访问同一个具备高扩展性、可靠性、安全性和快速价廉的数据存储基础设施, Amazon 用它...

    0x0 前言

    Amazon Simple Storage Service 是互联网存储解决方案。该服务旨在降低开发人员进行网络规模级计算的难度。

    Amazon S3 提供了一个简单 Web 服务接口,可用于随时在 Web 上的任何位置存储和检索任何数量的数据。此服务让所有开发人员都能访问同一个具备高扩展性、可靠性、安全性和快速价廉的数据存储基础设施, Amazon 用它来运行其全球的网站网络。此服务旨在为开发人员带来最大化的规模效益。

    0x1 名词介绍

    0x1_1 存储桶(Bucket)

    存储桶是 Amazon S3 中用于存储对象的容器。每个对象都储存在一个存储桶中。

    0x1_2 对象(Object)

    对象是 Amazon S3 中存储的基本实体。对象由对象数据和元数据组成。在存储桶中,对象将由键 (名称) 和版本 ID 进行唯一地标识。注:版本ID 非必须,需要再存储桶中配置启用版本控制

    0x1_3 键(Key)

    键是指存储桶中对象的唯一标识符。存储桶内的每个对象都只能有一个键。由于将存储桶、键和版本 ID 组合在一起可唯一地标识每个对象,可将 Amazon S3 视为一种“存储桶 + 键 + 版本”与对象本身间的基本数据映射。

    0x1_4 版本ID(VersionId)

    通过VersionId,可以在一个存储桶中保留多个对象版本,例如, report_201706.csv (版本 111111) 和 report_201706.csv (版本 222222)。可以启用版本控制来防止自己意外覆盖和删除版本,或存档对象,以便您可以检索早期版本的对象。如果不启用则相同Key 采用覆盖策略。

    • PUT(开启版本控制):

    PUT Version Enalbled

    • DELETE(开启版本控制)

    PUT Version Enalbled

    0x1_5 区域(Region)

    存储桶所在地理区域,在某一区域存储的对象将一直留在该区域,除非您特意将其传输到另一区域。定位对象需要 区域>账户授权(获取账户)>桶>key(+version id)来定位到存储对象。

    0x2 起步

    0x2_1 操作

    对存储桶、对象等操作可以有一下几种方式,本文主要讲解SDK操作:

    • AWS S3控制台操作
    • 命令行操作
    • SDK 操作(多种语言SDK,这里以JAVA SDK为例)
    • REST API

    0x2_2 依赖(Maven)

    • 引入AWS SDK 依赖管理
    <dependencyManagement>
        <dependencies>
            <!--AWS S3需要至少4.5.2以上版本-->
            <dependency>
                <groupId>org.apache.httpcomponents</groupId>
                <artifactId>httpclient</artifactId>
                <version>4.5.2</version>
            </dependency>
            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-bom</artifactId>
                <version>1.11.170</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    • 引入S3-SDK
    <dependencies>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
        </dependency>
    </dependencies>
    

    0x3 对象操作

    所有操作都基于com.amazonaws.services.s3.AmazonS3操作,com.amazonaws.services.s3.AmazonS3ClientAmazonS3的实现。

    鉴权对象SDK提供了多种实现:

    • ProfileCredentialsProvider 默认加载~/.aws/credentials 中accessKey 和secretKey
    • PropertiesCredentials 根据配置文件获取accessKey 和secretKey (支持文件和InputStream)
    • BasicAWSCredentials 直接传入accessKey 和secretKey

    0x3_1 开启桶版本控制

    存储桶版本控制状态分为三种:OFF,ENABLED,SUSPENDED。默认是OFF状态,开启(ENABLED)后,就不能再变更状态为OFF,只能变更为SUSPENDED 暂停版本控制,代码如下:

    public class Sample{
        private static String BUCKET_NAME     = "bucket-name";
        private static String ACCESS_KEY = "access_key_id";
        private static String SECRET_KEY = "secret_access_key";
        public static void main(String[] args){
            //初始化AmazonS3
            AmazonS3 s3client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY,SECRET_KEY));
            Region usWest2 = Region.getRegion(Regions.CN_NORTH_1);
            s3client.setRegion(usWest2);
            //开启版本控制
            BucketVersioningConfiguration config = new BucketVersioningConfiguration()
                                .withStatus(BucketVersioningConfiguration.ENABLED);
            SetBucketVersioningConfigurationRequest setBucketVersioningConfigurationRequest = 
                                    new SetBucketVersioningConfigurationRequest(BUCKET_NAME, config);
            s3client.setBucketVersioningConfiguration(setBucketVersioningConfigurationRequest);
        }
    }
    

    0x3_2 上传对象

    S3支持单次操作上传和多次操作分段上传(断点上传),这里只介绍单操作上传,分段上传详见:分段上传

    单操作上传:

    1. 创建 AmazonS3Client 的实例。
    2. 根据上传文件或流数据,执行任意一个 AmazonS3Client.putObject 重载。
    public class Sample{
        private final static Logger LOGGER = LoggerFactory.getLogger(Sample.class);
    
        private static String BUCKET_NAME   = "bucket-name";
        private static String ACCESS_KEY    = "access_key_id";
        private static String SECRET_KEY    = "secret_access_key";
        private static String KEY_NAME      = "/dev/report/余额月报_201705.csv";
        private static String FILE_NAME     = "/web/files/report/余额月报_201705.csv";
    
        public static void main(String[] args){
            //初始化AmazonS3
            AmazonS3 s3client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY,SECRET_KEY));
            Region usWest2 = Region.getRegion(Regions.CN_NORTH_1);
            s3client.setRegion(usWest2);
            PutObjectResult result = null;
            //上传文件
            try {
                File file = new File(FILE_NAME);
                LOGGER.debug("Uploading a new object:" + file.getName() + " to S3 from a file\n");
                result = s3client.putObject(new PutObjectRequest(
                        BUCKET_NAME, KEY_NAME, FILE_NAME));
            } catch (AmazonServiceException ase) {
                LOGGER.error(ase);
            } catch (AmazonClientException ace) {
                LOGGER.error(ace);
            }
        }
    }
    
    

    S3还提供了InputStream 上传对象,但是官方不建议使用InputStream 方式上传,容易引起OOM,建议采用本地生成临时文件上传文件的形式上传对象。

    0x3_3 获取对象列表

    public class Sample{
    
        private static String BUCKET_NAME   = "bucket-name";
        private static String ACCESS_KEY    = "access_key_id";
        private static String SECRET_KEY    = "secret_access_key";
        private static String KEY_NAME      = "/dev/report/余额月报_201705.csv";
        private static String FILE_NAME     = "/web/files/report/余额月报_201705.csv";
    
        public static void main(String[] args){
            //初始化AmazonS3
            AmazonS3 s3client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY,SECRET_KEY));
            Region usWest2 = Region.getRegion(Regions.CN_NORTH_1);
            s3client.setRegion(usWest2);
            //获取所有对象
            ObjectListing objects = s3client.listObjects(BUCKET_NAME);
            do {
                for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) {
                    System.out.println("Object: " + objectSummary.getKey());
                }
                objects = s3client.listNextBatchOfObjects(objects);
            } while (objects.isTruncated());    
        }
    }
    

    0x3_4 获取对象

    获取(下载)对象,可以获取对象所有元数据和流,建议先读取流内容,再处理其他逻辑,因为获取S3数据时,在读取或关闭输入流之前,网络连接一直保持打开状态。

    下载对象:

    1. 创建 AmazonS3Client 类的实例。
    2. 执行 AmazonS3Client.getObject() 方法之一。通过创建 GetObjectRequest 类传入存储桶 键名版本ID)。
    3. 对返回的对象执行 getObjectContent() 方法之一以获取关于对象数据的流,并处理响应。
    public class Sample{
        private final static Logger LOGGER = LoggerFactory.getLogger(Sample.class);
    
        private static String BUCKET_NAME   = "bucket-name";
        private static String ACCESS_KEY    = "access_key_id";
        private static String SECRET_KEY    = "secret_access_key";
        private static String KEY_NAME      = "/dev/report/余额月报_201705.csv";
        private static String FILE_NAME     = "/web/files/report/余额月报_201705.csv";
    
        public static void main(String[] args){
            //初始化AmazonS3
            AmazonS3 s3client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY,SECRET_KEY));
            Region usWest2 = Region.getRegion(Regions.CN_NORTH_1);
            s3client.setRegion(usWest2);
            PutObjectResult result = null;
            //获取文件
            try {
                LOGGER.debug("Downloading an object");
                S3Object s3object = s3Client.getObject(new GetObjectRequest(
                        BUCKET_NAME, KEY_NAME));
                LOGGER.debug("Content-Type: "  +
                        s3object.getObjectMetadata().getContentType());
                displayTextInputStream(s3object.getObjectContent());
            } catch (AmazonServiceException ase) {
                LOGGER.error(ase);
            } catch (AmazonClientException ace) {
                LOGGER.error(ace);
            }
        }
    
        private static void displayTextInputStream(InputStream input) throws IOException {
            BufferedReader reader = new BufferedReader(new InputStreamReader(input));
            LOGGER.debug("---------------Object display start--------------------")
            while (true) {
                    String line = reader.readLine();
                    if (line == null) break;
                    LOGGER.debug("    " + line);
            }
            LOGGER.debug("---------------Object display end--------------------")
        }
    }
    

    0x3_5 删除对象

    S3提供单个删除和批量删除两种删除操作,操作大同小异,在此以单对象删除为例,批量删除不在累述。删除操作分为开启版本控制的桶删除对象,和未开启版本控制的桶删除对象

    无版本控制删除

    1. 创建 AmazonS3Client 类的实例。
    2. 执行 AmazonS3Client.deleteObject 方法之一。可以提供存储桶名称和对象名称作为参数,或在 DeleteObjectRequest 对象中提供相同的信息,然后作为参数传递对象。
    public class Sample{
        private final static Logger LOGGER = LoggerFactory.getLogger(Sample.class);
    
        private static String BUCKET_NAME   = "bucket-name";
        private static String ACCESS_KEY    = "access_key_id";
        private static String SECRET_KEY    = "secret_access_key";
        private static String KEY_NAME      = "/dev/report/余额月报_201705.csv";
        private static String FILE_NAME     = "/web/files/report/余额月报_201705.csv";
    
        public static void main(String[] args) throws IOException {
            //初始化AmazonS3
            AmazonS3 s3client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY,SECRET_KEY));
            Region usWest2 = Region.getRegion(Regions.CN_NORTH_1);
            s3client.setRegion(usWest2);
            PutObjectResult result = null;
            //删除文件
            try {
                LOGGER.debug("Delete an object");
                s3Client.deleteObject(new DeleteObjectRequest(BUCKET_NAME, KEY_NAME));
            } catch (AmazonServiceException ase) {
                LOGGER.error(ase);
            } catch (AmazonClientException ace) {
                LOGGER.error(ace);
            }
        }
    }
    

    含版本控制的删除版本

    1. 创建 AmazonS3Client 类的实例。
    2. 对存储桶启用版本控制功能。
    3. 使用 deleteVersion 方法删除示例对象。DeleteVersionRequest 类将指定对象键名称和版本 ID。
    public class Sample{
        private final static Logger LOGGER = LoggerFactory.getLogger(Sample.class);
    
        private static String BUCKET_NAME   = "bucket-name";
        private static String ACCESS_KEY    = "access_key_id";
        private static String SECRET_KEY    = "secret_access_key";
        private static String KEY_NAME      = "/dev/report/余额月报_201705.csv";
        private static String FILE_NAME     = "/web/files/report/余额月报_201705.csv";
    
        public static void main(String[] args) throws IOException {
            //初始化AmazonS3
            AmazonS3 s3client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY,SECRET_KEY));
            Region usWest2 = Region.getRegion(Regions.CN_NORTH_1);
            s3client.setRegion(usWest2);
            PutObjectResult result = null;
            //删除文件
            try {
                LOGGER.debug("Delete an object");
                //开启版本控制
                enableVersioningOnBucket(s3Client, BUCKET_NAME);
                //上传一个对象并获取版本号
                String versionId = putAnObject(keyName);
    
                s3Client.deleteVersion(new DeleteVersionRequest(BUCKET_NAME, KEY_NAME, versionId));
            } catch (AmazonServiceException ase) {
                LOGGER.error(ase);
            } catch (AmazonClientException ace) {
                LOGGER.error(ace);
            }
        }
    
        //开启版本控制
        private static void enableVersioningOnBucket(AmazonS3Client s3client,String bucketName) {
    
            BucketVersioningConfiguration config = new BucketVersioningConfiguration()
                                .withStatus(BucketVersioningConfiguration.ENABLED);
            SetBucketVersioningConfigurationRequest setBucketVersioningConfigurationRequest = 
                                    new SetBucketVersioningConfigurationRequest(bucketName, config);
            s3client.setBucketVersioningConfiguration(setBucketVersioningConfigurationRequest);
        }
    
        private static String putAnObject(String keyName) {
            String content = "This is the content body!";
            String key = "ObjectToDelete-" + new Random().nextInt();
            ObjectMetadata metadata = new ObjectMetadata();
            metadata.setHeader("Subject", "Content-As-Object");
            metadata.setHeader("Content-Length", content.length());
            InputStream inputStream = new ByteArrayInputStream(content.getBytes());
            PutObjectRequest request = new PutObjectRequest(bucketName, key, inputStream, metadata)
                                                .withCannedAcl(CannedAccessControlList.AuthenticatedRead);
            PutObjectResult response = s3Client.putObject(request);
            return response.getVersionId();
        }
    
    }
    

    0x4 参考文献

    0x5 关于我

    • @Author:Zemo
    • @Email:zemochen#126.com
    • 欢迎转载,让更多的人学到东西
    展开全文
  • aws-sdk-go 上传图片 安装 go get github.com/aws/aws-sdk-go 代码示例 package aws import ( "github.com/aws/aws-sdk-go/aws" "github.... "github.com/aws/aws-sdk-go/service/s3/s3manager"

    aws-sdk-go 上传图片

    安装

    go get github.com/aws/aws-sdk-go

    代码示例
    package aws
    
    import (
       "github.com/aws/aws-sdk-go/aws"
       "github.com/aws/aws-sdk-go/aws/credentials"
       "github.com/aws/aws-sdk-go/aws/session"
       "github.com/aws/aws-sdk-go/service/s3/s3manager"
       "github.com/gin-gonic/gin"
       "net/http"
    )
    
    const AccessKeyID = "*******"
    const SecretAccessKey = "*********"
    const S3Region = "*****"
    const S3Bucket = "*******"
    
    func UploadToS3(c *gin.Context) {
       sess, _ := session.NewSession(&aws.Config{
          Credentials:      credentials.NewStaticCredentials(AccessKeyID, SecretAccessKey, ""),
          Region:           aws.String(S3Region),
          DisableSSL:       aws.Bool(true),
          S3ForcePathStyle: aws.Bool(false),
       })
    
       fileHeader, _ := c.FormFile("img")
       file, _ := fileHeader.Open()
    
       uploader := s3manager.NewUploader(sess)
       _, err := uploader.Upload(&s3manager.UploadInput{
          Bucket: aws.String(S3Bucket),
          Key: aws.String("file/"+ fileHeader.Filename),
          Body: file,
       })
    
       var path string
       if err == nil {
          path = "访问路径" + fileHeader.Filename
       }
    
       c.JSON(http.StatusOK, gin.H{
          "data": path,
       })
    }
    upload和putObject得区别

    (在使用upload之前是采用得putObject 没有上传成功,并且没有报错。不知道什么原因所以采用了upload)

    upload适用于比较大的文件,putObject适用于小的文件内容,upload支持自定义多线程并发上传

    展开全文
  • AWS S3 对象存储服务

    千次阅读 2019-05-31 18:51:44
    S3 Java SDK 分两个版本,1.0和2.0 , 1.0 的S3对象是AmazonS3 ,2.0的忘记了,下面的示例代码,是1.0版本的 1.2导包 <dependencyManagement> <dependencies> <dependency> <gr...

    1.图片上传

    1.1

    S3 Java SDK 分两个版本,1.0和2.0 , 1.0 的S3对象是AmazonS3 ,2.0的忘记了,下面的示例代码,是1.0版本的

    1.2导包

    <dependencyManagement>
            <dependencies>
                <dependency>
                    <groupId>com.amazonaws</groupId>
                    <artifactId>aws-java-sdk-bom</artifactId>
                    <version>1.11.558</version>
                    <type>pom</type>
                    <scope>import</scope>
                </dependency>
            </dependencies>
        </dependencyManagement>
    
        <dependencies>
            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-s3</artifactId>
            </dependency>
            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-s3control</artifactId>
            </dependency>
        </dependencies>
    

    1.3代码,已经可以运行了,但是还需要配置Id和秘钥

    public static void main(String[] args) throws Exception {
            String clientRegion = "ap-east-1";
            String bucketName = "hotupdates";
            String keyName = "abc/demo.txt";
            String filePath = "C:\\Users\\Owner\\Desktop\\demo.txt";
    
            try {
                AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                        .withRegion(clientRegion)
                        .withCredentials(new ProfileCredentialsProvider())
                        .build();
                TransferManager tm = TransferManagerBuilder.standard()
                        .withS3Client(s3Client)
                        .build();
                Upload upload = tm.upload(bucketName, keyName, new File(filePath));
                System.out.println("Object upload started");
                upload.waitForCompletion();
                tm.shutdownNow();
                System.out.println("Object upload complete");
            }
            catch(AmazonServiceException e) {
                e.printStackTrace();
            }
            catch(SdkClientException e) {
                e.printStackTrace();
            }
        }
    

    1.4 配置ID和秘钥

    Windows 在C:\Users\xxx\.aws这个文件下面创建两个文件,没有后缀的哟
    文件1:config

    [default]
    region = ap-east-1
    

    文件2:credentials

    [default]
    aws_access_key_id=xxxxxxxxxxxxxxxxxxx
    aws_secret_access_key=xxxxxxxxxxxxxxxxxxx
    

    1.5 到这就可以运行了

    1.6 想要在上传的时候打印进度条

    public class HighLevelMultipartUpload2 {
    
        public static void main(String[] args) throws Exception {
            String clientRegion = "ap-east-1";
            String bucketName = "hotupdates";
            String keyName = "abc/demo.txt";
            String filePath = "C:\\Users\\Owner\\Desktop\\demo.txt";
    
            try {
                AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                        .withRegion(clientRegion)
                        .withCredentials(new ProfileCredentialsProvider())
                        .build();
                TransferManager tm = TransferManagerBuilder.standard()
                        .withS3Client(s3Client)
                        .build();
    
                // TransferManager processes all transfers asynchronously,
                // so this call returns immediately.
                Upload upload = tm.upload(bucketName, keyName, new File(filePath));
                System.out.println("Object upload started");
    
                // Optionally, wait for the upload to finish before continuing.
                upload.waitForCompletion();
    
    
                tm.shutdownNow();
    //            XferMgrProgress.showTransferProgress(upload);
    //            XferMgrProgress.waitForCompletion(upload);
    
    
                System.out.println("Object upload complete");
            }
            catch(AmazonServiceException e) {
                e.printStackTrace();
            }
            catch(SdkClientException e) {
                e.printStackTrace();
            }
        }
    }
    
    public class XferMgrProgress
    {
        // waits for the transfer to complete, catching any exceptions that occur.
        public static void waitForCompletion(Transfer xfer)
        {
            try {
                xfer.waitForCompletion();
            } catch (AmazonServiceException e) {
                System.err.println("Amazon service error: " + e.getMessage());
                System.exit(1);
            } catch (AmazonClientException e) {
                System.err.println("Amazon client error: " + e.getMessage());
                System.exit(1);
            } catch (InterruptedException e) {
                System.err.println("Transfer interrupted: " + e.getMessage());
                System.exit(1);
            }
        }
    
        // Prints progress while waiting for the transfer to finish.
        public static void showTransferProgress(Transfer xfer)
        {
            // print the transfer's human-readable description
            System.out.println(xfer.getDescription());
            // print an empty progress bar...
            printProgressBar(0.0);
            // update the progress bar while the xfer is ongoing.
            do {
                try {
                    Thread.sleep(100);
                } catch (InterruptedException e) {
                    return;
                }
                // Note: so_far and total aren't used, they're just for
                // documentation purposes.
                TransferProgress progress = xfer.getProgress();
                long so_far = progress.getBytesTransferred();
                long total = progress.getTotalBytesToTransfer();
                double pct = progress.getPercentTransferred();
                eraseProgressBar();
                printProgressBar(pct);
            } while (xfer.isDone() == false);
            // print the final state of the transfer.
            Transfer.TransferState xfer_state = xfer.getState();
            System.out.println(": " + xfer_state);
        }
    
        // Prints progress of a multiple file upload while waiting for it to finish.
        public static void showMultiUploadProgress(MultipleFileUpload multi_upload)
        {
            // print the upload's human-readable description
            System.out.println(multi_upload.getDescription());
    
            Collection<? extends Upload> sub_xfers = new ArrayList<Upload>();
            sub_xfers = multi_upload.getSubTransfers();
    
            do {
                System.out.println("\nSubtransfer progress:\n");
                for (Upload u : sub_xfers) {
                    System.out.println("  " + u.getDescription());
                    if (u.isDone()) {
                        Transfer.TransferState xfer_state = u.getState();
                        System.out.println("  " + xfer_state);
                    } else {
                        TransferProgress progress = u.getProgress();
                        double pct = progress.getPercentTransferred();
                        printProgressBar(pct);
                        System.out.println();
                    }
                }
    
                // wait a bit before the next update.
                try {
                    Thread.sleep(200);
                } catch (InterruptedException e) {
                    return;
                }
            } while (multi_upload.isDone() == false);
            // print the final state of the transfer.
            Transfer.TransferState xfer_state = multi_upload.getState();
            System.out.println("\nMultipleFileUpload " + xfer_state);
        }
    
        // prints a simple text progressbar: [#####     ]
        public static void printProgressBar(double pct)
        {
            // if bar_size changes, then change erase_bar (in eraseProgressBar) to
            // match.
            final int bar_size = 40;
            final String empty_bar = "                                        ";
            final String filled_bar = "########################################";
            int amt_full = (int)(bar_size * (pct / 100.0));
            System.out.format("  [%s%s]", filled_bar.substring(0, amt_full),
                    empty_bar.substring(0, bar_size - amt_full));
        }
    
        // erases the progress bar.
        public static void eraseProgressBar()
        {
            // erase_bar is bar_size (from printProgressBar) + 4 chars.
            final String erase_bar = "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b";
            System.out.format(erase_bar);
        }
    
        public static void uploadFileWithListener(String file_path,
                                                  String bucket_name, String key_prefix, boolean pause)
        {
            System.out.println("file: " + file_path +
                    (pause ? " (pause)" : ""));
    
            String key_name = null;
            if (key_prefix != null) {
                key_name = key_prefix + '/' + file_path;
            } else {
                key_name = file_path;
            }
    
            File f = new File(file_path);
            TransferManager xfer_mgr = TransferManagerBuilder.standard().build();
            try {
                Upload u = xfer_mgr.upload(bucket_name, key_name, f);
                // print an empty progress bar...
                printProgressBar(0.0);
                u.addProgressListener(new ProgressListener() {
                    public void progressChanged(ProgressEvent e) {
                        double pct = e.getBytesTransferred() * 100.0 / e.getBytes();
                        eraseProgressBar();
                        printProgressBar(pct);
                    }
                });
                // block with Transfer.waitForCompletion()
                XferMgrProgress.waitForCompletion(u);
                // print the final state of the transfer.
                Transfer.TransferState xfer_state = u.getState();
                System.out.println(": " + xfer_state);
            } catch (AmazonServiceException e) {
                System.err.println(e.getErrorMessage());
                System.exit(1);
            }
            xfer_mgr.shutdownNow();
        }
    
        public static void uploadDirWithSubprogress(String dir_path,
                                                    String bucket_name, String key_prefix, boolean recursive,
                                                    boolean pause)
        {
            System.out.println("directory: " + dir_path + (recursive ?
                    " (recursive)" : "") + (pause ? " (pause)" : ""));
    
            TransferManager xfer_mgr = new TransferManager();
            try {
                MultipleFileUpload multi_upload = xfer_mgr.uploadDirectory(
                        bucket_name, key_prefix, new File(dir_path), recursive);
                // loop with Transfer.isDone()
                XferMgrProgress.showMultiUploadProgress(multi_upload);
                // or block with Transfer.waitForCompletion()
                XferMgrProgress.waitForCompletion(multi_upload);
            } catch (AmazonServiceException e) {
                System.err.println(e.getErrorMessage());
                System.exit(1);
            }
            xfer_mgr.shutdownNow();
        }
    
        public static void main(String[] args)
        {
            final String USAGE = "\n" +
                    "Usage:\n" +
                    "    XferMgrProgress [--recursive] [--pause] <s3_path> <local_path>\n\n" +
                    "Where:\n" +
                    "    --recursive - Only applied if local_path is a directory.\n" +
                    "                  Copies the contents of the directory recursively.\n\n" +
                    "    --pause     - Attempt to pause+resume the upload. This may not work for\n" +
                    "                  small files.\n\n" +
                    "    s3_path     - The S3 destination (bucket/path) to upload the file(s) to.\n\n" +
                    "    local_path  - The path to a local file or directory path to upload to S3.\n\n" +
                    "Examples:\n" +
                    "    XferMgrProgress public_photos/cat_happy.png my_photos/funny_cat.png\n" +
                    "    XferMgrProgress public_photos my_photos/cat_sad.png\n" +
                    "    XferMgrProgress public_photos my_photos\n\n";
    
            if (args.length < 2) {
                System.out.println(USAGE);
                System.exit(1);
            }
    
            int cur_arg = 0;
            boolean recursive = false;
            boolean pause = false;
    
            // first, parse any switches
            while (args[cur_arg].startsWith("--")) {
                if (args[cur_arg].equals("--recursive")) {
                    recursive = true;
                } else if (args[cur_arg].equals("--pause")) {
                    pause = true;
                } else {
                    System.out.println("Unknown argument: " + args[cur_arg]);
                    System.out.println(USAGE);
                    System.exit(1);
                }
                cur_arg += 1;
            }
    
            // only the first '/' character is of interest to get the bucket name.
            // Subsequent ones are part of the key name.
            String s3_path[] = args[cur_arg].split("/", 2);
            cur_arg += 1;
    
            String bucket_name = s3_path[0];
            String key_prefix = null;
            if (s3_path.length > 1) {
                key_prefix = s3_path[1];
            }
    
            String local_path = args[cur_arg];
    
            // check to see if local path is a directory or file...
            File f = new File(args[cur_arg]);
            if (f.exists() == false) {
                System.out.println("Input path doesn't exist: " + args[cur_arg]);
                System.exit(1);
            }
            else if (f.isDirectory()) {
                uploadDirWithSubprogress(local_path, bucket_name, key_prefix,
                        recursive, pause);
            } else {
                uploadFileWithListener(local_path, bucket_name, key_prefix, pause);
            }
        }
    }
    

    参考官网:https://docs.aws.amazon.com/zh_cn/sdk-for-java/v1/developer-guide/credentials.html

    2.其余的一些配置

    2.1跨域

    <CORSConfiguration>
     <CORSRule>
       <AllowedOrigin>*</AllowedOrigin>
       <AllowedMethod>PUT</AllowedMethod>
       <AllowedMethod>POST</AllowedMethod>
       <AllowedMethod>DELETE</AllowedMethod>
       <AllowedMethod>GET</AllowedMethod>
       <AllowedMethod>HEAD</AllowedMethod>
       <AllowedHeader>*</AllowedHeader>
      <MaxAgeSeconds>3000</MaxAgeSeconds>
     </CORSRule>
    </CORSConfiguration>
    

    2.2 存储桶策略

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Stmt1546506260886",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "*"
                },
                "Action": "s3:GetObject",
                "Resource": "arn:aws:s3:::apk-online/*"
            }
        ]
    }
    

    apk-online 是桶名

    展开全文
  • minio对象存储原理Object storage is amazingly versatile. We normally associate it with AWS S3, but Object Storage can also be run on your private network with MinIO. In this article, I’ll show you how...
  • 一、AWS S3 对象存储API 切换签名版本示例 示例代码摘抄自 DELL ECS EMC aws-java-workshop public static AmazonS3 getS3ClientWithV4Signatures() { System.out.println("Running with V4 Signatures:\n"); ...
  • AWS S3(对象存储)基本操作

    万次阅读 热门讨论 2018-06-14 20:28:43
    1 AWS S3对象存储 1.1 基本概念 1.1.1 存储桶(bucket) 1.1.2 对象 1.1.3 对象标签(tag) 1.2 S3存储类型 2 使用 2.1 安装AWS SDK 2.2 编写基础配置文件 3 基础功能 3.1 桶操作 3.1.1 创建桶 3.1.2 桶的生命周期...
  • 一简介说明COS 提供了 AWS S3 兼容的 API,因此当您的数据从 S3 迁移到 COS 之后,只需要进行简单的配置修改,即可让您的客户端应用轻松兼容 COS 服务。本文主要介绍不...
  • 1.需要的依赖jar包 1 <dependency> 2 <groupId>com.amazonaws</groupId> 3 <...aws-java-sdk-s3</artifactId> 4 <version>1.11.233</version> 5 <...
  • s3存储分段上传命令行和SDK使用

    万次阅读 2017-12-12 10:48:25
    s3是亚马逊退出的对象存储服务。我之前blog介绍过什么是对象存储,这里普通的对象操作在此略过,如果大家感兴趣可以看aws官网,说的很详细,在此我想介绍的是分段上传的使用方式,先看下面我画的图 文件从分片,...
  • 添加onUploadProgress方法,监控上传进度,方便进度条百分比的展示 AWS S3,JS 文档地址:https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html 完整代码: import axios from 'axios' axios.put...
  • 如何用 Minio SDK 访问滴滴云对象存储

    千次阅读 2018-12-19 13:14:06
    滴滴云对象存储 S3 定位分布式海量数据存取服务。S3 提供标准的对象存储服务,您可以通过网络随时随地方便地存储和获取图片/音频/视频/文本/大文件等各类数据文件。 Minio 是一个开源的对象存储项目,它兼容亚马逊 ...
  • 对象存储python SDK开发

    2021-08-01 18:47:28
    在这里插入代码片 #!/usr/bin/python -- coding:utf-8 -- ...Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License....
  • 我们在之前的文章中已经尝试获取s3的所有存储的文件大小和 最后修改时间 清洗入库。 现在可以 对 s3存储进行 精细化的 优化了。比如 对s3存储进行分层优化。 这样可以把我们的成本 明显的降低。 s3存储目前有6层,...
  • 对象存储S3访问姿势

    2021-05-28 17:59:19
    本质上,对象存储使用bucket,key来唯一标识一个对象,所以只要你告诉对象存储服务端这两个东西,那么理论上就能定位到这个数据。以上三种方式,总归都是为了获取到(bucket,object_key)。 PathStyle URL 在...
  • 本文介绍如何使用Go语言对京东云对象存储OSS进行基本的操作,帮助客户快速通过Go SDK接入京东云对象存储,提高应用开发的效率。 在实际操作之前,我们先看一下京东云OSS的API接口支持范围和能力。从对象存储的API...
  • How to use AWS SDK for PHP with Minio Serveraws-sdk-php is the official AWS SDK for the ...
  • Ceph-对象存储S3使用手册实践

    千次阅读 2020-07-06 11:05:20
    使用python s3 boto3 SDK访问对象存储服务, 包括查看Bucket列表, 上传文件, 下载文件, 查看文件列表等。 1. 查看bucket列表 $ python ls_bucket.py ['first-bucket'] #!/usr/bin/python # -*-
  • NoDB是基于Amazon的S3静态文件存储的一个非常简单的Pythonic对象存储
  • Amazon S3 Storage(亚马逊的对象存储

    千次阅读 2020-06-24 20:37:44
    亚马逊S3,全称亚马逊简易存储服务(Amazon Simple Storage Service),是一个由亚马逊网络服务(Amazon Web Services,简称AWS)提供的服务,它通过一个Web Service接口来提供对象存储。亚马逊S3使用与amazon.com...
  • 调整大小、重命名和上传图像到 Amazon S3 磁盘存储,承诺意识。 使用官方和进行图像处理。 安装 使用 npm npm install git://github.com/ventuso/s3-uploader.git 或将此行添加到package.json "ventuso-s3-...
  • AWS JavaScript S3 Explorer是一个JavaScript应用程序,它使用AWSJavaScript SDKS3 API使公共S3存储桶的内容易于通过Web浏览器进行浏览。 我们创建此文件是为了通过Amazon S3更轻松地共享公共对象和数据。 该...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 3,584
精华内容 1,433
热门标签
关键字:

s3对象存储sdk