精华内容
下载资源
问答
  • PHP多线程异步请求HTTP

    千次阅读 2018-03-06 15:58:22
    /** * 多线程请求链接 * 来自官方参考例程修改 * * @param array $urlArr * @author Xinwen Mao */ public static function curl_multi($urlArr) { // 创建批处理cURL句柄 $mh = curl...
        /**
         * 多线程请求链接
         * 来自官方参考例程修改
         *
         * @param array $urlArr
         * @author Xinwen Mao
         */
        public static function curl_multi($urlArr) {
            // 创建批处理cURL句柄
            $mh = curl_multi_init();
    
            for ($i = 0; $i < count($urlArr); $i++) {
                // 创建一对cURL资源
                $chs [$i] = curl_init();
    
                // 设置URL和相应的选项
                curl_setopt($chs [$i], CURLOPT_URL, $urlArr [$i]);
                curl_setopt($chs [$i], CURLOPT_HEADER, 0);
                curl_setopt($chs [$i], CURLOPT_RETURNTRANSFER, 1); // 不输出返回内容
                //  curl_setopt($chs [$i], CURLOPT_PROXY, '10.211.55.3:8888'); // 代理调试
    
                // 增加句柄
                curl_multi_add_handle($mh, $chs [$i]);
            }
    
            $active = null;
            do {
                curl_multi_exec($mh, $active);
            } while ($active > 0);
    
            // 关闭全部句柄
            foreach ($chs as $ch) {
                curl_multi_remove_handle($mh, $ch);
            }
            curl_multi_close($mh);
        }

    展开全文
  • 本篇文章给大家分享了java模拟多线程http请求的相关实例代码,对此有需要的可以跟着测试下。
  • httpclient 多线程请求

    2018-04-14 06:59:00
    线程请求执行 当配备一个线程池管理器后,如PollingClientConnectionManager,HttpClient就能使用执行着的多线程去执行并行的多请求。 PollingClientConnectionManager会基于它的配置去分配连接。如果一个指定的路由...

    线程请求执行

    当配备一个线程池管理器后,如PollingClientConnectionManager,HttpClient就能使用执行着的多线程去执行并行的多请求。

    PollingClientConnectionManager会基于它的配置去分配连接。如果一个指定的路由连接已经被租用了,连接请求会被阻塞直到有一个连接被释回到池里。
    可以给http.connmanager.timeout设定一个正值以确保连接管理器在连接请求操作里不会无限的阻塞下去。如果连接请求不能在指定的时间获取服务就抛出
    ConnectionPoolTimeoutException异常。
    PollingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
    CloseableHttpClient httpClient = HttpClients
                                     .custom()
                                     .setConnectionManager(cm)
                                     .build();
    //URIs to perform GETs on
    String[] urisToGet={
     "http://www.domain1.com/",
     "http://www.domain2.com/",
     "http://www.domain3.com/",
     "http://www.domain4.com/"
    };
    //create a thread for each URI
    GetThread[] threads = new GetThread[uriToGet.length];
    for(int i=0;i<thread.length;i++){
      HttpGet httpget = new HttpGet(urisToGet[i]);
      threads[i]=new GetThread(httpClient,httpget);
    }
    //start the threads
    for(int j=0;j<threads.length;j++){
      threads[j].start();
    }
    //join the threads
    for(int j=0;j<threads.length;j++){
      threads[j].join();
    }

    HttpClient实例是线程安全的并且可以在执行着的多线程之间共享,并强烈推荐每个线程维护自己的HttpContext专用实例。

    Static class GetThread extends Thread{
     private final CloseableHttpClient httpClient;
     private final HttpContext context;
     private final HttpGet httpget;
    ​
     public GetThread(CloseableHttpClient httpClient, HttpGet httpGet){
       this.httpClient = httpClient;
       this.context=HttpClientContext.create();
       this.httpget=httpget;
     }
    ​
     public void run(){
      try{
         CloseableHttpResponse response = httpClient.execute(httpGet,context);
         try{
           HttpEntity entity = response.getEntity();
          }finally{
            response.close();
          }
       }catch(ClientProtocolException ex){
        //handle protocol errors
       }catch(IOException ex){
         //Handle I/O errors
       }
     }
    }

    转载于:https://www.cnblogs.com/ssgao/p/8829320.html

    展开全文
  • httpClient多线程请求

    2016-01-20 16:33:00
    使用单线程的请求速度上会有一定的限制,参考了Apache给出的例子,自己做了测试实现多线程并发请求,以下代码需要HttpClient 4.2的包,可以在http://hc.apache.org/downloads.cgi下载 1、并发请求 package ...

    使用httpClient可模拟请求Url获取资源,使用单线程的请求速度上会有一定的限制,参考了Apache给出的例子,自己做了测试实现多线程并发请求,以下代码需要HttpClient 4.2的包,可以在http://hc.apache.org/downloads.cgi下载

    1、并发请求

    package generate.httpclient;
    
    import java.util.List;
    import java.util.concurrent.ExecutorService;
    import java.util.concurrent.Executors;
    
    import org.apache.http.HttpEntity;
    import org.apache.http.HttpResponse;
    import org.apache.http.client.HttpClient;
    import org.apache.http.client.methods.HttpGet;
    import org.apache.http.conn.ClientConnectionManager;
    import org.apache.http.conn.params.ConnManagerParams;
    import org.apache.http.conn.scheme.PlainSocketFactory;
    import org.apache.http.conn.scheme.Scheme;
    import org.apache.http.conn.scheme.SchemeRegistry;
    import org.apache.http.impl.client.DefaultHttpClient;
    import org.apache.http.impl.conn.PoolingClientConnectionManager;
    import org.apache.http.params.BasicHttpParams;
    import org.apache.http.params.HttpConnectionParams;
    import org.apache.http.params.HttpParams;
    import org.apache.http.protocol.BasicHttpContext;
    import org.apache.http.protocol.HttpContext;
    import org.apache.http.util.EntityUtils;
    
    public class ThreadPoolHttpClient {
        // 线程池
        private ExecutorService exe = null;
        // 线程池的容量
        private static final int POOL_SIZE = 20;
        private HttpClient client = null;
        String[] urls=null;
        public ThreadPoolHttpClient(String[] urls){
            this.urls=urls;
        }
        public void test() throws Exception {
            exe = Executors.newFixedThreadPool(POOL_SIZE);
            HttpParams params =new BasicHttpParams();
            /* 从连接池中取连接的超时时间 */ 
            ConnManagerParams.setTimeout(params, 1000);
            /* 连接超时 */ 
            HttpConnectionParams.setConnectionTimeout(params, 2000); 
            /* 请求超时 */
            HttpConnectionParams.setSoTimeout(params, 4000);
            SchemeRegistry schemeRegistry = new SchemeRegistry();
            schemeRegistry.register(
                    new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
    
            //ClientConnectionManager cm = new PoolingClientConnectionManager(schemeRegistry);
            PoolingClientConnectionManager cm=new PoolingClientConnectionManager(schemeRegistry);
            cm.setMaxTotal(10);
            final HttpClient httpClient = new DefaultHttpClient(cm,params);
    
            // URIs to perform GETs on
            final String[] urisToGet = urls;
            /* 有多少url创建多少线程,url多时机子撑不住
            // create a thread for each URI
            GetThread[] threads = new GetThread[urisToGet.length];
            for (int i = 0; i < threads.length; i++) {
                HttpGet httpget = new HttpGet(urisToGet[i]);
                threads[i] = new GetThread(httpClient, httpget);            
            }
            // start the threads
            for (int j = 0; j < threads.length; j++) {
                threads[j].start();
            }
    
            // join the threads,等待所有请求完成
            for (int j = 0; j < threads.length; j++) {
                threads[j].join();
            }
            使用线程池*/
            for (int i = 0; i < urisToGet.length; i++) {
                final int j=i;
                System.out.println(j);
                HttpGet httpget = new HttpGet(urisToGet[i]);
                exe.execute( new GetThread(httpClient, httpget));
            }
            
            
            //创建线程池,每次调用POOL_SIZE
            /*
            for (int i = 0; i < urisToGet.length; i++) {
                final int j=i;
                System.out.println(j);
                exe.execute(new Thread() {
                    @Override
                    public void run() {
                        this.setName("threadsPoolClient"+j);
                        
                            try {
                                this.sleep(100);
                                System.out.println(j);
                            } catch (InterruptedException e) {
                                // TODO Auto-generated catch block
                                e.printStackTrace();
                            }
                            
                            HttpGet httpget = new HttpGet(urisToGet[j]);
                            new GetThread(httpClient, httpget).get();
                        }
                        
                        
                    
                });
            }
            
            */
            //exe.shutdown();
            System.out.println("Done");
        }
        static class GetThread extends Thread{
            
            private final HttpClient httpClient;
            private final HttpContext context;
            private final HttpGet httpget;
            
            public GetThread(HttpClient httpClient, HttpGet httpget) {
                this.httpClient = httpClient;
                this.context = new BasicHttpContext();
                this.httpget = httpget;
            }
            @Override
            public void run(){
                this.setName("threadsPoolClient");
                try {
                    Thread.sleep(5000);
                } catch (InterruptedException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
                get();
            }
            
            public void get() {
                try {
                    HttpResponse response = this.httpClient.execute(this.httpget, this.context);
                    HttpEntity entity = response.getEntity();
                    if (entity != null) {
                        System.out.println(this.httpget.getURI()+": status"+response.getStatusLine().toString());
                    }
                    // ensure the connection gets released to the manager
                    EntityUtils.consume(entity);
                } catch (Exception ex) {
                    this.httpget.abort();
                }finally{
                    httpget.releaseConnection();
                }
            }
        }
    }
    并发请求

    2、多线程异步请求

    package generate.httpclient;
    
    import java.io.ByteArrayOutputStream;
    import java.io.IOException;
    import java.io.InputStream;
    import java.util.ArrayList;
    import java.util.HashMap;
    import java.util.List;
    import java.util.Map;
    import java.util.concurrent.CountDownLatch;
    
    import org.apache.http.HttpResponse;
    import org.apache.http.client.methods.HttpGet;
    import org.apache.http.concurrent.FutureCallback;
    import org.apache.http.impl.nio.client.DefaultHttpAsyncClient;
    import org.apache.http.nio.client.HttpAsyncClient;
    import org.apache.http.nio.reactor.IOReactorException;
    
    public class AsynClient{
        /**
         * @param args
         * @throws IOReactorException
         * @throws InterruptedException
         */
        private List<String> urls;
        private HandlerFailThread failHandler;
        public AsynClient(List<String> list){
            failHandler=new HandlerFailThread();
            urls=list;
        }
        public Map<String,String> asynGet() throws IOReactorException,
                InterruptedException {
            final HttpAsyncClient httpclient = new DefaultHttpAsyncClient();
            httpclient.start();
            int urlLength=urls.size();
            HttpGet[] requests = new HttpGet[urlLength];
            int i=0;
            for(String url : urls){
                requests[i]=new HttpGet(url);
                i++;
            }
            final CountDownLatch latch = new CountDownLatch(requests.length);
            final Map<String, String> responseMap=new HashMap<String, String>();
            try {
                for (final HttpGet request : requests) {
                    httpclient.execute(request, new FutureCallback<HttpResponse>() {
    
                        public void completed(final HttpResponse response) {
                            latch.countDown();
                            responseMap.put(request.getURI().toString(), response.getStatusLine().toString());
                            try {
                                System.out.println(request.getRequestLine() + "->"
                                        + response.getStatusLine()+"->");
                                //+readInputStream(response.getEntity().getContent())
                                
                            } catch (IllegalStateException e) {
                                failHandler.putFailUrl(request.getURI().toString(),
                                        response.getStatusLine().toString());
                                e.printStackTrace();
                            } catch (Exception e) {
                                failHandler.putFailUrl(request.getURI().toString(),
                                        response.getStatusLine().toString());
                                e.printStackTrace();
                            }
                        }
    
                        public void failed(final Exception ex) {
                            latch.countDown();
                            ex.printStackTrace();
                            failHandler.putFailUrl(request.getURI().toString(),
                                    ex.getMessage());
                        }
    
                        public void cancelled() {
                            latch.countDown();
                        }
    
                    });
                }
                System.out.println("Doing...");
            } finally {
                latch.await();
                httpclient.shutdown();
            }
            System.out.println("Done");
            failHandler.printFailUrl();
            return responseMap;
        }
        private String readInputStream(InputStream input) throws IOException{
            byte[] buffer = new byte[128];
            int len = 0;
            ByteArrayOutputStream bytes = new ByteArrayOutputStream();
            while((len = input.read(buffer)) >= 0) {
                bytes.write(buffer, 0, len);
            }
            return bytes.toString();
        }
        /**
         * Test
         * @param args
         */
        public static void main(String[] args) {
            List<String> urls=new ArrayList<String>();
            urls.add("http://127.0.0.1/examples/servlets/");
            urls.add("http://127.0.0.1/examples/servlets/");
            urls.add("http://127.0.0.1/examples/servlets/");
            for(int i=0;i<10;i++){
                urls.addAll(urls);
            }
            System.out.println(urls.size());
            AsynClient client=new AsynClient(urls);
            try {
                client.asynGet();
            } catch (IOReactorException e) {
                e.printStackTrace();
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            System.out.println("done");
        }
    }
    多线程异步请求

    3、创建一个线程记录失败的请求

    package generate.httpclient;
    
    import java.util.HashMap;
    import java.util.Map;
    
    public class HandlerFailThread  extends Thread{
        Map<String, String> failUrl=new HashMap<String, String>();
        public void putFailUrl(String url,String status){
            synchronized (failUrl) {
                failUrl.put(url,status);
            }
        }
        @Override
        public void run() {
            while(true){
                
            }
        }
        public void printFailUrl(){
            for(Map.Entry<String, String> m: failUrl.entrySet()){
                System.out.println("****fail:url:"+m.getKey()+ "  code :"+m.getValue());
            }
        }
    }
    线程记录失败的请求

     

    异步请求,也可通过pool管理,例如

     ConnectingIOReactor nio=new DefaultConnectingIOReactor();
      PoolingClientAsyncConnectionManager manager=new PoolingClientAsyncConnectionManager(nio);
      manager.setMaxTotal(1000);
      manager.setDefaultMaxPerRoute(100);
      HttpParams params=new BasicHttpParams();
      /* 连接超时 */ 
      HttpConnectionParams.setConnectionTimeout(params, 10000); 
      /* 请求超时 */
      HttpConnectionParams.setSoTimeout(params, 60*1000);
      DefaultHttpAsyncClient.setDefaultHttpParams(params);
      final HttpAsyncClient httpclient = new DefaultHttpAsyncClient(manager);
      httpclient.start();

     

    HttpClient相关可参看,里面有很多说明与例子

    http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html

     

    转载于:https://www.cnblogs.com/Struts-pring/p/5145694.html

    展开全文
  • 多线程进行http请求

    2016-02-02 14:15:00
    昨天需要一个线下脚本进行单播推送,大约有1kw个用户,考虑到推送速度就临时搞了个请求线上的一个脚本 /** * 临时支持invoke单播推送 */ #include <stdio.h> #include <stdlib.h> #include ...

    昨天需要一个线下脚本进行单播推送,大约有1kw个用户,考虑到推送速度就临时搞了个请求线上的一个脚本

     

    /**
     * 临时支持invoke单播推送
     */
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <pthread.h>
    #include "ghttp.h"
    
    #include "log.h"
    
    //调试模式
    #define DEBUGS
    //最大线程数
    #define MAX_THREADS 30
    
    #ifndef DEBUGS
    char *global_uri = "xxx";
    #else
    char *global_uri = "xxx";
    #endif
    
    struct active_t{
            pthread_mutex_t active_mutex;
            pthread_cond_t active_cond;
            int active;
    }active_struct;
    
    const char *global_request_conf = "conf/request.conf";
    
    void *work_fun(void *arg);
    
    static void _init_main(){
            open_log();
            print_log(DEBUG, "%s", "start...");
    }
    
    static void _shut_down(){
            print_log(DEBUG, "%s", "end...");
            close_log();
    }
    
    static void error_die(const char *msg){
            perror(msg);
            exit(1);
    }
    
    /**
     * 请求mapi
     */
    int mapi_push(char *data, char *ret, int ret_len){
            ghttp_request *request = NULL;
            request = ghttp_request_new();
            ghttp_set_uri(request, global_uri);
            ghttp_set_type(request, ghttp_type_post);
            ghttp_set_header(request, http_hdr_Connection, "close");
            ghttp_set_header(request, http_hdr_Content_Type, "application/x-www-form-urlencoded");
            ghttp_set_body(request, data, strlen(data));
            ghttp_prepare(request);
            int status = ghttp_process(request);
            if(status == ghttp_error){
                    return -1;
            }
            memset(ret, 0, ret_len);
            strncpy(ret, ghttp_get_body(request), ghttp_get_body_len(request));
            ret[strlen(ret)] = '\0';
            int http_code = ghttp_status_code(request);
            ghttp_request_destroy(request);
            return http_code; 
    }
    
    int main(){
            _init_main();
    
            FILE *fp = fopen(global_request_conf, "r");
            if(!fp){
                    error_die("fopen error");
            }
            char buf[2048];
            memset(buf, 0, 2048);
            pthread_t thid;
            while(!feof(fp) && fgets(buf, 2047, fp) != NULL){
                    //max thread
                    pthread_mutex_lock(&active_struct.active_mutex);
                    while(active_struct.active >= MAX_THREADS){
                            pthread_cond_wait(&active_struct.active_cond, &active_struct.active_mutex);
                    }
                    pthread_mutex_unlock(&active_struct.active_mutex);
                    //run
                    pthread_create(&thid, NULL, work_fun, (void *)buf);
                    if(!thid){
                            printf("create thread error");
                            continue;
                    }
                    /*active+1*/
                    pthread_mutex_lock(&active_struct.active_mutex);
                    active_struct.active++;
                    pthread_mutex_unlock(&active_struct.active_mutex);
            }
            //wait for all thread done
        pthread_mutex_lock(&active_struct.active_mutex);
        while(active_struct.active != 0){
            pthread_cond_wait(&active_struct.active_cond, &active_struct.active_mutex);
        }
        pthread_mutex_unlock(&active_struct.active_mutex);
            //clear
            _shut_down();
    }
    
    /**
    * thread fun
    */
    void *work_fun(void *arg){
            char *data = (char *)arg;
            pthread_detach(pthread_self());
            char ret[2048];
            int http_code = mapi_push(data, ret, 2048);
            if(http_code == 200){
                    if(strncmp(ret, "{\"errno\":0", 10) != 0){
                            print_log(DEBUG, "errno:[2] http_cdoe:[%d] data:[%s] ret:[%s]", http_code, data, ret);
                    }
            }else{
                    print_log(DEBUG, "errno:[1] http_cdoe:[%d] data:[%s] ret:[%s]", http_code, data, ret);
            }
    
            //printf("%d, %s, %s\n", http_code, data, ret);
            //notice main thread
            pthread_mutex_lock(&active_struct.active_mutex);
            active_struct.active--;
            pthread_cond_signal(&active_struct.active_cond);
            pthread_mutex_unlock(&active_struct.active_mutex);
    }

     其实还有好多可以优化的点,线下执行了一下,效果和速度还行

    转载于:https://www.cnblogs.com/bai-jimmy/p/5177433.html

    展开全文
  • 多线程实际应用处理多次Http请求欢迎使用Markdown编辑器多线程线程池的创建多线程任务的创建如实际@Service的开发 欢迎使用Markdown编辑器 公司里有一个需求就是批量保存用户信息。用一个采集设备,采取用户的身份证...
  • 业务场景:前端调Java接口,接口需要多次请求http请求,因为是顺序执行,每个http耗时很长,大概5秒左右,但是叠加起来是很恐怖的,有必要做成多线程去处理。 大体思路:多线程去do任务,使用CountDownLatch进行...
  • java模拟多线程http请求

    千次阅读 2018-10-10 15:21:57
    /**java 模拟测试多线程测试http请求 * @author wdh * */ public class ServiceThreadTest implements Runnable{ private String url; private Map,Object> paramStr; public ServiceThreadTest...
  • python伪造IP多线程发送http请求

    热门讨论 2013-11-22 18:10:16
    注,需要安全scapy 在centos下,请执行以下脚本安装scapy #!/usr/bin sh cd /tmp wget scapy.net unzip scapy-latest.zip cd scapy-2.* python setup.py install 需要手工启动和杀线程关闭 经测python2可用
  • 有时我们可能会碰到这样一种情况,我们有一个功能,这个功能对外提供了一个http接口,我们需要对这个http接口发起请求才能启动这个服务,但是这个服务功能可能会执行很久,这样如果等功能执行结束再返回请求结果,那...
  • 使用java 线程计数器进行HTTP请求同时调用处理 import java.util.concurrent.CountDownLatch; public void executeUserList(Date startDate , Date endDate){  logger.info("定时任务:aa 任务--->...
  • 发起http请求后,检查返回的数据是否含有特征码,几百w的数据跑了一天后也没跑完,尝试了下ruby多线程,发现并不能提高运行速度,果断换JAVA来写,ruby代码贴下: #coding:gbkrequire 'rubygems'require ...
  • org.apache.http.client.methods.HttpGet; import org.apache.http.impl.client.DefaultHttpClient; import org.apache.http.impl.conn.PoolingClientConnectionManager; import org.apache....
  • 0 概述 在写app后台的时候,需要调用另一个服务器上的算法服务,所以...所以使用多线程来处理这几个接口调用,以此减少消耗时间。(在发送一个请求后不盲等,继续发送另一个请求,这相当于一种异步请求) 1 po...
  • PHP需要同时批量请求多个URL时,可以使用curl_multi实现并发请求。 public static function rolling_curl($urls, $postData=array(),$custom_options = null){//个url访问 if (sizeof($urls)==0) return; // ...
  • 多线程请求乌云链接

    2015-07-12 12:02:00
    #coding=utf-8importrequestsimportthreadingimporttimeprint"start:"+(time.strftime("%H:%M:%S"))defaccess():foriinrange(1,11):url='http://drops.wooyun.org/papers/7049'r=...
  • C#(也称Csharp)在多线程下并发执行HTTP请求的实现,采用C#封装HttpWebRequest类开发的多线程并发采集程序源码文档,文档中详细说明了HttpWebRequest并发HTTP请求实现网站采集的方法,经过测试同时并发1000+不是问题...
  • 这一两个礼拜,一直在忙着做安卓课程设计,刚好昨天在做的时候碰到了一个多线程的应用问题。虽然之前有学习过多线程的知识,但是一直没有真正在自己的开发实践过程中应用过,再加上也有段时间没接触过多线程了,知识...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 3,679
精华内容 1,471
关键字:

多线程请求http