精华内容
下载资源
问答
  • k8s创建pod代码流程(1.18.x) 参考文档: https://www.bookstack.cn/read/source-code-reading-notes/kubernetes-kubelet_create_pod.md https://cloud.tencent.com/developer/article/1492108 一、写在前面 ...

    k8s创建pod代码流程(新版1.18.x)

    参考文档:
    https://www.bookstack.cn/read/source-code-reading-notes/kubernetes-kubelet_create_pod.md
    https://cloud.tencent.com/developer/article/1492108
    https://www.huweihuang.com/kubernetes-notes/code-analysis/kubelet/syncPod.html

    一、写在前面

    工欲善其事,必先利其器,一上来就硬啃我试了一下,好像不太行,能找到的源码分析文档都是老版本了,只能做参考。
    想想还是日志比较直接,整体思路就是:先创建一个简单的pod,然后跟随日志对应源码看创建的实际代码流程

    准备工作:

    1. 调高日志级别
    kubectl --v 6 
    

    –v=0 总是对操作人员可见。
    –v=1 合理的默认日志级别,如果您不需要详细输出。
    –v=2 可能与系统的重大变化相关的,有关稳定状态的信息和重要的日志信息。这是对大多数系统推荐的日志级别。
    –v=3 有关更改的扩展信息。
    –v=4 调试级别详细输出。
    –v=6 显示请求的资源。
    –v=7 显示HTTP请求的header。
    –v=8 显示HTTP请求的内容

    1. 创建测试pod

    2.1 检查镜像是否下载好

        docker image ls | grep  hello-world
    

    2.2 准备好yaml,如下

    apiVersion: apps/v1            #当前格式的版本,这是1.18的,和老版本的有出入
    kind: Deployment               
    metadata:                      #当前资源的元数据
      name: nginx-deployment       #当前资源的名字 必须项
      labels:
        app: nginx
    spec:                          #是当前Deployment的规格说明
      replicas: 1
      selector:
        matchLabels:
          app: nginx
      template:                    #定义pod的模板
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:latest
            ports:
            - containerPort: 80   #监听端口
    

    2.3 创建pod并保存日志

    kubectl apply -f create_pod.yaml --v 6  #创建deployment并打印日志
    
    journalctl -u kubelet --since "5 min ago" > kubelet.log  #5分钟内kubelet的日志
    

    二、kubenetes主要代码结构

    目录 说明
    api 输出接口文档用
    build 构建脚本
    cluster 适配不同I层的云,例如亚马逊AWS,微软Azure,谷歌GCE的集群启动脚本
    cmd 所有的二进制可执行文件入口代码,从这里看起
    contrib 项目贡献者
    docs 文档,包括了用户文档、管理员文档、设计、新功能提议
    example 使用案例
    Godeps 项目中依赖使用的Go第三方包,例如docker客户端SDK,rest等
    hack 工具箱,各种编译、构建、测试、校验的脚本都在这里面
    hooks git提交前后触发的脚本
    pkg 项目代码主目录,cmd的只是个入口,这里是所有的具体实现
    plugin 插件,k8s认为调度器是插件的一部分,所以调度器的代码在这里

    三、查看日志对应代码

    直接搜索pod名称(共计结果111处)

    Nov 01 18:30:29 k8s-cluster-0 kubelet[63707]: I1101 18:30:29.558356   63707 kubelet.go:1908] SyncLoop (ADD, "api"): "nginx-deployment-cc7df4f8f-tm57r_default(caa03e90-c1ad-44e4-a301-a0609f540023)"
    

    首次出现就是和SyncLoop函数一起(在硬啃和通过日志自己看之前,肯定要先捡一捡以前的人写过的帖子,看看参考,即使大版本更新有了许多变化,依旧有很多不变值得参考的地方,可以和日志一起作为阅读源码的佐证),这个函数是kubelet用来控制循环的,日常看看有没有pod的需要更新

    syncLoop的注释的一段,It watches for changes from three channels (file, apiserver, and http) and creates a union of them.,kubelet接收到的pod的变更来自于三类source,不管是通过文件还是apiserver还是http请求都可以,创建pod的yaml启动就相当于配置发生变化,从这里开始触发syncLoop

    func (kl *Kubelet) syncLoop(updates <-chan kubetypes.PodUpdate, handler SyncHandler) {
    	klog.Info("Starting kubelet main sync loop.")
        //syncTicker 轮询检测是否有需要同步的pod workers
    	syncTicker := time.NewTicker(time.Second)
        defer syncTicker.Stop()
        //housekeepingTicker检测有没有需要干掉的pod
    	housekeepingTicker := time.NewTicker(housekeepingPeriod)
        defer housekeepingTicker.Stop()
        //plegCh Watch pod的生命周期
    	plegCh := kl.pleg.Watch()
    	const (
    		base   = 100 * time.Millisecond
    		max    = 5 * time.Second
    		factor = 2
    	)
    	duration := base
    
    	if kl.dnsConfigurer != nil && kl.dnsConfigurer.ResolverConfig != "" {
    		kl.dnsConfigurer.CheckLimitsForResolvConf()
    	}
    
    	for {
    		if err := kl.runtimeState.runtimeErrors(); err != nil {
    			klog.Errorf("skipping pod synchronization - %v", err)
    			// exponential backoff
    			time.Sleep(duration)
    			duration = time.Duration(math.Min(float64(max), factor*float64(duration)))
    			continue
    		}
    		// reset backoff if we have a success
    		duration = base
    
            kl.syncLoopMonitor.Store(kl.clock.Now())
            
            //syncLoopIteration监听pod变化
    		if !kl.syncLoopIteration(updates, handler, syncTicker.C, housekeepingTicker.C, plegCh) {
    			break
    		}
    		kl.syncLoopMonitor.Store(kl.clock.Now())
    	}
    }
    
    
    Nov 01 18:30:29 k8s-cluster-0 kubelet[63707]: I1101 18:30:29.878055   63707 kuberuntime_manager.go:422] No sandbox for pod "nginx-deployment-cc7df4f8f-tm57r_default(caa03e90-c1ad-44e4-a301-a0609f540023)" can be found. Need to start a new one
    

    轮询之后找不到这个pod,所以认为需要start一个新的,开始创建,所以我们去找这个函数

    //这个函数主要是检查一下pod sandbox是不是和配置文件一样,没有的话就打印start nwe one 并返回true开始创建
    func (m *kubeGenericRuntimeManager) podSandboxChanged(pod *v1.Pod, podStatus *kubecontainer.PodStatus) (bool, uint32, string) {
    	if len(podStatus.SandboxStatuses) == 0 {
    		klog.V(2).Infof("No sandbox for pod %q can be found. Need to start a new one", format.Pod(pod))
    		return true, 0, ""
    	}
    
    	readySandboxCount := 0
    	for _, s := range podStatus.SandboxStatuses {
    		if s.State == runtimeapi.PodSandboxState_SANDBOX_READY {
    			readySandboxCount++
    		}
    	}
    
        //sandbox如果不是最新的或者readySandboxCount > 1那就重新建一个
    	sandboxStatus := podStatus.SandboxStatuses[0]
    	if readySandboxCount > 1 {
    		klog.V(2).Infof("Multiple sandboxes are ready for Pod %q. Need to reconcile them", format.Pod(pod))
    		return true, sandboxStatus.Metadata.Attempt + 1, sandboxStatus.Id
    	}
    	if sandboxStatus.State != runtimeapi.PodSandboxState_SANDBOX_READY {
    		klog.V(2).Infof("No ready sandbox for pod %q can be found. Need to start a new one", format.Pod(pod))
    		return true, sandboxStatus.Metadata.Attempt + 1, sandboxStatus.Id
    	}
    
    	// 网络namespace发生变化时,也要重建sandbox
    	if sandboxStatus.GetLinux().GetNamespaces().GetOptions().GetNetwork() != networkNamespaceForPod(pod) {
    		klog.V(2).Infof("Sandbox for pod %q has changed. Need to start a new one", format.Pod(pod))
    		return true, sandboxStatus.Metadata.Attempt + 1, ""
    	}
    
    	// 没有ip的时候也要重建sandbox
    	if !kubecontainer.IsHostNetworkPod(pod) && sandboxStatus.Network.Ip == "" {
    		klog.V(2).Infof("Sandbox for pod %q has no IP address. Need to start a new one", format.Pod(pod))
    		return true, sandboxStatus.Metadata.Attempt + 1, sandboxStatus.Id
    	}
    
    	return false, sandboxStatus.Metadata.Attempt, sandboxStatus.Id
    }
    
    
    Nov 01 18:30:29 k8s-cluster-0 kubelet[63707]: I1101 18:30:29.766819   63707 reconciler.go:269] operationExecutor.MountVolume started for volume "default-token-928mj" (UniqueName: "kubernetes.io/secret/caa03e90-c1ad-44e4-a301-a0609f540023-default-token-928mj") pod "nginx-deployment-cc7df4f8f-tm57r" (UID: "caa03e90-c1ad-44e4-a301-a0609f540023")
    Nov 01 18:30:29 k8s-cluster-0 kubelet[63707]: I1101 18:30:29.775930   63707 operation_generator.go:657] MountVolume.SetUp succeeded for volume "default-token-928mj" (UniqueName: "kubernetes.io/secret/caa03e90-c1ad-44e4-a301-a0609f540023-default-token-928mj") pod "nginx-deployment-cc7df4f8f-tm57r" (UID: "caa03e90-c1ad-44e4-a301-a0609f540023")
    

    为这个pod准备了一个新的volume

    Nov 01 18:30:30 k8s-cluster-0 kubelet[63707]: 2020-11-01 18:30:30.599 [INFO][38672] plugin.go 145: Extracted identifiers EndpointIDs=&utils.WEPIdentifiers{Namespace:"default", WEPName:"", WorkloadEndpointIdentifiers:names.WorkloadEndpointIdentifiers{Node:"k8s-cluster-0", Orchestrator:"k8s", Endpoint:"eth0", Workload:"", Pod:"nginx-deployment-cc7df4f8f-tm57r", ContainerID:"b01434f92e7465521bb9e83ece451c74e0cbc0dd342d80998c8d6cd225df6e87"}}
    

    顺着Node,Orchestrator,Endpoint,Workload,ContainerID这个日志,可以搜到syncLoopIteration函数,前面syncLoop最后也是调用的它

    //syncLoopIteration对多个管道进行遍历
    func (kl *Kubelet) syncLoopIteration(configCh <-chan kubetypes.PodUpdate, handler SyncHandler,
    	syncCh <-chan time.Time, housekeepingCh <-chan time.Time, plegCh <-chan *pleg.PodLifecycleEvent) bool {
    	select {
    	case u, open := <-configCh:
    		// Update from a config source; dispatch it to the right handler
    		// callback.
    		if !open {
    			klog.Errorf("Update channel is closed. Exiting the sync loop.")
    			return false
    		}
    
    		switch u.Op {
    		case kubetypes.ADD:
    			klog.V(2).Infof("SyncLoop (ADD, %q): %q", u.Source, format.Pods(u.Pods))
    			//kubelet将通过ADD获取所有现有的容器,HandlePodAdditions会处理新增的pod
    			handler.HandlePodAdditions(u.Pods)
    		case kubetypes.UPDATE:
    			klog.V(2).Infof("SyncLoop (UPDATE, %q): %q", u.Source, format.PodsWithDeletionTimestamps(u.Pods))
    			handler.HandlePodUpdates(u.Pods)
    		case kubetypes.REMOVE:
    			klog.V(2).Infof("SyncLoop (REMOVE, %q): %q", u.Source, format.Pods(u.Pods))
    			handler.HandlePodRemoves(u.Pods)
    		case kubetypes.RECONCILE:
    			klog.V(4).Infof("SyncLoop (RECONCILE, %q): %q", u.Source, format.Pods(u.Pods))
    			handler.HandlePodReconcile(u.Pods)
    		case kubetypes.DELETE:
    			klog.V(2).Infof("SyncLoop (DELETE, %q): %q", u.Source, format.Pods(u.Pods))
    			// DELETE is treated as a UPDATE because of graceful deletion.
    			handler.HandlePodUpdates(u.Pods)
    		case kubetypes.SET:
    			// TODO: Do we want to support this?
    			klog.Errorf("Kubelet does not support snapshot update")
    		default:
    			klog.Errorf("Invalid event type received: %d.", u.Op)
    		}
    
    		kl.sourcesReady.AddSource(u.Source)
    
    	case e := <-plegCh:
    		if e.Type == pleg.ContainerStarted {
    			// record the most recent time we observed a container start for this pod.
    			// this lets us selectively invalidate the runtimeCache when processing a delete for this pod
    			// to make sure we don't miss handling graceful termination for containers we reported as having started.
    			kl.lastContainerStartedTime.Add(e.ID, time.Now())
    		}
    		if isSyncPodWorthy(e) {
    			// PLEG event for a pod; sync it.
    			if pod, ok := kl.podManager.GetPodByUID(e.ID); ok {
    				klog.V(2).Infof("SyncLoop (PLEG): %q, event: %#v", format.Pod(pod), e)
    				handler.HandlePodSyncs([]*v1.Pod{pod})
    			} else {
    				// If the pod no longer exists, ignore the event.
    				klog.V(4).Infof("SyncLoop (PLEG): ignore irrelevant event: %#v", e)
    			}
    		}
    		if e.Type == pleg.ContainerRemoved {
    			kl.deletePodSandbox(e.ID)
    		}
    
    		if e.Type == pleg.ContainerDied {
    			if containerID, ok := e.Data.(string); ok {
    				kl.cleanUpContainersInPod(e.ID, containerID)
    			}
    		}
    	case <-syncCh:
    		// Sync pods waiting for sync
    		podsToSync := kl.getPodsToSync()
    		if len(podsToSync) == 0 {
    			break
    		}
    		klog.V(4).Infof("SyncLoop (SYNC): %d pods; %s", len(podsToSync), format.Pods(podsToSync))
    		handler.HandlePodSyncs(podsToSync)
    	case update := <-kl.livenessManager.Updates():
    		if update.Result == proberesults.Failure {
    			// The liveness manager detected a failure; sync the pod.
    
    			// We should not use the pod from livenessManager, because it is never updated after
    			// initialization.
    			pod, ok := kl.podManager.GetPodByUID(update.PodUID)
    			if !ok {
    				// If the pod no longer exists, ignore the update.
    				klog.V(4).Infof("SyncLoop (container unhealthy): ignore irrelevant update: %#v", update)
    				break
    			}
    			klog.V(1).Infof("SyncLoop (container unhealthy): %q", format.Pod(pod))
    			handler.HandlePodSyncs([]*v1.Pod{pod})
    		}
    	case <-housekeepingCh:
    		if !kl.sourcesReady.AllReady() {
    			// If the sources aren't ready or volume manager has not yet synced the states,
    			// skip housekeeping, as we may accidentally delete pods from unready sources.
    			klog.V(4).Infof("SyncLoop (housekeeping, skipped): sources aren't ready yet.")
    		} else {
    			klog.V(4).Infof("SyncLoop (housekeeping)")
    			if err := handler.HandlePodCleanups(); err != nil {
    				klog.Errorf("Failed cleaning pods: %v", err)
    			}
    		}
    	}
    	return true
    }
    

    于是我们跟到HandlePodAdditions中

    // HandlePodAdditions函数从配置源添加的pod的回调,并下发创建任务
    func (kl *Kubelet) HandlePodAdditions(pods []*v1.Pod) {
    	start := kl.clock.Now()
    	//对pod进行排序,先处理时间靠前的
    	sort.Sort(sliceutils.PodsByCreationTime(pods))
    	for _, pod := range pods {
    		existingPods := kl.podManager.GetPods()
    		//把pod添加到pod manager里面
    		kl.podManager.AddPod(pod)
    		//判断是不是静态pod(静态pod与普通pod不同,这个暂时放一放,以后再看)
    		if kubetypes.IsMirrorPod(pod) {
    			kl.handleMirrorPod(pod, start)
    			continue
    		}
    
    		if !kl.podIsTerminated(pod) {
    			//activepods里面就是所有已经接受的存活的pod,终止的不算
    			activePods := kl.filterOutTerminatedPods(existingPods)
    
    			//通过canAdmitPod方法校验Pod能否在该计算节点创建(取决于节点资源)
    			if ok, reason, message := kl.canAdmitPod(activePods, pod); !ok {
    				kl.rejectPod(pod, reason, message)
    				continue
    			}
    		}
    		mirrorPod, _ := kl.podManager.GetMirrorPodByPod(pod)
    		// 通过dispatchWork将创建pod的任务分发下去
    		kl.dispatchWork(pod, kubetypes.SyncPodCreate, mirrorPod, start)
    		kl.probeManager.AddPod(pod)
    	}
    }
    
    //dispatchWork启动Pod创建中Pod的异步同步。 如果pod已完成那dispatchWork就啥也不干
    func (kl *Kubelet) dispatchWork(pod *v1.Pod, syncType kubetypes.SyncPodType, mirrorPod *v1.Pod, start time.Time) {
    	// check whether we are ready to delete the pod from the API server (all status up to date)
    	containersTerminal, podWorkerTerminal := kl.podAndContainersAreTerminal(pod)
    	if pod.DeletionTimestamp != nil && containersTerminal {
    		klog.V(4).Infof("Pod %q has completed execution and should be deleted from the API server: %s", format.Pod(pod), syncType)
    		kl.statusManager.TerminatePod(pod)
    		return
    	}
    
    	// optimization: avoid invoking the pod worker if no further changes are possible to the pod definition
    	if podWorkerTerminal {
    		klog.V(4).Infof("Pod %q has completed, ignoring remaining sync work: %s", format.Pod(pod), syncType)
    		return
    	}
    
    	//启动podWorkers,把Pod,MirrorPod,UpdateType,OnCompleteFunc各项参数配好
    	kl.podWorkers.UpdatePod(&UpdatePodOptions{
    		Pod:        pod,
    		MirrorPod:  mirrorPod,
    		UpdateType: syncType,
    		OnCompleteFunc: func(err error) {
    			if err != nil {
    				metrics.PodWorkerDuration.WithLabelValues(syncType.String()).Observe(metrics.SinceInSeconds(start))
    			}
    		},
    	})
    	//记录容器数量
    	if syncType == kubetypes.SyncPodCreate {
    		metrics.ContainersPerPodCount.Observe(float64(len(pod.Spec.Containers)))
    	}
    }
    
    

    这个podWorkers比较难找,一搜一大堆,借助其他老前辈的帖子还是找到了
    podWorkers会处理pod的创建,删除,更新。podWorkers会为每一个Pod单独创建一个goroutine和更新事件的channel,goroutine 会阻塞式的等待 channel 中的事件,并且对获取的事件进行处理。podWorker自己还会下发任务下去

    // PodWorkers是个抽象接口,直接进入它的方法
    type PodWorkers interface {
    	UpdatePod(options *UpdatePodOptions)
    	ForgetNonExistingPodWorkers(desiredPods map[types.UID]sets.Empty)
    	ForgetWorker(uid types.UID)
    }
    
    //将new setting 给制定的pod
    func (p *podWorkers) UpdatePod(options *UpdatePodOptions) {
    	pod := options.Pod
    	uid := pod.UID
    	var podUpdates chan UpdatePodOptions
    	var exists bool
    
    	p.podLock.Lock()
    	defer p.podLock.Unlock()
    	//判断当前pod有没有启动goroutine,没有的话就启动并创建更新事件channel
    	//用到podUpdates要么是肯定是pod更新,新建或者删除
    	//这三种情况中,更新和删除都已经经过了启动goroutine和创建更新事件channel
    	//只有创建新pod会走true
    	if podUpdates, exists = p.podUpdates[uid]; !exists {
    		//创建更新事件channel
    		podUpdates = make(chan UpdatePodOptions, 1)
    		p.podUpdates[uid] = podUpdates
    		//启动goroutine
    		go func() {
    			defer runtime.HandleCrash()
    			p.managePodLoop(podUpdates)
    		}()
    	}
    	//下发更新事件
    	if !p.isWorking[pod.UID] {
    		p.isWorking[pod.UID] = true
    		podUpdates <- *options
    	} else {
    		// if a request to kill a pod is pending, we do not let anything overwrite that request.
    		update, found := p.lastUndeliveredWorkUpdate[pod.UID]
    		if !found || update.UpdateType != kubetypes.SyncPodKill {
    			p.lastUndeliveredWorkUpdate[pod.UID] = *options
    		}
    	}
    }
    
    

    我去吃个瓜再写……
    好了,我吃完了

    managePodLoop方法与UpdatePod都在pod_work.go中,managePodLoop会调syncPodFn方法去同步pod,这个方法很大,他的延展列到后面

    func (p *podWorkers) managePodLoop(podUpdates <-chan UpdatePodOptions) {
    	var lastSyncTime time.Time
    	for update := range podUpdates {
    		err := func() error {
    			podUID := update.Pod.UID
    			// This is a blocking call that would return only if the cache
    			// has an entry for the pod that is newer than minRuntimeCache
    			// Time. This ensures the worker doesn't start syncing until
    			// after the cache is at least newer than the finished time of
    			// the previous sync.
    			status, err := p.podCache.GetNewerThan(podUID, lastSyncTime)
    			if err != nil {
    				// This is the legacy event thrown by manage pod loop
    				// all other events are now dispatched from syncPodFn
    				p.recorder.Eventf(update.Pod, v1.EventTypeWarning, events.FailedSync, "error determining status: %v", err)
    				return err
    			}
    			//同步pod,这里的updateType会指明是创建还是删除
    			err = p.syncPodFn(syncPodOptions{
    				mirrorPod:      update.MirrorPod,
    				pod:            update.Pod,
    				podStatus:      status,
    				killPodOptions: update.KillPodOptions,
    				updateType:     update.UpdateType,
    			})
    			lastSyncTime = time.Now()
    			return err
    		}()
    		// notify the call-back function if the operation succeeded or not
    		if update.OnCompleteFunc != nil {
    			update.OnCompleteFunc(err)
    		}
    		if err != nil {
    			// IMPORTANT: we do not log errors here, the syncPodFn is responsible for logging errors
    			klog.Errorf("Error syncing pod %s (%q), skipping: %v", update.Pod.UID, format.Pod(update.Pod), err)
    		}
    		p.wrapUp(update.Pod.UID, err)
    	}
    }
    
    
    

    继续调用kubelet进行pod创建,syncPod方法是创建前的最后一次准备工作,首先就判断updateType,确定接下来的调用路线

    func (kl *Kubelet) syncPod(o syncPodOptions) error {
    	//获取pod信息
    	pod := o.pod
    	mirrorPod := o.mirrorPod
    	podStatus := o.podStatus
    	updateType := o.updateType
    
    	// 判断updateType是不是删除
    	if updateType == kubetypes.SyncPodKill {
    		killPodOptions := o.killPodOptions
    		if killPodOptions == nil || killPodOptions.PodStatusFunc == nil {
    			return fmt.Errorf("kill pod options are required if update type is kill")
    		}
    		apiPodStatus := killPodOptions.PodStatusFunc(pod, podStatus)
    		kl.statusManager.SetPodStatus(pod, apiPodStatus)
    		// we kill the pod with the specified grace period since this is a termination
    		if err := kl.killPod(pod, nil, podStatus, killPodOptions.PodTerminationGracePeriodSecondsOverride); err != nil {
    			kl.recorder.Eventf(pod, v1.EventTypeWarning, events.FailedToKillPod, "error killing pod: %v", err)
    			// there was an error killing the pod, so we return that error directly
    			utilruntime.HandleError(err)
    			return err
    		}
    		return nil
    	}
    
    	// Latency measurements for the main workflow are relative to the
    	// first time the pod was seen by the API server.
    	var firstSeenTime time.Time
    	if firstSeenTimeStr, ok := pod.Annotations[kubetypes.ConfigFirstSeenAnnotationKey]; ok {
    		firstSeenTime = kubetypes.ConvertToTimestamp(firstSeenTimeStr).Get()
    	}
    
    	// Record pod worker start latency if being created
    	// TODO: make pod workers record their own latencies
    	if updateType == kubetypes.SyncPodCreate {
    		if !firstSeenTime.IsZero() {
    			// This is the first time we are syncing the pod. Record the latency
    			// since kubelet first saw the pod if firstSeenTime is set.
    			metrics.PodWorkerStartDuration.Observe(metrics.SinceInSeconds(firstSeenTime))
    		} else {
    			klog.V(3).Infof("First seen time not recorded for pod %q", pod.UID)
    		}
    	}
    
    	// 使用Pod和Status Manager状态生成最终的API Pod状态
    	apiPodStatus := kl.generateAPIPodStatus(pod, podStatus)
    
        //将容器规范写入容器标签后,检查容器是否正在使用主机网络,然后在运行时直接将容器IP设置为hostIP
    	podStatus.IPs = make([]string, 0, len(apiPodStatus.PodIPs))
    	for _, ipInfo := range apiPodStatus.PodIPs {
    		podStatus.IPs = append(podStatus.IPs, ipInfo.IP)
    	}
    
    	if len(podStatus.IPs) == 0 && len(apiPodStatus.PodIP) > 0 {
    		podStatus.IPs = []string{apiPodStatus.PodIP}
    	}
    
    	// Record the time it takes for the pod to become running.
    	existingStatus, ok := kl.statusManager.GetPodStatus(pod.UID)
    	if !ok || existingStatus.Phase == v1.PodPending && apiPodStatus.Phase == v1.PodRunning &&
    		!firstSeenTime.IsZero() {
    		metrics.PodStartDuration.Observe(metrics.SinceInSeconds(firstSeenTime))
    	}
    
    	runnable := kl.canRunPod(pod)
    	if !runnable.Admit {
    		// pod如果没有运行,就将pod及容器的状态都置为Blocked
    		apiPodStatus.Reason = runnable.Reason
    		apiPodStatus.Message = runnable.Message
    		// Waiting containers are not creating.
    		const waitingReason = "Blocked"
    		for _, cs := range apiPodStatus.InitContainerStatuses {
    			if cs.State.Waiting != nil {
    				cs.State.Waiting.Reason = waitingReason
    			}
    		}
    		for _, cs := range apiPodStatus.ContainerStatuses {
    			if cs.State.Waiting != nil {
    				cs.State.Waiting.Reason = waitingReason
    			}
    		}
    	}
    
    	// 更新statusManager中pod的状态
    	kl.statusManager.SetPodStatus(pod, apiPodStatus)
    
    	// 如果runnable.Admit为假或者pod.DeletionTimestamp为假或者apiPodStatus.Phase的值是v1.PodFailed,就杀掉这个pod
    	if !runnable.Admit || pod.DeletionTimestamp != nil || apiPodStatus.Phase == v1.PodFailed {
    		var syncErr error
    		if err := kl.killPod(pod, nil, podStatus, nil); err != nil {
    			kl.recorder.Eventf(pod, v1.EventTypeWarning, events.FailedToKillPod, "error killing pod: %v", err)
    			syncErr = fmt.Errorf("error killing pod: %v", err)
    			utilruntime.HandleError(syncErr)
    		} else {
    			if !runnable.Admit {
    				// There was no error killing the pod, but the pod cannot be run.
    				// Return an error to signal that the sync loop should back off.
    				syncErr = fmt.Errorf("pod cannot be run: %s", runnable.Message)
    			}
    		}
    		return syncErr
    	}
    
    	// 如果网络组件没有就用主机网络
    	if err := kl.runtimeState.networkErrors(); err != nil && !kubecontainer.IsHostNetworkPod(pod) {
    		kl.recorder.Eventf(pod, v1.EventTypeWarning, events.NetworkNotReady, "%s: %v", NetworkNotReadyErrorMsg, err)
    		return fmt.Errorf("%s: %v", NetworkNotReadyErrorMsg, err)
    	}
    
        //cgroups-per-qos这个东西是kubelet拿来指定要不要给pod创建一个cgroup
        //cgroup是linux内核中用于实现资源使用限制和统计的模块
    	pcm := kl.containerManager.NewPodContainerManager()
    	// If pod has already been terminated then we need not create
    	// or update the pod's cgroup
    	if !kl.podIsTerminated(pod) {
    		// When the kubelet is restarted with the cgroups-per-qos
    		// flag enabled, all the pod's running containers
    		// should be killed intermittently and brought back up
    		// under the qos cgroup hierarchy.
    		// Check if this is the pod's first sync
    		firstSync := true
    		for _, containerStatus := range apiPodStatus.ContainerStatuses {
    			if containerStatus.State.Running != nil {
    				firstSync = false
    				break
    			}
    		}
    		// Don't kill containers in pod if pod's cgroups already
    		// exists or the pod is running for the first time
    		podKilled := false
    		if !pcm.Exists(pod) && !firstSync {
    			if err := kl.killPod(pod, nil, podStatus, nil); err == nil {
    				podKilled = true
    			} else {
    				klog.Errorf("killPod for pod %q (podStatus=%v) failed: %v", format.Pod(pod), podStatus, err)
    			}
    		}
    		// Create and Update pod's Cgroups
    		// Don't create cgroups for run once pod if it was killed above
    		// The current policy is not to restart the run once pods when
    		// the kubelet is restarted with the new flag as run once pods are
    		// expected to run only once and if the kubelet is restarted then
    		// they are not expected to run again.
    		// We don't create and apply updates to cgroup if its a run once pod and was killed above
    		if !(podKilled && pod.Spec.RestartPolicy == v1.RestartPolicyNever) {
    			if !pcm.Exists(pod) {
    				if err := kl.containerManager.UpdateQOSCgroups(); err != nil {
    					klog.V(2).Infof("Failed to update QoS cgroups while syncing pod: %v", err)
    				}
    				if err := pcm.EnsureExists(pod); err != nil {
    					kl.recorder.Eventf(pod, v1.EventTypeWarning, events.FailedToCreatePodContainer, "unable to ensure pod container exists: %v", err)
    					return fmt.Errorf("failed to ensure that the pod: %v cgroups exist and are correctly applied: %v", pod.UID, err)
    				}
    			}
    		}
    	}
    
    	//针对静态pod的以后再管,先跳过
    	if kubetypes.IsStaticPod(pod) {
    		podFullName := kubecontainer.GetPodFullName(pod)
    		deleted := false
    		if mirrorPod != nil {
    			if mirrorPod.DeletionTimestamp != nil || !kl.podManager.IsMirrorPodOf(mirrorPod, pod) {
    				// The mirror pod is semantically different from the static pod. Remove
    				// it. The mirror pod will get recreated later.
    				klog.Infof("Trying to delete pod %s %v", podFullName, mirrorPod.ObjectMeta.UID)
    				var err error
    				deleted, err = kl.podManager.DeleteMirrorPod(podFullName, &mirrorPod.ObjectMeta.UID)
    				if deleted {
    					klog.Warningf("Deleted mirror pod %q because it is outdated", format.Pod(mirrorPod))
    				} else if err != nil {
    					klog.Errorf("Failed deleting mirror pod %q: %v", format.Pod(mirrorPod), err)
    				}
    			}
    		}
    		if mirrorPod == nil || deleted {
    			node, err := kl.GetNode()
    			if err != nil || node.DeletionTimestamp != nil {
    				klog.V(4).Infof("No need to create a mirror pod, since node %q has been removed from the cluster", kl.nodeName)
    			} else {
    				klog.V(4).Infof("Creating a mirror pod for static pod %q", format.Pod(pod))
    				if err := kl.podManager.CreateMirrorPod(pod); err != nil {
    					klog.Errorf("Failed creating a mirror pod for %q: %v", format.Pod(pod), err)
    				}
    			}
    		}
    	}
    
    	// 给pod创建数据目录
    	if err := kl.makePodDataDirs(pod); err != nil {
    		kl.recorder.Eventf(pod, v1.EventTypeWarning, events.FailedToMakePodDataDirectories, "error making pod data directories: %v", err)
    		klog.Errorf("Unable to make pod data directories for pod %q: %v", format.Pod(pod), err)
    		return err
    	}
    
    	// Volume manager是不会为terminated 状态的pod安装卷的,因为这个函数其实创建删除更新都用的他,所以具体操作根据type来搞,只要不是删除状态的pod都为他挂卷
    	if !kl.podIsTerminated(pod) {
    		// Wait for volumes to attach/mount
    		if err := kl.volumeManager.WaitForAttachAndMount(pod); err != nil {
    			kl.recorder.Eventf(pod, v1.EventTypeWarning, events.FailedMountVolume, "Unable to attach or mount volumes: %v", err)
    			klog.Errorf("Unable to attach or mount volumes for pod %q: %v; skipping pod", format.Pod(pod), err)
    			return err
    		}
    	}
    
    	// 从pod那里拉取secrets
    	pullSecrets := kl.getPullSecretsForPod(pod)
    
    	// 调用containerRuntime.SyncPod
    	result := kl.containerRuntime.SyncPod(pod, podStatus, pullSecrets, kl.backOff)
    	kl.reasonCache.Update(pod.UID, result)
    	if err := result.Error(); err != nil {
            //不会返回error如果仅仅是pod失败
    		for _, r := range result.SyncResults {
    			if r.Error != kubecontainer.ErrCrashLoopBackOff && r.Error != images.ErrImagePullBackOff {
    				// Do not record an event here, as we keep all event logging for sync pod failures
    				// local to container runtime so we get better errors
    				return err
    			}
    		}
    
    		return nil
    	}
    
    	return nil
    }
    
    

    其中makePodDataDirs的实现

    // makePodDataDirs为容器数据创建目录。分别是getPodDir和getPodVolumesDir和getPodPluginsDir
    func (kl *Kubelet) makePodDataDirs(pod *v1.Pod) error {
    	uid := pod.UID
    	if err := os.MkdirAll(kl.getPodDir(uid), 0750); err != nil && !os.IsExist(err) {
    		return err
    	}
    	if err := os.MkdirAll(kl.getPodVolumesDir(uid), 0750); err != nil && !os.IsExist(err) {
    		return err
    	}
    	if err := os.MkdirAll(kl.getPodPluginsDir(uid), 0750); err != nil && !os.IsExist(err) {
    		return err
    	}
    	return nil
    }
    

    getPullSecretsForPod的实现

    // 检查pod并且获取 pull的Secret
    func (kl *Kubelet) getPullSecretsForPod(pod *v1.Pod) []v1.Secret {
    	pullSecrets := []v1.Secret{}
    
    	for _, secretRef := range pod.Spec.ImagePullSecrets {
            //secretManager的GetSecret去实现获取的逻辑
    		secret, err := kl.secretManager.GetSecret(pod.Namespace, secretRef.Name)
    		if err != nil {
    			klog.Warningf("Unable to retrieve pull secret %s/%s for %s/%s due to %v.  The image pull may not succeed.", pod.Namespace, secretRef.Name, pod.Namespace, pod.Name, err)
    			continue
    		}
    
    		pullSecrets = append(pullSecrets, *secret)
    	}
    
    	return pullSecrets
    }
    

    syncPod调用containerRuntime.SyncPod正式创建并启动pod,containerRuntime.SyncPod函数位于\pkg\kubelet\kuberuntime\kuberuntime_manager.go

    func (m *kubeGenericRuntimeManager) SyncPod(pod *v1.Pod, podStatus *kubecontainer.PodStatus, pullSecrets []v1.Secret, backOff *flowcontrol.Backoff) (result kubecontainer.PodSyncResult) {
    	// 检查sandbox和container的更改情况
    	podContainerChanges := m.computePodActions(pod, podStatus)
    	klog.V(3).Infof("computePodActions got %+v for pod %q", podContainerChanges, format.Pod(pod))
    	if podContainerChanges.CreateSandbox {
    		ref, err := ref.GetReference(legacyscheme.Scheme, pod)
    		if err != nil {
    			klog.Errorf("Couldn't make a ref to pod %q: '%v'", format.Pod(pod), err)
    		}
    		//如果podContainerChanges.SandboxID不是空,就杀死这个pod重建
    		if podContainerChanges.SandboxID != "" {
    			m.recorder.Eventf(ref, v1.EventTypeNormal, events.SandboxChanged, "Pod sandbox changed, it will be killed and re-created.")
    		} else {
    			klog.V(4).Infof("SyncPod received new pod %q, will create a sandbox for it", format.Pod(pod))
    		}
    	}
    
    	// 从podContainerChanges获取具体的需要kill的pod信息
    	// 调用killPodWithSyncResult进行kill操作
    	if podContainerChanges.KillPod {
    		if podContainerChanges.CreateSandbox {
    			klog.V(4).Infof("Stopping PodSandbox for %q, will start new one", format.Pod(pod))
    		} else {
    			klog.V(4).Infof("Stopping PodSandbox for %q because all other containers are dead.", format.Pod(pod))
    		}
    
    		killResult := m.killPodWithSyncResult(pod, kubecontainer.ConvertPodStatusToRunningPod(m.runtimeName, podStatus), nil)
    		result.AddPodSyncResult(killResult)
    		if killResult.Error() != nil {
    			klog.Errorf("killPodWithSyncResult failed: %v", killResult.Error())
    			return
    		}
    
    		if podContainerChanges.CreateSandbox {
    			m.purgeInitContainers(pod, podStatus)
    		}
    	} else {
    		// Step 3: kill any running containers in this pod which are not to keep.
    		// 杀死容器中所有不需要keep的pod,其中怎样判断不需要keep的有待研究
    		for containerID, containerInfo := range podContainerChanges.ContainersToKill {
    			klog.V(3).Infof("Killing unwanted container %q(id=%q) for pod %q", containerInfo.name, containerID, format.Pod(pod))
    			killContainerResult := kubecontainer.NewSyncResult(kubecontainer.KillContainer, containerInfo.name)
    			result.AddSyncResult(killContainerResult)
    			if err := m.killContainer(pod, containerID, containerInfo.name, containerInfo.message, nil); err != nil {
    				killContainerResult.Fail(kubecontainer.ErrKillContainer, err.Error())
    				klog.Errorf("killContainer %q(id=%q) for pod %q failed: %v", containerInfo.name, containerID, format.Pod(pod), err)
    				return
    			}
    		}
    	}
    
    	m.pruneInitContainersBeforeStart(pod, podStatus)
    	var podIPs []string
    	if podStatus != nil {
    		podIPs = podStatus.IPs
    	}
    
    	// 在必要的时候为pod创建Sandbox,所谓的必要就是指podContainerChanges.CreateSandbox不为空
    	podSandboxID := podContainerChanges.SandboxID   //获取podContainerChanges.SandboxID的值以局部变量的形式赋给podSandboxID
    	if podContainerChanges.CreateSandbox {
    		var msg string
    		var err error
    
    		klog.V(4).Infof("Creating PodSandbox for pod %q", format.Pod(pod))
    		createSandboxResult := kubecontainer.NewSyncResult(kubecontainer.CreatePodSandbox, format.Pod(pod))
    		result.AddSyncResult(createSandboxResult)
    		podSandboxID, msg, err = m.createPodSandbox(pod, podContainerChanges.Attempt)
    		if err != nil {
    			createSandboxResult.Fail(kubecontainer.ErrCreatePodSandbox, msg)
    			klog.Errorf("createPodSandbox for pod %q failed: %v", format.Pod(pod), err)
    			ref, referr := ref.GetReference(legacyscheme.Scheme, pod)
    			if referr != nil {
    				klog.Errorf("Couldn't make a ref to pod %q: '%v'", format.Pod(pod), referr)
    			}
    			m.recorder.Eventf(ref, v1.EventTypeWarning, events.FailedCreatePodSandBox, "Failed to create pod sandbox: %v", err)
    			return
    		}
    		klog.V(4).Infof("Created PodSandbox %q for pod %q", podSandboxID, format.Pod(pod))
    
    		podSandboxStatus, err := m.runtimeService.PodSandboxStatus(podSandboxID)
    		if err != nil {
    			ref, referr := ref.GetReference(legacyscheme.Scheme, pod)
    			if referr != nil {
    				klog.Errorf("Couldn't make a ref to pod %q: '%v'", format.Pod(pod), referr)
    			}
    			m.recorder.Eventf(ref, v1.EventTypeWarning, events.FailedStatusPodSandBox, "Unable to get pod sandbox status: %v", err)
    			klog.Errorf("Failed to get pod sandbox status: %v; Skipping pod %q", err, format.Pod(pod))
    			result.Fail(err)
    			return
    		}
    
    		//如果我们曾经允许将Pod从非主机网络更新为主机网络,那么就可以使用旧的IP。
    		if !kubecontainer.IsHostNetworkPod(pod) {
    			// Overwrite the podIPs passed in the pod status, since we just started the pod sandbox.
    			podIPs = m.determinePodSandboxIPs(pod.Namespace, pod.Name, podSandboxStatus)
    			klog.V(4).Infof("Determined the ip %v for pod %q after sandbox changed", podIPs, format.Pod(pod))
    		}
    	}
    
    	podIP := ""
    	if len(podIPs) != 0 {
    		podIP = podIPs[0]
    	}
    
    	// 获取podSandboxConfig启动容器
    	configPodSandboxResult := kubecontainer.NewSyncResult(kubecontainer.ConfigPodSandbox, podSandboxID)
    	result.AddSyncResult(configPodSandboxResult)
    	podSandboxConfig, err := m.generatePodSandboxConfig(pod, podContainerChanges.Attempt)
    	if err != nil {
    		message := fmt.Sprintf("GeneratePodSandboxConfig for pod %q failed: %v", format.Pod(pod), err)
    		klog.Error(message)
    		configPodSandboxResult.Fail(kubecontainer.ErrConfigPodSandbox, message)
    		return
    	}
    
    	// 创建init container,typeName会将init container标注出来
    	// init container一般用来做初始化,比如启动容器之前必须包含的一些前置条件的启动准备
    	start := func(typeName string, spec *startSpec) error {
    		startContainerResult := kubecontainer.NewSyncResult(kubecontainer.StartContainer, spec.container.Name)
    		result.AddSyncResult(startContainerResult)
    
    		isInBackOff, msg, err := m.doBackOff(pod, spec.container, podStatus, backOff)
    		if isInBackOff {
    			startContainerResult.Fail(err, msg)
    			klog.V(4).Infof("Backing Off restarting %v %+v in pod %v", typeName, spec.container, format.Pod(pod))
    			return err
    		}
            //调用startContainer创建pod
    		klog.V(4).Infof("Creating %v %+v in pod %v", typeName, spec.container, format.Pod(pod))
    		if msg, err := m.startContainer(podSandboxID, podSandboxConfig, spec, pod, podStatus, pullSecrets, podIP, podIPs); err != nil {
    			startContainerResult.Fail(err, msg)
    
    			switch {
    			case err == images.ErrImagePullBackOff:
    				klog.V(3).Infof("%v %+v start failed in pod %v: %v: %s", typeName, spec.container, format.Pod(pod), err, msg)
    			default:
    				utilruntime.HandleError(fmt.Errorf("%v %+v start failed in pod %v: %v: %s", typeName, spec.container, format.Pod(pod), err, msg))
    			}
    			return err
    		}
    
    		return nil
    	}
    
    	// 启动ephemeral containers,ephemeral containers需要在init containers之前就启动
    	if utilfeature.DefaultFeatureGate.Enabled(features.EphemeralContainers) {
    		for _, idx := range podContainerChanges.EphemeralContainersToStart {
    			start("ephemeral container", ephemeralContainerStartSpec(&pod.Spec.EphemeralContainers[idx]))
    		}
    	}
    
    	// 启动初始化容器
    	if container := podContainerChanges.NextInitContainerToStart; container != nil {
    		// 初始化容器运行成功则调用容器启动containerStartSpec
    		if err := start("init container", containerStartSpec(container)); err != nil {
    			return
    		}
    
    		// 有一个成功就行,然后把把失败的删除
    		klog.V(4).Infof("Completed init container %q for pod %q", container.Name, format.Pod(pod))
    	}
    
    	// 到这里意味着初始化容器已经执行成功,然后调用podContainerChanges.ContainersToStart正式启动容器
    	for _, idx := range podContainerChanges.ContainersToStart {
    		start("container", containerStartSpec(&pod.Spec.Containers[idx]))
    	}
    
    	return
    }
    

    关于startContainer函数,这里面进行了具体的pod创建,其中ImageService和RuntimeService两个接口用来拉镜像和创建pod。ImageService接口主要管镜像仓库拉取、查看、和移除镜像的RPC。RuntimeSerivce主要是Pods和容器生命周期管理的RPC,以及跟容器交互的调用(exec/attach/port-forward)。

    func (m *kubeGenericRuntimeManager) startContainer(podSandboxID string, podSandboxConfig *runtimeapi.PodSandboxConfig, spec *startSpec, pod *v1.Pod, podStatus *kubecontainer.PodStatus, pullSecrets []v1.Secret, podIP string, podIPs []string) (string, error) {
    	container := spec.container
    
    	// 拉取镜像
    	imageRef, msg, err := m.imagePuller.EnsureImageExists(pod, container, pullSecrets, podSandboxConfig)
    	if err != nil {
    		s, _ := grpcstatus.FromError(err)
    		m.recordContainerEvent(pod, container, "", v1.EventTypeWarning, events.FailedToCreateContainer, "Error: %v", s.Message())
    		return msg, err
    	}
    
    	// 这时候会检查RestartCount的值,必须要是0才会创建新的pod
    	restartCount := 0
    	containerStatus := podStatus.FindContainerStatusByName(container.Name)
    	if containerStatus != nil {
    		restartCount = containerStatus.RestartCount + 1
    	}
    
    	target, err := spec.getTargetID(podStatus)
    	if err != nil {
    		s, _ := grpcstatus.FromError(err)
    		m.recordContainerEvent(pod, container, "", v1.EventTypeWarning, events.FailedToCreateContainer, "Error: %v", s.Message())
    		return s.Message(), ErrCreateContainerConfig
    	}
    
        //containerConfig里面都是容器配置
    	containerConfig, cleanupAction, err := m.generateContainerConfig(container, pod, restartCount, podIP, imageRef, podIPs, target)
    	if cleanupAction != nil {
    		defer cleanupAction()
    	}
    	if err != nil {
    		s, _ := grpcstatus.FromError(err)
    		m.recordContainerEvent(pod, container, "", v1.EventTypeWarning, events.FailedToCreateContainer, "Error: %v", s.Message())
    		return s.Message(), ErrCreateContainerConfig
    	}
        //根据podSandboxID, containerConfig, podSandboxConfig信息,调用CreateContainer创建pod
    	containerID, err := m.runtimeService.CreateContainer(podSandboxID, containerConfig, podSandboxConfig)
    	if err != nil {
    		s, _ := grpcstatus.FromError(err)
    		m.recordContainerEvent(pod, container, containerID, v1.EventTypeWarning, events.FailedToCreateContainer, "Error: %v", s.Message())
    		return s.Message(), ErrCreateContainer
    	}
    	err = m.internalLifecycle.PreStartContainer(pod, container, containerID)
    	if err != nil {
    		s, _ := grpcstatus.FromError(err)
    		m.recordContainerEvent(pod, container, containerID, v1.EventTypeWarning, events.FailedToStartContainer, "Internal PreStartContainer hook failed: %v", s.Message())
    		return s.Message(), ErrPreStartHook
    	}
    	m.recordContainerEvent(pod, container, containerID, v1.EventTypeNormal, events.CreatedContainer, fmt.Sprintf("Created container %s", container.Name))
    
    	// 启动pod
    	err = m.runtimeService.StartContainer(containerID)
    	if err != nil {
    		s, _ := grpcstatus.FromError(err)
    		m.recordContainerEvent(pod, container, containerID, v1.EventTypeWarning, events.FailedToStartContainer, "Error: %v", s.Message())
    		return s.Message(), kubecontainer.ErrRunContainer
    	}
    	m.recordContainerEvent(pod, container, containerID, v1.EventTypeNormal, events.StartedContainer, fmt.Sprintf("Started container %s", container.Name))
    
    	// Symlink container logs to the legacy container log location for cluster logging
    	// support.
    	// TODO(random-liu): Remove this after cluster logging supports CRI container log path.
    	containerMeta := containerConfig.GetMetadata()
    	sandboxMeta := podSandboxConfig.GetMetadata()
    	legacySymlink := legacyLogSymlink(containerID, containerMeta.Name, sandboxMeta.Name,
    		sandboxMeta.Namespace)
    	containerLog := filepath.Join(podSandboxConfig.LogDirectory, containerConfig.LogPath)
    	// only create legacy symlink if containerLog path exists (or the error is not IsNotExist).
    	// Because if containerLog path does not exist, only dangling legacySymlink is created.
    	// This dangling legacySymlink is later removed by container gc, so it does not make sense
    	// to create it in the first place. it happens when journald logging driver is used with docker.
    	if _, err := m.osInterface.Stat(containerLog); !os.IsNotExist(err) {
    		if err := m.osInterface.Symlink(containerLog, legacySymlink); err != nil {
    			klog.Errorf("Failed to create legacy symbolic link %q to container %q log %q: %v",
    				legacySymlink, containerID, containerLog, err)
    		}
    	}
    
    	// 如果pod start失败就重启pod
    	if container.Lifecycle != nil && container.Lifecycle.PostStart != nil {
    		kubeContainerID := kubecontainer.ContainerID{
    			Type: m.runtimeName,
    			ID:   containerID,
    		}
    		msg, handlerErr := m.runner.Run(kubeContainerID, pod, container, container.Lifecycle.PostStart)
    		if handlerErr != nil {
    			m.recordContainerEvent(pod, container, kubeContainerID.ID, v1.EventTypeWarning, events.FailedPostStartHook, msg)
    			if err := m.killContainer(pod, kubeContainerID, container.Name, "FailedPostStartHook", nil); err != nil {
    				klog.Errorf("Failed to kill container %q(id=%q) in pod %q: %v, %v",
    					container.Name, kubeContainerID.String(), format.Pod(pod), ErrPostStartHook, err)
    			}
    			return msg, fmt.Errorf("%s: %v", ErrPostStartHook, handlerErr)
    		}
    	}
    
    	return "", nil
    }
    

    结束了~大致的主线流程就是这样,里面很多分支情况,比如静态pod以及各种异常情况的处理,还有展开的函数另外开新纪录贴慢慢展开看。

    展开全文
  • k8s创建pod出现Failed create pod sandbox.Pod sandbox changed, it will be killed and re-created.错误,如下图所示: 到相应子节点上查看kubelet日志: journalctl -u kubelet -n 1000 发现问题的原因...

    k8s创建pod出现Failed create pod sandbox.Pod sandbox changed, it will be killed and re-created.错误,如下图所示:

    到相应子节点上查看kubelet日志:

    journalctl -u kubelet -n 1000

    发现问题的原因如下:

    “network: open /run/flannel/subnet.env: no such file or directory”

    解决方法:将master上对应目录下的flannel文件复制一份到子节点上即可。

    (如果解决了你的问题请点个赞,谢谢,如不能解决有可能是configmap出了问题,具体解决方案后续放上来)

    展开全文
  • 前面我们已经将SpringBoot项目部署在K8S中,此时需要filebeat收集日志并通过ELK进行展示,以用于后续的问题排查及监控。与传统的日志收集不同:pod所在节点不固定,每个pod中运行filebeat,配置繁琐且浪费资源;pod...
    83147b16977d48949be6e29eaa66b933

    前面我们已经将SpringBoot项目部署在K8S中,此时需要filebeat收集日志并通过ELK进行展示,以用于后续的问题排查及监控。

    与传统的日志收集不同:

    • pod所在节点不固定,每个pod中运行filebeat,配置繁琐且浪费资源;
    • pod的日志目录一般以emptydir方式挂载在宿主机,目录不固定,filebeat无法自动匹配;
    • pod持续增多,filebeat需要做到自动检测并收集;

    因此最好的收集方式为node节点上的一个filebeat能够收集所有的pod日志,但是这就要求统一的日志收集规则、目录以及输出方式。

    下面我们就按这个思路进行思考。

    日志目录

    K8S中的日志目录有以下三种:

    • /var/lib/docker/containers/
    • /var/log/containers/
    • /var/log/pods/

    为什么会有这三种目录呢?这就要从容器运行时(Container Runtime)组件说起了

    • 当Docker 作为 k8s 容器运行时,容器日志的落盘将由 docker 来完成,保存在/var/lib/docker/containers/$CONTAINERID 目录下。Kubelet 会在 /var/log/pods 和 /var/log/containers 下建立软链接,指向 /var/lib/docker/containers/CONTAINERID 该目录下的容器日志文件。
    • 当Containerd 作为 k8s 容器运行时, 容器日志的落盘由 Kubelet 来完成,保存至 /var/log/pods/$CONTAINER_NAME 目录下,同时在 /var/log/containers 目录下创建软链接,指向日志文件。
    # 1.查看/var/log/containers目录下文件,已被软链到/var/log/pods中的xx.log文件按# cd /var/log/containers && lllrwxrwxrwx 1 root root 107 Jun 15 15:39 kube-apiserver-uvmsvr-3-217_kube-system_kube-apiserver-7fbb97008724e35427262c1ac294c24d7771365b9facf5f5a49c6b15f032e441.log -> /var/log/pods/kube-system_kube-apiserver-uvmsvr-3-217_837ea80229ea9cd5bbf448f4f0386cbc/kube-apiserver/1.loglrwxrwxrwx 1 root root 107 Aug 21 11:39 kube-apiserver-uvmsvr-3-217_kube-system_kube-apiserver-fd8b454b07b8701045f007bd55551aae60f376d1aee07729779ec516ea505239.log -> /var/log/pods/kube-system_kube-apiserver-uvmsvr-3-217_837ea80229ea9cd5bbf448f4f0386cbc/kube-apiserver/2.log...只列举部分...# 2.查看/var/log/pods目录下的文件,xx.log 最终又软链到/var/lib/docker/containers/ 下的xxx-json.log# cd /var/log/podsll /var/log/pods/kube-system_kube-apiserver-uvmsvr-3-217_837ea80229ea9cd5bbf448f4f0386cbc/kube-apiserver/1.loglrwxrwxrwx 1 root root 165 Jun 15 15:39 /var/log/pods/kube-system_kube-apiserver-uvmsvr-3-217_837ea80229ea9cd5bbf448f4f0386cbc/kube-apiserver/1.log -> /var/lib/docker/containers/7fbb97008724e35427262c1ac294c24d7771365b9facf5f5a49c6b15f032e441/7fbb97008724e35427262c1ac294c24d7771365b9facf5f5a49c6b15f032e441-json.log

    无论k8s使用哪种容器运行时,最终的日志都是读取的xxx-json.log,是由容器以json格式stdout输出的,了解这些后我们得到了统一的日志收集规则

    • 统一目录 :/var/log/containers
    • 统一的输出方式:stdout
    • 统一的日志格式:json

    filebeat

    在明确统一的日志收集规则后,我们就可以使用DaemonSet运行filebeat,收集k8s的所有pod日志。

    ELK官方提供filebeat在k8s中的运行配置文件。

    # 下载文件curl -L -O https://raw.githubusercontent.com/elastic/beats/7.9/deploy/kubernetes/filebeat-kubernetes.yaml

    其中filebeat的配置文件以configmap的方式挂载至容器的/etc/filebeat.yml,因此我们只需要修改configmap就可以实现配置文件的修改。

    默认的filebeat-kubernetes.yaml虽然能够实现pod日志的收集,但是最终在ELK展示时会有多余字段或字段不全的现象,如:

    5bc210d22f164042884d0ddcf84b6c60

    其中host字段、agent字段我们不需要,保留将会增加存储开销,而且没有关于K8S的描述信息,如namespace、node节点信息、image信息等是不显示的。因此我们还需要基于默认配置文件进行微调,来获得我们需要的日志。

    1.添加k8s描述信息

    k8s描述信息默认不添加,可通过以下方式开启:

    processors:  - add_kubernetes_metadata:      default_indexers.enabled: true      default_matchers.enabled: true

    效果如下:

    76e34b29d9d04d248b90b2a29e6d46df

    此时pod的基本信息都将会在ELK中展示,对我们排查问题非常便利。

    2.删除多余字段

    默认收集的字段包括host,agent等无用信息,我们可以自定义进行删除。

    processors:  - drop_fields:    #删除的多余字段    fields: ["host", "tags", "ecs", "log", "prospector", "agent", "input", "beat", "offset"]    ignore_missing: true

    此时可以通过drop_fields删除多余的字段,但是@timestamptype不能被删除的。经过测试ecsagenttags也未能通过filbebeat删除。

    注意:一个filebeat.yml可能由多个processors,如果你在其中一个processors中设置了drop_fields,而在其他processors没有设置,则最终可能导致需要删除的字段被删除后又自动添加。

    对于未删除的字段,我们还可以通过logstash进行删除,例如:

    remove_field => [ "agent" ]remove_field => [ "ecs" ]remove_field => [ "tags" ]

    3.多行合并

    对于springboot日志,我们一般需要将日志拆分成时间戳、日志级别、信息内容;对于ERROR级别日志还要多行合并,以便友好的展示错误信息。
    在filebeat中设置将每行开头时间格式为2020-09-06与之后不匹配此规则的行进行合并。

    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'multiline.negate: truemultiline.match: aftermultiline.timeout: 30

    4.message字段拆分

    默认filebeat收集的日志以json格式存储,日志内容存储为message,因此需要在logstash中过滤message并将其拆分后再写入elasticsearch。

        grok {      match => { "message" => "(%{TIMESTAMP_ISO8601:logdatetime}  %{LOGLEVEL:level} %{GREEDYDATA:logmessage})|%{GREEDYDATA:logmessage}" }    }

    将message中的日志内容再拆分为时间、日志级别、信息内容等字段,以便在ELK中进行搜索。
    效果为:

    7bffd584daba4d3ab194da6c6e25d158

    注意:stream:stdout表示日志以stdout输出到json格式的日志中。

    logstash自动创建索引

    logstash将filebeat收集的日志输出到elasticsearch并创建索引。对于不断增多的pod,我们不可能挨个去手动自定义索引,最好的方式就是logstash根据filebeat的字段自动创建索引

    默认logstash通过metadata中的字段自动创建索引,如:

    output {  if [fields][service] == "k8s-log" {    elasticsearch {      hosts => ["192.168.3.101:9200"]      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"    }    # 可通过以下方式,查看metadata信息    # stdout { codec => rubydebug { metadata => true}}  }}

    此时在elasticsearch中自动创建的索引为filebeat-7.9-2020.09.06,因为通过stdout打印的metadata内容为:

    {    ....,    "@metadata" => {        "beat" => "filebeat",        "version" => "7.9"    }}

    注意:metadata信息不作为最终的信息在ELK中展示。

    由于logtash通过metadata创建的filebeat-7.9-2020-8索引是固定的,因此收集的k8s中所有的pod日志都将写入这个索引。

    思考:logstash是否可以根据filebeat的字段自动创建索引,这样将日志写入不同的索引了。

    因此我尝试根据pod的命名空间和pod标签自动创建索引。

    output {  if [fields][service] == "k8s-log" {    elasticsearch {      hosts => ["192.168.3.101:9200"]      index => "k8s-%{[kubernetes][namespace]}-%{[kubernetes][labels][app]}-%{+YYYY.MM.dd}"      #user => "elastic"      #password => "changeme"    }    #stdout { codec => rubydebug { metadata => true}}  }}

    最终创建的索引为k8s-test-helloworld-2020.09.06,实现自动创建索引的需求了。

    最终整合

    1.filebeat

    # 1.下载初始配置文件curl -L -O https://raw.githubusercontent.com/elastic/beats/7.9/deploy/kubernetes/filebeat-kubernetes.yaml# 2.修改configmapapiVersion: v1kind: ConfigMapmetadata:  name: filebeat-config  namespace: kube-system  labels:    k8s-app: filebeatdata:  filebeat.yml: |-    filebeat.inputs:    - type: container      paths:        - /var/log/containers/api-*.log      #多行合并      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'      multiline.negate: true      multiline.match: after      multiline.timeout: 30      fields:        #自定义字段用于logstash识别k8s输入的日志        service: k8s-log      #禁止收集host.xxxx字段       #publisher_pipeline.disable_host: true      processors:        - add_kubernetes_metadata:            #添加k8s描述字段            default_indexers.enabled: true            default_matchers.enabled: true            host: ${NODE_NAME}            matchers:            - logs_path:                logs_path: "/var/log/containers/"        - drop_fields:            #删除的多余字段            fields: ["host", "tags", "ecs", "log", "prospector", "agent", "input", "beat", "offset"]            ignore_missing: true    output.redis:      hosts: ["192.168.3.44"]      #password: ""      key: "k8s-java-log_test"      db: 1      timeout: 5    #output.logstash:    #  hosts: ["192.168.3.101:5044"]

    2. logstash

    input {  beats {    port => 5044  }  redis {    host => "192.168.3.44"    port => "6379"    db => 1    data_type => "list"    key => "k8s-java-log_test"    type => "k8s-log"  }}filter {  if [type] == "k8s-log" {    grok {      match => { "message" => "(%{TIMESTAMP_ISO8601:logdatetime}  %{LOGLEVEL:level} %{GREEDYDATA:logmessage})|%{GREEDYDATA:logmessage}" }      remove_field => [ "message" ]      remove_field => [ "agent" ]      remove_field => [ "ecs" ]      remove_field => [ "tags" ]    }  }}output {  if [fields][service] == "k8s-log" {    elasticsearch {      hosts => ["192.168.3.101:9200"]      index => "k8s-%{[kubernetes][namespace]}-%{[kubernetes][labels][app]}-%{+YYYY.MM.dd}"    }    #stdout { codec => rubydebug { metadata => true}}  }}

    最终效果如下:

    3ab44bf79e514c9796bb527decb5afa6

    总结

    刚开始想到pod的特性时,最初想到使用sidecar模式,即每个pod中都运行一个filebeat进行日志收集,的确比较耗费资源。经过阅读了官方的k8s部署以及k8s的几个日志目录,才确定DaemonSet部署filebeat才是最优方案。

    另外,还尝试使用阿里开源的log-pilot,但最终还是选择了ELK官方的部署。

    展开全文
  • 使用k8s Elasticsearch查看pod日志的时候偶尔会遇到这样的情况,在创建完容器并运行后去查看日志的时候总是加载不出来,需要等待十几秒甚至一分钟才能加载。我“有幸”被分配来解决这个问题,经过一天的努力终于发现...

    使用k8s Elasticsearch查看pod日志的时候偶尔会遇到这样的情况,在创建完容器并运行后去查看日志的时候总是加载不出来,需要等待十几秒甚至一分钟才能加载。我“有幸”被分配来解决这个问题,经过一天的努力终于发现这个问题的原因,特与大家分享。

    k8s日志记录工作原理

    关于k8s日志的介绍可以参考这篇文章,这里简单总结几点:

    • 通过命令行和API查询的日志的生命周期与Pod相同,也就是说Pod重启之后之前的日志就不见了。
    • 为了解决上述问题,可以通过“集群级别日志”技术来获取Pod的历史日志。
    • k8s支持多种类型的“集群级别日志”,我们的产品使用的是Elasticsearch。

    关于k8s Elasticsearch日志的相关介绍可以看这篇,这里简单总结几点:

    • k8s会在每个node上启动一个Fluentd agent Pod来收集当前节点的日志。
    • k8s会在整个集群中启动一个Elasticsearch Service来汇总整个集群的日志。
    • 上述内容可以通过kubectl get pod/rc/service –namespace=kube-system来查看。
    • 我们可以通过Elasticsearch API来按条件查询日志。
    • Elasticsearch URL可以通过kubectl cluster-info指令查看

    问题定位

    Elasticsearch只是用来汇总和查询日志的,系统压力很小时Elasticsearch API查询不到日志基本是由于Fluentd agent没有将日志准时发送到Elasticsearch。所以现在要来了解Fluentd agent的工作机制。k8s github代码库中的Fluentd配置介绍了很多细节,可以参考这里

    简单说就是Fluentd会在node的/var/log/containers/目录中监控日志,这些文件名称包含了namespace、pod名、rc名等信息,文件内容是时间、日志级别和日志内容等信息。Fluentd以tail形式监控这些文件并将修改发送给Elasticsearch。

    在Fluentd配置中可以看到“flush_interval 5s”这样的信息,理论上在系统压力很小时应该几秒内就能看到日志,那为何会有文章开头提到的问题呢?原来是由于创建容器后,/var/log/containers/下会创建新的日志文件,而Fluentd对于新生成的日志不会立即进行监控,而是有一个时间间隔,具体可以参考这里。其中讲述的是Fluentd tail Input Plugin的配置,对该问题产生影响的就是refresh_interval配置。由于k8s使用的Fluentd配置文件中没有指定refresh_interval,因此使用的是60s的默认配置。这样就有可能导致在创建容器服务且运行之后最多要等待1分钟才能查到日志。

    验证方法

    验证方法有很多,我的方法比较笨拙:

    • 首先修改/etc/profile,添加export HISTTIMEFORMAT=”%Y-%m-%d %H:%M:%S “变量,并重新打开shell。该操作目的是让history命令可以显示历史操作的执行时间。
    • 然后准备好Elasticsearch的查询方法,如下所示,其中我用的rc名称是aaa和bbb。
    curl http://192.168.1.81:8080/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging/logstash-2016.05.26/_search?pretty -d '{"sort":[{"time_nano":{"order":"desc"}}],"query":{"bool":{"must":[{"match":{"kubernetes.namespace_name":"wangleiatest"}}, {"match": {"kubernetes.container_name":"rc名称"}}]}}}'
    • 创建aaa rc,并使用上述查询方法查看日志。我是间隔5秒进行一次查询。
    • 查询到aaa日志之后,马上创建bbb rc,并用上述方法查询bbb的日志。依然用5秒间隔查询,直到看到日志。
    • 这时用history 20来查看历史记录就可以看到最后一条查询aaa日志的操作与最后一条查询bbb日志操作的时间间隔约为1min。
    2010  2016-05-26 18:45:45  curl http://192.168.1.81:8080/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging/logstash-2016.05.26/_search?pretty -d '{"sort":[{"time_nano":{"order":"desc"}}],"query":{"bool":{"must":[{"match":{"kubernetes.namespace_name":"wangleiatest"}}, {"match": {"kubernetes.container_name":"aaa"}}]}}}'
    2011  2016-05-26 18:46:45  curl http://192.168.1.81:8080/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging/logstash-2016.05.26/_search?pretty -d '{"sort":[{"time_nano":{"order":"desc"}}],"query":{"bool":{"must":[{"match":{"kubernetes.namespace_name":"wangleiatest"}}, {"match": {"kubernetes.container_name":"bbb"}}]}}}'

    解决方法

    问题定位到了,解决就很简单了,只需要修改Fluentd配置,添加指定refresh_interval就好了。参考方法如下:

    • 登入fluentd pod所在node,查找fluentd的container id。并使用docker exec访问容器。
    $ docker ps|grep fluentd
    4bbd5f71ef85        index.tenxcloud.com/tenxcloud/fluentd-elasticsearch:hosting   "/run.sh"                2 weeks ago         Up 25 minutes                           k8s_fluentd-elasticsearch.5f73fa89_fluentd-elasticsearch-192.168.1.84_kube-system_39c01b250b373a72f082d04d80fa3216_aa1c08a2
    
    $ docker exec -ti 4bbd5f71ef85 /bin/bash
    root@fluentd-elasticsearch-192:/#
    • 编辑容器中td-agent.conf,找到path为/var/log/containers/*.log的source配置,添加refresh_interval 10配置项。保存并退出编辑,再退出容器。
    root@fluentd-elasticsearch-192:/# vi /etc/td-agent/td-agent.conf
    ...
    
    <source>
      type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      time_format %Y-%m-%dT%H:%M:%S
      tag kubernetes.*
      format json
      read_from_head true
      # refresh_interval是fluentd刷新path下文件列表的时间间隔的配置,这里改成了10秒
      refresh_interval 10
    </source>
    
    ...
    
    root@fluentd-elasticsearch-192:/# exit
    • 执行docker restart 4bbd5f71ef85来重启fluentd容器。

    修改之后在该node上新建pod的日志加载将在10秒内完成。

    展开全文
  • OpenStack虚拟机部署K8S集群,使用Cinder提供PV报错:mount: special device does not exist 2、K8S动态使用Cinder PV流程 2.1、创建带Cinder PV的POD # kubectl apply -f busybox-cinder-dym.yaml pod/testpvcpod-...
  • 前面我们已经将SpringBoot项目部署在K8S中,此时需要filebeat收集日志并通过ELK进行展示,以用于后续的问题排查及监控。 与传统的日志收集不同: pod所在节点不固定,每个pod中运行filebeat,配置繁琐且浪费资源; ...
  • 文章目录前言一:k8s工作分析1.1:k8s创建pod工作流程二 :调度方式示例1 --nodeName方式创建资源查看详细事件(发现未经过调度器)清空所有pod示例2:nodeSelector获取标签帮助给对应的node设置标签分别为sha=a和...
  • 同时需要收集pod 容器的标准输出日志环境:本次环境es、kibana 均部署在k8s 集群外,在物理机部署,只需要log-pilot 指定es 地址具体步骤:创建 daemonset log-pilotkubectl get daemonsets.apps log-pilot -o ...
  • pod基本操作2.1 pod创建2.2 pod删除,查看日志3. service和deployment(控制器)3.1 deployment基本操作3.2 Pod扩容与缩容3.3 expose暴露端口3.3.1 ClusterIP默认类型暴露端口3.3.2 NodePort类型暴露端口3.4 更新pod...
  • 文章目录服务器环境实验步骤Master2节点部署LB1,2负载均衡部署Node节点修改实验测试在LB1上查看nginx的K8S日志创建测试pod在Node节点上测试nginx 服务器环境 角色 IP master1 192.168.18.10 master2 192....
  • Spark有很多种部署的方式,...1 Spark on k8s原理spark-submit可以直接向k8s提交应用程序,提交的机制大致如下:第一步,spark在k8s创建driver,driver是一个pod;第二步,driver与k8s集群沟通创建需要的executor,...
  • 创建pod失败时候的日志如下: [root@user1-group1-295 ceph]# journalctl -u kubelet -f -- Logs begin at Tue 2017-07-11 06:42:53 UTC. -- Sep 04 06:23:21 user1-group1-295.novalocal kubelet[15976]: I0904...
  • Fluentd将docker日志目录/var/lib/docker/containers和/var/log目录挂载到Pod中,然后Pod会在node节点的/var/log/pods目录中创建新的目录,可以区别不同的容器日志输出,该目录下有一个日志文件链接到/var/lib/...
  • 用nfs作为StorageClass的后端存储,然后创建pvc挂载到每个微服务的pod中来做日志的集中化和持久化管理。 在使用的过程中发现在nfs服务的源目录中用 tailf 查看日志会有很大的延迟,延迟可以达到几十秒。 ...
  • k8s线上应用实践

    2021-03-10 20:13:19
    k8s 安装 忽略 k8s常用命令 查看pod状态 kubectl get pods -o wide 查看pod日志 kubectl logs promtail-daemonset-6dq5v 查看configmap配置 kubectl describe cm config-map 从文件创建config-map ...使用文件创建pod
  • k8s小手册

    2021-05-15 17:15:53
    k8s小手册K8s常用命令显示资源列表 get查看名称空间 资源隔离查看名称空间下的pod查看集群节点查看集群服务 一组相同服务的pod的对外访问接口 实现负载均衡查看pod中的容器的打印日志 logs在pod中的容器环境内执行...
  • 目前应用日志,tomcat日志 统一输出到 /data/logs/pod名字/目录下,并且/data/logs目录挂载...使用 cronjob创建一个pod,在每天2点开始清除日志。 apiVersion: batch/v1beta1 kind: CronJob metadata: name:...
  • K8S命令总结

    2020-08-28 18:04:00
    kubectl create -f xxx.yaml [-n $namespaceName] ...在k8s节点上查询指定pod日志,此命令只适用于pod中只有一个容器的日志查询 kubectl logs $podname -c $containerName 查询pod中某个容器的日...
  • k8s常用命令

    2019-03-01 14:55:13
    查看节点状态 kubectl get nodes 查看pod kubectl get pods 查看pod跑在哪个node上 kubectl get pod --all-namespaces -o wide ...查看pod日志 kubectl logs 查看系统运行日志 journalctl -f 创建pod ...
  • k8s部署服务流程

    2021-03-16 09:55:24
    k8s创建并部署yaml服务------>k8s控制器管理pod------->暴露应用------->对外发布应用------->日志监控 手动部署 运行环境镜像:安装java,maven,mysql环境 检测是否安装成功并能够使用: 在k8s
  • 该操作员创建一个记录pod,将corev1.Event信息记录为结构化json日志。 crd允许配置要记录的事件。 安装 操作员 操作员不知所措。 helm upgrade --install eventlogger ./helm/ 自定义资源定义(CRD) apiVersion :...
  • k8s考证-CKA真题

    2021-03-27 16:37:55
    2.找出pod中的错误日志 #要求是把错误内容输出到某个文件中,可以粘贴,也可以直接重定向文件 $ kubectl logs mypod-798fcd9949-lk9rc | grep error > xx.log 3.创建一个pod ,并调度到某个节点上 $ cat > ...
  • 1、创建 Pod 两个容器的写法; 一个 Volume 同时挂载到 两个 容器的 写法; 查看某 Pod 中的 某 容器的输出日志: kubectl logs Pod名 -c 容器名 vim pod.yaml apiVersion: v1 kind: Pod metadata: name: demo ...
  • 而之前没有指定,所以 Spring Cloud Data Flow 在跑 Task 时失败了,无法创建 Pod 。按照 Spring 官方文档配置也一直没用,后面查看源码、修改源码增加日志后终于解决了。2 配置无法生效在自己定义 yaml 文件,并...
  • 首先k8s常用命令 创建部署 kubectl create -f 部署yml文件 更新部署配置(亦可用于创建部署) kubectl apply -f 部署yml文件 查看已部署pod kubectl get pod [-o wide] 查看pod详细信息 kubectl describe...
  • k8s问题收集

    2018-06-28 17:08:12
    1、集群创建完后,创建Pod时提示[ Warning FailedCreatePodSandBox 1s kubelet, {ip} Failed create pod sandbox.] 【定位手段】kubectl describe po {pod-name} 或查看kubelet日志 【问题分析】docker内缺少...
  • 整个K8S体系涉及到的技术众多,包括存储、网络、安全、监控、日志、DevOps、微服务等,很多刚接触K8S的初学者,都会感到无从下手,为了能让大家系统地学习,克服这些技术难点,推出了这套K8S架构师课程。Kubernetes...
  • 文章目录下载安装包制作adminservice镜像准备资源配置清单应用资源配置清单dashboard查看验证查看pod日志浏览器访问验证 下载安装包 下载官方release包: ...
  • k8s常用命令记录

    2019-10-08 12:24:04
    目录 kubectl常用命令 kubectl get pod -n dev 查看日志 查看pod详情 删除pod 删除job 进入pod里面 查看namespace 创建namespace 删...
  • 文章目录一、概述二、kubectl命令的使用1、pod创建2、pod发布3、更新版本4、回滚5、查看资源的详细信息(可以看到该资源日志信息)6、进入pod7、删除pod资源(删除控制器和nginx服务) 一、概述 Kubectl是管理k8s...

空空如也

空空如也

1 2 3 4 5
收藏数 85
精华内容 34
热门标签
关键字:

k8s创建pod日志