2024-10-29 10:05:30,002 INFO org.apache.flink.client.cli.CliFrontend [] - -------------------------------------------------------------------------------- 2024-10-29 10:05:30,004 INFO org.apache.flink.client.cli.CliFrontend [] - Starting Command Line Client (Version: 1.19.1, Scala: 2.12, Rev:5edb5a9, Date:2024-06-06T14:45:33+02:00) 2024-10-29 10:05:30,004 INFO org.apache.flink.client.cli.CliFrontend [] - OS current user: root 2024-10-29 10:05:30,005 INFO org.apache.flink.client.cli.CliFrontend [] - Current Hadoop/Kerberos user: 2024-10-29 10:05:30,005 INFO org.apache.flink.client.cli.CliFrontend [] - JVM: OpenJDK 64-Bit Server VM - Red Hat, Inc. - 11/11.0.22+7-LTS 2024-10-29 10:05:30,005 INFO org.apache.flink.client.cli.CliFrontend [] - Arch: amd64 2024-10-29 10:05:30,005 INFO org.apache.flink.client.cli.CliFrontend [] - Maximum heap size: 30688 MiBytes 2024-10-29 10:05:30,005 INFO org.apache.flink.client.cli.CliFrontend [] - JAVA_HOME: 2024-10-29 10:05:30,005 INFO org.apache.flink.client.cli.CliFrontend [] - No Hadoop Dependency available 2024-10-29 10:05:30,005 INFO org.apache.flink.client.cli.CliFrontend [] - JVM Options: 2024-10-29 10:05:30,006 INFO org.apache.flink.client.cli.CliFrontend [] - -XX:+IgnoreUnrecognizedVMOptions 2024-10-29 10:05:30,006 INFO org.apache.flink.client.cli.CliFrontend [] - -Da=b 2024-10-29 10:05:30,006 INFO org.apache.flink.client.cli.CliFrontend [] - -Dlog.file=/root/deploy/ob-agent-stream/log/flink-root-client-k8smaster.log 2024-10-29 10:05:30,006 INFO org.apache.flink.client.cli.CliFrontend [] - -Dlog4j.configuration=file:/root/deploy/ob-agent-stream/conf/log4j-cli.properties 2024-10-29 10:05:30,006 INFO org.apache.flink.client.cli.CliFrontend [] - -Dlog4j.configurationFile=file:/root/deploy/ob-agent-stream/conf/log4j-cli.properties 2024-10-29 10:05:30,006 INFO org.apache.flink.client.cli.CliFrontend [] - -Dlogback.configurationFile=file:/root/deploy/ob-agent-stream/conf/logback.xml 2024-10-29 10:05:30,006 INFO org.apache.flink.client.cli.CliFrontend [] - Program Arguments: 2024-10-29 10:05:30,008 INFO org.apache.flink.client.cli.CliFrontend [] - run-application 2024-10-29 10:05:30,008 INFO org.apache.flink.client.cli.CliFrontend [] - --target 2024-10-29 10:05:30,008 INFO org.apache.flink.client.cli.CliFrontend [] - kubernetes-application 2024-10-29 10:05:30,008 INFO org.apache.flink.client.cli.CliFrontend [] - -Dkubernetes.cluster-id=ob-agent-stream 2024-10-29 10:05:30,008 INFO org.apache.flink.client.cli.CliFrontend [] - -Dkubernetes.container.image.ref=reg.cestong.com.cn/cecf/ob-agent-stream:v2.4.3 2024-10-29 10:05:30,008 INFO org.apache.flink.client.cli.CliFrontend [] - -Dkubernetes.namespace=data 2024-10-29 10:05:30,008 INFO org.apache.flink.client.cli.CliFrontend [] - local:///opt/flink/usrlib/ob-agent-stream.jar 2024-10-29 10:05:30,009 INFO org.apache.flink.client.cli.CliFrontend [] - Classpath: /root/deploy/ob-agent-stream/lib/flink-cep-1.19.1.jar:/root/deploy/ob-agent-stream/lib/flink-connector-files-1.19.1.jar:/root/deploy/ob-agent-stream/lib/flink-csv-1.19.1.jar:/root/deploy/ob-agent-stream/lib/flink-json-1.19.1.jar:/root/deploy/ob-agent-stream/lib/flink-scala_2.12-1.19.1.jar:/root/deploy/ob-agent-stream/lib/flink-table-api-java-uber-1.19.1.jar:/root/deploy/ob-agent-stream/lib/flink-table-planner-loader-1.19.1.jar:/root/deploy/ob-agent-stream/lib/flink-table-runtime-1.19.1.jar:/root/deploy/ob-agent-stream/lib/hadoop-client-2.10.2.jar:/root/deploy/ob-agent-stream/lib/log4j-1.2-api-2.17.1.jar:/root/deploy/ob-agent-stream/lib/log4j-api-2.17.1.jar:/root/deploy/ob-agent-stream/lib/log4j-core-2.17.1.jar:/root/deploy/ob-agent-stream/lib/log4j-slf4j-impl-2.17.1.jar:/root/deploy/ob-agent-stream/lib/flink-dist-1.19.1.jar::: 2024-10-29 10:05:30,009 INFO org.apache.flink.client.cli.CliFrontend [] - -------------------------------------------------------------------------------- 2024-10-29 10:05:30,011 INFO org.apache.flink.configuration.GlobalConfiguration [] - Using standard YAML parser to load flink configuration file from /root/deploy/ob-agent-stream/conf/config.yaml. 2024-10-29 10:05:30,110 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.annotations.prometheus.io/scrape, true 2024-10-29 10:05:30,110 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: env.java.opts.jobmanager, -javaagent:/opentelemetry-javaagent.jar -Dotel.exporter.otlp.protocol=grpc -Dotel.service.name=ob-agent-stream -Dotel.resource.attributes=key=value -Dotel.logs.exporter=otlp -Dotel.traces.exporter=none -Dotel.metrics.exporter=none -Dotel.exporter.otlp.endpoint=http://otel-collector-opentelemetry-collector.observe.svc.cluster.local:4317 2024-10-29 10:05:30,110 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: rest.flamegraph.enabled, true 2024-10-29 10:05:30,110 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.taskmanager.annotations.prometheus.io/port, 9249 2024-10-29 10:05:30,111 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.annotations.prometheus.io/port, 9249 2024-10-29 10:05:30,111 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region 2024-10-29 10:05:30,111 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost 2024-10-29 10:05:30,111 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.task.off-heap.size, 1gb 2024-10-29 10:05:30,111 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.reporter.prom.factory.class, org.apache.flink.metrics.prometheus.PrometheusReporterFactory 2024-10-29 10:05:30,111 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.savepoints.dir, s3p://oas/savepoints 2024-10-29 10:05:30,111 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.bind-host, localhost 2024-10-29 10:05:30,111 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: s3.secret-key, ****** 2024-10-29 10:05:30,111 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: s3.endpoint, http://minio.base.svc.cluster.local:9000 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.host, localhost 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.namespace, data 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.taskmanager.annotations.prometheus.io/scrape, true 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 2 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.reporter.prom.scope.variables.additional.biz, ob-agent-stream 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: env.java.opts.all, -Da=b 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend.type, hashmap 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 6gb 2024-10-29 10:05:30,112 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.reporter.prom.port, 9249 2024-10-29 10:05:30,113 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.bind-host, localhost 2024-10-29 10:05:30,113 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.pod-template-file.taskmanager, ./conf/taskmanager-pod-template.yaml 2024-10-29 10:05:30,113 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.annotations.prometheus.io/path, / 2024-10-29 10:05:30,113 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.latency.interval, 15 2024-10-29 10:05:30,113 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m 2024-10-29 10:05:30,114 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.network.fraction, 0.1 2024-10-29 10:05:30,114 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123 2024-10-29 10:05:30,114 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.checkpointing.interval, 1min 2024-10-29 10:05:30,115 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: s3.access-key, BYerwM0myK9T6eNI4iuj 2024-10-29 10:05:30,115 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.serialization-config, [{io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest: {type: kryo, kryo-type: registered, class: com.twitter.chill.protobuf.ProtobufSerializer}}] 2024-10-29 10:05:30,115 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.managed.size, 300mb 2024-10-29 10:05:30,115 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: rest.profiling.enabled, true 2024-10-29 10:05:30,115 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: env.java.opts.taskmanager, -javaagent:/opentelemetry-javaagent.jar -Dotel.exporter.otlp.protocol=grpc -Dotel.service.name=ob-agent-stream -Dotel.resource.attributes=key=value -Dotel.logs.exporter=otlp -Dotel.traces.exporter=none -Dotel.metrics.exporter=none -Dotel.exporter.otlp.endpoint=http://otel-collector-opentelemetry-collector.observe.svc.cluster.local:4317 2024-10-29 10:05:30,115 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.pod-template-file.jobmanager, ./conf/jobmanager-pod-template.yaml 2024-10-29 10:05:30,115 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.dir, s3p://oas/checkpoints 2024-10-29 10:05:30,116 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: s3.path.style.access, true 2024-10-29 10:05:30,168 INFO org.apache.flink.client.cli.CliFrontend [] - Loading FallbackYarnSessionCli 2024-10-29 10:05:30,204 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. 2024-10-29 10:05:30,229 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: external-resource-gpu 2024-10-29 10:05:30,239 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-influx 2024-10-29 10:05:30,239 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-jmx 2024-10-29 10:05:30,239 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-slf4j 2024-10-29 10:05:30,240 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-prometheus 2024-10-29 10:05:30,240 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-statsd 2024-10-29 10:05:30,240 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-datadog 2024-10-29 10:05:30,240 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-graphite 2024-10-29 10:05:30,241 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: s3-fs-presto 2024-10-29 10:05:30,387 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. 2024-10-29 10:05:30,405 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-13371056665245982461.conf. 2024-10-29 10:05:30,411 INFO org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. 2024-10-29 10:05:30,413 INFO org.apache.flink.client.cli.CliFrontend [] - Running 'run-application' command. 2024-10-29 10:05:30,424 INFO org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer [] - Submitting application in 'Application Mode'. 2024-10-29 10:05:30,426 INFO org.apache.flink.client.deployment.DefaultClusterClientServiceLoader [] - Could not load factory due to missing dependencies. 2024-10-29 10:05:31,786 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead 2024-10-29 10:05:32,345 ERROR org.apache.flink.client.cli.CliFrontend [] - Error while running the command. org.apache.flink.client.deployment.ClusterDeploymentException: The Flink cluster ob-agent-stream already exists. at org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:191) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:214) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1273) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) [flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367) [flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335) [flink-dist-1.19.1.jar:1.19.1]