解决引入skywalking后kafka报UnknowServerException问题

以javaagent方式引入skywalking后,kafka报错,org.apache.kafka.common.errors.UnknownServerException,协助排查了一下记录一下问题原因。

简略报错信息如下:
error
org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.

详细异常栈如下:

org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.

    at org.springframework.kafka.core.KafkaTemplate.lambdabuildCallback4(KafkaTemplate.java:602)
    at org.apache.skywalking.apm.plugin.kafka.CallbackAdapter.onCompletion(CallbackAdapter.java:45)
    at org.springframework.kafka.core.DefaultKafkaProducerFactoryCloseSafeProducer1.onCompletionoriginalzBFqxPGz(DefaultKafkaProducerFactory.java:855)
    at org.springframework.kafka.core.DefaultKafkaProducerFactoryCloseSafeProducer1.onCompletionoriginalzBFqxPGzaccessorvQAeqrUE(DefaultKafkaProducerFactory.java)
    at org.springframework.kafka.core.DefaultKafkaProducerFactoryCloseSafeProducer1auxiliaryGL7GI6dN.call(Unknown Source)
    at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
    at org.springframework.kafka.core.DefaultKafkaProducerFactoryCloseSafeProducer1.onCompletion(DefaultKafkaProducerFactory.java)
    at org.apache.kafka.clients.producer.KafkaProducerInterceptorCallback.onCompletionoriginali8ShL01z(KafkaProducer.java:1356)
    at org.apache.kafka.clients.producer.KafkaProducerInterceptorCallback.onCompletionoriginali8ShL01zaccessorP8WlJZIT(KafkaProducer.java)
    at org.apache.kafka.clients.producer.KafkaProducerInterceptorCallbackauxiliaryoCQlmKZA.call(Unknown Source)
    at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
    at org.apache.kafka.clients.producer.KafkaProducerInterceptorCallback.onCompletion(KafkaProducer.java)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:231)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:197)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:676)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:662)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:620)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:554)
    at org.apache.kafka.clients.producer.internals.Sender.access100(Sender.java:69)
    at org.apache.kafka.clients.producer.internals.Sender1.onComplete(Sender.java:745)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:572)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:564)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:324)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.

问题解决

初步分析是skywalking中对spring-kafka版本的不支持,当前用的spring boot 2.2.x,对应spring-kafka是2.5.x,但是skywalking 中agent对应支持的中间件版本为 1.3.x – 2.3.x ,其中2.0.x-2.3.x是未经测试版本,回退到1.3.x版本是可以正常支撑的,所以暂时agent中去掉此kafka apm插件,后续详细分析再进行补充,附skywalking最新支持列表。

使用agent方式比较快捷,也是无入侵的,这是非常好的优势,但是对中间件的支持就需要各种插件官方来保持更新,这算是美中不足吧

The post 解决引入skywalking后kafka报UnknowServerException问题 first appeared on 司马他.

文章来源:

Author:司马他
link:https://www.congcong.us/post/skywalking_kafka_unknowserverexception_problem.html