[QUESTION] NodeNotReadyError: Attempt to send a request to node which is not ready (node id 4)

[saper]

When sending events, from time to time we get the following error:

NodeNotReadyError: Attempt to send a request to node which is not ready (node id 4)

and aiokafka does not seem to be able to recover from it. All subsequent uses cause ProducerClosed error.

We are still investigating what could be wrong on our side (we do not run the Kafka broker in question, only the client), but maybe somebody have encountered this already.

[vmaurin]

Hi @saper
Are you using transaction while producing. I have a similar issue here #1134

[saper]

No, just a simple send and wait... but your theory from #1134 looks very plausible to me. May be in our case this is not a coordinator...

At some point the connection is simply removed:

aiokafka/aiokafka/client.py

Lines 484 to 486 in 5ec91e7

	async def ready(self, node_id, *, group=ConnectionGroup.DEFAULT):
	conn = await self._get_conn(node_id, group=group)
	return conn is not None

this can be a factor

aiokafka/aiokafka/conn.py

Line 507 in 5ec91e7

return bool(self._reader is not None and not self._reader.at_eof())

metadata refresh is likely requested

aiokafka/aiokafka/client.py

Line 479 in 5ec91e7

self.force_metadata_update()

but that does not seem to help (need to get more logging on that)

[vmaurin]

I have a PR that should fix the issue https://github.com/aio-libs/aiokafka/pull/1135/files
The main issue is that some errors are just not handled other than "retry", while when it comes to query to a coordinator, it should always force an invalidiation of the failing coordinator (so we can find a new one on the next retry)